Monitoring Airbnb reviews over COVID-19 with folium HeatMapWithTime

Introduction

This artile will walk through visualizing the fluctuations of Airbnb busineess affected by COVID-19 pandemic. Intuitively, we might have a rough guess what the curves will be looking like, however, I thought this would be interesting to practice both data processing and visulization when time-series attributes involved, what’s more important, to better explore the data and express the insight to a wide variety of audience in a more approachable manner, an interactive visulization significantly helps.

Data

The dataset was downloaded from InsideAirbnb-Hawaii-data, the analysis results were based on a span of two year from July, 2019 to July, 2021.

Prerequisite

In addition to commonly used python data science packages(numpy, pandas, matplotlib, seaborn), here we also need install plotly, folium, chart-studio, those can be easily installed via pip under conda environment:

$ pip install plotly
$ pip install folium
$ pip install chart-studio

Part 1. Visualizing with matplotlib and plotly

Inspecting Data

I will mainly working on reviews.csv and listings.csv, merging operations will be included later for visulization.

df_review = pd.read_csv('reviews.csv')
df_listing = pd.read_csv('listings.csv')

df_listing contains all the metadata for all the listings and df_review has all the review information for each review associated with its list id. Noticed listing_id in df_review is id in df_listing, it will be the key for our merging.

Data processing: the number of reviews

Apparently, there is no obvious feature indicating the fluctuations of Airbnb busineess activity, however, from df_review we can tell that each listing has received multiple comments in various time, with that being said, we can get all the listings that have received any comment on different dates, then the count of the total comments on each date will be good enough to indicate the change of business activity. In addition, we can add any metadata associated with listing based on our needs, for instance, we may consider adding geographic features(neighbourhood,latitude,longitude) since we are creating a folium map later, also this kind of feature helps to segragate the data to better examine the change focusing on a particular area.

Inspect the number of reviews over time

def process_count(df_review, df_listing):
    df1 = df_review.drop(['reviewer_id', 'reviewer_name', 'comments'], axis=1)
    df2 = df_listing[['id', 'neighbourhood_group_cleansed']]
    df3 = pd.merge(df2, df1, left_on='id', right_on='listing_id')
    df3 = df3[['date', 'listing_id', 'neighbourhood_group_cleansed']]
    df3.date = pd.to_datetime(df3.date, format="%Y-%m-%d")
    df3 = df3[df3['date'].isin(pd.date_range("2019-07-10", "2021-07-10"))]
    df3 = df3.set_index(df3.date).drop('date', axis=1)
    #df3.to_pickle("ts.pkl")
    return df3

listing_reveived_review = process_count(df_review, df_listing)
listing_reveived_review

output:

Then we specify the area to Honolulu and count how many listings that have received comments on each date.

def count_listings(df, loc='Honolulu'):
    df = df[df.neighbourhood_group_cleansed == loc]
    df = df.groupby('date', sort=True)['listing_id'].count().rename('review_count').reset_index().set_index('date')
    return df

count_per_day_honolulu = count_listings(listing_reveived_review, loc='Honolulu')
count_per_day_honolulu

output:

Create time variable

# define time variable
ts = count_per_day_honolulu.review_count

Fit moving average

def fit_moving_avg(series, window=5):
    """calculate moving average number of reviews
    """
    return series.rolling(window, center=True).mean()

avg_ts = fit_moving_avg(ts)

Visualization

via matplotlib

# plot time variable
ts.plot(figsize=(16, 4), label='Number of reviews', title='Number of Reviews over time', fontsize=14, alpha=0.6)
# plot moving average
avg_ts.plot(label='Average number of reviews', fontsize=14)
plt.legend(fontsize=14)
#plt.savefig('moving_avg.png')

output:

Insights

• This analysis takes the number of reviews per day as an indicator of Airbnb business activity. It dramaticly decreased after outbreak of COVID-19 in March, 2020.
• The number of reviews keeps increasing which indicates the popularity of Airbnb was thriving before the pandemic. Meanwhile there is a clear seasonality pattern before the mid of Feb, 2020.

Interactive visualization with Plotly

# set up for plotly
r = go.Scatter(x=ts.index, y=ts.values, 
               line=go.scatter.Line(color='red', width = 1), opacity=0.8, 
               name='Reviews', text=[f'Reviews: {x}' for x in ts.values])

layout = go.Layout(title='Number of Reviews over time', xaxis=dict(title='Date'),
                   yaxis=dict(title='Count'))

fig1 = go.Figure(data=[r],layout=layout)
iplot(fig1)

output:

Following snippet is used to push the plotly object to plotly express, which generates embedding information for hosting the interactive image on web pages.

username = '' # your username
api_key = '' # your api key for plotly express - go to profile > settings > regenerate key
chart_studio.tools.set_credentials_file(username=username, api_key=api_key)

py.plot(fig1, filename = 'review_over_time', auto_open=True)

Part 2. Reviews via folium HeatMapWithTime plugin

Data processing

Count reviews received for each listing each month

As you probably already know, folium creates great interactive maps for visualization, here I am going to create a heat map along with time stamp by using its HeatMapWithTime plugin, before that there still some processing work need to complete in order to fit our data to the plugin. A simple Demo.

First, count how many reviews each listing received each day, then change the time range to Month, that is to record how many reviews each listing received each month. The reason that change time range from day to month is we don’t want the final display moving too frequently so that we can clearly spot the moving trend.

Also, features like latitude and longitude will be kept for rendering the map.

def process_listing_total_count(df_review, df_listing):
        # process data for timestamped folium map
        df2 = df_review.groupby(['listing_id', 'date'])['id'].count().rename('review_count').reset_index()
        df2['date'] = pd.to_datetime(df2['date'])
        df2 = df2[df2['date'].isin(pd.date_range("2019-07-10", "2021-07-10"))]
        df2 = df2.groupby(['listing_id', pd.Grouper(key='date', freq='M')])['review_count'] \
            .sum().reset_index()

        merged = pd.merge(df_listing, df2, left_on='id', right_on='listing_id')
        #merged.to_pickle("timestamped_review.pkl")
        return merged

listings_with_total_review_count = process_listing_total_count(df_review, df_listing)
listings_with_total_review_count[['id','date','review_count','latitude','longitude']]

output:

Transform data to the form that folium can take

For next step, our dataframe should be looking like below:

review_count_time_map = listings_with_total_review_count.drop(['neighbourhood_group_cleansed', 'listing_id'], axis=1)
review_count_time_map = review_count_time_map.groupby('date').agg(lambda x: list(x))
review_count_time_map[['latitude','longitude','review_count']]

output:

Lastly, we just extract the repective latitude, longitude, and review_count to a final form for pass in to the plugin.

# transform data for folium
def generate_location_points(all_points):
    # all_points = pd.read_pickle("timestamped_review.pkl")
    # loc_points = all_points[all_points.neighbourhood_group_cleansed == location]
    loc_points = all_points.drop(['neighbourhood_group_cleansed', 'listing_id'], axis=1)
    loc_points = loc_points.groupby('date').agg(lambda x: list(x))

    to_draw = []
    for i in range(loc_points.shape[0]):
        single_draw = []
        for j in list(zip(loc_points.iloc[i].latitude, loc_points.iloc[i].longitude, loc_points.iloc[i].review_count)):
            single_draw.append(list(j))
        to_draw.append(single_draw)

    time_index = []
    for t in loc_points.index:
        time_index.append(t.strftime("%Y-%m-%d"))

    return to_draw, time_index

points, indices = generate_location_points(listings_with_total_review_count)[0], \
                      generate_location_points(listings_with_total_review_count)[1]

Visualization - click display button

# create folium object and add timestamp object
time_map = folium.Map([21.3487, -157.944], zoom_start=10.5)
hm = plugins.HeatMapWithTime(points, index=indices, auto_play=True, max_opacity=0.6)
hm.add_to(time_map)

# display map
#time_map
#time_map.save("index.html")

output:

Insight

• The heat map above indicates how the number of reviews of Airbnb listings changed on Oahu island during 2019-7 to 2021-7
• Time variable is incremented by month, this can be adjusted to a bigger of smaller increment.
• The trend showed from above matches the result acquired by matplotlib.
• Moreover, timestamped visualization delivers a more approachable result to a variaty of audiences.
• By taking advantage of geographaic information, we could monitor many other attributes as long as we render them with suitable weights.