How to Scrape Rentals Websites Using BeautifulSoup and Python?

Browsing manually using listings on the rental websites can be very time-consuming. So, the better option is scrape rental websites using web scraping Python. This blog shows How to Scrape Rentals Websites Using BeautifulSoup and Python.

March 29, 2022

Our achievements in the field of business digital transformation.

Scraping Rental Website Data through Web scraping using BeautifulSoup and Python

We have decided to extract data from TorontoRentals.com with Python and BeautifulSoup. This website has lists for Toronto as well as many suburbs like Brampton, Scarborough, Mississauga, and Vaughan etc. This has various kinds of listings like apartment, house, condo, as well as basement.

Initially, we have imported the necessary Python libraries.

# Import Python Libraries
# For HTML parsing
from bs4 import BeautifulSoup 
# For website connections
import requests 
# To prevent overwhelming the server between connections
from time import sleep 

# Display the progress bar
from tqdm import tqdm
# For data wrangling
import numpy as np
import pandas as pd
pd.set_option('display.max_columns', 500)
pd.set_option('display.width', 1000)
# For creating plots
import matplotlib.pyplot as plt
import plotly.graph_objects as go

Next, we have written the function named get_page to have soup objects for every page (iteration). Functions accept 4 user inputs — type, city, beds, and page. Function consists of logic for checking HTTP response status codes for finding if HTTP requests have been completed successfully. A get_page function is named from the key function named page_num.

def get_page(city, type, beds, page):
  
  url = f'https://www.torontorentals.com/{city}/{type}?beds={beds}%20&p={page}'
  # https://www.torontorentals.com/toronto/condos?beds=1%20&p=2
  
  result = requests.get(url)
  
  # check HTTP response status codes to find if HTTP request has been successfully completed
  if result.status_code >= 100  and result.status_code <= 199:
      print('Informational response')
  if result.status_code >= 200  and result.status_code <= 299:
      print('Successful response')
      soup = BeautifulSoup(result.content, "lxml")
  if result.status_code >= 300  and result.status_code <= 399:
      print('Redirect')
  if result.status_code >= 400  and result.status_code <= 499:
      print('Client error')
  if result.status_code >= 500  and result.status_code <= 599:
      print('Server error')
      
  return soup

Our plan is scraping the given information from every listing — City, Zip, Street, Rent, Dimensions, Bed, and Bath. We have assigned an empty listing for every variable having scraped. Seven empty listings are created.

The complete scripting grabs the City, Zip, Street, Rent, Bath, Dimensions, and Bed for every listing with a nested FOR LOOP logic as well as consistent HTML tags.

for page_num in tqdm(range(1, 250)):
    sleep(2)
    
    # get soup object of the page
    soup_page = get_page('toronto', 'condos', '1', page_num)

    # grab listing street
    for tag in soup_page.find_all('div', class_='listing-brief'):
        for tag2 in tag.find_all('span', class_='replace street'):
            # to check if data point is missing
            if not tag2.get_text(strip=True):
                listingStreet.append("empty")
            else:
                listingStreet.append(tag2.get_text(strip=True))

After scripts complete execution, observe the length of all seven listings to ensure all have similar lengths. After that, make a pandas DF using the listing. Save a DF to the csv file.

# create the dataframe
df_Toronto_Condo = pd.DataFrame({'city_main':'Toronto', 'listing_type': 'Condo', 'street': listingStreet, 'city': listingCity, 'zip': listingZip, 'rent': listingRent, 'bed': listingBed,'bath': listingBath, 'dimensions': listingDim})

# saving the dataframe to csv file
df_Toronto_Condo.to_csv('df_Toronto_Condo.csv')

WIth page_num functions as well as changing different parameters of get_page function, we have collected data for various housing kinds — apartment, house, condo, as well as basement for Toronto as well as suburban cities. We have created the pandas DF for every housing kind as well as saved that to the CSV file.

Data Preparation & Cleaning with Pandas

The main part of different data science projects include data colection, cleaning, and preparation. So, we have united the DFs proudced from web scraping for getting one key DF having all listings. After that, start data wrangling.

⭢ Search for different missing values in DF

Secret HTML wrappers inhabit like empty listings.

Missing data in a few listings for bath, bed, or dimensions.

⭢ Cope with the missing data

Secret HTML wrappers, which populate like empty listings got dropped from a DF.

If details on bath, bed, or dimensions for the list got missing, after that this was set at zero.

⭢ Get data issues as well as fix them

For different listings, rent get specified as the range. For all the listings, even bath, bed, dimensions are identified as ranges. For e.g., one listing, rent is among $1795–2500, bed differs between 1 to 3, bath ranges between 1 to 2, dimensions differ between 622 to 955 ft2. With this ‘range’ listings look like 1 listing on a website, it looks as if they are promoting for different units in the listing — however, we don’t understand how many units might be accessible within every listing or individual specifications about rent, baths, bed, and dimensions of separate units. Making speculations or getting averages doesn’t seem right in this situation. As such, these rows got dropped from this analysis.

Find and examine larger outliers in the DF.

⭢ Complete data transformations

Clean City features through deleting ‘, ON’ from entries.

Clean particular characters including $, -, as well as,

Data kind conversions: to do data analysis, Rents & Dimensions got converted into numeric data types.

After completing web preparation and cleaning, we get a clear dataset, which we can analyze more to draw helpful insights.

Insights Produced from Data

Here is a count of total number of listings through Type and City.

Sample dataframe indicating the information and listings collected on various features.

Insights from different plots produced with Plotly and Matplotlib

Condos are having maximum number of lists on a website, given by Apartments.

Renters looking for other kinds of accommodation including Basements or Houses should perhaps search other rental websites.

Toronto is having the maximum listings on a website given by Etobicoke and North York.

Suburb cities are having fewer listings associated to Toronto.

For the majority of cities, Condos have the maximum listing types. The next one is Apartments.

For Scarborough, this appears as there are equivalent number of listings given for Apartments, Condos, and Houses.

Brampton looks to have maximum Basements and Houses listed.

The majority of listings either get one or two bedrooms. This trend gets observed across various cities.

Listings having one-bath are most usual.

Toronto looks to have good % of lists having two-baths.

Mean gets affected by the outliers in a dataset. Median is the better statistic because it is robust to the outliers.

For Rent: Important difference between the Median and Mean for Etobicoke, Toronto, Mississauga, and North York.

For Dimensions: Important difference between Median and Mean for Richmond Hill, Vaughan, Markham, and Toronto.

Additional investigation in data to know why Vaughan has enormously higher values for %difference between Median and Mean: Two lists for Vaughan have the dimensions of 800,899 SQFT!! Renting of ~$2300 for 2B-2B condo. So, it looks like these listings have typos for dimensions.

Toronto, Mississauga, as well as Etobicoke have the maximum Median Rents among all cities.

This was expected that the costing of rent in Toronto might be considerably higher compared to other cities. However, to do data analysis, this looks as if Toronto, Etobicoke, and Mississauga have related median Rents.

Scarborough and Suburbs Brampton have the lowermost median Rents.

Using Median Rents for Mississauga and Toronto are alike, the median dimensions about lists in Mississauga is ~100 SQFT larger than those within Toronto.

Lists in Brampton are having lowest Rents however, largest dimensions compared with other cities.

Investigation of Relationship Between Dimensions and Rent

Scatter plot A: with original DF

No relationships between Dimensions and Rent.

A few larger outliers of Dimensions and Rent skew a plot.

Scatter plot B: with DF after reducing rows having outlier Dimensions

Weaker positive correlation between Dimensions and Rent.

Larger Rent outliers skew a plot.

Scatter plot C: With DF after dipping rows having outlier Rent

A bit more positive association between Dimensions and Rent.

Note: Missing data of dimensions got replaced using zeros. Therefore, in the given three plots, we have observed many listings that look to get zero dimensions.

For more information about scraping rental website using Python and BeautifulSoup, contact 3i Data Scraping or ask for a free quote!