How To Scrape YouTube Data For Smart Business Strategies?

    
     1. Start Setting Up 

You need to install Python 3+ and Python IDE to handle the process. You can use Visual Studio or PyCharm Community Edition, which are two free extensions that are available. Start initializing the Python project: 
mkdir youtube-scraper
cd youtube-scraper
python -m venv env

Then, open this in IDE and create a scraper .py file as follows: 
python scraper.py

2. Install Libraries For Scraping 

As an interactive platform, tools must be used dynamically to render data. So integrate Webdriver Manager and Selenium packages:
pip install selenium webdriver-manager

Webdriver Manager makes data management more effortless. Redirect to Selenium present in the scraper.py file. The script below will create an instance of the Chrome WebDriver object that will control the Chrome window:
from selenium import webdriver
from Selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager
from Selenium.webdriver.chrome.options import Options

# initialize a web driver instance to control a Chrome window
# in headless mode
options = Options()
options.add_argument('--headless=new')

driver = webdriver.Chrome(
    service=ChromeService(ChromeDriverManager().install()),
    options=options
)
# scraping logic

# close the browser and free up the resources
driver.quit()

3. Connect With YouTube
Choosing the video from where you want to gather data is important. You can copy the URL. Now, assign the URL to a variable:
url = 'Paste Your Chosen YouTube URL’

Now, let Selenium connect with the target web page using get() function that assigns you to the particular page for visiting using the URL parameter:  
driver.get(url)

4. Check Target Page
A consent dialog might be displayed once you access YouTube, which asks for the accessibility of the data. You must click on "Accept All". Then follow the below process: 
● Open the YouTube platform in the incognito tab. 
● Then, right-click at consent modal, choose "Inspect."
● Find the id attribute, which is used to define a selector strategy. 

Then use the below coding lines to handle cookies: 
try:
    # wait up to 15 seconds for the consent dialog to show up
    consent_overlay = WebDriverWait(driver, 15).until(
        EC.presence_of_element_located((By.ID, 'dialog'))
    )

    # select the consent option buttons
    consent_buttons = consent_overlay.find_elements(By.CSS_SELECTOR, '.eom-buttons button.yt-spec-button-shape-next')
    if len(consent_buttons) > 1:
        # retrieve and click the 'Accept all' button
        accept_all_button = consent_buttons[1]
        accept_all_button.click()
except TimeoutException:
    print('Cookie modal missing')

The WebDriverWait is used to look for a condition to occur; if there is no action, it will show a TimeoutException. 

5. Add Imports

It is essential to add imports to let the script run successfully:
from Selenium.webdriver.support.ui import WebDriverWait
from Selenium.webdriver.support import expected_conditions as EC
from Selenium.webdriver.common.by import By
from Selenium.common import TimeoutException

6. Start Scraping And Compiling

Now, look at the complete code for the .py file:
from selenium import webdriver
from Selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager
from Selenium.webdriver.chrome.options import Options
from Selenium.webdriver.support.ui import WebDriverWait
from Selenium.webdriver.support import expected_conditions as EC
from Selenium.webdriver.common.by import By
from Selenium.common import TimeoutException
import json

# enable the headless mode
options = Options()
# options.add_argument('--headless=new')

# initialize a web driver instance to control a Chrome window
driver = webdriver.Chrome(
    service=ChromeService(ChromeDriverManager().install()),
    options=options
)

# the URL of the target page
url = ‘PASTE YOUR URL LINK HERE'
# visit the target page in the controlled browser
driver.get(url)

try:
    # wait up to 15 seconds for the consent dialog to show up
    consent_overlay = WebDriverWait(driver, 15).until(
        EC.presence_of_element_located((By.ID, 'dialog'))
    )

    # select the consent option buttons
    consent_buttons = consent_overlay.find_elements(By.CSS_SELECTOR, '.eom-buttons button.yt-spec-button-shape-next')
    if len(consent_buttons) > 1:
        # retrieve and click the 'Accept all' button
        accept_all_button = consent_buttons[1]
        accept_all_button.click()
except TimeoutException:
    print('Cookie modal missing')

# wait for YouTube to load the page data
WebDriverWait(driver, 15).until(
        EC.visibility_of_element_located((By.CSS_SELECTOR, 'h1.ytd-watch-metadata'))
)

# initialize the dictionary that will contain
# the data scraped from the YouTube page
video = {}

# scraping logic
title = driver \
    .find_element(By.CSS_SELECTOR, 'h1.ytd-watch-metadata') \
    .text

# dictionary where to store the channel info
channel = {}

# scrape the channel info attributes
channel_element = driver \
    .find_element(By.ID, 'owner')

channel_url = channel_element \
              .find_element(By.CSS_SELECTOR, 'a.yt-simple-endpoint') \
              .get_attribute('href')
channel_name = channel_element \
              .find_element(By.ID, 'channel-name') \
              .text
channel_image = channel_element \
              .find_element(By.ID, 'img') \
              .get_attribute('src')
channel_subs = channel_element \
              .find_element(By.ID, 'owner-sub-count') \
              .text \
              .replace(' subscribers', '')

channel['url'] = channel_url
channel['name'] = channel_name
channel['image'] = channel_image
channel['subs'] = channel_subs

# click the description section to expand it
driver.find_element(By.ID, 'description-inline-expander').click()

info_container_elements = driver \
    .find_elements(By.CSS_SELECTOR, '#info-container span')
views = info_container_elements[0] \
    .text \
    .replace(' views', '')
publication_date = info_container_elements[2] \
    .text

description = driver \
    .find_element(By.CSS_SELECTOR, '#description-inline-expander .ytd-text-inline-expander span') \
    .text

likes = driver \
    .find_element(By.ID, 'segmented-like-button') \
    .text

video['url'] = url
video['title'] = title
video['channel'] = channel
video['views'] = views
video['publication_date'] = publication_date
video['description'] = description
video['likes'] = likes

# close the browser and free up the resources
driver.quit()

# export the scraped data to a JSON file
with open('video.json', 'w') as file:
    json.dump(video, file, indent=4)

Launch the above script. Then video.json file will appear inside your project's root folder.