Some pages build their content with JavaScript after loading — requests only sees the empty shell. Selenium drives a real browser, so you scrape what a user actually sees.

Setup (Selenium 4.6+)

Terminal

pip install selenium beautifulsoup4

Scrolling a dynamic page

Python

import time
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("--headless=new")        # no visible window
driver = webdriver.Chrome(options=options)    # Selenium Manager finds a matching driver
try:
    driver.get("https://quotes.toscrape.com/scroll")
    last = driver.execute_script("return document.body.scrollHeight")
    while True:                               # scroll until the page stops growing
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight)")
        time.sleep(1.5)
        height = driver.execute_script("return document.body.scrollHeight")
        if height == last:
            break
        last = height
    soup = BeautifulSoup(driver.page_source, "html.parser")
    print(len(soup.find_all("div", class_="quote")))
finally:
    driver.quit()                             # always close the browser

Waiting for elements

Python

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# Better than time.sleep: wait up to 10 s, continue as soon as the element appears.
first = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.CSS_SELECTOR, "div.quote span.text"))
)
print(first.text)

Selenium	Does
`driver.get(url)`	Open a page
`driver.find_element(By.CSS_SELECTOR, …)`	Find one element (`find_elements` for all)
`element.click()` / `.send_keys("text")`	Interact like a user
`driver.execute_script(js)`	Run JavaScript (e.g. scroll)
`driver.page_source`	The current HTML — hand it to BeautifulSoup
`driver.quit()`	Close the browser (use `try/finally`)