Web scraping is a powerful way to keep an eye on competitor prices, promotions or any public catalog. But a good scraper isn’t just one that gets the data. It’s one that doesn’t hammer the site, plays by the rules, and tells you when it breaks. Here are the building blocks of a scraper in Python that actually holds up.
1. Read the rules first
Before you write a line of code, check the site’s terms of use and its robots.txt. If there’s an API or a data export, always use that instead. Checking robots.txt takes a few lines with the standard library:
from urllib import robotparser
rp = robotparser.RobotFileParser()
rp.set_url("https://www.example-store.com/robots.txt")
rp.read()
url = "https://www.example-store.com/category/dairy"
if not rp.can_fetch("PriceWatchBot", url):
raise SystemExit("Crawling this page is not allowed.")
delay = rp.crawl_delay("PriceWatchBot") or 2 # secondsIf you hit a CAPTCHA, a login wall or bot protection, don’t treat it as an obstacle to get around. Treat it as the site saying “no”, and stop.
2. Static pages only need requests
If the prices are already in the HTML, there’s no need to spin up a browser. requests plus BeautifulSoup is much faster and lighter. Send a User-Agent that identifies you, and wait between requests:
import time
import requests
from bs4 import BeautifulSoup
session = requests.Session()
session.headers["User-Agent"] = "PriceWatchBot/1.0 (+https://www.atilayberber.com)"
def fetch_page(url):
response = session.get(url, timeout=20)
response.raise_for_status()
time.sleep(delay)
return BeautifulSoup(response.text, "html.parser")3. JavaScript-rendered pages: Selenium 4
If products are loaded by JavaScript after the page opens, you need Selenium. The old find_element_by_* methods are gone in Selenium 4; everything goes through By now. Use an explicit wait instead of time.sleep. It’s both faster and more reliable:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options) # Selenium Manager finds the driver for you
try:
driver.get(url)
WebDriverWait(driver, 15).until(
EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".product-card"))
)
cards = driver.find_elements(By.CSS_SELECTOR, ".product-card")
for card in cards:
name = card.find_element(By.CSS_SELECTOR, ".product-name").text
price = card.find_element(By.CSS_SELECTOR, ".product-price").text
finally:
driver.quit()4. Parse prices properly
Turkish sites write prices like “1.249,90 TL”, with a dot for thousands and a comma for decimals. Throwing that at float either fails or quietly loses precision. Use Decimal for money:
from decimal import Decimal, InvalidOperation
import re
def parse_price(text: str) -> Decimal | None:
clean = re.sub(r"[^\d,\.]", "", text) # "1.249,90"
clean = clean.replace(".", "").replace(",", ".") # "1249.90"
try:
return Decimal(clean)
except InvalidOperation:
return None
parse_price("1.249,90 TL") # Decimal('1249.90')5. Write in batches
Running a separate INSERT and commit for every product is slow. Collect the rows and write them in one go. If you’re on SQL Server, pyodbc‘s fast_executemany makes a huge difference. And always close your connections and cursors:
import pyodbc
from contextlib import closing
with closing(pyodbc.connect(CONNECTION_STRING)) as conn:
with closing(conn.cursor()) as cursor:
cursor.fast_executemany = True
cursor.executemany(
"INSERT INTO dbo.Price (Source, ProductName, Price, ScrapedAt) VALUES (?, ?, ?, SYSDATETIME())",
rows,
)
conn.commit()6. Know when it breaks
- Log how many products each run picked up. If the number suddenly drops, the site’s HTML has almost certainly changed.
- Deduplicate by product code or URL so the same product doesn’t come in twice across pages.
- Keep your selectors (the CSS classes) in one config section instead of scattering them through the code. When the site changes, you fix it in one place.
Scraping is only the first step. The real value comes from writing that data to a database on a schedule and making the changes over time visible in a tool like Power BI.
Common mistakes
- Ignoring 429 responses: A “Too Many Requests” (429) response means the site wants you to slow down. Respect the
Retry-Afterheader and back off further on each retry instead of hammering the server. - Brittle selectors: Long XPath chains and auto-generated class names break on the first redesign. Prefer stable IDs,
data-*attributes or the structured data (JSON-LD) on the page. - Failing silently: Writing an unreadable price as
0corrupts your report. Log the failures separately and track their count. - Collecting personal data: Do not collect reviews, user names or other personal data. Product and price information is all you need for price monitoring, and personal data brings privacy law into play.
Frequently asked questions
Is web scraping legal?
This is not legal advice, but the general picture is this: pulling public data while following the site’s terms and robots.txt, without overloading it and without collecting personal data, carries far less risk. When in doubt, ask the site owner or use an official API if one exists.
How do I run the scraper on a schedule?
Task Scheduler on Windows or cron on Linux is enough. Log the start time, end time and row count of every run to a table so you spot problems early. You can then calculate price changes in SQL Server with LAG and visualize them in Power BI.
Related posts
- SQL Server Window Functions: calculating price changes with LAG
- Why Your Power BI Model Needs a Date Table
- Cleaning product names with RegEx in T-SQL
STAY IN THE LOOP
New guides, sorted by topic in a single email.
Every Monday, the week's new posts arrive grouped by topic. No new posts, no email.