4.5Automation & web scraping
The essentials in 30 seconds
- ROI first: 30-min/week task + 2-hour script = paid off in one month; pathlib, openpyxl/pandas, logging in the toolbox.
- Static duo:
requests.getfetches (timeout, user-agent),BeautifulSoupparses (select); Selenium drives a browser for JavaScript. - Nabil's reflex: robots.txt + ToS BEFORE the first request, delays between requests, official API preferred over scraping.
Running mini-case — Nabil automates his price watch
Every Monday, Nabil, in Algiers, manually collects public prices for his competitive watch: 30 minutes of copy-paste. He writes a polite script: he reads the site's robots.txt and ToS, spaces his requests by 2 seconds with an identified user-agent, extracts prices with BeautifulSoup and stores them in CSV with logs. When the site offers an official API, he switches to it. Last milestone of the running thread: the analyst who won his Mondays back — cleanly.
Last case of the 4.1-4.5 running thread — Nabil completes his specializations.
1. Automate what pays off
Automate what is repetitive: files, spreadsheets (openpyxl/pandas), reports, emails — with pathlib for paths. The rule: compute ROI before coding (30 min/week versus 2 h of dev = profitable in one month). And make it reliable: logging to trace, targeted try with alerts, tests on copies (never on originals), then scheduling (cron, Scheduler) to run alone. Nabil tests his renaming on copies before touching the real folders.
2. Polite scraping: requests, BeautifulSoup, Selenium
For static pages: requests.get downloads (with timeout and identified User-Agent), BeautifulSoup analyses the HTML (select(".prix")), we store in CSV. Politeness: read robots.txt, space requests (time.sleep), never hammer — otherwise ban. For JavaScript and infinite scroll: Selenium drives a real browser, heavier. And the master rule: if an official API exists (or if login/ToS forbid it), use it — scraping comes last, never against the rules.
🔎 Concrete example — 🔎 Concrete example — Nabil's price watch
Nabil's polite scraper, with the real APIs:
r = requests.get(url, timeout=10, headers={"User-Agent": "veille-perso"})
prix = BeautifulSoup(r.text, "html.parser").select_one(".prix").text
logging.info("%s : %s", url, prix) # everything is logged
time.sleep(2) # politeness: space requestsReading: fetch, parse, log, wait.
Lesson: timeout + user-agent + delay = polite scraper.
Section vocabulary
- Automation / ROI
- Entrusting repetitive tasks to a script (files, spreadsheets, reports) after checking the saved time exceeds dev time.
Simple example: Nabil: 30 min/week saved for 2 h of dev, paid off in one month.
What it is not: automating a task done twice a year.
Reflex: compute ROI before coding.
- requests + BeautifulSoup
- The static-pages duo: requests downloads the page, BeautifulSoup analyses the HTML (select) to extract fields.
Simple example: Nabil extracts public prices with select_one and stores them in CSV.
What it is not: a driven browser for a simple static page.
Reflex: timeout + user-agent + spaced requests, always.
- Selenium
- Driving a real browser for JavaScript pages (loaded content, infinite scroll) — powerful but heavier.
Simple example: Nabil uses it when prices are not in the initial HTML.
What it is not: hammering requests on JS-generated content.
Reflex: Selenium only if static is not enough.
- robots.txt / ToS
- A site's access rules (robots.txt file, terms of use): what may be collected, at which pace — to read before the first request.
Simple example: Nabil reads both, spaces his requests and switches to the official API.
What it is not: bulk-scraping a site forbidding it.
Reflex: allowed + spaced + logged, or abstain.
In practice — In practice — Automate then scrape politely
- Choose ONE repetitive task, compute its ROI (time/week versus dev time) and automate it with pathlib/pandas.
- Add logging, targeted try with alerts, and test on copies — never on originals.
- Before any scraping: read ToS + robots.txt, look for an official API, and plan timeout, user-agent and delays.
- Scrape (requests + BeautifulSoup), store in CSV, schedule (cron/Scheduler) and monitor logs.
Ready-to-say script — justifying your collection (30 s)
“This is Nabil: ToS and robots.txt read before the first request, official API preferred, spaced requests with identified user-agent, everything logged. Our Mondays are automated — and our collection is defensible.”
Say before any scraper goes live; show ToS + logs.
Copy-paste template — polite scraper (tested code)
import csv
import logging
import time
import requests
from bs4 import BeautifulSoup
logging.basicConfig(level=logging.INFO)
URLS = ["https://example.com/page-1", "https://example.com/page-2"]
HEADERS = {"User-Agent": "veille-perso (contact : exemple@domaine.test)"}
# Prerequisite: read https://example.com/robots.txt + ToS,
# prefer the official API if it exists.
with open("prix.csv", "w", newline="", encoding="utf-8") as f:
out = csv.writer(f)
out.writerow(["url", "prix"])
for url in URLS:
r = requests.get(url, timeout=10, headers=HEADERS)
r.raise_for_status()
prix = BeautifulSoup(r.text, "html.parser").select_one(".prix").text.strip()
out.writerow([url, prix])
logging.info("%s : %s", url, prix)
time.sleep(2) # politeness: space requests
Office task
Renaming, reports, emails: pathlib + pandas + logging, tested on copies, then scheduled.
Occasional watch
Like Nabil: a few pages, spaced, logged — the script runs on Monday, nobody thinks about it.
JavaScript page
Content missing from HTML? Selenium drives a browser — heavier, reserved for that case.
Regular collection
Official API as soon as it exists: stable, allowed, faster than any scraping.
✓ Best practices
- Compute ROI before automating.
- Read ToS + robots.txt before the first request.
- Logs + tests on copies + spaced requests.
✗ Common pitfalls
- Scraping without reading ToS or robots.txt.
- Hammering with no delay (guaranteed ban).
- Launching a destructive script untested.
Key takeaways
- AUTOMATE profitable repetition: pathlib, openpyxl/pandas, smtplib — computed ROI.
- Static DUO: REQUESTS (downloads) + BEAUTIFULSOUP (analyses HTML).
- SELENIUM = driven browser for JS; OFFICIAL API preferred over scraping.
- POLITENESS: robots.txt, delays, user-agent; RELIABILITY: logs, test copies, scheduling.
48h challenge — apply tomorrow
Automate 1 repetitive task (computed ROI, tested on copies) OR scrape 2 allowed pages (ToS read) with delays + logs.
- I can name ROI, requests, BeautifulSoup, Selenium.
- I can explain ToS/robots.txt + delays.
- I have 1 reliable script or 1 polite collection running.
Frequently asked questions
Is web scraping legal?
It depends: the site's ToS and robots.txt, data rights (including personal data), collection pace — all frame what is allowed. Nabil's rules: read ToS + robots.txt before the first request, space requests, prefer the official API, never collect personal data without a legal basis, and get validation from a lawyer if in doubt or for professional use.
Is requests enough, or is Selenium needed?
requests + BeautifulSoup suffice when data is in the initial HTML (static pages): fast and light. Selenium becomes needed when content is generated in JavaScript (dynamic loading, infinite scroll): it drives a real browser, heavier and slower. Start static, move to the browser only if needed.
Which duo for scraping static pages?