Dmytro Galko · open-source tools

web-scraper-toolkit

Name: web-scraper-toolkit
Author: Dmytro Galko

A paginated listing to a clean CSV, built the way a paid job is

Most scraping scripts work on the author's machine and fail on the fifth page: no pacing, no retry, no idea what to do with a row that is missing a field. This is the same job done the way a paid one has to be — request pacing, retries with exponential backoff, user-agent rotation, field normalisation, deduplication — and its tests run without touching the network.

Source on GitHub Work with me

What it does

Politeness is a setting, not an afterthought: pacing and rotation are on by default, because a blocked scraper returns no data at all.
A row that does not parse is reported, not silently dropped — a CSV that is quietly short is worse than one that says what is missing.
The tests run offline against recorded pages, so the suite does not depend on a site nobody here controls.

How it is proved

21 tests pass without a network. Targets are the two public sandboxes published for scraping practice.

Pythonoffline tests21 testsCSV output

People find this looking for

polite web scraper cli python, retries exponential backoff user agent rotation, paginated listing to csv, deduplicated scrape output, scraper tests without network.

If that is your week and you would rather someone else did it, write to hello@dkautomation.dev or open an issue.