Web Scraping
Web scraping is the automated collection of data from web pages: software fetches each page, reads its HTML and pulls out specific fields, such as prices, listings or published notices, into structured records. It is also called web data scraping. Unlike an official API, a scraper depends on page layouts the site owner can change at any time.
Key takeaways
- Software fetches pages, parses the HTML and extracts named fields into records
- Crawling finds pages; scraping pulls data out of them
- robots.txt rules, a clear User-Agent and low request rates are basic engineering norms
- An official API or a licensed feed is usually more stable when one exists
- Layout changes break scrapers quietly, so every run needs checks and alerts
What Web Scraping means
A scraper does a narrow job. It visits a known set of pages, finds the same elements on each one and turns them into rows: a product name and price, a court filing date, a list of clinic locations. The output is structured data that a database, a spreadsheet or an analytics pipeline can use.
Web scraping is often confused with crawling. A crawler follows links to discover pages, as a search engine does. A scraper extracts content from pages it already knows about. Many projects use both: a small crawler finds new listing pages and a scraper reads each one.
Agnotic Technologies builds data extraction software for clients who need public web data in their own systems. Before writing a scraper, we check whether the source offers an API or a data feed, and we ask the client's counsel to confirm the source can be used, because site terms and the law vary and that decision belongs to the client.
How does a web scraper work?
Most scrapers run the same six stages in order. Each stage is a place where a run can fail, so each one logs what it did.
- Fetch: request the page over HTTP with an identifying User-Agent, within an agreed request rate.
- Render: if the content is built by JavaScript, load it in a headless browser; otherwise skip this costly step.
- Parse: turn the HTML into a document tree the code can query.
- Extract: select each field with CSS selectors or XPath, then normalize dates, currencies and units.
- Validate: check each record against an expected schema and reject rows with missing or malformed fields.
- Store: write the records with the source URL and fetch time, so every value can be traced back to its page.
What does a responsible scraper do?
A responsible scraper reads the site's robots.txt before fetching, names itself in the User-Agent header with a way to contact its operator, and keeps its request rate low enough that the site barely notices the load.
When a server answers with HTTP 429 Too Many Requests, the scraper stops and waits, using the Retry-After value if one is sent, instead of retrying at once.
Robots rules are a request, not a lock, so honoring them is a choice the operator makes. Whether scraping a given site is lawful depends on its terms and the jurisdiction, and the client's counsel makes that call.
Why do scrapers break, and how do you keep them working?
Scrapers break when a site changes its layout. The worst failures are silent: the selector still matches something, so the job succeeds but writes empty or wrong values.
We guard against that with schema checks on every run, alerts when the record count or the share of empty fields moves outside its normal range, and saved copies of sample pages, so a fix can be tested against both the old and the new layout.
How do site owners see scraping?
From the site owner's side, scraping is an automated threat to plan for. The OWASP Automated Threats to Web Applications project lists it as OAT-011, next to other bot activity such as credential stuffing.
This matters when Agnotic builds a client's own portal or public API. Rate limits, bot detection and clear terms of use protect the data the client publishes.
Web scraping, an official API or a licensed data feed: which fits?
All three get outside data into your systems. They differ in who controls the format and in how they fail.
| Option | What you get | Who controls changes | How it breaks | Choose it when |
|---|---|---|---|---|
| Web scraping | Whatever the page shows, in the shape the page shows it | The site owner, without notice to you | Layout changes, blocking or rate limits, often silently | No API or feed exists and the data is public |
| Official API | Documented fields in a stable format, usually behind authentication | The provider, through versioned releases and deprecation notices | Retired versions, quota limits or revoked keys, with errors you can catch | The provider offers one that covers the fields you need |
| Licensed data feed | A bulk file or stream under contract, often with history | The vendor, under the terms of the license | Late or incomplete deliveries, which the contract covers | You need full coverage, history or the right to redistribute |
Facts and primary sources
RFC 9309 standardizes the Robots Exclusion Protocol, the robots.txt rules that crawlers are requested to honor, and states that these rules are not a form of access authorization.
Source: IETF RFC 9309, Robots Exclusion ProtocolRFC 6585 defines HTTP status 429 Too Many Requests for a client that has sent too many requests in a given time, and says the response may include a Retry-After header saying how long to wait.
Source: IETF RFC 6585RFC 9110 defines the User-Agent request header, which carries information about the client software making the request.
Source: IETF RFC 9110, HTTP SemanticsThe OWASP Automated Threats to Web Applications project lists scraping as OAT-011: collecting application content or other data for use elsewhere.
Source: OWASP Automated Threats to Web Applications
Explore this on Agnotic
Where Web Scraping shows up in how we build.