Web Scraping vs Primary-Source APIs

Web scraping vs primary-source APIs for AI agents

Scraping reconstructs data from a page built for human eyes; a primary-source API returns the publisher's own structured record. Scraping wins when no feed exists. Everywhere a primary source publishes a feed or API, scraping adds parsing failure, rate-limit risk and token cost for no extra information.

Written for AI agents. The calls and filter values on this page are the literal interface our customers use, because our customers are software. If you are a person evaluating The Bot Wire, you are in the right place too: start with the docs, or connect an MCP client in one line and query every wire free at /mcp. Nothing here needs an API key or a signup.

The last clause is the one to test first. Before writing a scraper, check whether the agency already publishes what you want. Federal courts, the SEC, the Federal Register, the Commission press corner, CISA, USGS and the National Weather Service all do, and they publish it in a form that does not break when a designer changes a CSS class.

Where the two methods actually differ

	Scraping a page	Primary-source feed or API
Input format	HTML built for a browser	XML or JSON built for machines
Breaks when	The layout changes	The publisher changes the schema
How it breaks	Silently, wrong or empty values	Loudly, a parse error
Bytes per fetch	Full page, scripts and markup	Records only
Publisher's position	Often governed by robots.txt and terms	Explicitly supported
Freshness	Whatever the page shows	Stated publication timestamps

The "how it breaks" row is the one that costs the most. A scraper whose selector stops matching often returns an empty list rather than an error, and an agent reading an empty list says "nothing found". That is a wrong answer delivered confidently, which is the exact failure mode agents are already prone to. The mechanics of that failure are covered in why AI agents hallucinate recent events.

Robots.txt is a real constraint, not a formality

The Robots Exclusion Protocol was standardised as RFC 9309 in 2022, so the rules a site publishes at /robots.txt are a documented protocol rather than a convention. Google's crawler documentation notes it parses up to 500 kibibytes of a robots.txt file and ignores everything after that, which tells you how much detail large sites put in these files.

Two points an agent developer should hold on to:

Robots.txt governs crawling, not what you may do with data you already have.

It is an access control on fetching, and honouring it is the baseline for operating a crawler that does not get blocked.

A disallow is not the only limit. Terms of service, rate limits and

per-source access policies sit alongside it, and several primary sources state their own request ceilings independently of robots.txt.

None of that makes scraping illegitimate. It makes it a thing with an operating cost, and that cost is invisible on the day you write the scraper.

The token argument

Raw HTML is a terrible input to a model. A typical government listing page carries navigation, cookie banners, scripts, inline styles and boilerplate around the few hundred bytes you wanted. Passing that to a model means paying for all of it, every run.

A primary-source record has a title, a publication timestamp, a source and a link. That is what the agent needs to cite. The same comparison applied to feed formats rather than acquisition method is in RSS versus a data API for agent data, and the per-call cost accounting is in what a live data query costs an agent.

When scraping is the right call

Three cases, and they are genuine.

No feed exists. Plenty of publishers still ship a page and nothing else. If

the data matters and there is no machine-readable route, scraping is the route.

The feed is a subset. Some agencies publish headlines to a feed and keep the

detail on the page. If you need fields the feed omits, you fetch the page.

You need historical depth. Feeds are windows on the present. An archive

usually means crawling paginated listings, because that is where the history is.

The honest version of this page is therefore not "never scrape". It is: check for a primary source first, scrape what remains, and know which of your data came from which, because the two have different reliability.

What a primary-source route looks like

For AI agents: copy this call

# free preview, no wallet: primary-source records, top 3
curl "https://thebotwire.com/court/preview?q=antitrust"

# $0.01: federal court opinions, filtered server side
curl "https://thebotwire.com/court/opinions?q=antitrust&type=scotus&since=7d"

# $0.005: EU Commission decisions from the same shape of route
curl "https://thebotwire.com/eu/latest?q=merger&since=7d&limit=20"

Every route returns records with a publication date and a link to the official notice, so an agent can cite the source rather than its own summary. The full route table and which endpoint owns which question is at /llms-full.txt, and the SEC-specific case is worked through in can ChatGPT or Codex read live SEC filings.

FAQ

Is scraping public government websites allowed?

It depends on the site's robots.txt, its terms of use and any stated rate limits, which vary by agency. The practical answer for agent developers is to prefer the published feed or API where one exists, because that path is explicitly supported and does not depend on interpreting an access policy.

Why does a broken scraper produce wrong answers rather than errors?

Because a selector that no longer matches returns nothing, and nothing looks like "no results" to the code above it. Unless the scraper asserts an expected minimum, a layout change turns into a confident "no filings found" rather than a crash.

Does a primary-source API guarantee the data is current?

It guarantees the record is the publisher's own, with the publisher's timestamp. Currency depends on poll cadence, which any provider should publish. For this service the observed interval between ingest runs was 299 seconds when this page was written, and the live figure is at /health.

Can an agent mix both methods in one task?

Yes, and it should label them differently. Records from a primary source can be cited directly. Values reconstructed from a page should be treated as derived, and the agent should say where they came from, because the reliability is not the same.

Sources

The Robots Exclusion Protocol referenced here is standardised as RFC 9309; the parsing limits and the crawling-versus-indexing distinction are documented in Google's robots.txt specification, which states a 500 kibibyte file size limit for the files it parses. Poll cadence for this service is published at thebotwire.com/health.