Freshness windows: how recent does agent data need to be?

How recent does data for an AI agent need to be?

Match the window to how fast the source itself publishes, not to how fast you can poll. A severe weather alert is stale in minutes; a quarterly filing stays fresh for months. The useful question is not how real-time an API claims to be, but how much delay it adds on top of the source's own.

"Real-time" as a marketing word means almost nothing, because it is measured against nothing. Freshness is only meaningful as a comparison between when the issuing body published and when your agent could act on it.

Freshness is a property of the source

A data supplier cannot be fresher than what it reads. If an agency posts once a day, polling that agency every ten seconds produces one new record per day and 8,639 wasted requests. Conversely, if a source emits continuously, a supplier polling hourly is an hour stale no matter what its homepage says.

So the first thing to establish about any feed is the publishing rhythm of the body behind it. The SEC disseminates filings continuously and states a typical processing delay of under a second for its submissions API. USGS updates its earthquake feeds every minute. The Bureau of Labor Statistics publishes major releases on a schedule announced a year ahead, at a fixed time. Those three demand completely different polling strategies and completely different notions of "late".

Four freshness classes

Most agent tasks are Event class, and Event class is where over-engineering happens. A research agent summarising this week's enforcement actions does not need sub-second delivery. It needs to not be reading last month.

Scheduled data is the class people most often get wrong in the other direction. A CPI print does not need constant polling; it needs one call at a known time, because the number does not move again once published. That pattern is worked through in how agents get US economic data releases.

The three delays that stack

Total staleness is not one number. It is the sum of three, and suppliers typically quote only the second.

1. Source publish delay. From the event to the issuer posting it. Outside anyone's control and often the largest term. A federal award appears in public data long after the award decision. 2. Poll interval. From the issuer posting to your supplier noticing. This is the number that gets advertised. 3. Cache and delivery. From the supplier having it to your agent seeing it. A cheerful "updated every minute" with a fifteen-minute CDN cache in front is a fifteen-minute feed.

Worst-case staleness is roughly the sum of all three, and average is about half the poll interval plus the other two. An agent that needs a hard bound should work from the worst case.

How to check a supplier's real cadence

Ask for the observed interval, not the advertised one. Any supplier can put "real-time" on a page; far fewer will publish a timestamp of their last successful poll and let you compare it to the clock.

This service publishes both. The /health endpoint returns last_poll, an observed_interval_sec of 300, and a breakdown of which wires run on which cadence: 9 breaking wires on every cron run, 17 timely wires at most every 5 minutes, and 32 sparse wires at most every 15 minutes because the government and science sources behind them publish daily at most. The tiering is not a limitation being dressed up, it is the publishing rhythm of the sources.

# the supplier's own freshness, checkable before you buy anything
curl "https://thebotwire.com/health"

# freshness window as a query parameter, not a plan feature
curl -i "https://thebotwire.com/cve/latest?since=24h"

# free preview carries timestamps, so you can judge lag at zero cost
curl "https://thebotwire.com/quakes/preview"

Two habits follow from all this. Pass an explicit since window rather than fetching everything and filtering, so the freshness requirement lives in the request. And check timestamps in the payload rather than trusting the endpoint name, since the item's own publication time is the only figure that survives contact with a cache. The underlying reason an agent needs any of this is in why agents hallucinate recent events, and the general retrieval pattern is in how AI agents get real-time data.

FAQ

Is polling faster than my source publishes ever worth it?

Rarely. If a source posts daily, sub-minute polling buys you nothing and burns rate limit you may need during a genuine burst. The exception is bursty sources with a fixed release time, where polling tightens briefly around the announced moment and relaxes afterwards.

What is an acceptable freshness window for a research agent?

For most Event-class work, minutes. A research agent assembling an answer for a person is competing with that person's own reading speed, not with a trading system. Choose the window that makes the answer correct, then stop paying for more.

How do I tell whether a feed is stale or just quiet?

Compare the supplier's last successful poll time against the clock, not the newest item against the clock. An empty result from a poll thirty seconds ago means nothing happened. An empty result with no recent poll means the pipeline is down, and those look identical in the payload.

Does a bulk download count as fresh data?

Only within its republish cycle. The SEC's bulk ZIP archives, for instance, are republished nightly at roughly 3:00 a.m. ET, so they are the right tool for backfill and the wrong one for anything an agent needs today. Bulk for history, a live route for now.

Sources

Real-time dissemination behaviour, the sub-second submissions processing delay, and the nightly republish schedule for bulk archives are documented at SEC EDGAR application programming interfaces. The one-minute update cadence for earthquake feeds is stated in the USGS GeoJSON summary feed documentation. The poll interval, wire tiering and last poll timestamp for this service are published live at thebotwire.com/health.

Related: Why do AI agents hallucinate recent events, and how do you fix it? · How AI agents get real-time data past the training cutoff