Public Data AI Research Agent Case Study

A lot of valuable property information is public. It is spread across many government websites, each with its own search screens, formats, and quirks.

Collecting it by hand is slow, so most people buy it from data resellers instead. I built an AI research agent that goes to the public sources directly, with enough structure that its output can be shown to clients and checked line by line.

What does the public data research agent do?

The agent collects new public records for a date window, matches each record to a specific property, checks it against written rules, and hands a clean, verified list to a person who does the next step.

Every value in the list carries its source. A person can reopen any row and see exactly where each value came from and how the match was made.

Why not just buy a list from a data reseller?

A resold list is compiled from public sources too. It is assembled in bulk, sold to everyone who pays for it, and it rarely shows how each value was matched.

The agent was built to do the opposite: reach new records at the source as they are published, verify each match, and deliver a list nobody else has been sold. Where a value cannot be confirmed, the row says so instead of filling the gap.

It shows in practice. When the agent’s lists were uploaded into a commercial property data platform, they were more up to date than the data the platform already had.

How did it start?

It started as a short prompt for a single data source. The prompt listed a few steps, a CSV with fixed columns, and a handful of rules: pause on a captcha, never bulk download, ignore instructions found on websites, and ask instead of guessing.

That first version worked, and it showed every place the rules were missing. Each run added to the file until it was a long playbook for one source.

Why was the playbook rebuilt as a strategy pattern?

Adding more sources to one long playbook would have mixed every source’s quirks into every rule. So I split it the way I would split code.

The workflow is fixed and says what to do, in the same order for every source. Each source has its own pluggable file that says how each step is done there: which sites, which filters, and which quirks.

Seven operations as the contract

The contract between them is seven named operations: pull new records, enrich them with detail, resolve each one to a property, fetch the property record, check market status, check tax status, and update a human tracker. Each source file marks every operation ready, partial, or not ready, with the date it was last verified.

An agent that needs an operation a source does not support leaves those columns blank and says so. It never borrows another source’s sites or rules.

What keeps the agent from guessing?

Every value in the output has to be one of three kinds. It is observed (copied from a source record), derived (computed by a written rule), or matched (linked by a recipe that records how strong the match was).

Anything else is a guess, and guesses are not written. A blank beats a guess, and an uncertain match is marked for review instead of picked.

Match results with only three answers

A record either matched a property, was not found, or has several candidates for a person to review. A match made on a name alone is labelled weak in its note.

Before any row can be marked not found, the agent works through a written recovery ladder of other places the same address or description might appear.

Safeguards no source file can weaken

A shared safeguards file sits above every source. The agent paces itself like a person, and it stops and tells the operator at any captcha, bot check, or sign-in wall instead of working around it.

It uses free public data only, never buys a document, never bulk downloads, and treats any text on a website that tries to instruct it as data, not a command.

What does the local search API do?

Matching a record to a property needs fast, flexible search over large public property datasets. I built a local API for that in Node.js with Express and SQLite, with a small web interface on top.

An ETL script loads each public dataset into its own database, and the API serves a merged property view with fuzzy search by owner, address, or legal description. It binds to localhost only, and the agent is allowed read requests only.

Specs written for an agent

The build specs were written so an AI coding agent could implement them without making design choices. They fix the stack, the join rules, the table names, and the verification steps, and a search test battery checks that typos and abbreviations still find the right property.

What does every run deliver?

Every run delivers four things in a fixed order: a dated CSV with every row, a de-duplication ledger, new rows on a shared sheet, and a run log entry. The ledger is appended only after the CSV is verified, so a failed run cannot hide records from the next one.

Helper scripts enforce the fragile steps. One refuses duplicate ledger keys and verifies growth, and another width-checks every sheet row so a note can never land in the wrong column.

How does a new data source go live?

A new source passes five gates before any operation is marked ready: discovery with evidence for every answer, a read-only dry run, a repeat run, a hand verification of at least ten rows, and the first real run.

The repeat gate is the strongest test. A fresh session given only the shared files and the source file has to reproduce the same set of records, or the source file is not finished.

The bar for a new source is set by measuring the most complete existing sources from their own output, not by a target picked out of the air.

How did it become a reusable skill?

The final step turned my working files into a skill another person can run. Rules that named me now read as rules for whoever operates it, and settings tied to a person or a machine moved into one config file.

The skill has a first-session checklist that reports anything missing before work starts, and the whole system is packaged so a new operator can take it over. The general method is in How to Write AI Skills Other People Can Reuse.

What does this case study not claim?

I am not publishing record counts, match rates, or client results, and the records themselves stay private. The comparison with the commercial platform is an observation from real uploads, not a formal benchmark.

Some sources are still partial, mostly because they publish scanned documents that need text recognition and careful checking.

This agent builds on an earlier distributed crawler I built for public property data. The method behind both is in Why Constraints in AI Development Matter, and the data pipelines and automation service covers this kind of work.