Give it one URL, and this is what it actually produces — not a JSON file. An ad.

generated end to end from a single input URL — this scraper's output, fed straight into GliaCloud's internal ad-production tools, no manual editing in between

This pipeline exists to make money, not to make scrapers — the business makes ads for other companies' webpages, and the video above is what it actually makes. Everything below is how, end to end.

A domain that used to cost an engineer half a day to a full day — reading the page, writing selectors field by field, handling the edge cases, testing across a handful of real articles — now takes about five minutes, unattended, start to finish.

measured: run_all_openai_log.txt · avg 297s, range 156–763s, n=27 successful generations

The bigger saving isn't the first five minutes, though — it's the ones that never happen. A hand-written scraper that breaks needs a person to notice, diagnose, and rewrite it. This one notices itself, on every call and again every day — the mechanics are in Exhibit G, further down.

A single GitHub issue is where all of this starts. The map below follows every path from there — a new domain becoming a working scraper, a broken one healing itself, and the two separate checks running underneath, catching drift before anyone has to go looking for it.

flowchart TD
    T1["GitHub issue: scrape<br/>(new domain)"]
    T2["GitHub issue: regenerate<br/>(existing domain, broken)"]
    T3["GitHub issue: story-generation<br/>(new source URLs)"]
    T4["Daily health check<br/>(scheduled, not issue-driven)"]
    T5["Production /scrape request<br/>(external caller)"]

    T1 --> EXIST{"Scraper already exists for this domain?"}
    T3 --> EXIST
    EXIST -- no --> S0
    EXIST -- yes --> S3

    S0["Step 0 — fetch<br/>playwright/chromium → zyte fallback"]
    S1["Step 1 — read<br/>pluckr LLM tool loop<br/>hardcoding + schema checks"]
    S2["Step 2 — write<br/>generate bridge scraper,<br/>commit to repo"]
    S3["Step 3 — run<br/>execute scraper on URL(s)"]

    S0 --> S1 --> S2 --> S3
    T5 --> S3
    S3 --> OUT["Structured JSON output"]

    OUT --> TYPE{"Per-call type check passes?"}
    TYPE -- yes --> RESP["Return data to caller"]
    TYPE -- no --> AI1["Auto-file regenerate issue"]
    AI1 --> T2

    S2 --> BRANCH{"Which issue triggered this run?"}
    BRANCH -- scrape --> AM["PR auto-approved & merged<br/>(nothing to lose)"]
    BRANCH -- regenerate --> HR["PR waits for human review"]
    BRANCH -- story-generation --> VID["Call video production API<br/>(poll up to 20 min)"]
    VID --> POST["Post finished video link<br/>back to the issue"]

    T4 --> LOG["Query Cloud Logging:<br/>domains scraped, past 7 days"]
    LOG --> RS["Re-scrape each via<br/>production API"]
    RS --> CMP["Compare vs baseline<br/>saved at scraper creation"]
    CMP --> SCORE{"Weighted score ≥ 90%?"}
    SCORE -- yes --> OK["Pass — no action"]
    SCORE -- no --> AI2["Auto-file regenerate issue"]
    AI2 --> T2
      

Two different failures feed back into the same place — a failed per-call type check and a failed daily health check both land on a regenerate issue, the one path in this whole diagram that waits for a human.

The business makes ads for other companies' webpages — you just saw one. A good ad starts with actually knowing what's on the page it runs next to: the headline, the body, the cover image, the category. Getting that content means reading the page first, off whatever CMS the publisher happens to run.

Do that for one domain and it's a Tuesday afternoon of writing CSS selectors. Do it for every domain an ads business actually needs, and two problems show up that no amount of elbow grease makes go away: there is no one scraper that fits every website, and there is no scraper that stays right on one website forever. Every domain hides its byline in a different unlabeled <span>. Every domain, eventually, redesigns the page a working scraper was built for.

The LLM is the apprentice who learns the trade on day one, once per domain — that's the first problem handled. After that it clocks out, and a plain, deterministic script does the job — watched over for as long as that page keeps changing underneath it, which is the second problem, and the reason this report doesn't stop once the scraper is written.

Before any of this ran in production, something had to decide which model does the reading in Step 1. A handful of candidates got tried early on — GitHub Copilot, Claude, and GPT among them — before the field narrowed to a real head-to-head: Claude Sonnet 4.5 against GPT-5.2-codex.

The test: 100 domains, three separate pages each — a fixed exam, graded against hand-verified ground truth using the same weighted rubric that scores every scrape today (Exhibit F) — content worth 40% of the score, title/images/keywords/categories at 10% each, videos/tags/cover_image at 5%, use_contained_media at 3%, subtitle at 2%.

A score above 75% passed. Everything below that got sorted into what specifically went wrong — missing content, a wrong URL, stray ads pulled into the body, bot detection, and so on.

No tradeoff to argue about: GPT was both more accurate and cheaper. It became the production model. Generation has since moved forward again, to gpt-5.4.

Every URL that comes in runs the same four-step routine — but for any domain that's been seen before, three of those four steps are skipped entirely.

Step 1 can run two ways. Shown the correct answer for one sample article — ground truth — the model converges faster and generally more accurately: a guided apprentice. Shown nothing but the schema, it has to reason its way to a selector from scratch: slower, but the result is honest, because there was never an answer key to peek at.

That's the loop working as intended. It doesn't always go this cleanly.

With the right answer sitting right there, the fastest way to pass validation isn't to find the byline in the DOM — it's to write the answer as the answer. A transform that just returns "ja" technically produces the correct output for the one page the model is looking at. It also produces "ja" for every article on that domain forever, including the English ones.

{
  field: "language",
  selector: ".lang-flag",
  transform: "'ja'"
}

{
  field: "language",
  selector: ".lang-flag",
  transform:
    "/[ぁ-ん]/.test(value)
      ? 'ja' : 'en'"
}

The fix wasn't a policy — it was a referee built into the loop itself. Every submitted transform is scanned for pure literals that never touch the actual scraped value. Catch one, and the model doesn't get a pass — it gets the answer bounced back with a written objection, inside the very same conversation, and has to try again:

When a domain still comes back hardcoded after that, the fallback is blunter: run it again with no answer key at all, and let the model earn a real selector the hard way.

The hardcoding check is only half the gate. Every submission also has to clear a schema check — content is a non-empty string, language matches a real language-code pattern, images and videos are arrays of URL-shaped strings, and so on. A selector can be perfectly honest and still get bounced back for returning the wrong shape.

None of this runs forever. The tool-call budget isn't per field — it's one shared pool sized to the schema: three tries times however many fields are being extracted, so a 6-field page gets up to 18 tool-calls total, and a field that lands cleanly on the first try leaves more room for a stubborn one. There are exactly three ways the loop ends:

That last one exists so a domain that keeps failing stops quietly burning API calls on the sixth, seventh, and eighth attempt — it fails fast and loud instead.

Laid out end to end, from a cleaned page to a committed scraper file:

flowchart TD
    START["Cleaned HTML + target schema"]
    MODE{"GT-guided or no-GT mode?"}
    START --> MODE
    MODE -- "default: GT-guided" --> LOOP
    MODE -- "fallback: still hardcoding<br/>after retries — rerun manually" --> LOOP

    LOOP["Tool loop begins<br/>shared budget = 3 × field count"]
    CB{"4 consecutive failed<br/>attempts on this domain?"}
    LOOP --> CB
    CB -- yes --> PERM["PERMANENT_FAILURE<br/>skip the LLM entirely"]
    CB -- no --> CALL["Model calls a tool"]

    WHICH{"Which tool?"}
    CALL --> WHICH

    TEST["testSelector:<br/>try a CSS selector against<br/>the real rendered page"]
    SHOW["Model sees the actual<br/>extracted value"]
    WHICH -- testSelector --> TEST
    TEST --> SHOW
    SHOW --> BUDGETCHECK

    HONEST["reportNoData:<br/>field genuinely not on this page"]
    WHICH -- reportNoData --> HONEST
    HONEST --> FIELDSDONE

    HC{"Hardcoding check:<br/>pure literal, no real logic?"}
    WHICH -- submitResult --> HC
    BOUNCE["Bounced back with a written<br/>error, same conversation"]
    HC -- caught --> BOUNCE
    BOUNCE --> BUDGETCHECK

    SCHEMA{"Schema check:<br/>Zod types valid?"}
    HC -- clean --> SCHEMA
    SCHEMA -- no --> BOUNCE

    FIELDSDONE{"Every field resolved?"}
    SCHEMA -- yes --> FIELDSDONE
    FIELDSDONE -- no --> BUDGETCHECK
    FIELDSDONE -- yes --> SUCCESS["Selectors + transforms JSON"]

    BUDGETCHECK{"Tool-call budget remaining?"}
    BUDGETCHECK -- yes --> CALL
    BUDGETCHECK -- no --> FAILED["EXTRACTION_FAILED"]

    SUCCESS --> BUILD["generate_bridge_scraper.py:<br/>selectors JSON → Selenium scraper"]
    BUILD --> CHROME["Auto-detect installed Chrome version,<br/>pin version_main to match"]
    CHROME --> COMMIT["Commit scraper_domain_final.py —<br/>this domain's scraper, from now on"]
      

The GT → no-GT fallback is a human rerunning a different script, not something the loop switches to on its own — worth being precise about, since everything else on this diagram happens without a person watching.

Two things that went wrong while building this, both about the scraping itself, not the plumbing around it.

This is the check that compares a scrape against a saved baseline and reduces it to a single weighted number — the one the daily health check runs on every known domain (Exhibit G), and the one the model bake-off used back in Exhibit B. Not every field counts the same.

A scraper that nails the headline and mangles the article body has failed, full stop — which is why the body text alone outweighs the next four fields combined.

Ninety percent is the bar for staying in production unattended. Below it, a site doesn't get a shrug — it gets a repair request filed against itself, automatically. Which is Exhibit G.

Nothing here runs by a human clicking "scrape." It runs by someone filing a GitHub issue — and by two separate watchdogs, on two different clocks, that never stop checking on the rest.

A scraper degrades quietly on some Tuesday — a redesign, a new bot check, a moved byline. One of the two watchdogs notices before a person does, and by the time anyone looks, there's already a pull request waiting for review.

A third loop, Story Generation, chains the same scraper step into an entirely different job: turn a list of source URLs into a produced video story, polling an external production API for up to twenty minutes and posting the finished link back on the issue that requested it.