Build a competitor-teardown agent with Claude Code (UK, 2026)

Validation has three legs. Demand - is anyone searching for this? You read that for free with keyword data. Willingness to pay - will they buy? You test that with a smoke-test landing page and pre-sale. The third leg is the competitive picture - and it does not stop mattering once you have built. Competitors move. The free tier you assumed was permanent disappears. A funded rival adds an enterprise plan and reshapes the market you just validated into.

You could pay GBP 100+/month for a competitor-monitoring SaaS that watches a hundred metrics you do not care about. Or you could build a small Claude Code agent that watches the three competitors who actually matter, extracts exactly the facts that move your decision, and costs pennies per run. This post is that agent, end to end.

What the agent does

Once a week it:

1. Scrapes each competitor's pricing and positioning pages.
2. Extracts the structured facts - tiers, prices, headline positioning, key features - with Claude.
3. Diffs this week against last week.
4. Alerts you when something semantically changed: a free tier removed, a price up 15%, a new enterprise plan, a repositioning.

It does not flag that "some bytes moved". It tells you, in a sentence, what changed and why you might care. That semantic-change detection is the entire value, and it is exactly what an LLM is good at.

The stack and what it costs

Three layers, all AI-native, all cheap:

• Firecrawl for scraping - renders JS-heavy pages, returns clean markdown. Free tier covers 500 pages/month; paid from ~GBP 16/month if you outgrow it. (Add it as an MCP server or call its API directly.)
• Claude for extraction and diffing - reliable structured JSON from messy pages, ~GBP 0.02-0.05 per page analysed.
• A flat JSON file or free Supabase table for the weekly snapshots.

Item	Cost (GBP/month)
Firecrawl (3-5 pages/week, free tier)	0.00
Claude extraction (~20 page-analyses/month)	~1.00
Storage (JSON file or Supabase free tier)	0.00
Total	under 5.00

Versus GBP 100+/month for a dedicated tool. For a solo UK builder watching a handful of rivals, the free tiers alone cover it.

Step 1: scrape the pages with Firecrawl

Firecrawl returns clean markdown, which is far easier for Claude to read than raw HTML. A small fetcher:

# scrape.py
import os, requests

FIRECRAWL_KEY = os.environ["FIRECRAWL_API_KEY"]
WATCHLIST = {
    "rival-a": "https://rival-a.com/pricing",
    "rival-b": "https://rival-b.co.uk/pricing",
    "rival-c": "https://rival-c.com/pricing",
}

def scrape(url):
    r = requests.post(
        "https://api.firecrawl.dev/v1/scrape",
        headers={"Authorization": f"Bearer {FIRECRAWL_KEY}"},
        json={"url": url, "formats": ["markdown"]},
        timeout=60,
    )
    r.raise_for_status()
    return r.json()["data"]["markdown"]

def scrape_all():
    return {name: scrape(url) for name, url in WATCHLIST.items()}

Three to five public pages, once a week. That is trivially light traffic - behave like a careful human analyst, just automated.

Step 2: extract structured facts with Claude

This is why you use an LLM and not a CSS-selector scraper. Competitor pages reshuffle their markup constantly; a brittle parser breaks the moment a rival redesigns their pricing table. Claude reads the page the way a person does and returns clean JSON regardless of structure.

# extract.py
import os, json, anthropic

client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])

SCHEMA_PROMPT = """Extract the pricing and positioning from this competitor page
as JSON with exactly these keys:
{
  "tiers": [{"name": str, "price_gbp": number|null, "period": "month"|"year"|null}],
  "has_free_tier": boolean,
  "headline_positioning": str,
  "notable_features": [str]
}
Convert any non-GBP prices to GBP and note the original in the tier name.
Return only the JSON, no prose."""

def extract(markdown):
    msg = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=1024,
        messages=[{"role": "user", "content": f"{SCHEMA_PROMPT}\n\nPAGE:\n{markdown}"}],
    )
    return json.loads(msg.content[0].text)

You now have a structured snapshot per competitor, in your own schema, ready to diff.

Step 3: diff against last week and detect semantic change

Store this week's snapshot, then ask Claude to compare it with last week's and report only what matters:

# diff.py
import json, anthropic, os

client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])

DIFF_PROMPT = """You are a competitive analyst. Compare LAST WEEK and THIS WEEK
for one competitor. Report only material changes a founder would care about:
price changes (with % and direction), a free tier added or removed, a new or
removed plan, a clear repositioning. Ignore cosmetic wording tweaks.
Output 0-3 bullet points, or 'No material change.' Be terse, EN-UK."""

def diff(name, last, this):
    msg = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=512,
        messages=[{"role": "user", "content":
            f"{DIFF_PROMPT}\n\nCOMPETITOR: {name}\n"
            f"LAST WEEK:\n{json.dumps(last)}\n\nTHIS WEEK:\n{json.dumps(this)}"}],
    )
    return msg.content[0].text.strip()

This is the step a string diff cannot do. "Rival A removed their free tier and introduced an Enterprise plan at GBP 290/year" is a sentence you act on. A byte-level diff would just shrug.

Step 4: alert and schedule

Wire the diffs into an alert - reuse the Resend pattern from the morning-brief agent, or a Slack chat.postMessage if you live there. Then schedule the whole pipeline weekly. The Linux scheduling matrix covers your options; a systemd timer or a plain cron entry is plenty for once a week:

0 8 * * 1 cd /opt/competitor-agent && /usr/bin/python3 run.py >> /var/log/comp.log 2>&1

Monday 8am, every week, a short brief lands telling you whether the competitive ground moved. Structure the scrape/extract/diff steps as a Claude Code subagent with its own context and tools if you want to run on-demand deep-dives alongside the weekly sweep.

The ethics, briefly

Scraping public pages for your own research is fair. Do it responsibly:

• Public pages only - never anything behind a login or paywall.
• Respect robots.txt and keep the request rate low. You are pulling a handful of pages once a week.
• Extract facts, do not republish. Prices, tiers, and positioning for your internal decision is research; copying their content is not.

Firecrawl respects these norms by default, which is another reason to use it over raw requests.

The validation step - do not skip week one

LLM extraction is reliable in 2026 but not infallible, and pricing is the field where an error costs you most. So:

• Week one: manually spot-check every extraction against the live page. Confirm each price.
• After week one: automate a spot-check on 10-20% of extractions - re-fetch a sample, compare.
• Always: treat a flagged semantic change as a prompt to look yourself before acting, not as gospel.

Once you have seen a clean fortnight, trust the weekly alert and only eyeball what it flags.

The point

The competitive picture is the leg of validation people check once and never again - right up until a competitor's move blindsides them. A GBP 5/month Claude Code agent turns "I should keep an eye on the competition" into a Monday-morning sentence that tells you when the ground actually moved. You own the code, you choose the watchlist, and you tune it to the questions that matter to your build. That is the difference between renting someone else's idea of what to monitor and owning yours.

---

New here? IdeaStack publishes one deeply researched UK business opportunity every Thursday - real keyword data, competitor analysis, builder prompts. See the latest free report.