IndexNow, Sitemaps, and Why Agentic Browsing Cared About llms.txt

TL;DR

• Sitemaps tell crawlers where pages live; llms.txt tells AI systems what to cite; IndexNow pings partners when URLs change. They belong in the same publish chain, not separate chores.
• Lighthouse Agentic Browsing exposed a real llms.txt bug (future-dated posts leaking in) that sitemap logic already handled. Fix the generator, not the prose.

When discovery files fall out of sync

Search crawlers, AI citation systems, and partner indexers each read a different discovery surface: sitemaps, llms.txt, and IndexNow pings. When those files disagree with what the live site shows, you ship the wrong map. Lighthouse's Agentic Browsing category now audits llms.txt structure, which is how a date-parsing bug in my generator showed up on the same scorecard as performance: broken discovery files, not a philosophy debate.

Who it is for: Founders, operators, and site owners who already publish on a static or Next.js stack and want search and AI discovery to reflect what they meant to ship. You do not need an agency retainer to get the basics aligned.

What you will learn: how sitemaps, IndexNow, and llms.txt fit one pipeline, what the Agentic Browsing llms.txt audit checks, the date-parsing lesson from my generator, and an honest limitation: IndexNow still returns 403 on my production key.

Three discovery files, three audiences

Screenshot: petralian.com/llms.txt excerpt — Petralian (2026)

Traditional SEO and GEO overlap but do not collapse into one file.

Google still wants Search Console URL Inspection for many sites. IndexNow does not replace Google. llms.txt does not replace seo_description. Each layer answers a different bot.

Photo: Pexels — Petralian (2026); search terms in frontmatter body_images slot 04

I documented the full vault-to-Vercel chain in how I built an SEO and GEO publish pipeline. This post zooms in on the discovery trio plus the Lighthouse lesson.

What Lighthouse Agentic Browsing checks on llms.txt

Screenshot: Lighthouse Agentic Browsing / llms.txt — Petralian (2026)

Chrome's Lighthouse Agentic Browsing audits include llms.txt recommendations: the file should be Markdown with at least one H1, readable structure, and guidance that matches the llms.txt convention.

That matters because:

1. AI discovery is no longer invisible. A failing llms.txt audit sits beside performance and SEO audits in the same run.
2. GEO structure is machine-testable. Vague marketing copy in llms.txt fails the same way a missing alt tag fails accessibility.
3. Regeneration must be deterministic. If your generator drifts from your sitemap rules, Lighthouse catches the lie.

Screenshot: Chrome Lighthouse — Agentic Browsing llms.txt — Petralian (2026)

I run npm run audit:lighthouse after meaningful publish batches. Agentic Browsing turned llms.txt from a nice idea into a release gate.

The date-parsing fix

petralian.com hides future-dated posts from the live site until their editorial date (calendar day in Asia/Hong_Kong). getAllPosts() in the app already enforced that rule. My early generate-llms-txt.mjs script did not.

Gray-matter parses YAML date fields into JavaScript Date objects. A string compare on 2026-08-04 works until the value is an object or an ISO string with timezone offset. The llms.txt list then included scheduled posts early. Crawlers and AI indexes saw URLs the homepage did not promote. That is worse than missing llms.txt: it is wrong llms.txt.

The fix mirrored src/lib/posts.ts:

1. Normalize every post date to a YYYY-MM-DD key in editorial timezone.
2. Filter with date <= today before writing article lines.
3. Regenerate llms.txt on every publish sync, not only on manual runs.

After the fix, Lighthouse llms.txt audit passed and the live file matched sitemap intent. The bug was not philosophical. It was parser parity between app and generator.

Screenshot: petralian.com repo — scripts/generate-llms-txt.mjs — Petralian (2026); redact if needed

For broader search shift context, see how LLMs are reshaping search and SEO.

IndexNow in the pipeline (and the 403 I still have open)

Screenshot: IndexNow response (keys redacted) — Petralian (2026)

IndexNow fits after deploy: POST changed URLs to api.indexnow.org with a key file hosted at https://petralian.com/{key}.txt. My request-indexing.mjs script prints Google Search Console inspection links because Google still lacks a simple public "index this URL" API for most publishers.

Honest limitation: production IndexNow calls still return 403 from api.indexnow.org even after the key file ships. That loop is open in my repo memory (memories/repo/open-loops.md). Possible causes I am still verifying:

Screenshot: scripts/request-indexing.mjs output — Petralian (2026); redact keys

I do not treat IndexNow as blocked forever. I also do not claim it works today. Sitemap plus GSC manual inspection remains the reliable path. IndexNow is a best-effort accelerator, not the spine of discovery.

GitHub Actions parity matters: publishing Obsidian drafts through GitHub Actions runs the same generator and indexing scripts so laptop and CI do not disagree.

How this relates to tools you already use

If you already run a CMS or static generator, you likely have sitemap generation. Add llms.txt as a sibling artifact, not a replacement:

• SEO tools (Semrush, Ahrefs) optimize pages humans click.
• llms.txt optimizes passages machines quote.
• IndexNow notifies partners of deltas.

Skipping llms.txt does not break Google rankings tomorrow. Shipping a wrong llms.txt after Lighthouse starts auditing Agentic Browsing is a measurable hygiene failure.

Limitations

llms.txt is not a ranking hack. Google has not replaced sitemaps with it. Some AI crawlers may ignore the file entirely.

Agentic Browsing audits are also Chrome-channel dependent. Pin your Lighthouse version when you benchmark.

My stack is Next.js on Vercel with Obsidian vault sync. Hugo, Astro, and WordPress need the same parity principle (app filter == generator filter) even if filenames differ.

Path A: wire discovery in one afternoon

Photo by Mohammad Hedayet Sarker from Pexels: https://www.pexels.com/photo/aerial-view-of-highway-interchange-in-tokyo-38751509/ Photo: TBD — Pexels "highway interchange aerial"; credit photographer; Petralian (2026)

1. Confirm https://yoursite.com/sitemap.xml lists only URLs you want indexed.
2. Add or regenerate public/llms.txt with H1, site sections, and per-article lines with seo_description or excerpt text.
3. Compare llms.txt article list to sitemap blog URLs. Mismatch means generator bug, not content bug.
4. Register an IndexNow key file at https://yoursite.com/{key}.txt.
5. POST one new URL after publish. Log status code. If 403 persists, keep GSC manual inspection and fix key hosting before you trust automation.
6. Run Lighthouse with Agentic Browsing. Fix llms.txt structure before you tune hero images.

Schedule llms.txt regen on the same hook as sitemap refresh so scheduled posts cannot leak early.

FAQ

What is the difference between sitemap.xml and llms.txt?

A sitemap lists URLs for classic search crawlers. llms.txt is a curated markdown index that helps AI systems find high-value passages without scraping your whole site. They serve different audiences and should stay in sync on which articles exist.

Does llms.txt replace SEO or sitemaps?

No. Google still relies on sitemaps and page metadata for ranking. llms.txt is a GEO layer for AI discovery and citation. Skipping it does not break rankings tomorrow; shipping a wrong llms.txt after Lighthouse audits Agentic Browsing is a hygiene failure.

What does Lighthouse Agentic Browsing check on llms.txt?

Chrome Lighthouse can audit llms.txt structure (H1, sections, link patterns) as part of Agentic Browsing. Treat it as a scorecard for machine-readable discovery, not a Google ranking factor.

Why might IndexNow return 403?

Common causes include key file not reachable at the exact keyLocation URL, host mismatch between POST body and served key, or API-side rejection without a detailed body. Until resolved, use Google Search Console URL inspection and sitemap refresh as the reliable path.

How often should I regenerate llms.txt?

Regenerate on the same hook as sitemap refresh—especially when scheduled posts sync early. Parser parity between your app date filter and generator must match or llms.txt will list articles search should not see yet.

What changed for me

Discovery is part of publish, not a post-publish memory task. Sitemaps, llms.txt, and IndexNow belong in one script chain with the same date logic as the app. Lighthouse Agentic Browsing gave me a scorecard reason to enforce that parity.

IndexNow 403 is still an open loop on my domain. I report it because honest limits beat pretending every automation line is green. Until that clears, I click GSC inspection links and keep llms.txt truthful. Next: match your llms.txt article list to sitemap URLs after every publish, then fix generator parity before you chase IndexNow status codes.