Repository navigation
fix(seo): stale facts + AI crawler grouping in robots.txt - #361
Merged
Merged
Conversation
- robots.ts: split AI crawlers into answer-engine/search (OAI-SearchBot, Claude-SearchBot, PerplexityBot), on-demand fetchers (ChatGPT-User, Claude-User, Perplexity-User), and training/extended-use (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended) groups so the policy matches its comments - TuiBoardMockup: replace the July 6 live-board snapshot (real operator names, stale ranks/yields) with clearly synthetic example rows - /about, page-faqs, yield-cascade, benchmark blog: drop the incorrect "yield maps to class tier / >1000 = upper tiers" claim; class tier is total-token based. Adopt the canonical rough yield bands - /faq: refresh Last updated date; replace stale ccusage-paste workflow with the canonical npx sigrank -> enroll -> submit flow - page-faqs: local agent exposes 30+ tools (was 25) Gates: tsc 0 errors, canonical 11/11, touched lint clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
robots.tsregroups AI crawlers by function: answer-engine/search (OAI-SearchBot, Claude-SearchBot, PerplexityBot), on-demand fetchers (ChatGPT-User, Claude-User, Perplexity-User), and training/extended-use (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended — kept at parity with*, no policy change). Comments now match actual behavior.TuiBoardMockupreplaces the hardcoded July-6 board snapshot (real operator names, stale ranks/yields) with clearly synthetic example rows — removes contradiction with the live board and the real-name opt-out scrubbing liability./about,page-faqs.ts,/metrics/yield-cascade, benchmark blog, and legacy board prose. All now use the canonical rough yield bands./faq: refreshed Last updated date, replaced stale ccusage-paste workflow with canonicalnpx sigrank→enroll→submit, added enroll step to "How does SigRank work" answers, corrected local-agent tool count (25 → 30+).Test plan
npx tsc --noEmit— 0 errorsnpm run test:canonical— 11/11 passGenerated with Devin