chore: reduce Contentful load from bot-driven ISR cache misses - #25
chore: reduce Contentful load from bot-driven ISR cache misses#25gsteenkamp89 wants to merge 1 commit into
Conversation
Three small changes to keep crawler traffic from spamming Contentful via ISR cache-key inflation, and to dedupe per-request fetches. - robots.txt → robots.ts: blocks `/?` query variants (search, tag, product, page, utm_*, anything). The canonical `/` and every /articles/[slug] stay crawlable. Also fixes a host typo by routing through SITE_BASE_URL. - Listing page (app/page.tsx): redirects unknown query params to the canonical URL and clamps out-of-range `?page=N` to the first page, so bots can't mint unbounded ISR entries with `?cb=12345` or `?page=99999` style URLs. - getBlogPostBySlug wrapped in React `cache()`: contentful 10.9.2 uses axios, so Next.js's automatic fetch dedup doesn't apply — cache() collapses the duplicate call between generateMetadata and the page render down to one Contentful request.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6954edf6c9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const hasUnknownParams = Object.keys(searchParams).some( | ||
| (key) => | ||
| !ALLOWED_SEARCH_PARAMS.includes( | ||
| key as (typeof ALLOWED_SEARCH_PARAMS)[number], | ||
| ), | ||
| ); |
There was a problem hiding this comment.
The whitelist redirect treats every unknown query key as invalid, but Next.js App Router issues internal requests with an _rsc query parameter for Flight/RSC payloads. With the current check, those framework requests are redirected to a URL without _rsc, which can cause extra 307 hops on every client navigation/prefetch and, in some deployments, broken route transitions or redirect loops when the server/CDN expects _rsc to be preserved. This regression is introduced by the new unknown-param canonicalization logic and should explicitly allow _rsc (or skip redirecting framework-internal requests).
Useful? React with 👍 / 👎.
2 participants
Summary
UMA blog turned out to be ~22% of total Contentful CDA volume on the shared space — confirmed by SDK fingerprint (
contentful 10.9.2 + Node 20) and by being the only consumer ofcontent_type=blogPost, the highest-frequency single query in support's report. Most of that volume is ISR cache misses from crawlers hitting parameterized listing URLs.This PR closes the same three gaps we just fixed in across.to#135 and the dapp PR:
What changed
app/robots.txtwas empty (Disallow:) and pointed at the wrong host (blog.uma.xyzwhileSITE_BASE_URLisuma.blog.xyz). Newapp/robots.tsdisallows/?so crawlers can't probe/?search=*,/?tag=*,/?product=*,/?page=*,/?utm_*, or anything else with query strings on the root. Articles at/articles/[slug]and the canonical/remain crawlable, and the dynamic sitemap continues to give Google a deterministic discovery path.app/page.tsxnow whitelistspage,search,tag,product,limit— anything else triggers a redirect to the canonical URL. Out-of-range?page=Nredirects to the first page. Together these prevent bots from minting unbounded ISR cache entries.getBlogPostBySlug. Wrapped in Reactcache(). The contentful SDK at^10.9.2uses axios, so Next.js's automatic fetch dedup doesn't apply — bothgenerateMetadataand the page component were independently hitting Contentful for the same slug. Cuts article-page renders from 3 Contentful calls to 2.Why now
Contentful support flagged limit pressure on the shared space; the across.to fix is in flight. The CSV showed
content_type=blogPost&fields.content[exists]=true&limit=10&order=-fields.publishDate(164 calls — exact match forgetBlogEntrieswith default args) as the single most-requested query, plus 1774 calls overall for this app. With ISR already in place (revalidate = 1800on/), the leak is unique URL variants forcing cache misses, not the canonical render itself.Test plan
pnpm buildsucceeds locally (verified —/robots.txtand/sitemap.xmlshow as static routes in the build output).curl https://<deploy>/robots.txtshowsDisallow: /?and the correctSitemap:reference./?utm_source=foo(or any unknown param) 307-redirects to/./?page=99999redirects to/(or to the canonical filtered URL if other filters are set)./articles/<slug>still renders correctly with metadata, image, and related articles.app/sitemap.tsand lists every published article.Related
🤖 Generated with Claude Code