DES-17: sitemap, canonical tags, robots.txt and a real 404 [render preview] - #96
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
There was a problem hiding this comment.
🟡 Changes recommended
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR addresses SEO/indexing correctness for the docs site by ensuring every rendered page declares the correct canonical URL (always on https://docs.zerodev.app), generating a sitemap/robots.txt from the built output, and providing a styled 404 page for unmatched routes.
Changes:
- Add per-page canonical +
og:url(andnoindexfor unmaintained React hooks pages) viavocs.config.tsx. - Generate
docs/dist/sitemap.xmlpost-build and restorerobots.txt; add a standalone404.html. - Move non-page shared MDX/TSX out of
docs/pagesintodocs/componentsand repoint imports to the@componentsalias.
File summaries
| File | Description |
|---|---|
| vocs.config.tsx | Adds canonical + dynamic og:url, derives v5.3.x canonicals, and introduces @components alias. |
| scripts/sitemap.mjs | Adds build-derived sitemap generation based on redirect/noindex/canonical checks. |
| scripts/check-redirects.mjs | Adds validation ensuring v5.3.x pages map to current equivalents via redirects. |
| package.json | Runs sitemap generation after vocs build. |
| docs/public/robots.txt | Restores robots rules and points to the sitemap URL. |
| docs/public/404.html | Adds a styled standalone 404 page served for unmatched routes. |
| docs/pages/sdk/v5_3_x/signers/web3auth.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/signers/turnkey.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/signers/privy.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/signers/portal.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/signers/particle.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/signers/magic.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/signers/lit-protocol.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/signers/intro.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/signers/fireblocks.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/signers/eoa.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/signers/dynamic.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/signers/dfns.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/signers/custom-signer.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/signers/capsule.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/signers/arcana.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/presets/zerodev.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/presets/intro.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/permissions/transaction-automation.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/permissions/signers/passkeys.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/permissions/signers/multisig.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/permissions/signers/ecdsa.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/permissions/signers/build-your-own.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/permissions/policies/timestamp.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/permissions/policies/sudo.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/permissions/policies/signature.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/permissions/policies/rate-limit.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/permissions/policies/gas.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/permissions/policies/call.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/permissions/policies/build-your-own.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/permissions/intro.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/permissions/actions/build-your-own.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/permissions/1-click-trading.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/infra/zerodev.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/infra/pimlico.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/infra/intro.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/infra/coinbase.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/index.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/getting-started/tutorial.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/getting-started/tutorial-passkeys.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/getting-started/quickstart.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/getting-started/migration.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/faqs/use-with-react-native.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/faqs/use-with-gelato.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/faqs/use-with-ethers.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/faqs/debug-userop.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/faqs/chains.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/faqs/audits.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/core-api/using-plugins.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/core-api/sponsor-gas.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/core-api/sign-and-verify.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/core-api/send-transactions.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/core-api/pay-gas-with-erc20s.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/core-api/deploy-contract.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/core-api/delegatecall.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/core-api/create-account.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/core-api/batch-transactions.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/advanced/wallet-connect.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/advanced/supported-defi-tokens.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/advanced/supported-base-tokens.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/advanced/social-login.mdx | Update imports to @components (shared MDX + VersionWarning). |
| docs/pages/sdk/v5_3_x/advanced/session-keys.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/advanced/recovery.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/advanced/passkeys.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/advanced/parallel-orders.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/advanced/multisig.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/advanced/multi-chain-signing.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/advanced/key-storage.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/advanced/fallback-providers.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/advanced/defi.mdx | Update VersionWarning import to @components. |
| docs/pages/sdk/v5_3_x/advanced/chain-abstraction.mdx | Update VersionWarning import to @components. |
| docs/pages/onboarding/social-login.mdx | Update shared MDX import to @components. |
| docs/pages/advanced/react-hooks/use-switch-chain.mdx | Update shared MDX import to @components. |
| docs/pages/advanced/react-hooks/use-set-kernelclient.mdx | Update shared MDX import to @components. |
| docs/pages/advanced/react-hooks/use-session-kernelclient.mdx | Update shared MDX import to @components. |
| docs/pages/advanced/react-hooks/use-send-useroperation.mdx | Update shared MDX import to @components. |
| docs/pages/advanced/react-hooks/use-send-useroperation-with-session.mdx | Update shared MDX import to @components. |
| docs/pages/advanced/react-hooks/use-send-transaction.mdx | Update shared MDX import to @components. |
| docs/pages/advanced/react-hooks/use-send-transaction-with-session.mdx | Update shared MDX import to @components. |
| docs/pages/advanced/react-hooks/use-kernelclient.mdx | Update shared MDX import to @components. |
| docs/pages/advanced/react-hooks/use-disconnect-kernelclient.mdx | Update shared MDX import to @components. |
| docs/pages/advanced/react-hooks/use-create-session.mdx | Update shared MDX import to @components. |
| docs/pages/advanced/react-hooks/use-create-kernelclient-social.mdx | Update shared MDX import to @components. |
| docs/pages/advanced/react-hooks/use-create-kernelclient-passkey.mdx | Update shared MDX import to @components. |
| docs/pages/advanced/react-hooks/use-create-kernelclient-eoa.mdx | Update shared MDX import to @components. |
| docs/pages/advanced/react-hooks/use-create-basic-session.mdx | Update shared MDX import to @components. |
| docs/pages/advanced/react-hooks/use-balance.mdx | Update shared MDX import to @components. |
| docs/components/VersionWarning.tsx | New shared warning component (moved out of pages). |
| docs/components/shared/social-login-setup.mdx | New shared MDX snippet (moved out of pages). |
| docs/components/shared/query-result.mdx | New shared MDX snippet (moved out of pages). |
| docs/components/shared/mutation-result.mdx | New shared MDX snippet (moved out of pages). |
Review details
- Files reviewed: 92/96 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Tested on the Render preview
Canonical spot checks:
The last four One note on the preview itselfThe preview serves CorrectionMy first run reported 330 redirects wrong. That was the test, not the site: |
Every page now declares one absolute address on docs.zerodev.app. og:url was a fixed https://zerodev.app on all 249 pages; it follows the canonical now. The v5.3.x set credits its current equivalent, derived from the redirect table rather than a hand-written map, so the two cannot drift. The 20 unmaintained @zerodev/waas pages carry noindex. scripts/sitemap.mjs runs after the build and reads the built HTML, so the sitemap can never contradict what a page declares: 135 of 246 pages, with 22 redirected, 20 noindex and 69 old-version duplicates left out. Moves VersionWarning.tsx and the three shared snippets out of docs/pages, which vocs was turning into pages of their own.
The head block was a JSX element, and vocs checks typeof head === object before it checks for an element, so it took the path-prefix-map branch and rendered nothing. og:type, og:title and og:url never reached a page. Keeping our own og:type/og:title/og:description now would duplicate the per-page ones vocs emits, so drop them and keep the canonical, og:url and the noindex flag. Also records why /global-address takes two hops: a Cloudflare rule answers it before the request reaches this server.
72 of the 135 indexable pages carry no description, so Google writes its own snippet and a shared link previews bare. Vocs only emits one when the page supplies it, through frontmatter or the bracketed suffix on the H1. Collects the routes that already have one and fills only the gap, so no page ends up with two. A stopgap: one description across 72 pages is weak, and each page still needs its own (DES-25).
c4e60a6 to
9f04467
Compare
A page with no canonical tag was listed, which contradicts the rule the script states and hides the exact regression that already happened once: vocs drops the whole head config silently. Also trims the comments and drops the ticket numbers.
Retested on the Render preview
The description line is the new one: every indexable page now serves exactly one |
Part of DES-17
#95 has merged, so this now targets
main. It was rebased rather than merged, because #95 was squash-merged and its commits are not inmain's history.Problem
Google had no correct address for any page on this site.
docs.zerodev.appandnew-docs.zerodev.app, so Google could hold two copies of all 249 pages and split the rank between them.og:url.robots.txt, so nothing told Google which pages are the real ones.docs/pages, extension ignored, soVersionWarning.tsxbecame a 43 KB page with no text and the threeshared/*.mdxsnippets became pages of their own.Fix
docs.zerodev.app. This is what makes thenew-docscopy credit the real site instead of competing with it (DES-13 chose to leave that host alone). A path-only canonical would resolve against whichever host served the page and fix nothing.og:urlon every page, following the canonical.scripts/sitemap.mjs, run fromnpm run buildaftervocs build.robots.txtrestored fromaf0655c, pointing at the sitemap.404.html, a standalone styled page.server.mjsalready looked for this file and never found it.docs/pagesintodocs/components, and their 86 importers repointed at the existing@componentsalias.The
headblock invocs.config.tsxnever renderedWorth reading before reviewing the diff, because it explains why tags appear to be deleted.
headwas a JSX element. Vocs checkstypeof head === "object"before it checks for an element, and a JSX element is an object, so it took the "map of path prefix to element" branch, looked for a key matching the route, found none, and rendered nothing:So the
og:type,og:title,og:urlandog:descriptionin that block never reached a single page. The og tags in production today come from vocs itself, generated per page fromtitle,descriptionandogImageUrl.Making
heada function activates the block. Keeping the old contents would then have given every page twoog:titletags, vocs's per-page one and a hardcoded site-wide one, and the same forog:description. They are removed for that reason, not to drop metadata. Only the tags vocs does not emit stay: the canonical, theog:url, and thenoindexflag.Before and after, on
/get-started/quickstart:Descriptions
Vocs emits a description only when the page supplies one, through frontmatter or the bracketed suffix on the H1 (
# Social Login [Social login lets users...]). 72 of the 135 indexable pages do neither.This branch collects the routes that already have one and fills only the gap, so no page ends up with two:
A stopgap, not a fix. One description across 72 pages is a weak signal and Google may still write its own snippet. Each page needs its own, tracked in DES-24. The fallback uses the copy that was sitting in the dead
og:descriptiontag, so this is the first time it ships.The sitemap reads the build, not the sources
scripts/sitemap.mjswalksdocs/distand lists a page only when the page itself asks to be indexed: nonoindex, and a canonical pointing at itself.That means the sitemap cannot contradict the pages. A sitemap entry says "index this"; a
noindexor a canonical elsewhere says the opposite, and Google reports the conflict as an error. Deriving one from the other removes the chance of them disagreeing.A page with no canonical fails the build rather than being listed. That is the same failure mode as the
headbug above: vocs drops the whole head config without a word, and a sitemap that kept working while every canonical vanished would hide it. Verified by stripping the tag from one built page:@zerodev/waastree (DES-21).DES-14 predicted 221 URLs. It was written before DES-10 and DES-21 decided the old-version and unmaintained trees, and both of those decisions remove pages from the sitemap. 135 is the number those decisions imply.
The v5.3.x canonical map is derived, not written by hand
DES-10 chose to keep all 70 v5.3.x pages and point each at the current equivalent. Rather than hand-write 70 pairs:
/sdk/v5_3_x/xis the old version of/sdk/x, and every/sdk/xalready has a target inredirects.config.jsthat was checked page by page in #95. Reusing it means the canonical map and the redirect table cannot drift apart.69 of 70 map cleanly. Only
/sdk/v5_3_xitself has no equivalent and credits itself.This corrects a claim I made on DES-10: I guessed that
signers/capsule,signers/arcanaandsigners/web3authhad no successor. All three do (/onboarding/capsule,/onboarding/arcana,/onboarding/web3auth).npm run check:redirectsnow fails if a v5.3.x page stops mapping, because the silent failure mode is a page falling back to crediting itself and quietly competing again.Verification
Against a local
npm run build:On the Render preview for the earlier revision of this branch, which runs
server.mjs:npm run check:redirects:353 redirects ok, 246 routes.Found while testing production:
/global-addresstakes two hopsNot fixed here, because it cannot be fixed in this repository.
A Cloudflare rule answers
/global-addressbefore the request reaches Render, so the rule for it inredirects.config.jsnever runs. The response carries nox-render-origin-serverheader, while every rule we own does.Testing all 353 rules that way found exactly one in this state, so there is no wider hidden layer. DES-25 has the dashboard steps. A comment now records this above the rule, so nobody debugs it twice.
Not in this PR
Step 9, the ten wrong links in
zerodevapp/zerodev-website. That is a different repository and needs its own pull request. Split into its own ticket rather than dropped.Step 6, the
Hostcheck inserver.mjs. DES-13 decided to change nothing onnew-docs.zerodev.app. The canonical tags in step 1 solve the duplicate instead.Step 10, the link to the v5.3.x set on
/advanced/migration. Written, then removed at the author's request; that guide stays as it was. Worth knowing what this costs: the set is now reachable only from the navigation menu and from a direct link, and DES-10 kept those 70 pages on the promise that a reader on the old SDK could still find them. The canonical tags make Google credit the current pages either way, so the ranking side of DES-10 is unaffected. Reopened on DES-10 as a question about reader navigation, not as blocking work.No sidebar entries were added. The v5.3.x set and the react-hooks tree stay where they are.
Note on the preview
Vercel serves
docs/diststatically and never runsnpm start. The redirects and the 404 page do not apply on the preview URL. Production is the Render web service. Verify after merge againsthttps://docs.zerodev.app.