fix: resolve /forex 404 by adding browser headers and KV fallback fetch - #751
Closed
irishmicoletcando wants to merge 1 commit into
Closed
irishmicoletcando wants to merge 1 commit into
irishmicoletcando wants to merge 1 commit into
Conversation
BSP's WAF was rejecting the Worker-originated scrape (403), leaving the FOREX_KV cache empty so /forex always 404'd. Add browser-like User-Agent/ Referer headers to the BSP fetch, and have /forex fall back to fetching fresh data directly (and repopulating KV) instead of hard-404ing on a cache miss.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Author
|
Closing in favor of a different approach — testing showed the BSP block is not header-based; it's Cloudflare Workers' fetch client itself being blocked (reproduced locally with zero custom headers still returning 403, vs 200 from plain curl on the same machine). No header change can fix that. Redesigning to scrape BSP from a GitHub Actions cron job instead, writing directly to the Cloudflare KV store via the Cloudflare API, bypassing the Workers runtime entirely. New PR incoming. |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
FOREX_KVnever got populated and/forexalways returned 404.User-Agent/Refererheaders to the BSP fetch infunctions/api/forex.tsso the upstream request succeeds.functions/forex.tsnow falls back to fetching fresh data directly (and repopulating KV) on a cache miss, instead of hard-404ing the client.Test plan
npx tsc --noEmit -p functions/tsconfig.jsonpasses/forexand/api/forexreturn live data once deployed (BSP headers depend on network conditions in production)