Skip to content

Early responses to large uploads hang when the relay forwards a cloned request body #737

Description

@Luke-Rouse

Early upload rejection can stall

No credentials, account identifiers, project references, private content or actual photos are included.

Problem

An Edge Function returning an early error for a finite 2,097,153-byte upload can leave the HTTP client waiting until its own timeout. This prevents a bounded photo endpoint from delivering its HTTP 413 response. A minimal local function with no Auth, database, Storage or Flare code reproduces the issue. Even an immediate response without reading or explicitly cancelling the body can stall.

Environment and observations

  • Windows host with Docker Desktop; Supabase CLI 2.110.0.
  • CLI reports supabase-edge-runtime-1.74.2, compatible with Deno 2.1.4.
  • Normal CLI-generated relay, with no runtime patches, used for the failing tests.
  • Finite 2 MiB + 1 body: immediate return, awaited cancellation, unawaited cancellation and a streamed error response timed out after 5 seconds in the minimal reproduction. Connection: close on the response did not resolve it.
  • Fully consuming that same finite body before returning produced HTTP 413 in approximately 60 ms. This is a diagnostic control, not a proposed production mitigation: unrestricted draining would defeat the resource bound.
  • A bounded-drain control returned 413 for 2 MiB + 1 in 77 ms but a finite 4 MiB request timed out after 5 seconds when that drain budget was exceeded. This just moves the failure to a larger request.
  • The real authenticated photo endpoint independently times out after 20 seconds for both fixed-length and chunked 2 MiB + 1 requests. Exact 2 MiB reaches JPEG validation; ordinary photo upload and removal work.
  • Earlier hosted Beta checks showed the same two authenticated boundary timeouts. We cannot inspect the managed relay and do not claim its implementation has been proved identical to the local cause.

Local cause and counterfactual

The CLI-generated relay forwards new Request(t,e.clone()), leaving the original request branch unused. In Edge Runtime v1.74.2, UserWorker.fetch pipes the forwarded body to the user worker and waits for both the body-pipe promise and the response promise before releasing the response:

Exact upstream implementation.

When the worker stops reading early, piping tries to cancel its source. Cancellation of the cloned branch can remain pending while the other branch is unconsumed. A standalone Web Streams model reproduces that pending wait and releases it when the unused branch is cancelled.

An earlier isolated diagnostic removed request cloning only in the generated local relay; all 26 endpoint checks then passed with unchanged application handlers. That diagnostic change was restored and was never deployed. Direct Deno with the same application handlers also passed those checks. These controls isolate the local forwarding/cancellation interaction; they do not establish a supported hosted workaround.

Minimal reproduction

The index.ts included below is the exact local diagnostic function. It uses no credentials or database. Its drain modes exist only as controls and must not be deployed as a public function.

  1. In a disposable local Supabase project, put it at supabase/functions/photo-body-diagnostic/index.ts and configure [functions.photo-body-diagnostic] with verify_jwt = false.
  2. Use local API port 58121, start the local stack and run supabase functions serve. The probe refuses bodies larger than 5 MiB and targets only 127.0.0.1:58121.
  3. Wait until OPTIONS /functions/v1/photo-body-diagnostic returns 204. Do not edit watched function files during checks, because the CLI restarts the runtime.
  4. Run node transport-probe.mjs return cancel cancel-nowait drain response-stream. Default body size is exactly 2 MiB + 1.
  5. The early-response cases should promptly return 413; they instead time out. The finite full-drain control returns 413.
  6. The bounded-drain mode also demonstrates that limiting discard work moves the same failure to larger bodies. Set PHOTO_PROBE_BYTES=4194304 for that finite 4 MiB control.

Assistance requested

Please confirm whether the managed Edge Functions gateway has the same forwarding/cancellation issue, and provide either a provider fix or a supported way to deliver an early 413 while preserving strict byte/time bounds on rejected input. Increasing the accepted photo limit, unrestricted body draining, editing generated local runtime files, and relying solely on client-side validation are not acceptable production fixes for this endpoint.

index.ts (complete reproduction file)
// Local-only transport experiment. Never package or deploy this function.
Deno.serve(async (request) => {
  const mode = new URL(request.url).searchParams.get('mode') ?? 'return';
  if (request.method === 'OPTIONS') return new Response(null, {status: 204});
  const response = () => new Response('too large', {status:413,headers:{'Content-Type':'text/plain','Connection':'close'}});
  if (mode === 'return') return response();
  if (mode === 'cancel') { await request.body?.cancel(); return response(); }
  if (mode === 'cancel-nowait') { request.body?.cancel().catch(() => {}); return response(); }
  if (mode === 'drain') { const reader = request.body?.getReader(); while (reader && !(await reader.read()).done) {} return response(); }
  if (mode === 'bounded-drain') {
    let readBytes = 0;
    const reader = request.body?.getReader();
    while (reader) {
      const {done,value} = await reader.read();
      if (done) break;
      readBytes += value.length;
      if (readBytes > 2*1024*1024+65536) { await reader.cancel(); break; }
    }
    return response();
  }
  if (mode === 'drain-response') {
    return new Response(new ReadableStream({async start(controller) {
      controller.enqueue(new TextEncoder().encode('too large'));
      const reader = request.body?.getReader(); while (reader && !(await reader.read()).done) {}
      controller.close();
    }}), {status:413});
  }
  if (mode === 'response-stream') {
    return new Response(new ReadableStream({start(controller) {
      controller.enqueue(new TextEncoder().encode('too large'));
      controller.close();
    }}), {status:413});
  }
  if (mode === 'throw') throw new Error('too large');
  return response();
});
transport-probe.mjs (complete reproduction file)
import http from 'node:http';
const modes=process.argv.slice(2);
const byteCount=Number(process.env.PHOTO_PROBE_BYTES || (2*1024*1024+1));
if (!Number.isSafeInteger(byteCount) || byteCount<1 || byteCount>5*1024*1024) throw Error('Local probe size guard');
for (const mode of modes.length?modes:['return','cancel','cancel-nowait','drain','drain-response','response-stream']) {
  const started=Date.now();
  const result=await new Promise(resolve=>{
    const body=Buffer.alloc(byteCount);
    const req=http.request(`http://127.0.0.1:58121/functions/v1/photo-body-diagnostic?mode=${mode}`,{method:'POST',headers:{'Content-Type':'image/jpeg','Content-Length':body.length,Connection:'close'}},res=>{
      console.log(JSON.stringify({mode,headersAt:Date.now()-started,status:res.statusCode}));
      res.resume();res.on('end',()=>resolve({mode,status:res.statusCode,ms:Date.now()-started}));
    });
    req.on('error',e=>resolve({mode,error:e.message,ms:Date.now()-started}));
    req.setTimeout(5000,()=>req.destroy(new Error('timeout')));
    req.end(body);
  });
  console.log(JSON.stringify(result));
}

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions