Lesson 2 — Why the reserved-URL path can't stream

Grounded in a production incident · media-svc · streaming OOM fix
← Lesson 1: io.Pipe & io.LimitReader · Reference cheatsheet →

1. The setup: two upload paths in a production incident

a production incident introduced streaming download→upload for videos so the pod never buffers an entire 80 MB file in memory. But the fix only worked on one of the two upload paths. Let's look at both.

The reserved path is used when the caller pre-allocates a GCS object with a known destination URL and a signed upload policy (e.g. for CDN pre-warming or exact-key uploads). It calls DownloadFile (returns []byte) then UploadToReservedURL.

2. Constraint 1 — the httpkit API itself

UploadToReservedURL calls request.PostFormWithContext. That function's File struct looks like this (from the actual httpkit source):

// httpkit/request/request.go — File struct
type File struct {
    Key      string
    Content  []byte       // ← must be fully in memory. No io.Reader.
    Filename string
    MimeType string
}

There is no overload that accepts io.Reader. The library was designed for regular JSON API uploads where the caller already has the bytes. To pass a stream, you'd have to bypass the library entirely and write your own http.NewRequestWithContext call — which is what DownloadAndUploadFile does for the normal path, using request/v2.Post(ctx, url, pr, ...).

3. Constraint 2 — GCS pre-signed POST requires Content-Length

This is the deeper protocol constraint. A GCS pre-signed POST URL is not a regular REST endpoint. It's an HTML-form-style upload backed by a signed policy document. Look at what FormFields contains at runtime:

// Fields sent to GCS pre-signed POST URL (req.FormFields)
{
  "key":              "videos/2026/abc123.mp4",   // exact object path
  "content-type":    "video/mp4",              // must match policy
  "policy":          "eyJleHBpcm...",           // base64 policy doc
  "x-goog-signature":"4a8bc...",              // HMAC of the policy
  "x-goog-algorithm":"GOOG4-RSA-SHA256",
  "x-goog-credential":"sa@project.iam...",
  "x-goog-date":     "20260617T000000Z",
}

When GCS receives this POST, it decodes and validates the policy. That policy is a JSON document signed at URL-generation time. It contains a content-length-range condition:

// Inside the base64-decoded policy
{
  "conditions": [
    {"bucket": "my-bucket"},
    {"key": "videos/2026/abc123.mp4"},
    ["content-length-range", 1, 209715200]  // 1B – 200MB
  ]
}

GCS validates that the actual byte count of the uploaded file falls within content-length-range. The only way for GCS to enforce this during the upload is to know the Content-Length of the body being streamed.

Now here is the problem. When Go's http.Client sends a request whose body is an *io.PipeReader (which has no Len() method), it sends:

POST /upload/storage/v1/b/my-bucket/o?uploadType=multipart
Transfer-Encoding: chunked
Content-Type: multipart/form-data; boundary=abc

The normal path (upload-svc) does accept chunked encoding because it's our own server. It reads the stream and buffers internally at its discretion. GCS is not our server; we cannot change what it accepts.

4. Constraint 3 — we don't know the file size upfront

Even if we could bypass the library (constraint 1) and GCS accepted chunked encoding (constraint 2), there's a third problem: the provider's download URL often doesn't include a Content-Length response header.

To set a correct Content-Length on the GCS upload, we'd need to know the video's byte count before starting the upload. Without it from the provider response headers, the only way to know is to buffer the entire file — which is exactly what we're trying to avoid.

5. Why the normal path doesn't have these constraints

For the normal path, we explicitly bypassed httpkit's multipart helper and built the pipe manually — that was the key architectural choice in a production incident. For the reserved path, there's no equivalent bypass that would work against GCS.

6. Could we ever fix this?

Three conditions would all need to be true simultaneously:

7. Quick check — test your understanding