Lesson 3 — Streaming the reserved URL + OOM improvements

Grounded in a production incident · media-svc · real GCS policy decoded live
← Lesson 1 · ← Lesson 2 · Reference →

0. Correcting lesson 2 — GCS does NOT reject chunked encoding

Here is the actual httpkit PostForm source — note the io.Pipe:

// httpkit/request/request.go — PostForm
pr, pw := io.Pipe()
writer := multipart.NewWriter(pw)
ct := writer.FormDataContentType()
go func() {
    for _, form := range forms {
        _ = writer.WriteField(form.Key, form.Value)
    }
    fw, _ := writer.CreatePart(createFileMimeHeader(...))
    _, _ = io.Copy(fw, bytes.NewReader(file.Content))  ← file must be []byte already
    pw.CloseWithError(writer.Close())
}()
req, _ := http.NewRequestWithContext(ctx, "POST", url, pr)  ← body is PipeReader = chunked

The HTTP request body is a *io.PipeReader — Go sends it chunked. GCS accepts it. The only real barrier is file.Content []byte — the library forces us to buffer the download before we can hand it to the library. Remove the library, and streaming works.

1. What the real policy tells us

Decoding the live GCS policy from the example reveals something important:

{
  "conditions": [
    ["content-length-range", 1, 10485760],   ← max 10 MB
    {"content-type": "image/png"},               ← reserved path = IMAGES, not videos
    {"bucket": "example-assets-bucket"},
    {"key": "upload/2026/.../01KV9...png"}
  ],
  "expiration": "2026-06-17T04:42:02Z"          ← 1-hour window
}

The reserved upload path handles images (PNG, max 10 MB), not videos. The video OOM risk — 80 MB per pod — was already fixed in the normal path. The reserved path's unbuffered cost is at most 10 MB per worker iteration.

2. How to stream the reserved path — concrete implementation

Add a new method DownloadAndUploadToReservedURL to the upload repository. It follows the same io.Pipe goroutine pattern as DownloadAndUploadFile, bypassing httpkit's File.Content []byte requirement entirely:

// app/repository/uploadsvc/upload/download_and_upload_reserved.go

func (r *uploadRepository) DownloadAndUploadToReservedURL(
    ctx context.Context, downloadURL string, req ReservedUploadRequest,
) error {
    // Phase A: open the download stream
    dlResp, err := request.Get(ctx, downloadURL)
    if err != nil {
        return apierror.Error("0500", fmt.Errorf("download failed: %w", err))
    }
    defer dlResp.Close()
    if !dlResp.Ok() {
        return apierror.Error("0500", fmt.Errorf("download status %d", dlResp.StatusCode))
    }

    // Fast-fail: if provider sends Content-Length, check it before starting the pipe.
    limit := maxDownloadBytesLimit()
    if dlResp.ContentLength > limit {
        return apierror.Error("0500", fmt.Errorf("file too large: %d bytes", dlResp.ContentLength))
    }

    // Phase B: construct multipart body via pipe
    pr, pw := io.Pipe()
    mw := multipart.NewWriter(pw)
    contentType := mw.FormDataContentType()

    go func() {
        // All form fields FIRST — file must be the final part (GCS policy)
        for key, val := range req.FormFields {
            if err := mw.WriteField(key, val); err != nil {
                pw.CloseWithError(err); return
            }
        }

        // Override MIME type with policy's content-type if present
        mimeType := req.File.MimeType
        if ct := req.FormFields["content-type"]; ct != "" {
            mimeType = ct
        }

        h := make(textproto.MIMEHeader)
        h.Set("Content-Disposition",
            fmt.Sprintf(`form-data; name="file"; filename="%s"`, req.File.Filename))
        h.Set("Content-Type", mimeType)
        fw, err := mw.CreatePart(h)
        if err != nil {
            pw.CloseWithError(err); return
        }

        // Pipe download → multipart file part with size limit
        n, err := io.Copy(fw, io.LimitReader(dlResp.Body, limit+1))
        if n > limit {
            pw.CloseWithError(fmt.Errorf("file exceeds maximum allowed size")); return
        }
        if err != nil {
            pw.CloseWithError(err); return
        }
        pw.CloseWithError(mw.Close())
    }()

    // Phase C: POST to GCS pre-signed URL — body is PipeReader (chunked) = works
    resp, err := http.Post(req.UploadURL, contentType, pr)
    // or: use httpkit/request/v2.Post(ctx, req.UploadURL, pr, ...)
    if err != nil {
        return apierror.Error("0500", fmt.Errorf("upload to GCS failed: %w", err))
    }
    defer resp.Body.Close()
    if resp.StatusCode >= 400 {
        body, _ := io.ReadAll(resp.Body)
        return apierror.Error("0500", fmt.Errorf("GCS upload status %d: %s", resp.StatusCode, body))
    }
    io.Copy(io.Discard, resp.Body)
    return nil
}

Then in status_checker.go, replace the two-step reserved path with this single call:

// Before (buffers up to 10 MB)
fileBytes, err := s.mediauploadRepo.DownloadFile(ctx, res.OutputURL)
err = s.mediauploadRepo.UploadToReservedURL(ctx, upload.ReservedUploadRequest{
    File: upload.File{Data: fileBytes, ...}, ...
})

// After (zero-copy streaming)
err = s.mediauploadRepo.DownloadAndUploadToReservedURL(ctx, res.OutputURL,
    upload.ReservedUploadRequest{
        File:       upload.File{Filename: "image.png", MimeType: "image/png"},
        UploadURL:  reserved.UploadURL,
        FormFields: reserved.FormFields,
    })

3. Do BytePlus / Hedra tell us the file size?

Neither GetStatusResponse struct from Hedra or BytePlus contains file size. The status payload only carries a URL and token counts. But there is a second place to look: the actual download HTTP response headers.

// httpkit/request/v2 Response embeds *http.Response
dlResp, _ := request.Get(ctx, res.OutputURL)
dlResp.ContentLength   // -1 if provider omits Content-Length, actual bytes if set

Use ContentLength for fast-fail only — not as a dependency for streaming. If it's -1, stream anyway with io.LimitReader as the safety valve. If it's set and over the limit, reject before starting the pipe.

4. Full OOM & performance improvement map

5. One dangerous edge case in the streaming reserved path

The req.FormFields is a map[string]string. Map iteration order in Go is randomized deliberately at runtime. The form fields written to the multipart body will be in an unpredictable order on every call.

The content-type form field has one extra subtlety: it appears both as a form field (text part in multipart, value image/png) AND as the Content-Type MIME header of the file part. Both must match the policy. Our implementation handles this with the req.FormFields["content-type"] override on the file part header.

6. Quick check