Lesson 4 — DownloadAndUploadFile vs DownloadAndUploadToReservedURL

Grounded in a production incident · media-svc · both files read verbatim
← L1: io.Pipe · ← L2: Reserved path · ← L3: Streaming fix · Reference →

0. The one-sentence summary

Both functions do the same thing: stream a provider video URL into a multipart POST without buffering. They diverge because their upload targets are completely different servers with different protocols, auth schemes, and response shapes.

1. Why different HTTP clients?

uploadResp, err := request.Post(ctx, r.endpoint, pr,
    request.WithHeader(request.Header{
        "Content-Type":  contentType,
        "Authorization": "Bearer " + authKey,
    }),
)

httpReq, err := http.NewRequestWithContext(
    ctx, http.MethodPost, req.UploadURL, pr)
httpReq.Header.Set("Content-Type", contentType)
// No Authorization header — auth is in the URL
resp, err := http.DefaultClient.Do(httpReq)

Why httpkit for upload-svc? Our internal service expects Authorization: Bearer <key> — the httpkit wrapper provides WithHeader, OTel tracing, and structured logging for free.

Why bare stdlib for GCS? A GCS pre-signed URL encodes the authorization inside the query string (x-goog-signature=…). The httpkit wrapper unconditionally injects OTel tracing headers (traceparent, tracestate) which are irrelevant to GCS. Bare http.DefaultClient sends exactly and only what we specify.

2. Why different form field sources?

for _, form := range req.buildFormData() {
    mw.WriteField(form.Key, form.Value)
}
// Returns an ordered []FormData slice
// Fields: account_id, apply_thumbnail, etc.

for key, val := range req.FormFields {
    mw.WriteField(key, val)
}
// map[string]string — random iteration
// Fields: key, policy, x-goog-*, content-type

buildFormData() returns a slice because upload-svc's field names are compiled into the code (account_id, apply_thumbnail, etc.). An ordered slice is the natural representation. FormFields is a map because the GCS policy fields come from the caller at runtime — we don't know in advance which fields will be present, only that they came from a signed URL generator.

3. Why does MIME type override come from FormFields?

In the reserved path, the GCS signed policy includes a {"content-type": "video/mp4"} condition. GCS validates that the Content-Type header of the file part matches this condition exactly. The override:

mimeType := req.File.MimeType          // caller hint (fallback)
if ct := req.FormFields["content-type"]; ct != "" {
    mimeType = ct                      // always wins — matches the signed policy
}

If the caller passed MimeType: "video/mp4" and the policy also says "video/mp4", the override is a no-op. If they ever differ, FormFields["content-type"] is authoritative because it's what GCS expects. Without this override, a mismatch would cause GCS to return HTTP 400 with a policy condition failure.

4. Why is the filename derived from FormFields["key"]?

filename := path.Base(req.FormFields["key"])
// "upload/2026/06/17/01KV9X8G86DAERW5E7GJ9YHHG6.mp4"
//  → path.Base → "01KV9X8G86DAERW5E7GJ9YHHG6.mp4"
if filename == "" || filename == "." {
    filename = req.File.Filename       // fallback only
}

The key field in FormFields is the GCS object path that was signed when the pre-signed URL was generated. It always carries the correct filename including extension (.mp4 for video, .png for image). If we used req.File.Filename directly, the caller would have to know and supply the exact filename. Instead, the GCS key is the single source of truth.

5. The limit vs limit+1 difference

This is a subtle correctness fix applied in the reserved path that was not backfilled to the standard path.

n, err := io.Copy(fw, io.LimitReader(
    dlResp.Body, limit))  // reads up to limit

if n == limit {       // ← ambiguous!
    pw.CloseWithError(...)
}
// A file EXACTLY limit bytes long
// will be falsely rejected here.

n, copyErr := io.Copy(fw, io.LimitReader(
    dlResp.Body, limit+1))  // reads one extra byte

if n > limit {        // ← unambiguous
    pw.CloseWithError(...)
}
// n > limit only if file was truly larger.
// n == limit means "exactly limit bytes" → OK.

6. Why different return types and response handling?

if !uploadResp.Ok() {
    body, _ := io.ReadAll(
        io.LimitReader(uploadResp.Body, 4096))
    var errResp UploadErrorResponse
    json.Unmarshal(body, &errResp)
    // Returns structured apierror with error code
}
var result UploadResponse
uploadResp.JSON(&result)
return result, nil // URL, thumbnail, duration

if resp.StatusCode >= 400 {
    body, _ := io.ReadAll(
        io.LimitReader(resp.Body, 4096))
    // No JSON parsing — GCS returns XML on error
    log.Error(...)
    return apierror.Error("0500", ...)
}

io.Copy(io.Discard, resp.Body)
return nil // no useful metadata returned

upload-svc returns a JSON body with the CDN URL, thumbnail info, and video duration — data the caller needs to store in the database. The response is worth parsing.

By default GCS pre-signed POST returns HTTP 204 with an empty body on success. On failure it returns XML (not JSON). The resultURL was already known before the upload started — it's reserved.ResultURL, which was part of the original reserved upload slot. There is nothing useful to parse.

7. Quick check