Lesson 1 of ?  ·  Go I/O Primitives  ·  media-svc context

Why io.Pipe and io.LimitReader matter for memory

This lesson is grounded in the OOM fix we just shipped in a real production incident. Before that fix, every incoming Replicate webhook caused the pod to download an entire 80 MB video into RAM before uploading it. This lesson explains why that was the problem, and exactly how io.Pipe and io.LimitReader solved it.

1. The problem: io.ReadAll as an invisible buffer

The original code in downloadfile.go was:

resp, _ := http.Get(fileURL)
fileBytes, _ := io.ReadAll(resp.Body)  // ← full file in RAM
return fileBytes, nil

io.ReadAll reads from a reader until EOF and returns all the bytes as a single []byte slice. For a small JSON response this is harmless. For an 80 MB video it allocates 80 MB on the heap and keeps it there until the slice is garbage-collected.

2. The fix: connect two streams with io.Pipe

io.Pipe() creates a synchronised in-memory channel between a writer and a reader. No data is buffered — bytes written on one end are immediately readable on the other end, and the writer blocks until the reader consumes them.

// io.Pipe signature
func Pipe() (*PipeReader, *PipeWriter)

Think of it as a Unix pipe, but between two goroutines in the same process:

In the new DownloadAndUploadFile:

pr, pw := io.Pipe()
mw := multipart.NewWriter(pw)

go func() {
    // goroutine A: write the multipart form into pw
    for _, form := range req.buildFormData() {
        mw.WriteField(form.Key, form.Value)
    }
    fw, _ := mw.CreatePart(h)               // file part header
    io.Copy(fw, resp.Body)                   // stream download into pw
    pw.CloseWithError(mw.Close())
}()

// goroutine B (main goroutine): read from pr → upload request body
uploadReq, _ := http.NewRequestWithContext(ctx, "POST", endpoint, pr)
http.DefaultClient.Do(uploadReq)             // reads from pr as it sends

3. io.LimitReader: the safety valve

Even in the streaming path, a malformed response could theoretically stream forever. io.LimitReader wraps any reader and returns EOF after at most n bytes:

// io.LimitReader signature
func LimitReader(r Reader, n int64) Reader

It is just a struct holding a counter:

type limitedReader struct {
    r Reader
    n int64   // remaining bytes
}

func (l *limitedReader) Read(p []byte) (n int, err error) {
    if l.n <= 0 {
        return 0, io.EOF
    }
    if int64(len(p)) > l.n {
        p = p[:l.n]       // shrink the slice to remaining limit
    }
    n, err = l.r.Read(p)
    l.n -= int64(n)
    return
}

In downloadfile.go and download_raw_url.go the fix was:

const maxDownloadBytes int64 = 2 * 1024 * 1024 * 1024  // 2 GB

fileBytes, err := io.ReadAll(io.LimitReader(resp.Body, maxDownloadBytes))
if int64(len(fileBytes)) == maxDownloadBytes {
    return nil, apierror.Error("0500", fmt.Errorf("file exceeds maximum allowed size"))
}

4. How they compare

5. The goroutine lifetime rule

When you use io.Pipe, the goroutine writing to pw must always call pw.CloseWithError — even on success. If it exits without closing, the reader blocks forever.

go func() {
    var err error
    // ... do work ...
    pw.CloseWithError(err)   // nil = EOF (success), non-nil = error
}()

The pattern pw.CloseWithError(mw.Close()) used in our code is idiomatic: mw.Close() writes the multipart boundary terminator and returns any write error; that error (or nil) is forwarded to the pipe reader.

6. When not to stream

Streaming via io.Pipe is the wrong choice when:

• You need to read the bytes twice (e.g. watermarking: read once to apply watermark, then upload). Piped data is consumed once and gone.
• The upload target requires knowing the content-length upfront and you don't know the file size (some S3-compatible APIs). Use buffering or chunked transfer encoding.
• You need to inspect or transform the bytes in memory (e.g. image processing, encryption). Buffer first, then operate.

This is exactly why the reserved (pre-signed GCS) upload path in a production incident was left on the buffered path — GCS pre-signed POST requires specific field ordering that's easier to guarantee when you have the full bytes.