Go `io` Primitives — Quick Reference

media-svc context · → Lesson 1

Decision Tree

Do you need the bytes in memory for processing (watermark, encrypt, decode)?
  YES → io.ReadAll  +  io.LimitReader (always cap)
  NO  → io.Pipe  (stream through, zero heap)
        use io.Copy inside the goroutine to move bytes

Core Signatures

Function / Type	Signature	Memory cost
`io.ReadAll`	`func ReadAll(r Reader) ([]byte, error)`	= full content size
`io.LimitReader`	`func LimitReader(r Reader, n int64) Reader`	~0 (counter only)
`io.Pipe`	`func Pipe() (PipeReader, PipeWriter)`	~32 bytes sync buffer
`io.Copy`	`func Copy(dst Writer, src Reader) (int64, error)`	32 KB stack buffer (reused)
`io.TeeReader`	`func TeeReader(r Reader, w Writer) Reader`	= whatever you write to w

Canonical Patterns

Safe buffered download

const maxBytes int64 = 2 << 30 // 2 GB

data, err := io.ReadAll(
    io.LimitReader(resp.Body, maxBytes),
)
if int64(len(data)) == maxBytes {
    return nil, errors.New("file too large")
}

Streaming download → upload (io.Pipe)

pr, pw := io.Pipe()

go func() {
    _, err := io.Copy(pw, src)
    pw.CloseWithError(err)
}()

req, _ := http.NewRequestWithContext(
    ctx, "POST", uploadURL, pr,
)
client.Do(req)

io.Pipe Rules (must memorise)

1. Writer always calls pw.CloseWithError(err) — even on success (pass nil).

2. Writer and reader run in separate goroutines. Calling both in the same goroutine deadlocks.

3. If the reader closes (e.g. upload fails), the next writer Write returns io.ErrClosedPipe. Handle it and exit.

4. Pipe data is consumed once. You cannot rewind or re-read.

LimitReader Gotcha

LimitReader returns EOF silently at the limit — no error.
Always check len(data) == limit after ReadAll to detect truncation.

Memory Comparison (80 MB file, 10 concurrent requests)

Pattern	Per-request peak	10× concurrent
`io.ReadAll` → upload	~160 MB	~1.6 GB (OOM)
`io.Pipe` streaming	~32 KB	~320 KB