Object storage and CDNs

Photos, videos, backups, logs and data lakes all end up in object storage, and anything users download a lot goes out through a CDN. The products look alike from a distance, but they differ in limits, storage classes, what you pay to read data out, and how their caches fill and purge. This guide explains how they work, how they differ, and which to choose, with a chooser and code you can run.

1. Two halves of one job

Module 10 teaches the concepts; this guide is about the products.

2. How an upload and a download work

The usual pattern: clients upload straight to the bucket with a presigned URL, and downloads go through a CDN with a shield layer in front of the bucket.

3. The products

4. Differences at a glance

5. When to choose which, and why

1. Store files in your cloud's object store (S3, Cloud Storage, Blob). Keep metadata (owner, size, status) in a database and only the bytes in the bucket.
2. Upload directly from clients with presigned (signed) URLs, in parts for anything large, so your servers never carry the bytes.
3. Serve popular content through a CDN, with a shield layer when there are many edges, and versioned URLs (logo.v42.svg) so you rarely need purges.
4. Move cold data to cheaper classes with lifecycle rules; remember archive tiers charge for and delay retrieval.
5. If egress dominates the bill (large downloads to many users), compare an egress-free store such as R2, or negotiated CDN pricing.

Try it: choose for your requirements

6. CDN edges and an origin shield, in code

One request a second for five minutes, for ten images, with a 60-second TTL. With three edges, half the requests hit the edge cache. With twelve edges, each edge sees so little of the traffic that its copies expire before they're reused: no hits at all, and every request goes to the origin. A shield layer between the edges and the origin keeps origin fetches at 50 either way (ten images, refreshed every minute). The last line works out multipart upload parts. The program was run in all three languages; each prints the output below it.

import math

TTL = 60  # seconds an edge or shield may serve a cached copy


class Cache:
    def __init__(self):
        self.expires = {}

    def fresh(self, obj, now):
        return self.expires.get(obj, -1) > now

    def store(self, obj, now):
        self.expires[obj] = now + TTL


def run(edges, use_shield):
    """One request a second for 300 s, spread over the edges, for 10 objects."""
    edge_caches = [Cache() for _ in range(edges)]
    shield = Cache()
    edge_hits = origin = 0
    for t in range(300):
        obj, edge = f"img{t * 7 % 10}", edge_caches[t % edges]
        if edge.fresh(obj, t):
            edge_hits += 1
            continue
        if use_shield and shield.fresh(obj, t):
            edge.store(obj, t)                    # filled from the shield, not the origin
            continue
        origin += 1
        if use_shield:
            shield.store(obj, t)
        edge.store(obj, t)
    return edge_hits, origin


for edges in (3, 12):
    hits, without = run(edges, False)
    _, with_shield = run(edges, True)
    print(f"{edges} edges: edge hit ratio {hits / 3:.0f}%, origin fetches {without} without a shield, {with_shield} with one")

# Multipart upload: S3 allows up to 10,000 parts, each 5 MiB to 5 GiB (the last part may be smaller).
size_gib = 200
part_mib = max(5, math.ceil(size_gib * 1024 / 10_000))
print(f"a {size_gib} GiB upload needs parts of at least {part_mib} MiB: {math.ceil(size_gib * 1024 / part_mib)} parts")

const TTL = 60; // seconds an edge or shield may serve a cached copy

class EdgeCache {
  expires = new Map<string, number>();
  fresh(obj: string, now: number): boolean {
    return (this.expires.get(obj) ?? -1) > now;
  }
  store(obj: string, now: number): void {
    this.expires.set(obj, now + TTL);
  }
}

// One request a second for 300 s, spread over the edges, for 10 objects.
function run(edges: number, useShield: boolean): [number, number] {
  const edgeCaches = Array.from({ length: edges }, () => new EdgeCache());
  const shield = new EdgeCache();
  let edgeHits = 0, origin = 0;
  for (let t = 0; t < 300; t++) {
    const obj = `img${(t * 7) % 10}`, edge = edgeCaches[t % edges];
    if (edge.fresh(obj, t)) { edgeHits++; continue; }
    if (useShield && shield.fresh(obj, t)) { edge.store(obj, t); continue; } // filled from the shield, not the origin
    origin++;
    if (useShield) shield.store(obj, t);
    edge.store(obj, t);
  }
  return [edgeHits, origin];
}

for (const edges of [3, 12]) {
  const [hits, without] = run(edges, false);
  const [, withShield] = run(edges, true);
  console.log(`${edges} edges: edge hit ratio ${(hits / 3).toFixed(0)}%, origin fetches ${without} without a shield, ${withShield} with one`);
}

// Multipart upload: S3 allows up to 10,000 parts, each 5 MiB to 5 GiB (the last part may be smaller).
const sizeGib = 200;
const partMib = Math.max(5, Math.ceil((sizeGib * 1024) / 10000));
console.log(`a ${sizeGib} GiB upload needs parts of at least ${partMib} MiB: ${Math.ceil((sizeGib * 1024) / partMib)} parts`);

import java.util.HashMap;
import java.util.Map;

public class Main {
    static final int TTL = 60; // seconds an edge or shield may serve a cached copy

    static class EdgeCache {
        final Map<String, Integer> expires = new HashMap<>();
        boolean fresh(String obj, int now) { return expires.getOrDefault(obj, -1) > now; }
        void store(String obj, int now) { expires.put(obj, now + TTL); }
    }

    // One request a second for 300 s, spread over the edges, for 10 objects.
    static int[] run(int edges, boolean useShield) {
        EdgeCache[] edgeCaches = new EdgeCache[edges];
        for (int i = 0; i < edges; i++) edgeCaches[i] = new EdgeCache();
        EdgeCache shield = new EdgeCache();
        int edgeHits = 0, origin = 0;
        for (int t = 0; t < 300; t++) {
            String obj = "img" + (t * 7 % 10);
            EdgeCache edge = edgeCaches[t % edges];
            if (edge.fresh(obj, t)) { edgeHits++; continue; }
            if (useShield && shield.fresh(obj, t)) { edge.store(obj, t); continue; } // filled from the shield, not the origin
            origin++;
            if (useShield) shield.store(obj, t);
            edge.store(obj, t);
        }
        return new int[] {edgeHits, origin};
    }

    public static void main(String[] args) {
        for (int edges : new int[] {3, 12}) {
            int[] without = run(edges, false), with = run(edges, true);
            System.out.printf("%d edges: edge hit ratio %d%%, origin fetches %d without a shield, %d with one%n",
                    edges, Math.round(without[0] / 3.0), without[1], with[1]);
        }
        // Multipart upload: S3 allows up to 10,000 parts, each 5 MiB to 5 GiB (the last part may be smaller).
        int sizeGib = 200;
        int partMib = Math.max(5, (int) Math.ceil(sizeGib * 1024 / 10000.0));
        System.out.printf("a %d GiB upload needs parts of at least %d MiB: %d parts%n", sizeGib, partMib, (int) Math.ceil(sizeGib * 1024.0 / partMib));
    }
}

3 edges: edge hit ratio 50%, origin fetches 150 without a shield, 50 with one
12 edges: edge hit ratio 0%, origin fetches 300 without a shield, 50 with one
a 200 GiB upload needs parts of at least 21 MiB: 9753 parts

7. Say it in the interview

8. Common mistakes

9. Check yourself

10. Sources and further reading

Product facts checked on 3 November 2026. Limits, classes and retirement dates change; check before relying on them.