The object storage model

Every product eventually stores bytes that don't belong in a database. Profile photos, uploaded PDFs, video and backups are examples. Object storage holds those bytes. Services such as Amazon S3, Google Cloud Storage and Azure Blob store each file as one whole object, found by a name, and reached over HTTP.

The big idea

Object storage has a deliberately small model. Its limits are what let it scale.

The keyspace is flat. users/42/avatar/8f3a.jpg is one string. The slashes only look like folders. "Listing a folder" means listing every key that starts with a prefix. A closer picture is a set of labeled lockers. The labels can contain slashes, but no drawer holds another drawer. Renaming a folder means writing new keys, because each object is named by its full key.

Objects are also immutable. You can replace an object or delete it. You can't change byte 1,000 in place. Those limits let the store spread data across many machines and keep it safe.

Walking through it

The model comes down to a few facts.

• A bucket holds objects. An object can be anywhere from 0 bytes to 50 TB, the limit at the time of writing.
• Each object carries system metadata, such as size, content type and checksum, plus a few custom key-value pairs.
• Access is over HTTP: PUT, GET (including byte ranges), HEAD, DELETE and LIST by prefix.
• Versioning, if enabled, keeps older copies of an object.

Compare the three storage types.

Databases run on block storage, because they need fast in-place updates. Their backups go to object storage, because it is cheap, durable and never needs in-place changes.

Common mistakes

• Treating it like a disk. You can't edit an object, so write the whole new object instead.
• Putting media in database rows. A 20 MB video in a row bloats backups and replication, and nobody filters by the bytes.
• Assuming a folder is real. A "folder" is only a shared prefix. Moving one means rewriting every key under it.

Check yourself

1. Can you append bytes to an existing object? No. You replace the whole object with a new one.
2. Why does a database run on block storage and not on object storage? Databases need fast, in-place updates. Object storage offers neither.

Key takeaways

• Object storage holds whole, immutable blobs under flat keys behind an HTTP API.
• It gives up in-place edits and low latency to gain near-unlimited scale and low cost.
• Keep bytes in object storage, and keep the things you query in a database.

Further learning

The Amazon S3 User Guide covers buckets, keys and the HTTP operations in detail.

Next, read about durability and erasure coding, which explains how the store avoids losing your bytes.