Storage and Data: Three Shapes, and Choosing a Database

Every storage service is one of three shapes, and the shape determines the access pattern it can serve efficiently.

On AWS these are S3, EBS and EFS respectively. The distinctions are not marketing: they follow from what each one lets you do to part of an item.

Object storage has no concept of editing byte 4,096 of an object. You replace the whole object. That constraint is exactly what allows it to be enormously scalable, cheap, and reachable over HTTP from anywhere.

Block storage lets you write any block at any offset, which is what a database needs to update a row in place, and that requirement is why a block volume attaches to one machine rather than being shared freely.

Gotcha: using object storage as a filesystem is the classic mistake. Code that opens a file, seeks, and writes a few bytes will either fail or silently rewrite the entire object every time, turning a cheap store into an expensive and slow one.

Storage services quote two numbers that sound alike and mean entirely different things. Conflating them leads directly to a backup strategy that does not protect against the thing that actually happens.

Definition: durability is the probability your data still exists. Availability is the probability you can reach it right now. Data can be perfectly durable and temporarily unreachable, which is an outage. Data that is not durable is gone, which is not an outage but a loss.

AWS designs S3 for 99.999999999 percent durability, eleven nines, and 99.99 percent availability in its standard class. The durability comes from storing data redundantly across a minimum of three availability zones by default, with checksum verification, erasure coding and background auditing that repairs damage without anyone asking.

The asymmetry in those two numbers is deliberate and worth reading. Four nines of availability permits roughly an hour of unreachability a year, which is survivable for most systems. Eleven nines of durability is the promise that the bytes do not vanish, because unreachable data comes back and lost data does not.

In practice: when a design says "we need high availability", ask which of the two is meant. The answers lead to completely different work: availability wants replicas, failover and retries, while durability wants backups, versioning and copies in another blast radius.

Durability figures are quoted per object per year, so the risk you actually carry scales with how many objects you hold. Working that out is more useful than the marketing number.

Eleven nines means each object has a one in one hundred billion chance of being lost in a year. The chance that at least one object in a collection is lost follows directly.

AWS's own illustration matches this arithmetic: store 10 million objects and you might expect to lose one roughly every 10,000 years.

Key idea: the number is extraordinary, and it protects against exactly one thing, hardware and media failure. It does not protect against you. A deletion, an overwrite, a bad migration, a compromised credential or a buggy script will destroy data at eleven nines of reliability, faithfully and immediately.

Which is why the real controls are versioning, so an overwrite is recoverable; object lock, so a deletion can be made impossible for a retention period; and a copy in a separate account, so a credential compromise cannot reach both.

Object storage is not one price. It is a set of classes trading storage cost against retrieval cost and speed, and choosing among them is one of the few pure-savings levers in cloud.

The shape of the trade, from most to least expensive to store:

• Standard. Frequent access, no retrieval fee, highest storage price. For anything actively served.
• Infrequent access. Cheaper to store, with a per-retrieval charge and a minimum storage duration. For data you keep but rarely read.
• Archive classes. Dramatically cheaper storage, with retrieval measured in minutes to hours and a higher retrieval cost. For compliance retention and cold backups.
• Intelligent tiering. AWS moves objects between tiers based on observed access patterns, for a small monitoring fee per object.

Gotcha: the minimum-duration and retrieval charges are what catch people. Moving a large volume of small objects to a colder class and then reading them frequently can cost more than leaving them in standard, and deleting an object before its minimum duration still bills the remainder. Intelligent tiering exists precisely because guessing access patterns is hard, and its per-object fee makes it a poor fit for enormous numbers of tiny objects.

The durable rule: match the class to how the data is read, not to how old it is. Age is a proxy for access frequency, and it is often a bad one.

Database choice is usually argued as relational versus NoSQL, which is the wrong axis. The useful question is what shape your reads and writes have, because that is what the engines actually differ on.

A relational database is the right default when your data has relationships you will query in ways you cannot fully predict, when you need transactions across several tables, and when correctness under concurrent writes matters more than raw scale. Ad hoc queries and joins are exactly what it is built for. AWS provides managed engines through RDS, and Aurora as a cloud-native implementation with storage that replicates across availability zones.

A key-value store like DynamoDB is right when access is predictable and by key, when you need single-digit millisecond reads at any scale, and when you can design the keys around the queries in advance. Its price is that a query it was not designed for is expensive or impossible: there is no equivalent of adding a join later.

A cache such as ElastiCache sits in front of either, holding hot values in memory. It is not a database, because losing it must be survivable, and treating it as one is how a cache eviction becomes an outage.

In practice: most systems end up with several, and that is fine. The failure is choosing a key-value store for its scaling story when the workload is ad hoc analytical queries, then rediscovering joins by writing them in application code.

A managed database service takes over the operational work that has nothing to do with your product, and it is worth being precise about which work, because the remainder is still yours.

What AWS takes: provisioning, engine patching, automated backups and point-in-time recovery, failover to a standby in another availability zone, read replicas on request, and monitoring hooks. Each is a task that is boring, easy to get subtly wrong, and painful to discover you got wrong.

What stays yours entirely: the schema, the indexes, the queries, the connection management, capacity choices, and knowing whether your access pattern suits the engine at all.

Given eleven nines and automated backups, data loss should be a solved problem. It is not, because the mechanisms that cause it in practice are not the ones the guarantees address.

The recurring causes, in rough order of how often they bite:

• Deletion by a person or a script, replicated faithfully to every replica within seconds. Replication is not backup: it copies your mistake.
• Backups nobody ever restored. An untested backup is a hypothesis. Restore drills are the only thing that converts it into a fact, and they routinely surface missing dependencies, wrong retention, or a restore that takes far longer than the business assumed.
• A compromised credential with delete permission reaching both the data and the backups, which is exactly why the backup copy belongs in a different account.
• Retention quietly misconfigured, so the backup you need was aged out before you needed it.

Key idea: durability guarantees cover the storage layer's failures. Every remaining loss mechanism is an access-control and process problem, which means the tools that actually protect you are versioning, object lock, cross-account copies, least-privilege permissions, and a restore you have performed at least once.

One cost note that closes this lesson and opens the next: keeping data is cheap, and moving it is not. That asymmetry drives more architecture, and more surprising invoices, than any storage decision, which is where networking comes in.