What Is S3-Compatible Storage? – Definition and How It Works

Learn what S3-compatible object storage is, how keys and prefixes work, and how to lay out collected data in it.

S3-compatible storage is any object store that implements Amazon S3's HTTP API. Because the API became a de facto standard, one client library works against many providers and self-hosted systems.

1. Objects, Not Files

An object store is a flat key-value map from a string key to a blob plus metadata. It is not a filesystem.

The / in a key is a convention, not a structure. raw/2026-08-18/page.html is one key that happens to contain slashes, and "directories" are a listing convenience.

The consequences that bite: you cannot append to an object, and a rename costs a full copy.

2. Why It Suits Collected Data

• Effectively unbounded capacity without capacity planning.
• Low cost per byte, with cheaper tiers for archival.
• Write-once, read-many matches raw payload storage exactly.
• Durability across devices without your involvement.
• Direct querying by engines that read objects in place.

Raw HTML and JSON payloads are the canonical fit: written once, occasionally reprocessed, never modified.

3. Key Layout Is the Design Decision

The key structure determines what is cheap to list and what is expensive to find.

raw/dt=2026-08-18/site=example.com/<sha256-of-url>.html.gz

Three properties make this work: date first so a day's data lists with one prefix, site second so a single source is isolable, and a hash of the URL as the filename so the key is deterministic — rewriting replaces rather than accumulating, which is what makes the write idempotent.

Avoid deep prefixes for their own sake, and avoid keys beginning with a random value if you need date-range listing.

4. Basic Operations

import boto3

s3 = boto3.client(
    "s3",
    endpoint_url="https://storage.example.com",   # non-AWS providers
)

s3.put_object(Bucket="collect", Key=key, Body=gzipped,
              ContentType="text/html", ContentEncoding="gzip")

obj = s3.get_object(Bucket="collect", Key=key)

paginator = s3.get_paginator("list_objects_v2")
for page in paginator.paginate(Bucket="collect", Prefix="raw/dt=2026-08-18/"):
    for item in page.get("Contents", []):
        print(item["Key"], item["Size"])

Use the paginator rather than a single list call. A plain list returns a bounded page, and code that ignores the continuation token silently processes only the first slice — a classic partial-data bug.

5. Multipart and Large Objects

Objects above a threshold are uploaded in parts, which allows parallelism and retry of a single part rather than the whole transfer.

The operational detail: an abandoned multipart upload leaves parts that consume storage but appear in no listing. A lifecycle rule to abort incomplete uploads after a few days is the fix, and its absence is a common source of unexplained storage cost.

6. Lifecycle and Tiers

Lifecycle rules move or delete objects by age. A typical arrangement keeps recent raw payloads in standard storage, transitions older ones to a colder tier, and expires them after a retention period.

Set the retention deliberately. Raw payloads are irreplaceable when the source cannot be re-read, so an expiry rule is a decision about what you are willing to lose.

7. Consistency and Concurrency

Modern S3 and most compatible stores offer read-after-write consistency for new objects, so a read following a write returns the data.

What they do not offer is any locking. Two writers to one key means last-write-wins, with no error. Coordination must live outside the store — which is why deterministic keys and idempotent writes matter more here than in a filesystem.

8. Summary

S3-compatible storage is a flat key-value object store with an API that became a standard, offering cheap, durable, effectively unbounded capacity for write-once data.

Design the key layout deliberately — date prefix, then source, then a hash of the URL so writes are idempotent — page through listings rather than trusting a single call, add a lifecycle rule to abort incomplete multipart uploads, and coordinate concurrent writers outside the store, because it provides no locks.

Potential Related Glossary

• Data Structure
• Web Crawling
• Unstructured Data
• Data Wrangling