File Uploads and Object Storage
Why uploaded files belong in object storage, not your database, and how presigned URLs keep bytes off your own server.
File Uploads and Object Storage
The player loads from youtube-nocookie.com when you press play. Nothing reaches YouTube before that.
Not started
What you'll learn
- A multipart/form-data request is how a browser sends a file alongside regular form fields in one HTTP request, with each part separated by a boundary marker and its own small header.
- Buffering an entire upload into memory before processing it works fine for a 50KB avatar and falls over for a 2GB video — streaming processes the file in chunks as bytes arrive, so memory use stays flat regardless of file size.
- Object storage (S3 and its equivalents) is built for exactly one job — storing arbitrary byte blobs behind a key, cheaply, at any scale — while a relational database is built and tuned for structured, queryable rows, which makes it a poor and expensive place to put binary file data.
- A presigned URL is a time-limited, cryptographically signed link that grants temporary permission to upload or download one specific object, which lets a client talk directly to object storage without your server ever touching the file's bytes.
- The pattern that combines all of this: your server only ever handles metadata and hands out short-lived presigned URLs; the actual file bytes flow directly between the client and the object store, both ways.
The naive version, and where it breaks
The simplest possible file upload feature looks like this: a client sends a file in a request, your API server receives it, and you store the file’s bytes directly in a database column. It works for a demo. It breaks down in roughly this order as real usage shows up: large files, many concurrent uploads, and the realization that your database — the thing holding your most important, most carefully backed-up data — is now also holding a few hundred gigabytes of video files nobody queries by content.
Fixing this properly means getting four separate things right: how the file actually arrives over HTTP, how your server processes it without falling over, where the bytes end up living long-term, and how clients get to upload and download them without your server standing in the middle of every byte. Each piece solves a distinct problem.
How a file actually travels in an HTTP request
When a browser submits a form containing a file input, it typically encodes the request body as multipart/form-data rather than the more familiar application/x-www-form-urlencoded. The reason is simple: URL-encoding is built for short text values, and it doesn’t have a sane way to carry raw binary data of arbitrary length alongside normal fields. Multipart solves this by splitting the body into separate parts, each with its own header and separated by a unique boundary string your server can use to tell where one part ends and the next begins:
POST /upload HTTP/1.1
Content-Type: multipart/form-data; boundary=----Boundary7MA4YWxkTrZu0gW
------Boundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="title"
Vacation photo
------Boundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="file"; filename="beach.jpg"
Content-Type: image/jpeg
<raw JPEG bytes here>
------Boundary7MA4YWxkTrZu0gW--
Your web framework almost certainly parses this for you — you call some request.files['file'] equivalent and get a usable object back. What’s worth understanding is what that framework is doing underneath: it’s reading the stream, finding boundaries, and handing you each part as it’s identified.
Stream it, don’t buffer it
The question that matters for anything beyond small files is: does your server read the entire upload into memory before doing anything with it, or does it process bytes as they arrive?
Buffering the whole thing is simpler to write and fine for small, predictable files — a profile picture, a document under a few megabytes. It falls apart once files get large or uploads get concurrent: ten users uploading 500MB files simultaneously means 5GB sitting in memory at once, on top of whatever else your server process is doing. Do that on a server with a few gigabytes of RAM and you’ll see out-of-memory crashes under exactly the load pattern you most need to handle gracefully.
Streaming avoids this by processing the upload in chunks as bytes arrive off the network socket — reading a chunk, writing it onward (to disk, or to object storage, or through a hashing function to verify integrity), then reading the next chunk, never holding more than one chunk’s worth of the file in memory at a time. Memory use for a streaming upload handler stays roughly flat whether the file is 1MB or 10GB. This is also exactly the shape of problem background processing exists for, if resizing, transcoding, or scanning the file takes real time: accept and store the upload quickly, then hand the heavier processing off as a background job rather than making the uploading client wait on it synchronously.
Why the file doesn’t belong in your database
Once you’re not buffering the whole file in your web server’s memory, the next question is where it ends up living. A relational database is a remarkable piece of engineering, but it’s engineered for a specific shape of data: structured, relatively small rows that get indexed, joined, and queried by their actual content. Every assumption behind a database’s performance — index structures, page caching, replication strategy, backup tooling — is built around that shape.
A multi-megabyte binary blob doesn’t fit it. Storing files as BLOB or BYTEA columns bloats your database’s physical size far beyond your actual structured data, slows down backups and replication (which now have to move those bytes too), and wastes a resource you’re paying a premium for — database compute and fast storage — on something that doesn’t need any of what makes a database expensive.
Object storage — Amazon S3, and its equivalents like Google Cloud Storage or Azure Blob Storage — is purpose-built for the opposite shape: store an arbitrary blob of bytes behind a simple key, cheaply, at a scale of billions of objects, with no concept of rows, columns, or queries. Your database still has a role: it stores the metadata — the object’s key, its owner, upload timestamp, file size, content type — exactly the kind of structured, queryable data it’s good at. The file’s actual bytes live in object storage; your database just holds a pointer to where they are.
Presigned URLs: getting your server out of the way
Even once you’ve decided the file belongs in object storage, there’s still a design choice: does every upload and download route through your API server on its way to and from the object store? That works, but it means your server pays the bandwidth and compute cost of every byte, twice — once coming in from the client, once going out to storage — and it means your server’s capacity caps how many concurrent uploads you can handle, even though the actual bottleneck (object storage) could handle far more.
A presigned URL removes your server from that path entirely. It’s a URL your server generates — using its own storage credentials, without the file’s bytes ever touching your server — that grants temporary, scoped permission to upload or download one specific object directly. The client’s browser then talks straight to the object store:
- Client tells your API: “I want to upload
beach.jpg.” - Your API checks the user is authorized, then asks the storage provider’s SDK to generate a presigned URL for that exact object key, with an expiration (typically minutes).
- Your API returns that URL to the client. It never sees the file.
- The client uploads the file directly to the object store using that URL.
- The client tells your API the upload finished; your API records the metadata.
The same pattern runs in reverse for downloads — a presigned URL that’s valid for a few minutes, handed to the client, rather than your server reading the file from storage and re-serving it itself. This is also the standard answer to “how do private files stay private without public bucket access”: the bucket stays locked down, and access is granted per-object, per-request, for a short window, through the signed URL rather than a bucket-wide policy. For files large enough that a single upload request isn’t practical — multi-gigabyte video, for instance — the same idea extends to a presigned URL per chunk, letting the client upload a large file in parts that the storage provider reassembles afterward.
What your server keeps doing, in all of this, is the part that actually needs a server: deciding who’s allowed to upload or download what, and keeping the metadata straight. What it stops doing is sitting in the middle of every file’s bytes — which is the whole point.
Check yourself
What is multipart/form-data actually doing, structurally, when a browser uploads a file?
It packages each form field — including the file's raw bytes — as a separate 'part' within one HTTP request body, with each part separated by a unique boundary string and preceded by its own small header naming the field and, for a file, its filename and content type. It's how one request carries both ordinary text fields and binary file data together.
Why does streaming an upload matter once files get large?
Buffering means loading the entire file into memory before you do anything with it — fine at a few hundred KB, but a server handling many concurrent large uploads this way can exhaust its memory and crash. Streaming processes the file in chunks as they arrive over the network, so memory use stays roughly constant no matter how big the file is or how many uploads are happening at once.
Why not just store an uploaded file as a BLOB column in your primary database?
A relational database is optimized for structured, queryable, relatively small rows — indexes, transactions, query planning all assume that shape. A database storing large binary blobs bloats your backups, slows down replication, and wastes an expensive, carefully-tuned resource on something it wasn't built for. Object storage is purpose-built for cheaply storing arbitrary-sized blobs behind a key, at effectively unlimited scale, and that's the better fit for the data, not just a style preference.
What problem does a presigned URL solve that a plain upload-through-your-server doesn't?
Without one, every uploaded byte has to pass through your application server on its way to storage — doubling the bandwidth your server pays for and making your server the bottleneck for every concurrent upload. A presigned URL is a short-lived, signed link your server generates (without needing to touch the file itself) that lets the client's browser upload or download directly to or from object storage, so your server only ever deals with small metadata requests.
What does your own backend still need to do in this pattern, if it's not handling the file bytes?
It authenticates the user, decides whether they're allowed to upload this particular file, generates the presigned URL (and records the resulting object's metadata, like its key and owner, once the client confirms the upload finished) — all the authorization and bookkeeping logic, just none of the actual byte-pushing.
Notes
Notes are saved to your account.
Everything else on this page works without one — progress and bookmarks keep themselves in this browser and come with you when you sign in. Notes are the one thing worth not trusting to a browser you might clear.
Your notes could not be loaded.