Presigned and multipart uploads

A user uploads a 2 GB video. If the bytes pass through your app server, that server holds the connection, the memory and the bandwidth, twice. If the server restarts mid-upload, the upload dies. Presigned uploads avoid this. The server hands the client a special URL, and the client sends the bytes straight to object storage. Multipart upload then splits big files into parts.

The big idea

Think of a building with a one-time pass. The front desk checks you in and issues a pass for one room, valid for 10 minutes. The pass opens only that room, for that purpose. The building's master key never leaves the desk.

The server does the same. It signs the URL with its own credentials. The client gets a URL that allows one PUT to one key until it expires. No secret reaches the client.

Walking through it

The server chooses the key, records the intent, and returns the URL:

// Sketch: `s3` and `db` are assumed. The server picks the key, not the client.
import { PutObjectCommand } from "@aws-sdk/client-s3"
import { getSignedUrl } from "@aws-sdk/s3-request-presigner"
import { randomUUID } from "node:crypto"

export async function createUpload(userId: string, contentType: string) {
  const key = `uploads/${userId}/${randomUUID()}`
  const url = await getSignedUrl(
    s3,
    new PutObjectCommand({ Bucket: "media-prod", Key: key, ContentType: contentType }),
    { expiresIn: 600 }, // 10 minutes
  )
  await db.files.insert({ key, userId, status: "PENDING" }) // record intent first
  return { key, url }
}

For uploads, 5 to 15 minutes is a typical expiry. S3 presigned URLs can last at most 7 days.

For large files, one PUT is fragile. One network blip near the end restarts everything. Multipart upload fixes that.

• CreateMultipartUpload returns an UploadId.
• Each part, at least 5 MiB except the last and at most 10,000 parts, gets its own presigned UploadPart URL and returns an ETag.
• CompleteMultipartUpload takes the part numbers and ETags. The object appears only then.

Work through a 4 GB video on a 50 Mbit/s upload link:

Single PUT: 4 GB × 8 bits / 50 Mbit/s = 640 s ≈ 11 min
  One drop at minute 10 → start over.

Multipart, 64 MiB parts: 4,096 MiB / 64 = 64 parts, 4 in parallel
  Total time is still about 11 min on a saturated link.
  A drop costs at most one part: 64 × 8 / 50 ≈ 10 s of re-upload.

Parallel parts also use the client's bandwidth fully.

Set a lifecycle rule to abort incomplete multipart uploads after a few days. Orphaned parts are billed as storage, but they're invisible in normal listings.

Common mistakes

• Letting clients choose keys. The server should choose, so it knows what it created and can avoid collisions.
• Long-lived URLs. Use short expiries. A leaked URL should stop working quickly.
• Forgetting to abort failed multipart uploads. Parts you never complete keep costing money.

Check yourself

1. Why does the server record the file row before returning the URL? So a cleanup job can find and delete a row whose upload never finished.
2. A single PUT of 4 GB drops at minute 10. What's lost? The whole upload. With 64 MiB parts, a drop costs at most one part, about 10 seconds.

Key takeaways

• Clients upload to storage directly, using URLs the server signs.
• Presigned URLs grant one operation on one key until they expire.
• Multipart uploads limit the cost of a failure to one part.
• Abort incomplete uploads, or you pay for their parts.

Further learning

The Amazon S3 User Guide has the full multipart and presigned URL reference.

Next, read about metadata and object consistency, which keeps your database and the store in agreement after an upload.