Base64 and URL encoding

Some channels carry only text. JSON strings, URLs, HTTP headers and email all expect printable characters. Binary data such as images, keys and encrypted blobs has to be converted before it can ride in those channels. Two encodings do most of that work: Base64 for bytes and URL encoding for characters that have meaning inside a URL.

The big idea

Base64 maps any bytes to 64 safe ASCII characters: A–Z, a–z, 0–9, + and /. It reads the input 3 bytes (24 bits) at a time and writes 4 characters, each carrying 6 bits.

URL encoding (also called percent-encoding) does the opposite job. It keeps the bytes but escapes the characters that a URL uses for structure, such as &, ? and spaces.

Walking through Base64

Encode the three bytes of Man.

"Man"            = 4D 61 6E
in binary        = 01001101 01100001 01101110
regroup 6 bits   = 010011 010110 000101 101110
as numbers       = 19     22     5      46
Base64 letters   = T      W      F      u
result           = "TWFu"

Base64 output is 4/3 the size of the input, about 33% overhead. When the input is not a multiple of 3 bytes, the output ends with = padding. You can check this in Python:

import base64

print(base64.b64encode(b"Man").decode())   # TWFu

Output:

TWFu

JWTs use base64url, which swaps + and / for - and _ so the output is safe in URLs.

Walking through URL encoding

Each unsafe byte becomes % followed by two hex digits. A space becomes %20, & becomes %26, and é (UTF-8 bytes C3 A9) becomes %C3%A9. HTML form encoding writes a space as + instead.

from urllib.parse import urlencode

print(urlencode({"q": "cats & dogs", "page": 2}))

Output:

q=cats+%26+dogs&page=2

The & inside the search term is escaped as %26. Without that, the server would read it as a separator and see a separate parameter.

The bug from string concatenation

Build a URL by pasting strings together and the characters can change its meaning.

bad:   /search?q=cats & dogs
server reads:  q = "cats "   and a stray parameter "dogs"
good:  /search?q=cats%20%26%20dogs

In worse cases, an attacker's input adds extra parameters to a request you never meant to send. Use your library's URL builder instead: URLSearchParams in JavaScript, urllib.parse.urlencode in Python, url.Values in Go.

Encode exactly once. Double-encoding turns %20 into %2520, and the server then sees a literal %20 instead of a space.

Common mistakes

• Treating Base64 as protection. Anyone can decode it. Secrets that are only Base64-encoded are a common security review finding.
• Concatenating URL strings. Use the library builder, so every value is escaped.
• Encoding twice. Encode each value once, at the point it enters the URL.
• Forgetting the 33% cost. Base64 inside JSON makes payloads about a third larger. For big binary data, use a separate upload or a binary format.

Check yourself

1. Why is Base64 output longer than its input? It packs 6 bits per character instead of 8, so every 3 input bytes become 4 output characters.
2. A search for cats & dogs returns results for cats only. What is likely wrong? The & was not URL-encoded, so the server split it into a separate parameter. Encode the value as cats%20%26%20dogs.

Key takeaways

• Base64 turns 3 bytes into 4 safe characters, at a cost of about 33% more size.
• Base64 is encoding, not encryption. Anyone can decode it.
• URL encoding escapes structural characters. Build URLs with the library helper, and encode each value exactly once.

Further learning

Next, read timestamps and time zones for the last text-heavy topic: how to store and send times without losing their meaning.