pako1324

Notes on compression and binary formats. Mostly for myself.

Compression levels

Level 9 is almost never worth it. Going from 6 to 9 typically buys two or three percent of size while roughly tripling CPU time. For anything served over the network the extra latency costs more than the saved bytes, and the difference disappears entirely once you account for TCP window behaviour on the first round trip.

When not to compress

Already-compressed payloads are the obvious case: JPEG, PNG, most video containers, anything gzipped upstream. Less obvious is small text. Below roughly 150 bytes the deflate header and the dictionary overhead make the output larger than the input, and the check happens after the work is done. Setting a minimum size threshold before invoking the compressor saved more CPU here than any level tuning.

Streaming versus buffering

Buffering the whole payload before compressing is simpler and produces slightly better ratios, since the dictionary sees more context. Streaming keeps memory flat and lets the first bytes leave earlier. For request bodies of unknown size the memory argument wins — a single oversized upload with the buffered approach is enough to take the process down, and it will happen on the day nobody is watching.

Dictionary reuse

If the payloads share structure — repeated JSON keys, a common header, the same schema over and over — a preset dictionary is the cheapest improvement available. Ratios improved by about a third on small messages here. The catch is versioning: both sides must agree on the dictionary, and there is no graceful way to detect a mismatch other than the output being garbage.