Facebook engineers come up with superior compression algorithm

A team of Facebook engineers have come up with a new lossless compression algorithm that is much faster than zlib and xz at compression and decompression, while also offering a slightly better compression ratio than lz4 and zlib.

The algorithm is called Zstandard, Facebook shared the source code but the library comes with an important extra condition: that you won't sue Facebook for patent infringement, nor countersue the social network if Facebook sues you first.

As The Register reports, the news sounds a bit like the initial story of US TV sitcom Silicon Valley, where the protagonist develops an innovative compression algorithm. Here's a bit more info about how Zstandard works:

The idea, basically, is to replace zlib with code that is mostly branchless and is optimized for parallel execution, which boosts performance. The library also benefits from finite state entropy, which is a state-of-the-art probability compressor design, among other improvements over zlib. Zlib is used in pretty much everything, including compressing zip and gzip archives and in-transit web traffic, so Facebook is keen to get its superior library adopted.

Facebook also shared MyRocks, a new open source project that uses the company's RocksDB as a new MySQL storage engine. With MyRocks, you get all the benefits from all the features in MySQL, combined with RocksDB as the backend storage engine. According to Facebook, the deployment of MyRocks enabled the social network to reduce its storage requirements by 50 percent thanks to the system's improved data compression.