What's purpose of the new function BinarySerialize?

Question

11.1 introduced a new function BinarySerialize, but I don't know what it can do better than the traditional method.Its behavior is very similar to Compress,though I cannot find any advantage of it.It even consumes more space than Compress,such as

BinarySerialize[Range[100]] // Normal // Length

805

Compress[Range[100]] // ToCharacterCode // Length

290

And ByteCount[BinarySerialize[Range[100]]] is also greater than ByteCount[Compress[Range[100]]].So what's purpose of this function? Anyone can provide a good example to use it?

Answer 1

Disclaimer: This answer is written from a user's point of view. For useful insider information on this topic see this discussion with Mathematica developers on Community Forums.

Introduction

Binary serialization is rewriting expressions as an array of bytes (list of integers from Range[0, 255]). Binary representation of expression takes less space than a textual one and also can be exported and imported faster than text.

How do Compress and BinarySerialize functions work?

Compress (with default options) always does three steps:

1. It performs binary serialization.
2. It deflates result using zlib.
3. It transforms deflated result to a Base64-encoded text string.

BinarySerialize performs only binary serialization and sometimes deflates the result using zlib. With default options it will decide itself if it wants to deflate or not. With an option PerformanceGoal -> "Speed" it will avoid deflation. With an option PerformanceGoal -> "Size" it will likely deflate. BinarySerialize returns a ByteArray object. ByteArray is something like a packed array of 8-bit integers. However FullForm of ByteArray is visualized as Base64-encoded text string. This visualization can be somewhat misleading, because internally ByteArrays are stored and operated in binary form, not as text strings.

Binary serialization algorithms of Compress and BinarySerialize

Original serialization algorithm of Compress is described in this answer. That algorithm is not very optimized for size and produces larger-then-necessary output for many typical expressions. For example, it has no support for packed arrays of integers and rewrites such arrays as nested lists, which take a lot bytes.

BinarySerialize uses a more size-optimized binary serialization algorithm compared to what Compress (with default options) does. This algorithm supports packed arrays of integers, has optimizations for integers of different size (8,16,32 bit), stores big integers in binary form (not as text strings), and has other optimizations.

Applications of BinarySerialize

Using BinarySerialize we can write our own Compress-like functions with better compression. For example we can write myCompress function which does the same three steps as original Compress, but uses BinarySerialize for the serialization step:

myCompress[expr_]:=Module[
    {compressedBinaryData},
    compressedBinaryData = BinarySerialize[expr, PerformanceGoal->"Size"];
    Developer`EncodeBase64[compressedBinaryData]
    ];

myUncompress[string_]:=Module[
    {binaryData},
    binaryData = Developer`DecodeBase64ToByteArray[string];
    BinaryDeserialize[binaryData]
    ];

Even for simple integer list we can see size reduction.

Compress[Range[100]] // StringLength
(* 290 *)

myCompress[Range[100]] // StringLength
(* 244 *)

myUncompress[myCompress[Range[100]]] === Range[100]
(* True *)

If we take an expression with large number of small integers we get much more noticeable improvement:

bitmap = Rasterize[Plot[x, {x, 0, 1}]];

StringLength[Compress[bitmap]]
(*31246*)

StringLength[myCompress[bitmap]]
(*17820*)

myUncompress[myCompress[bitmap]] === bitmap
(* True *)

Conclusion

The example above shows that the result of a simple user-defined function myCompress based on a BinarySerialize can be almost twice more compact than the result of Compress.

Outlook

To decrease the output size even further one can use a compression algorithm with higher compression settings (in the second step) or use Ascii85-encoding instead of Base64 in the third step.

Appendix 1: Undocumented options of Compress

I have noticed that in Version 11.1 Compress has more undocumented options than in previous versions. Those options allows one to:

• Disable both compression and Base64 encoding and return a binary serialized result as a string with unprintable characters:

  Compress[Range[100], Method -> {"Version" -> 4}]

• Change binary serialization algorithm to a more efficient one, but not exactly to BinarySerialize.

  Compress[Range[100], Method -> {"Version" -> 6}] // StringLength

  (* 254 *)

There is also a "ByteArray" option shown in usage message ??Compress but it does not work in Version 11.1.

Note that this behavior is undocumented and may change in future versions.

Appendix 2: Compression option of BinarySerialize

Just for fun one can manually compress result of BinarySerialize[..., PerformanceGoal -> "Speed"] to get the same output as BinarySerialize[..., PerformanceGoal -> "Size"] produces. This can be done with the following code:

myBinarySerializeSize[expr_]:=Module[
    {binaryData, dataBytes, compressedBytes},
    binaryData = Normal[BinarySerialize[expr, PerformanceGoal->"Speed"]];
    dataBytes = Drop[binaryData, 2]; (*remove magic "7:"*)
    compressedBytes = Developer`RawCompress[dataBytes];
    ByteArray[Join[ToCharacterCode["7C:"], compressedBytes]]
    ]

We can check that it gives the same result as PerformanceGoal -> "Size" option

data = Range[100];
myBinarySerializeSize[data] === BinarySerialize[data, PerformanceGoal -> "Size"]

Appendix 3: zlib compression functions

Description of undocumented zlib compression/decompression functions Developer`RawCompress and Developer`RawUncompress can be found in this answer.

Appendix 4: Base64 encoding functions

Usage of Base64 encoding/decoding functions from the Developer` context can be explained using the following code:

binaryData = Range[0, 255];

Normal[
    Developer`DecodeBase64ToByteArray[
        Developer`EncodeBase64[binaryData]
        ]
    ] == binaryData

(* True *)

Answer 2

Measure the difference in bytes with ByteCount.

BinarySerialize[Range[100]] // ByteCount

901

Range[100] // ByteCount

936

BinarySerialize will occupy fewer bytes. Not a big difference in this trivial example but the difference generally widens as the object to be serialised gets larger.

Also you must Compress both for apples-to-apples comparison.

Compress@BinarySerialize[Range[100]] // ByteCount

336

Compress@Range[100] // ByteCount

368

Hope this helps.

Answer 3

Here's an interesting complement to Edmund's answer. It's not always the case that the ByteArray form will be more compact. DocFind is just a table of reflinks I use for doc searching.

In[155]:= bs = BinarySerialize@DocFind[]; // RepeatedTiming
Out[155]= {1.8, Null}
In[156]:= cp = Compress@DocFind[]; // RepeatedTiming
Out[156]= {1.74, Null}
In[157]:= bs // ByteCount
Out[157]= 3027593
In[158]:= cp // ByteCount
Out[158]= 233104

On the other hand it's minimally faster to BinaryDeserialize:

In[159]:= Uncompress@cp; // RepeatedTiming
Out[159]= {0.34, Null}
In[160]:= BinaryDeserialize@bs; // RepeatedTiming
Out[160]= {0.30, Null}

If we use a much larger input we get a much more interesting result, though:

In[145]:= bs2 = BinarySerialize@Range[10000000]; // AbsoluteTiming
Out[145]= {0.16928, Null}
In[153]:= cp2 = Compress@Range[10000000]; // AbsoluteTiming
Out[153]= {4.92081, Null}

My intuition would suggest that BinarySerialize should almost always be as fast or faster than Compress but I'd be interested to be proven wrong.

Here the serialized version is much more compact:

In[161]:= bs2 // ByteCount
Out[161]= 80000104
In[162]:= bs3 // ByteCount
Out[162]= 31276128

And it's so much faster to use BinaryDeserialize here than Uncompress:

In[163]:= BinaryDeserialize@bs2; // RepeatedTiming
Out[163]= {0.13, Null}
In[164]:= Uncompress@cp2; // AbsoluteTiming
Out[164]= {29.8372, Null}

Again, because it's a byte format, my intuition would suggest Binary* will be faster and often more compact than a compressed string.

Update: yode confirmed that it is platform independent

On the other hand, since it's a byte format I would also believe it's platform dependent -- i.e., I can't take my serialized ByteArray and mail it to you and expect it to work if we have different operating systems/architecture, the same way I can't expect that of .mx files.

Answer 4

In general, serialization is widely used for two purposes:

1. To send large objects "over the wire" by TCP/IP from one computer to another. For example, a company might have a programming object such as a BillOfMaterials containing a large number of part numbers. The company would serialize the BillOfMaterials, send it to a vendor, and request a quote and an estimate of parts availability. The vendor would de-serialize the BillOfMaterials, assign prices to the part numbers and indicate whether in stock, serialize the completed BillOfMaterials, and return the quote to the originating company.

2. Memoization. For recursive algorithms using bimomial (Binomial[...]) or multinomial (Multinomial[...]) coefficients, the integers involved become very large very fast. Even computers with large amounts of RAM cannot keep intermediate results in core memory and still do useful work. It is common to keep a dictionary or map (in Mathematica, an Association) of the results of computing a binomial or multinomial coefficient as a serialized object on disk. The dictionary key is a concatenation of the binomial or multinomial arguments, and the dictionary value is the file name of the serialized result. So, instead of recomputing the coefficient, it is read from disk and de-serialized. The same basic idea could be used in Dynamic Programming, in which a large problem is split into many smaller problems. Since the smaller problems may have to be solved over and over again, their solutions can be serialized to disk and recalled as needed.