1

To use newer accelerators like this, I need to perform matmul in 4-bit precision. How do I tell whether this operation is stable? Wondering if there well common heuristics in terms of properties of matrices $\{A_i\}$.

For classification tasks, matrix multiplication is stable if $\langle y, y_{\text{4-bit}}\rangle>0$ with high probability for random $x$ and $y=A_d\ldots A_2 A_1 x$.

Orthogonal matrices seem more likely to work in 4-bits, but that's too hard of a restriction.

Yaroslav Bulatov
  • 2,655
  • 11
  • 23
  • 1
    What is "4-bit precision"? Are these integers? I'm not aware of 4-bit floating point numbers. – Wolfgang Bangerth Feb 21 '24 at 00:57
  • Yes, they 4 bit integer approximations of floating point (denominator needs to be stored separately as, say, one extra float32 per matrix) – Yaroslav Bulatov Feb 21 '24 at 04:05
  • 1
    That's wild.... – Wolfgang Bangerth Feb 21 '24 at 13:18
  • 1
    @WolfgangBangerth This is what majority of worlds compute will soon be used for (if not already) - super low precision matmuls for AI models – Yaroslav Bulatov Feb 21 '24 at 18:45
  • @WolfgangBangerth This probably comes from Quantization of neural networks. In addition to low memory and training costs, it probably improves generalization (reduces overfitting) of networks as well. https://stackoverflow.blog/2023/08/23/fitting-ai-models-in-your-pocket-with-quantization/ – NNN Feb 22 '24 at 09:53
  • 1
    Using 4-bit numbers you can either represent 16 non-negative numbers, or 8 positive and 8 non-positive numbers. The moment you multiply two number above 4, it will overflow. Matrix-matrix multiplication would be very unstable with these numbers – Abdullah Ali Sivas Feb 25 '24 at 23:17
  • @AbdullahAliSivas AI models (sometimes) run successfully at 4-bit quantization, so the question is, what is special about those matrices to make 4-bit multiplication stable in those cases – Yaroslav Bulatov Feb 26 '24 at 00:46
  • 1
    That is a fairer question. If the input is normalized and convolution kernel is relatively small (or if you are working with finite fields -think of modular arithmetic-) then I would expect it to be reasonably stable. For example, if the input is the output of a logit layer and the convolution is 3x3 (as it is a classic scenario in image classifiers), this quantization would be sufficient. It is a very operator and domain dependent question. – Abdullah Ali Sivas Feb 26 '24 at 04:40
  • @AbdullahAliSivas in the post I give a definition of "stability" which is more appropriate for this kind of doman, so the interesting question is what are the relevant properties of the matrix which makes this kind of stability possible – Yaroslav Bulatov Feb 26 '24 at 05:59

0 Answers0