How does Hash calculate hash for strings?

Question

I have this code

Hash["a","SHA"]
Hash[ToCharacterCode["a"],"SHA"]

and I was expecting to get the same result, but somehow it is different.

My question is: how does Mathematica calculate hashing for strings? From common sense it should convert a string to integer and calculate it for some integer. If so, how does it convert the string to integer?

Answer 1

As of version 11.3.0, the documentation (and implementation) of Hash have been updated.

When given a string, the hash algorithm works on the bytes of that string's UTF-8 representation.

For an example, below hashing the string "a" is the same as hashing the single byte 97 (decimal):

$Version

(* "11.3.0 for Mac OS X x86 (64-bit) (March 5, 2018)" *)

Hash["a", "SHA"] === Hash[StringToByteArray["a"], "SHA"] === Hash[ByteArray[{97}], "SHA"]

(* True *)

It is also worth noting that unlike previous versions, ByteArray does have a special meaning and the hashing is performed directly on the underlying bytes.

That applies for all hash types except the one argument form Hash[e] which can also be specified as Hash[e, "Expression"].

Addendum

Per @Kuba's request, I am adding some code that allows reproducing the results of string hashing as done in previous releases (back to version 9 anyway).

That is, of course, nothing like the actual implementation, just a mock-up intended to give the same hash values.

It has no error checking to speak of and probably is slower than it needs to be.

Clear[byteHash, stringHash];

byteHash[bytes_List, method_: "MD5"] := If[$VersionNumber >= 11.3
  ,
  Hash[ByteArray[bytes], method]
  ,
  Module[{tmp = CreateTemporary[], hash},
   BinaryWrite[tmp, bytes];
   Close[tmp];
   hash = FileHash[tmp, method];
   DeleteFile[tmp];
   hash]
]

stringHash[str_?StringQ, method_: "MD5"] := 
 Module[{bytes, cc = ToCharacterCode[str]},
  bytes = Which[
    $VersionNumber >= 11.3, 
       ToCharacterCode[str, "UTF-8"],
    $VersionNumber >= 11.2 && Max[cc] >= 512, 
       ToCharacterCode[ToString[str, CharacterEncoding -> "ISO8859-1"]],
    $VersionNumber >= 9.0, 
       Mod[cc, 256],
    True, 
       $Failed];
  byteHash[bytes, method]
]