Can I construct a feasible stream cipher out of HMAC and a secure hash algorithm?

Question

I have constructed a stream cipher from a secure hash algorithm and a HMAC. Here is a brief description of the algorithm:

Let: (Actually Objective-C styled pseudo code)

• [data SHA512Hash]: SHA-512 hash of data,
• [data SHA512HMAC:key]: SHA-512 HMAC of data with key,
• [data xor:another]: bitwise XOR of two data and another which have the same length,
• [data bytesTo:index]: first index bytes of data,
• [data removeFirstBytes:count]: remove first count bytes from data,
• [data length]: length of data.
• [data concat:other]: Concatenate other after data.
• [Data new]: allocate an empty buffer.

Pseudocode: (Or actual Objective-C code?)

Data hkey = [key SHA512]; // Hash to make lengths match.
Data segKey = [IV SHA512];

Data dest = [Data new]; // This will be the output
Data last = (Some constant)

do {
    // Cut off hash-sized chunks of source data
    Data segment = [data bytesTo:[hkey length]];
    [data removeFirstBytes:[hkey length]];

    // Derive a new segment key
    segKey = [segKey SHA512HMAC:hkey]; // or [segKey SHA512HMAC:[hkey concat:last]]? If so, how to decrypt?

    // XOR
    last = [segment xor:[segKey bytesTo:[segment length]]] // Truncates to make length match
    dest = [dest concat:last];
} while ([data length] > 0)

From some other questions I am suggested that this is SHA-512 HMAC running in OFB mode but what about the change in the code comment?

This code seem to decipher itself when given the same key.

Answer 1

Here is your algorithm in formula form, which might be easier to understand for non-objective-C-experts:

Encryption:

• $P_1 \mathbin\Vert P_2 \mathbin\Vert \dots \mathbin\Vert P_n := P$ (split plaintext in blocks, same size as hash output, except the last one)
• $K := \operatorname{SHA256}(\mathit{key})$
• $O_0 := \mathit{IV} (= K)$
• $O_i := \operatorname{HMAC}_{\operatorname{SHA256}}(K, O_{i-1})$ for all $i \in \{ 1, \dots, n\}$ (key stream)
• $C_i := P_i \oplus O_i$ for all $i \in \{ 1, \dots, n\}$ (the usual stream cipher XOR)
• $C := C_1 \mathbin\Vert \dots \mathbin\Vert C_n$

(where the last block $O_n$ is truncated to match the length of $P_n$)

Decryption: same, but with $C_i$ and $P_i$ swapped.

• $P_i := C_i \oplus O_i$ for all $i \in \{ 1, \dots, n\}$

As you remarked right, this is output feedback mode (OFB) with HMAC-SHA-512, with the key reused as the IV. (The value you named IV – $K$ here – is actually the effective key, derived by SHA-256 from the original key, the IV is your first segkey, initialized to the same value).

If you use the commented variant with

• $O_i := \operatorname{HMAC}_{\operatorname{SHA256}}(K \mathbin\Vert C_{i-1}, O_{i-1})$,

you have some new mode which is like a mixture of cipher feedback mode and output feedback mode, which uses the cipher feedback as part of the key (?). It is not a synchronous stream cipher anymore, as the keystream depends on the previous ciphertext. You can still decrypt it, you just have to use the previous version of segment instead of the previous version of last as input in the decryption function (i.e. you need one more variable in your program).

If you want to stay with established modes of operation, here is cipher feedback mode (CFB):

• $O_i := \operatorname{HMAC}_{\operatorname{SHA256}}(K, C_{i-1})$

This is Counter mode (CTR):

• $O_i := \operatorname{HMAC}_{\operatorname{SHA256}}(K, i)$

The analogons to ECB, CBC, PCBC don't work with a MAC (or other PRF like a keyed hash), as there is no inverse which would be needed for decryption.

Also, as Ivella remarked, your stream cipher is a lot slower than dedicated stream ciphers, or the same modes applied to a block cipher.

Answer 2

For clarity, we can rewrite this scheme as a keystream S generated by S[i] = HMAC-SHA512(K, S[i-1]), for a key K and with the initialization that S[0] = iv (the used keystream starts at S[1]). Basically, iteratively hash the last output to produce the keystream.

The choice of IV for the variable name is confusing. The first two lines are:

Data IV = [key SHA512]; // Should I derive this? If not, then how?
Data segKey = IV;

Based on the usage here:

segKey = [segKey SHA512HMAC:IV];

and the definition of [X SHA512HMAC:Y], "IV" both used as the HMAC key and the initialized value of segKey is set to IV. (Was the SHA512HMAC line supposed to be segKey = [segKey SHA512HMAC:key] instead?)

This is indeed OFB mode using a PRF (HMAC-SHA512) instead of a block cipher, which should be a suitable substitution.

The alternative proposal is

 [segKey SHA512HMAC:[IV concat:last]]

which concatenates IV and the previous ciphertext block for use as the HMAC key.

This is very close to CFB mode. CFB uses the ciphertext as the data input, whereas this mode used it as an addition to the key. To use CFB, change the line to segKey = [last SHA512HMAC:[IV]]. Both versions can be used for decryption: As you decrypt ciphertext blocks to plaintext, store the ciphertext block in an additional variable to use during the next iteration.

That said about the algorithm, there's a critical flaw. In the beginning, note that the IV is always derived from the key. (Technically, the IV is the key, but regardless, the problem is that they are a deterministic pair.) This means that every keystream produced with that key will be identical since the key will always produce the same IV. Reusing a PRG stream is very bad, since it's easily defeated with a known-plaintext attack and can even be defeated by just a ciphertext-only attack (assuming moderate, but not explicit, knowledge of the plaintext). This means the key can never be safely re-used. Allowing only one-time secure use of a key is probably not a desirable property.

Instead, the IV should be generated separately from the key and should be unique for each encryption with the key. Perhaps start with a random IV and increment by one for each encrypted message.