-3

There is a supposed NIST tool called SP800-90B_EntropyAssessment on that Git thing. It's designed to be used in accordance with NIST's Recommendation for the Entropy Sources Used for Random Bit Generation. Essentially the tools measure entropy in datasets from entropic data sources. It's meant to output 0 - 8 bits/byte, (but be warned that the Python tests take many hours to complete).

I don't know if that Git repository is officially sanctioned by the US Department of Commerce, or if these tools have been written by hobbyists during their lunch breaks.

  • The C++ IID tool doesn't compile at all (bzlib.h: No such file or directory)

  • The C++ non-IID tool gives the included 1.2MB $ \pi $ sequence an entropy of 0.08 bits/byte, which is based on a Compression Test Estimate. That's a tenth of what I measure it at, and it's hard to reconcile with the scientific consensus that the constant is perfectly random. The expected value should be ~1 bit/byte give or take the min-entropy variance.

  • It's not clear as to whether their $ \pi $ sequence is IID or non-IID as it's an odd file format and the definition of IID is vague regarding the degrees of a Markov sequence.

  • For some combinations of test and data source, you can get entropy > 8 bits/byte which infers a very good entropy generator or poorly validated code (definitely missing assert statements).

Does anyone have experience of this tool? I've never read any TRNG paper that's ever mentioned using it to estimate their entropy. And I've read a lot of them. Perhaps I've misunderstood what the tool does. Can this tool be relied upon to measure entropy correctly and securely?

Paul Uszak
  • 15,390
  • 2
  • 28
  • 77
  • Comments are not for extended discussion; this conversation has been moved to chat. – Ella Rose Jul 10 '19 at 15:08
  • @Owl I've pretty much sussed 90B in the last year, and the answer is that no, it's not reliable. In fact 90B is pretty useless in most real world entropy source designs due to it's silly input distribution requirement. There's an analysis here and some further links. No one really uses 90B for those reasons. Ad-hoc (and wacky) decorrelation + Shannon is the commonest technique, but the easiest is to just compress your raw sample with CMIX. – Paul Uszak Aug 24 '19 at 21:31
  • 1
    @Owl Why does the question need to be re-opened? It has an accepted answer. Do you intend to post a competing answer? As for re-opening: If the community feels a question should be re-opened, then they can cast votes to cause it to become so. Closing questions works similarly. This was closed by a former mod, but the community can still vote to re-open it. On that note... 1/2 – Ella Rose Aug 25 '19 at 03:09
  • 1
    @Owl I'd like to take this opportunity to mention that "question closing gestapo" is a slight against every user that participates in curating the community, and is tip-toeing very closely to the "not nice" side of things. If you really want this re-opened, then make positive, helpful contributions to the community until you have 3,000 rep, then vote for it. – Ella Rose Aug 25 '19 at 03:10
  • @EllaRose Just a point of clarification. The answer is only accepted as appreciation for the help Mr. new gave. It hasn't answered the core issue (which is now moot though). It's actually my fault, as I've inadvertently phrased the question as programmatic, rather than focusing on the underlying problem of reliable entropy measurement of TRNG designs. – Paul Uszak Aug 25 '19 at 13:13
  • 1
    @EllaRose closing relevant questions down isn't nice either. By the way, do you get points for closing down questions?? If you want to complain about my comments to whoever manages stack overflow, then feel free. This is an incredibly frustrating problem. – Owl Aug 27 '19 at 10:54
  • 1
    @Owl Obviously, no, you do not get points for closing questions. That would create an incentive structure that motivates users to close as many questions as possible, rather than only those they feel are inappropriate. I don't have to complain to stackexchanges management; It is my job to keep crypto.se clean, focused, and friendly. This is the reason why I mentioned I mentioned "be nice" and asked for why the question should be re-opened. If there was a convincing argument as to why this should be re-opened, it could be done. I asked you for such arguments, but I have yet to receive any. – Ella Rose Aug 27 '19 at 14:20
  • 1
    Anyways: Comments should not be used for extended discussion, and we're creating notifications in Paul's inbox by having this conversation here. If this is to continue, please use The Side Channel or meta rather than posting content that does not directly address the question/questioner here. – Ella Rose Aug 27 '19 at 14:22
  • 1
    The question was shut down. No reason was given. The question was voted down, to discourage people from asking any of the 2 other questions in OP. There is no link or reference or anything else on the CLOSED message saying how to reopen it, how to contest it. It's just an absolute, indisputable brick wall and that's why this is so frustrating. It's not even clear how to complain about these practices. – Owl Aug 28 '19 at 14:03
  • 1
    @Owl Paul knows how to contest a closed question. However, this question, while vaguely related to cryptography, isn't precisely on-topic for this site. The majority of the question is about programming. – forest Aug 29 '19 at 06:42
  • 1
    The question is very pertinent to those working in the field. Things have changed with the code and the answer needs amendment, particularly with respect to libdivsufsort and details on the counterintuitive behavior of the truncate options. Closing this was not the best idea. – David Johnston Mar 29 '21 at 18:09
  • @DavidJohnston Absolutely! The nexus of cryptography is the provision of highly entropic keys (typically generated via TRNGs). If you can't measure a thing, you can't manage a thing. And the linked documents indicate that NIST entropy measurement is very, very poor. It's only right that we explore why that is, and what decent people can do to mitigate. Qui bono? – Paul Uszak Mar 29 '21 at 19:45

1 Answers1

3

The C++ IID tool doesn't compile at all (bzlib.h: No such file or directory)

From the code:

 #include "bzlib.h" // sudo apt-get install libbz2-dev

Authors are indicating to install the bzip2 development package from your package manager, since is not installed by default. Their command will work on Debian-based distros.

You just need to install the package.
For me worked installing sudo eopkg install bzip2-devel (i'm on Solus), but it will vary upon OS and distro. Search for the bzip2 development package.

valerio_new
  • 268
  • 3
  • 13
  • You've helped but made it more confusing :-) It compiles now. Thanks, but now I can't get a sensible h value for $ \pi $ with either Python or C++ tools . Must just be me :-( – Paul Uszak Mar 31 '18 at 00:16
  • You can still edit /add if you have further insights... – Paul Uszak Mar 31 '18 at 02:01