How does OpenTimestamps work technically?

Question

The website doesn't explain anything, it just points to a bunch of command-line clients. I'm trying to write a client, and using a library that exists, but still the thing is cryptic to me.

For example, when a call to /digest with a binary payload 3c367e3dcc8171c287f300e4650f887aa36a046b68257158ae691fb9a9aa5078 (in hex) is made the server returns a binary message that translates to something like

message 3c367e3dcc8171c287f300e4650f887aa36a046b68257158ae691fb9a9aa5078
APPEND bdddce3c57eebd3145db11c7f8a88ade
 message 3c367e3dcc8171c287f300e4650f887aa36a046b68257158ae691fb9a9aa5078bdddce3c57eebd3145db11c7f8a88ade
 SHA256
  message 7f0d2f1a6654bf56a061886afd4d9434ea20cdf5bf0af35c5c539c2eff9ac725
  PREPEND 6508c034
   message 6508c0347f0d2f1a6654bf56a061886afd4d9434ea20cdf5bf0af35c5c539c2eff9ac725
   APPEND 580adf5d1d0cc157
    message 6508c0347f0d2f1a6654bf56a061886afd4d9434ea20cdf5bf0af35c5c539c2eff9ac725580adf5d1d0cc157
    VERIFY pending(url=https://bob.btc.calendar.opentimestamps.org)

What does that mean?

If do another call with 5f2c0653c7f703abf9ac2b083c256a7fbcb61984833d45e6449579dc4bd71ff1 I get:

message 5f2c0653c7f703abf9ac2b083c256a7fbcb61984833d45e6449579dc4bd71ff1
APPEND f0740a9b96d84df43c845b51274a375e
 message 5f2c0653c7f703abf9ac2b083c256a7fbcb61984833d45e6449579dc4bd71ff1f0740a9b96d84df43c845b51274a375e
 SHA256
  message a3bdd6448fdb88bb1a67f933c19c8c6a5a4094b075368a1c774ff8fabd6f3da2
  PREPEND 6508c495
   message 6508c495a3bdd6448fdb88bb1a67f933c19c8c6a5a4094b075368a1c774ff8fabd6f3da2
   APPEND c7eac04764d882a4
    message 6508c495a3bdd6448fdb88bb1a67f933c19c8c6a5a4094b075368a1c774ff8fabd6f3da2c7eac04764d882a4
    VERIFY pending(url=https://bob.btc.calendar.opentimestamps.org)

The final message has the same prefix in both cases, but the rest is different. I don't understand the logic for the PREPEND/APPEND stuff. Are these APPEND and PREPEND messages things submitted by others to be timestamped? Why is the server concatenating multiple messages like that instead of making a simpler merkle tree?

After that step it is possible to call the server at /timestamp/6508c0347f0d2f1a6654bf56a061886afd4d9434ea20cdf5bf0af35c5c539c2eff9ac725580adf5d1d0cc157 and it returns "Pending confirmation in Bitcoin blockchain", which makes me think that is the merkle root the server is trying to publish to Bitcoin on an OP_RETURN.

But it also returns that same message when I call the same endpoint with the second final message, i.e. after the merkle root was supposedly updated in the server.

UPDATE:

After many hours, both the timestamps above were confirmed on Bitcoin and now the OTS server has returned an answer that translates to:

message 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
SHA256
 message 541b3e9daa09b20bf85fa273e5cbd3e80185aa4ec298e765db87742b70138a53
 APPEND 250009906654d59cea5374a77edd8682ea94b0d77a637a146416e3ba09104a95
  message 541b3e9daa09b20bf85fa273e5cbd3e80185aa4ec298e765db87742b70138a53250009906654d59cea5374a77edd8682ea94b0d77a637a146416e3ba09104a95
  SHA256
   message 01cde257282d7bdb41f0c99f6e9c0a8113074c28adf66ea6623c932fbc75e427
   PREPEND a103433c90fd18c0e4637fb877521cec93c523fe0d22c760395d9146dec02bc9
    message a103433c90fd18c0e4637fb877521cec93c523fe0d22c760395d9146dec02bc901cde257282d7bdb41f0c99f6e9c0a8113074c28adf66ea6623c932fbc75e427
    SHA256
     message fa769d538d837a656af05ea214419171dd2557a5a4679d24fd21634496af0270
     APPEND cfb4f52de85cc1acde4fbcafe8e3cb22f536ffebddfff6cbe93d6d6d1c86a5f4
      message fa769d538d837a656af05ea214419171dd2557a5a4679d24fd21634496af0270cfb4f52de85cc1acde4fbcafe8e3cb22f536ffebddfff6cbe93d6d6d1c86a5f4
      SHA256
       message 32866a97a201463cb7831d9eb75f0c768d6cc5bfdb333f485089ab2a97198442
       APPEND 4b958aaa5ffdd07b996f5306490c07c2135fc7ab94c2254b030568f60b0edec1
        message 32866a97a201463cb7831d9eb75f0c768d6cc5bfdb333f485089ab2a971984424b958aaa5ffdd07b996f5306490c07c2135fc7ab94c2254b030568f60b0edec1
        SHA256
         message 940bfd44326bfcbad05d44146413ce994d456952dd0477c7f6f6c9c081484658
         APPEND 2b0b056026cb4b1870120f7d43f0f618ca5eb165f6196b3ff72ff4eb1087df6c
          message 940bfd44326bfcbad05d44146413ce994d456952dd0477c7f6f6c9c0814846582b0b056026cb4b1870120f7d43f0f618ca5eb165f6196b3ff72ff4eb1087df6c
          SHA256
           message 28ed9e3d6b57ede4813a5a774aabd3f3073612c2b25fcc35a001d7df3167d5c7
           PREPEND 5bcb0f5cf7f50364e579a25de1e3f4e504e1b51aaf20bd5e18f0bf5a9f7f7f5c
            message 5bcb0f5cf7f50364e579a25de1e3f4e504e1b51aaf20bd5e18f0bf5a9f7f7f5c28ed9e3d6b57ede4813a5a774aabd3f3073612c2b25fcc35a001d7df3167d5c7
            SHA256
             message c7d9192d9970a7f6d3b6a6ea07be48567d2dd4d7be21ad295b0672f168055225
             PREPEND f5f0aeacc5aba12b3c9a3a7b1a9e2181689e5543220a3f0b175fc4a023296842
              message f5f0aeacc5aba12b3c9a3a7b1a9e2181689e5543220a3f0b175fc4a023296842c7d9192d9970a7f6d3b6a6ea07be48567d2dd4d7be21ad295b0672f168055225
              SHA256
               message 60af325e7fa27152c79d8dd5c162f4cd484ee0c37597a069b020d183457c2470
               APPEND 984fd8a7d4f66e0a9e5fc82b7b73fe0d93f440a8587630fbb86ff91962e1be98
                message 60af325e7fa27152c79d8dd5c162f4cd484ee0c37597a069b020d183457c2470984fd8a7d4f66e0a9e5fc82b7b73fe0d93f440a8587630fbb86ff91962e1be98
                SHA256
                 message 00dffc1cbb9acb380df47631c8765a56221cd233e4eb0d14550d532aea37c9f3
                 APPEND 4930d3c401e9c155b005f73b68a2f26b978f5992f91397f82e34e470e593ea5f
                  message 00dffc1cbb9acb380df47631c8765a56221cd233e4eb0d14550d532aea37c9f34930d3c401e9c155b005f73b68a2f26b978f5992f91397f82e34e470e593ea5f
                  SHA256
                   message 8dc0351ea72247a104986e03cb9d52e804ff6d964a3a6ed63d25d5d485ef1e9a
                   APPEND b1ff48793f99802fd4fc063902da38716d3b7e13248ed8ab1ec4236ba83e7501
                    message 8dc0351ea72247a104986e03cb9d52e804ff6d964a3a6ed63d25d5d485ef1e9ab1ff48793f99802fd4fc063902da38716d3b7e13248ed8ab1ec4236ba83e7501
                    SHA256
                     message f56ce0318b66aee3bd6c6dd4775dc37913e6373f4a9891bda9e402ffb9b53671
                     APPEND fe12f09cdfe6bbe7591966070dfb2fda2f519ba7c7acdfe70d0478157feae86b
                      message f56ce0318b66aee3bd6c6dd4775dc37913e6373f4a9891bda9e402ffb9b53671fe12f09cdfe6bbe7591966070dfb2fda2f519ba7c7acdfe70d0478157feae86b
                      SHA256
                       message 283fd28cde2cba9a722f0f56a59b35db339531a253b5e6ad463a07fb7df74e7a
                       APPEND 244fa5bc8e7223d6272645426024ce9d8a4465ab5901613732160ae38185b62c
                        message 283fd28cde2cba9a722f0f56a59b35db339531a253b5e6ad463a07fb7df74e7a244fa5bc8e7223d6272645426024ce9d8a4465ab5901613732160ae38185b62c
                        SHA256
                         message f7cc2fd78a1a97a9054d55e64aa37f42f6e2d263e77e5a82d5c3bc02901d0d75
                         APPEND 3bcfd92d8fa55165665ca74b376b752eb4375c41791ad49ec275e4785585eea5
                          message f7cc2fd78a1a97a9054d55e64aa37f42f6e2d263e77e5a82d5c3bc02901d0d753bcfd92d8fa55165665ca74b376b752eb4375c41791ad49ec275e4785585eea5
                          SHA256
                           message 027b866868f87f0fc171390fc5168a75015136a3a8ff7274fef445c681034b74
                           PREPEND 854ce08b66850863967945e870b0189503abb1699445004546f8cc5095670738
                            message 854ce08b66850863967945e870b0189503abb1699445004546f8cc5095670738027b866868f87f0fc171390fc5168a75015136a3a8ff7274fef445c681034b74
                            SHA256
                             message 08dff6f8e74c94315efb3c6d26bd72c79a29cc14cba7c05d4263579a6fc9c10b
                             PREPEND 0100000001b9a03d9e2f04f72a3e905d9b88ee4e65538396c22c27519b2b3af3c1435c951c0000000000feffffff027920040000000000160014cf1eca98d98649bae0d4879d6b48e09ae4713b010000000000000000226a20
                              message 0100000001b9a03d9e2f04f72a3e905d9b88ee4e65538396c22c27519b2b3af3c1435c951c0000000000feffffff027920040000000000160014cf1eca98d98649bae0d4879d6b48e09ae4713b010000000000000000226a2008dff6f8e74c94315efb3c6d26bd72c79a29cc14cba7c05d4263579a6fc9c10b
                              APPEND b9550c00
                               message 0100000001b9a03d9e2f04f72a3e905d9b88ee4e65538396c22c27519b2b3af3c1435c951c0000000000feffffff027920040000000000160014cf1eca98d98649bae0d4879d6b48e09ae4713b010000000000000000226a2008dff6f8e74c94315efb3c6d26bd72c79a29cc14cba7c05d4263579a6fc9c10bb9550c00
                               SHA256
                                message 84816d5322d4344b5282886825811f0918b8efe2943499a595dbd95fdceb32d8
                                SHA256
                                 message 698814afec9f3fdcf7b3ab9739280a5a9123ee9bff8952649299f76eb2f99dd8
                                 PREPEND 8f73e0ac409d12fbb3bb13e487727f1ac818353b39bd5ae8e77276142849aeca
                                  message 8f73e0ac409d12fbb3bb13e487727f1ac818353b39bd5ae8e77276142849aeca698814afec9f3fdcf7b3ab9739280a5a9123ee9bff8952649299f76eb2f99dd8
                                  SHA256
                                   message 127b0510549272af35d7f8ba789a0f44cfba700a2790e634543541e3dd3581a8
                                   SHA256
                                    message 10850bb1c8154b79d5502502d6e759072e7225f3daf7649ca4c9b86445408593
                                    PREPEND e5316f3fbbc3e3c1372e6e10aec446402f5332b9d1b4e57a56101fc6894ea53e
                                     message e5316f3fbbc3e3c1372e6e10aec446402f5332b9d1b4e57a56101fc6894ea53e10850bb1c8154b79d5502502d6e759072e7225f3daf7649ca4c9b86445408593
                                     SHA256
                                      message d260097e8381b1d44e54366e271eda39f4eeab4d2f290585385e1344cfc190ee
                                      SHA256
                                       message ec0127bc24041cecdf5af99cb701340e00b9cf248648682fd40474e4b4307dcf
                                       APPEND f3c07fc356be819696e598ddccb19bf157d89f8563944cd5b8567fb436188d76
                                        message ec0127bc24041cecdf5af99cb701340e00b9cf248648682fd40474e4b4307dcff3c07fc356be819696e598ddccb19bf157d89f8563944cd5b8567fb436188d76
                                        SHA256
                                         message 37a89acfebd20ed206b917703c4966721b669d3f9eedbfb9ec572567f6e4fe8f
                                         SHA256
                                          message ec2b59a232a120274b80f21d9a4f7c0b34e969349102b203c26ffb88fd9b421f
                                          PREPEND bdbaf3a4688157ea8f6badcfbbadeb0c3835b4c893d87ce1846bb5fdb0fdf68a
                                           message bdbaf3a4688157ea8f6badcfbbadeb0c3835b4c893d87ce1846bb5fdb0fdf68aec2b59a232a120274b80f21d9a4f7c0b34e969349102b203c26ffb88fd9b421f
                                           SHA256
                                            message b4b334c82b606e2a16bf0b4a31d9803cb5ed3bda64bc21915984c921975ceda5
                                            SHA256
                                             message 56e773e1da7d62d57fbf110627429f1ed5a506c0b3d5acce3b90ff9b22cce71b
                                             PREPEND fd01da859f54fc2e9238a5812756f6280188bb1d71e1e49036725eb2cdc881d1
                                              message fd01da859f54fc2e9238a5812756f6280188bb1d71e1e49036725eb2cdc881d156e773e1da7d62d57fbf110627429f1ed5a506c0b3d5acce3b90ff9b22cce71b
                                              SHA256
                                               message 14d4d3e728511926f8ef87327112fa2898a6189038062bbd0093bc53248a3bb2
                                               SHA256
                                                message 734ac4b2bd892a19be81ee79690edc8e134435bc26694a9e677a88152cb3dd0d
                                                APPEND 5b488ab71dc10b34fca7020b4212f4d5e3ea635c1ea825d4283e2f4237f57e99
                                                 message 734ac4b2bd892a19be81ee79690edc8e134435bc26694a9e677a88152cb3dd0d5b488ab71dc10b34fca7020b4212f4d5e3ea635c1ea825d4283e2f4237f57e99
                                                 SHA256
                                                  message 0549e254e7b4ee305677dfd842528c9d97c4ebf2e0163cb9597fe18739fb727e
                                                  SHA256
                                                   message 34e5b4c7c8915fd981c1254dbd02a94debe55d1e04b380d0ed0c07dc5254ffe0
                                                   APPEND 2851d711964fac11fbc25656a733b3efef04edc3354f127b0939122e11363cb6
                                                    message 34e5b4c7c8915fd981c1254dbd02a94debe55d1e04b380d0ed0c07dc5254ffe02851d711964fac11fbc25656a733b3efef04edc3354f127b0939122e11363cb6
                                                    SHA256
                                                     message 31bbb0549dfe7603ca4cab419840e354e4668cde899a9b937ce17c179b7b6cd9
                                                     SHA256
                                                      message 0026274a40df16c7c7d903171f6774297833d88a8db3feffb54a8ca9b62817f9
                                                      APPEND 4adf2c5c03ef40ee15c0acec545f451d6c1014b1914e48908c0267fd3603e90e
                                                       message 0026274a40df16c7c7d903171f6774297833d88a8db3feffb54a8ca9b62817f94adf2c5c03ef40ee15c0acec545f451d6c1014b1914e48908c0267fd3603e90e
                                                       SHA256
                                                        message 8d3bbbb50823f284115f3d57ba33c27bc40af35a5ce69f90cb78fe8289be6314
                                                        SHA256
                                                         message 58d4b9662efa9ed0a5875ea6a0fb656da5f4fbbaf519ab145849831a25f45e51
                                                         PREPEND 412da24682f42a33b637ece80480c1d81598ea518f6dad8afc079e9de2d86bda
                                                          message 412da24682f42a33b637ece80480c1d81598ea518f6dad8afc079e9de2d86bda58d4b9662efa9ed0a5875ea6a0fb656da5f4fbbaf519ab145849831a25f45e51
                                                          SHA256
                                                           message e3debb7e1d8ae886a81f942880e0e075975d3ec5688c974654f6815c9c076fd6
                                                           SHA256
                                                            message ddf91a40c06c5586152d555572718254f7821efb5b888572428e0ae851536d56
                                                            PREPEND 21d5b70e6069499f440d0aeb2dacdd0375eec60c5aafd23035d32ed032601086
                                                             message 21d5b70e6069499f440d0aeb2dacdd0375eec60c5aafd23035d32ed032601086ddf91a40c06c5586152d555572718254f7821efb5b888572428e0ae851536d56
                                                             SHA256
                                                              message cbd33d36c6f47c8797c6af07721d98097c3d31c3d3b0cd4376d3a7b71e42b2d4
                                                              SHA256
                                                               message b06ff0d3395b68ba0fc30faca9780e8cae14694c68d8db59c3adf3fc3d0546a8
                                                               PREPEND 9a0c30eb61aa3011dc548920238092b5c3d1b51ab9f52aa6d1a724c53392cafe
                                                                message 9a0c30eb61aa3011dc548920238092b5c3d1b51ab9f52aa6d1a724c53392cafeb06ff0d3395b68ba0fc30faca9780e8cae14694c68d8db59c3adf3fc3d0546a8
                                                                SHA256
                                                                 message d4fd41141017e3503e6d3eaa542ef1b321b302b7a7faa67620212eed882bc2aa
                                                                 SHA256
                                                                  message 6de1bd24d522a1e8c012cbb8a6d417868b594a594aa10f3489e5ce36e410650d
                                                                  APPEND 917a55457c6043f2f8411a08b80c9cab1d35b488e6c28113b7d1b815dcc1fe29
                                                                   message 6de1bd24d522a1e8c012cbb8a6d417868b594a594aa10f3489e5ce36e410650d917a55457c6043f2f8411a08b80c9cab1d35b488e6c28113b7d1b815dcc1fe29
                                                                   SHA256
                                                                    message be9e0aa1ad814c061b500d28c1340135fd7c9fa96cb60869944f7b8d13ef6b26
                                                                    SHA256
                                                                     message 8b2eb6ece0b438a365dd205e98282a26f2c9c40298c828804debbc27e0df52ef
                                                                     APPEND cf3be3ad43b516e2fab3297c457db3f34bc2cecc381396fdfd5cf8bb5175797f
                                                                      message 8b2eb6ece0b438a365dd205e98282a26f2c9c40298c828804debbc27e0df52efcf3be3ad43b516e2fab3297c457db3f34bc2cecc381396fdfd5cf8bb5175797f
                                                                      SHA256
                                                                       message d8c80fb963b3bc4b42b7b822ddc91b83dc8810cf8cc4407d6a59f7013e304647
                                                                       SHA256
                                                                        message eba11fb133ff341e85d3c55fc4c04f1dd88f3a0a1e9984a731e8ed96fe58f9bf
                                                                        VERIFY bitcoin(height=808378)

Which makes zero sense to me, because:

I can't see the hash I wanted to attest in this tree;
I can't see the partial hash the server had given me (or whatever that is) that I was using to query the server at /timestamp;
the final message there is nowhere to be seen in the Bitcoin block 808378.

RedGrittyBrick · Answer 1 · 2023-09-19T11:24:00.207

The website doesn't explain anything

Scroll down to "How it works" and click "more"

That takes you to Peter Todd's 2016 description of the technical details.

[...]

An important way we use the flexibility commitment operations provides us is for scalability. For example, suppose you want to timestamp 10,000 different files: with most existing Bitcoin timestamping solutions, that would require 10,000 Bitcoin transactions - inefficient and expensive. But with OpenTimestamps, you can instead create a merkle tree of those 10,000 files and timestamp the tip of that tree in one transaction. Each per-file timestamp is simply the list of commitment operations that comprise the path up the first merkle tree, then up the Bitcoin block’s merkle tree, to finally get to the block header. The verifier doesn’t care: to it it’s just a series of operations like any other timestamp.

Secondly, we further improve this with a system of aggregation servers: publically available “meeting points” where anyone can submit a digest to be timestamped. As of writing, there are two public aggregation servers, a.pool.opentimestamps.org, and b.pool.opentimestamps.org

As digests are submitted for aggregation, they’re added to a list of pending digests. Periodically that list is combined into a single merkle tree, and then the tip of that tree is timestamped with Bitcoin

[...]

So, if you are going to use the Bitcoin blockchain as a notary for timestamping, this method, using aggregation servers, seems like it might be the least disruptive to Bitcoin.

score 4 · Accepted Answer · edited Sep 23 '23 at 14:18

So I think the key conceptual thing you are missing is that an OpenTimestamps proof is a series of commitment operations that start with your message, and eventually end up at a Bitcoin block header. Yes, under the hood the system uses merkle trees. But the proof format itself has no concept of a merkle tree for flexibility: it doesn't actually matter how the operations were derived. It just matters that they're mathematically correct and eventually lead to a merkle root in a valid Bitcoin block header.

How does OpenTimestamps work technically?

2 Answers2