0

Is the second-by-second output in curl accurate "enough"? What is the period that the "current rate" is calculated over?

I am currently using the second-by-second status output in Curl as a general measure of current bandwidth on that connection.

If you don't know what this "status output" means, then there's no point in describing it :-)

I figure it's "good enough"...spot tests have shown it to be pretty close, but I want to make sure. Also, is the second-by-second current rate over the last second? I am assuming so, although the manuals seem somewhat unclear about this.

Mark Gerolimatos
  • 1,679
  • 2
  • 16
  • 20
  • What's accurate to you? The rate over the last second, or the rate from the whole download? Maybe somewhere in between? – Brad Mar 02 '15 at 05:25
  • The claim is that the bitrate is over the last second. It could be that it's just random stuff that is printed out to keep the downloader happy that something is happening. The best possible result would be the actual-bits-from-the-last-60-seconds/60-seconds – Mark Gerolimatos Mar 03 '15 at 01:16
  • It's accurate. You can test this yourself with a packet sniffer. – Brad Mar 03 '15 at 01:17
  • Thanks! That's EXACTLY what I needed to know! If you state it in the form of an answer, I'll give you credit :-) – Mark Gerolimatos Mar 03 '15 at 01:18
  • Maybe, but what I can't answer is what is the time period being quantized. And, I see now that you edited your comment asking about the data from the last 60 seconds. I think your first comment is correct, that it's probably the average bitrate over the last second as I see it swing up and down wildly sometimes. But, you would have to inspect the source to be sure. In any case, I promise you that it isn't just some made up number. It's a count of data transferred over time... just what time period I'm not sure. – Brad Mar 03 '15 at 01:20
  • Thank you. I don't have the bandwidth (excuse the pun) for verification, and frankly do not need super accuracy, just want to make sure that the RELATIVE numbers are accurate. When you graph this data, it often becomes immediately apparent as to the cause of the download slowness (ISP limiting, tail drop, good ol' congestion or CDN funny business). And that is why accuracy was somewhat important, so that statements made about the graphs would hold. From what I see here, that is true. – Mark Gerolimatos Mar 03 '15 at 20:56

1 Answers1

2

I looked at this question a few days ago and I must say that I have wondered the same at times. By the way, it isn't nearly as accurate when you analyse it at packet level. To top that, even the values within the status output do contradict. See below for example:(check the values of Current Speed in Bytes/sec and relate with Average Download Speed and Received Data in Bytes)

% Total     % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
0     0     0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
0     0     0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2 10.0M     2  234k    0     0   161k      0  0:01:03  0:00:01  0:01:02  161k
7 10.0M     7  769k    0     0   314k      0  0:00:32  0:00:02  0:00:30  314k
10 10.0M   10 1102k    0     0   319k      0  0:00:32  0:00:03  0:00:29  319k
16 10.0M   16 1671k    0     0   368k      0  0:00:27  0:00:04  0:00:23  368k
23 10.0M   23 2371k    0     0   435k      0  0:00:23  0:00:05  0:00:18  480k
28 10.0M   28 2905k    0     0   450k      0  0:00:22  0:00:06  0:00:16  534k
31 10.0M   31 3269k    0     0   434k      0  0:00:23  0:00:07  0:00:16  492k
38 10.0M   38 3973k    0     0   470k      0  0:00:21  0:00:08  0:00:13  574k
43 10.0M   43 4476k    0     0   473k      0  0:00:21  0:00:09  0:00:12  569k
47 10.0M   47 4814k    0     0   459k      0  0:00:22  0:00:10  0:00:12  486k
54 10.0M   54 5571k    0     0   486k      0  0:00:21  0:00:11  0:00:10  533k
59 10.0M   59 6089k    0     0   474k      0  0:00:21  0:00:12  0:00:09  532k
63 10.0M   63 6522k    0     0   478k      0  0:00:21  0:00:13  0:00:08  492k
68 10.0M   68 6992k    0     0   481k      0  0:00:21  0:00:14  0:00:07  496k
74 10.0M   74 7599k    0     0   485k      0  0:00:21  0:00:15  0:00:06  536k
80 10.0M   80 8255k    0     0   489k      0  0:00:20  0:00:16  0:00:04  494k
84 10.0M   84 8648k    0     0   491k      0  0:00:20  0:00:17  0:00:03  535k
89 10.0M   89 9202k    0     0   493k      0  0:00:20  0:00:18  0:00:02  534k
95 10.0M   95 9745k    0     0   495k      0  0:00:20  0:00:19  0:00:01  536k
100 10.0M 100 10.0M    0     0   507k      0  0:00:20  0:00:20 --:--:--  584k

Now look at the sniffer trace analytical summary of the first and second second of this stream against the entire stream in pictures below: (Difference between these values and curl's status output is indeed remarkable)

First second:

First second data rate

Second second:

Second-second data rate

I believe this does prove the point that what curl reports isn't what you'd call accurate but does serve the purpose more or less. The average values that it reports are averaged over past 5 seconds from fifth second onwards and even the average calculation isn't precise.

Another point worth noticing is how curl always shows upload speed as 0 whereas sniffer trace shows the average upload speed to be around 22558 Bytes/sec for this 20 second 11MB transaction.

Upload speed for e

Hope that helps!

Donbhupi
  • 136
  • Thank you for the excellent analysis! The takeway would then be that the CURL progress meter is best limited to comparisons between downloads, or as what I am currently using it for, to generate "download bitrate shape" graphs. As with any user-level calculation, the bitrate is a function not just of the network, but also the delivery to the application, as well as the quirks of the application's measurement methodology. – Mark Gerolimatos Mar 07 '15 at 23:53
  • 1
    I believe one reason for the different results is that curl measured the application/TCP payload traffic, while wireshark measured raw traffic including Ethernet+IP+TCP header. The upload traffic is just due to pure ACKs. – Trần Việt Hoàng Apr 18 '19 at 09:25