Evaluate large file with ^H and ^M characters

Question

I have a logfile that contains many ^H and ^M characters, as the process that produces this file updates a text based progress bar.

When using cat the output is evaluated and appears human readable and concise. Below is an example output.

Epoch 11/120
4355/4355 [==============================] - ETA: 0s - loss: 0.0096   
Epoch 00011: val_loss did not improve from 0.00992
4355/4355 [==============================] - 1220s 280ms/step - loss: 0.0096 - val_loss: 0.0100

However, the file itself is huge compared to what the actual printed text from cat would suggest (70MB for around 900 lines) of text like above.

Below is a snippet of the actual text contained within the logfile.

1/Unknown - 0s 81us/step - loss: 0.5337^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^M  2/Unknown - 1s 438ms/step - loss: 0.5299^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^
H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^M      3/Unknown - 1s 386ms/step - loss: 0.5286^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^M      4/Unknown - 1s 357ms/step - loss: 0.5289^H^H^H^H^H^H^H^H^H^
H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^M      5/Unknown - 2s 339ms/step - loss: 0.5277^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^M      6/Unknown - 2s 327ms/
step - loss: 0.5258^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^M      7/Unknown - 2s 318ms/step - loss: 0.5250^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^
H^H^H^H^H^M      8/Unknown - 2s 312ms/step - loss: 0.5260^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^M      9/Unknown - 3s 307ms/step - loss: 0.5265^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^
H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^M     10/Unknown - 3s 303ms/step - loss: 0.5257^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^M     11/Unknown - 3s 299ms/step - loss: 0.5258^H^H^H^

Basically, I want to create a file that looks just like what cat produces.

Here's a few things i've tried with limited success:

tr -d '\b\r' < logfile > new_file removes all of the characters, but therefore leaves all of the undesired text.
cat logfile > new_file effectively just copies the file verbatim with no evaluation of the special characters.
cat logfile | col -b > new_file is pretty close, but does something odd on one of the repeated lines:

4355/4355 [==============================] - ETA: 0ss--loss::0.0096557
Epoch 00011: val_loss did not improve from 0.00992
4355/4355 [==============================] - 1220s 280ms/step - loss: 0.0096 - val_loss: 0.0100

Any help would be appreciated.

Thanks

"I want to create a file that looks just like what cat produces" – It's not what cat produces. It's what your terminal (terminal emulator) shows after it interprets ^H and such. What cat does is exactly like in cat logfile > new_file, except there's the terminal instead of new_file. — Kamil Maciorowski, Aug 13 '21 at 11:44
@Quasímodo Good point, i've uploaded a sample of 20 lines from the logfile here https://drive.google.com/file/d/1KhZbEoFLQ0JCmPlw5rPKaVeWk-rAicq4/view?usp=sharing hope that helps — Matt Lyon, Aug 13 '21 at 12:34
@rowboat your answer works perfectly, thank you! If you want to submit as an answer I will mark as the accepted answer. — Matt Lyon, Aug 13 '21 at 12:39
A general workaround may be to use an actual terminal emulator that allows you to save the content of its scrollback buffer. You cat the file to the terminal and save the ultimate result. Example for tmux: here. It seems "Save Output As…" in the "File" menu of konsole can do this. Other terminal emulators may provide similar functionality. Notes: (1) You need to make sure in advance the scrollback buffer is large enough. (2) I guess some terminal emulators may generate output with newlines where they wrap lines due to their width. — Kamil Maciorowski, Aug 13 '21 at 13:13
It's really what col -b is for. You see a difference from when dumping the file to a terminal because backspace on your terminal when the cursor is at the beginning of the line happens to cause the cursor to move back to the end of the previous line, but try it on a terminal that doesn't such as the linux virtual consoles and you'll see the same (see the bw terminfo capability) — Stéphane Chazelas, Aug 13 '21 at 15:24

Matt Lyon · Accepted Answer · 2021-08-24T10:33:57.357

Posting this as an answer for clarity.

As rowboat pointed out, in this case the command awk -F '\r' '{print $NF}' file works as intended, by removing everything after the last carriage return. Though this is not robust as zevzek pointed out.

I wrote a more robust solution in C++ below.

#include <fstream>
#include <string>
#include <iostream>
using namespace std;
string filter_string(string line, const char bspace, const char creturn){
string new_str;

for(string::size_type i = 0; i &lt; line.size(); ++i) {
    // Step back if current string not empty
    if (line[i] == *bspace){
        if (new_str.size() != 0){
            new_str.pop_back();
        };
    // Reset on carriage return
    } else if (line[i] == *creturn){
        new_str = &quot;&quot;;
    } else {
        new_str += line[i];
    };
}

return new_str;

};
int main(int argc, char* argv[]){
    const char backspace = '\x08';
    const char creturn = '\r';
if (argc != 2){
    cerr &lt;&lt; &quot;USAGE: &quot; &lt;&lt; argv[0] &lt;&lt; &quot; [src]&quot; &lt;&lt; endl;
    return 1;
}

// Filter lines in file
string line;
ifstream infile(argv[1]);
while (getline(infile, line)){
    cout &lt;&lt; filter_string(line, &amp;backspace, &amp;creturn) &lt;&lt; endl;
};

return 0;

};

Here each character in each line is iterated over, and if a ^H is present, the string is pushed back one (if not already empty), and if a ^M carriage return is present then the string is reset. The output is sent to stdout which can then be piped to a file.

const char newline = '\r'... yngh, that's the carriage return. Newline is \n. With the input data you have here, just removing up to the last CR of each line seems to be fine, since the backspaces are there just to clear the line before each time the cursor is moved back. — ilkkachu, Aug 23 '21 at 20:56

score 2 · Answer 2 · answered Aug 23 '21 at 21:38

2

sed 's/.*\x0d//' logfile

Seems to do what you ask for.

Note that col -b fails because it is ignoring the whitespace:

$ echo $'--------\r1st try\r2nd   \r3rd\n' | col -b
3rd-try-

answered Aug 23 '21 at 21:38

Evaluate large file with ^H and ^M characters

2 Answers2