How to align similar TimeSeries like ImageAlign?

Question

I have three audio files.

human=Audio["http://home.ustc.edu.cn/~xiaozh/SE/Audio/human.wav"];
hus=Audio["http://home.ustc.edu.cn/~xiaozh/SE/Audio/hus.wav"];
parm=Audio["http://home.ustc.edu.cn/~xiaozh/SE/Audio/parm.wav"];

The first one is generated by human.The second and third one are generated by computer using different methods.

The texts are the same:

联合国秘书长特使、前南地区维和部队最高长官明石康，5日书面回答了本报记者的提问。

You can hear these three sounds are very similar even if you don't speak Chinese.

AudioPlot@{human, hus, parm}

Spectrogram[#, PlotRange -> {Automatic, {0, 2000}}] & /@ {human, hus, parm}

If I want to compare there RMSAmplitude:

AudioLocalMeasurements[#, "RMSAmplitude"] & /@ {human, hus, parm}

And then plot it.

ListLinePlot[#, PlotRange -> All, PlotLegends -> {"human", "hus"}, AspectRatio ->  1/5] &[
AudioLocalMeasurements[#, "RMSAmplitude"] & /@ {human, hus}]

ListLinePlot[#, PlotRange -> All, PlotLegends -> {"human", "parm"}, AspectRatio -> 1/5] &[
AudioLocalMeasurements[#, "RMSAmplitude"] & /@ {human, parm}]

But actually,I want to align these curves,that is the second one may be shift to left a little and the third one may be shift to right a little:

There is a function can do align for pictures named ImageAlign even if their size,Translation are not equal.

PS:the picture of first coefficient of MFCC may be more clear

ListLinePlot[#, PlotRange -> All, PlotLegends -> {"human", "hus"},AspectRatio -> 1/5] &[
TimeSeriesMap[First, AudioLocalMeasurements[#, "MFCC"]] & /@ {human, hus}]

ListLinePlot[#, PlotRange -> All, PlotLegends -> {"human", "parm"},AspectRatio -> 1/5] &[
TimeSeriesMap[First, AudioLocalMeasurements[#, "MFCC"]] & /@ {human, parm}]

So I wonder if there is some mothods can align TimeSeries object like ImageAlign?

Should the audiofiles be just shifted, or also parts removed/contracted, etc? — Feyre, Nov 20 '16 at 19:42
@Feyre audio shifted and multiplied by some scaling factor at some time intervals are preferred. — partida, Nov 21 '16 at 03:17
I'm sorry,I missed something..And I have made some edit.If you mind that,you can rollback. — yode, Jun 21 '17 at 10:48
@AlexeyPopkov If the threads mean this question?if yes,the image works fine here. — partida, Jun 21 '17 at 12:31

bill s · Accepted Answer · 2016-11-24T18:33:08.023

One way to approach this is with "Dynamic Time Warping". First, preprocess your data to get the MFCC coefficients and extract the data from the time series:

human = Audio["http://home.ustc.edu.cn/~xiaozh/SE/Audio/human.wav"];
hus = Audio["http://home.ustc.edu.cn/~xiaozh/SE/Audio/hus.wav"];
{humMFCC, husMFCC} = AudioLocalMeasurements[#, "MFCC"] & /@ {human, hus};
x = TimeSeriesMap[First, humMFCC]["Values"]; nX = Length[x];
y = TimeSeriesMap[First, husMFCC]["Values"]; nY = Length[y];

Now define the DTW method:

(*distance function*)
dist[s_, t_] := Abs[s - t];
(*boundary conditions*)
Clear[dtw];
dtw[1, 1] = dist[x[[1]], y[[1]]];
dtw[1, j_] := dtw[1, j] = dist[x[[1]], y[[j]]] + dtw[1, j - 1];
dtw[i_, 1] := dtw[i, 1] = dist[x[[i]], y[[1]]] + dtw[i - 1, 1];
(*main recursion*)   
dtw[i_, j_] := dtw[i, j] = dist[x[[i]], y[[j]]] + 
                           Min[dtw[i - 1, j - 1], dtw[i - 1, j], dtw[i, j - 1]];
(*finding best path through dtwMatrix*) 
pathFind[{i_, j_}] := Module[{nbhd}, 
   nbhd = {{i, Max[j - 1, 1]}, {Max[i - 1, 1], j}, {Max[i - 1, 1], Max[j - 1, 1]}}; 
   nbhd[[First[Ordering[Map[dtwMat[[#[[1]], #[[2]]]] &, nbhd]]]]]];

Finally, apply the DTW to your data:

distMat = Outer[dist, x, y];
dtwMat = dtwPath = Outer[dtw, Range[nX], Range[nY]];
bestPath = NestWhileList[pathFind, {nX, nY}, (#[[1]] > 1) || (#[[2]] > 1) &];
ArrayPlot[Reverse@#, Frame -> False] &@ ReplacePart[dtwPath, {{x_, y_} /; 
    MemberQ[bestPath, {x, y}] -> 0, {x_, y_} /; ! MemberQ[bestPath, {x, y}] -> 1}]

The picture represents one MFCC on the horizontal axis and the other on the vertical. The best path is the jagged diagonal line where the two time series are best aligned. This lines up the MFCCs of the two audio streams. The bestPath variable contains a collection of pairs {indexX, indexY} which show the optimal correspondence in the original sequences x and y, thus they can be used to index into x and y and so demonstrate the alignment. For example, here is a plot of the aligned first coefficients of the MFCCs:

indX = Reverse[Transpose[bestPath][[1]]];
indY = Reverse[Transpose[bestPath][[2]]];
ListLinePlot[{x[[indX]], y[[indY]]}, PlotStyle -> {Blue, Green}]

To realign the audio itself then requires taking this mapping and resampling the audio.

Update: thanks to partida for pointing out some indexing issues in the DTW.

score 4 · Answer 2 · answered Jun 21 '17 at 12:41

In MMA11.0,there is a new function：WarpingCorrespondence

It makes the DTW(Dynamic time warping) very easy.

human = Audio["http://home.ustc.edu.cn/~xiaozh/SE/Audio/human.wav"];
hus = Audio["http://home.ustc.edu.cn/~xiaozh/SE/Audio/hus.wav"];
{humMFCC, husMFCC} = AudioLocalMeasurements[#, "MFCC"] & /@ {human, hus};
x = TimeSeriesMap[First, humMFCC]["Values"]; nX = Length[x];
y = TimeSeriesMap[First, husMFCC]["Values"]; nY = Length[y];

{n, m} = WarpingCorrespondence[x, y];
ListLinePlot[{x[[n]], y[[m]]}, PlotStyle -> {Blue, Green}]

But it seems in 11.2.0 the result is changed... – partida Dec 15 '17 at 02:11 — partida, Dec 15 '17 at 02:11

How to align similar TimeSeries like ImageAlign?

2 Answers2