How to find the distance of two lists?

Question

I have two lists as the following:

ab = {1, -1, -1, -1, 1, 1};
ac = {1, -1, -1,  1, 1, 1};

How I can find the difference (more precisely, the edit distance) between them? In this case the result should be 1, since there is one item difference between ab and ac.

Note: in my case, the list elements only take the values 1 and -1, and both lists are one-dimensional of the same length, but it is always nice to see more general solutions (elements are of Reals, lists are matrices, etc.).

Thank you.

Then you are in for a treat (see answers)! Please edit your question to include the 1/-1 information. — Yves Klett, Sep 06 '13 at 14:03

score 22 · Answer 1 · answered Sep 06 '13 at 12:00

22

There is an appropriate metrics:

HammingDistance[ab, ac]

one could use also (but in general it yields different results since it counts transpositions, deletions etc.)

DamerauLevenshteinDistance[ab, ac]

answered Sep 06 '13 at 12:00

Artes

57,212
12
157
245

Strangely HammingDistance doesn't work for mixed data i.e having both Integer and Symbols. – Pankaj Sejwal Sep 06 '13 at 16:20
1

@Blackbird What problems have you encountered? How many should it be HammingDistance[{3, 2, a, 10, b, 5, 11, c}, {3, 2, a, 11, b, Pi, 11, 3}] ? 3, shouldn't it? – Artes Sep 06 '13 at 16:50
I am sorry for reading the definition in wrong way.My bad, you are correct. – Pankaj Sejwal Sep 06 '13 at 18:05
1

@Blackbird Note that HammingDistance works also for strings (there is useful the IgnoreCase option) unlike the other methods. – Artes Sep 06 '13 at 18:31

Mr.Wizard · Answer 2 · 2013-09-06T16:59:39.397

16

For Integer data we also could write:

 Tr @ Unitize @ BitXor[ab, ac]

For Real data we can use the slightly slower but also shorter:

Tr @ Unitize[ab - ac]

Blackbird challenged me to provide a method that works on all input types. My approach is to select between methods depending on data.

diff[a__?(VectorQ[#, IntegerQ] &)] := Tr @ Unitize @ BitXor @ a
diff[a__?(VectorQ[#, NumericQ] &)] := Tr @ Unitize @ Subtract @ a
diff[a_, b_] := HammingDistance[a, b]

Timings for some of the methods posted so far (search the site for timeAvg):

{ab, ac} = List @@ RandomInteger[2, {2, 250000}];  (* List @@ to prevent unpacking *)

HammingDistance[ab, ac]                  // timeAvg
Count[MapThread[Equal, {ab, ac}], False] // timeAvg
Tr @ Unitize @ BitXor[ab, ac]            // timeAvg
diff[ab, ac]                             // timeAvg

0.009984

0.05428

0.0005488

0.0005744

Now with Real data:

{ab, ac} = N /@ {ab, ac};

HammingDistance[ab, ac]                  // timeAvg
Count[MapThread[Equal, {ab, ac}], False] // timeAvg
Tr @ Unitize[ab - ac]                    // timeAvg
diff[ab, ac]                             // timeAvg

0.01872

0.0748

0.00312

0.0021728

(I learned something from this test: Subtract[a,b] is faster than a-b on packed reals.)

Now something unpackable:

{ab, ac} = RandomChoice[CharacterRange["a", "z"], {2, 250000}];

HammingDistance[ab, ac]                  // timeAvg
Count[MapThread[Equal, {ab, ac}], False] // timeAvg
diff[ab, ac]                             // timeAvg

0.005488

0.0524

0.005496

edited Sep 06 '13 at 16:59

answered Sep 06 '13 at 12:17

Mr.Wizard

271,378
34
587
1,371

3

I knew this was one of those explode-in-your-face threads ;P – Yves Klett Sep 06 '13 at 12:25
@Yves Please define that. – Mr.Wizard Sep 06 '13 at 12:30
Simple/trivial question, one gazillion possible (and sometimes surprising and/or creative) answers. Oh, and my timings are usually abysmal... – Yves Klett Sep 06 '13 at 12:31
@Yves Ah, one of those. – Mr.Wizard Sep 06 '13 at 12:35
Is the term unsuitable / lost in translation? – Yves Klett Sep 06 '13 at 12:38
@Yves no, I'm just thick. – Mr.Wizard Sep 06 '13 at 12:41
Anyway, now you´ll have to do the honors and get timings for all solutions to come. – Yves Klett Sep 06 '13 at 12:41
This bitwise operation is ultrafast but it doesn't work for Symbol based list, this will really limit its use. I know OP has not asked this, but I was just thinking about variations. – Pankaj Sejwal Sep 06 '13 at 16:21
@Blackbird It would be quite simple to write a master function to select between these methods depending on the data. I sought to provide the best solutions for specific cases. – Mr.Wizard Sep 06 '13 at 16:25
@Mr.Wizard : I doubt it wil be this fast, its lightening fast. – Pankaj Sejwal Sep 06 '13 at 16:26
@Blackbird Are you saying you feel the overhead of the selector will be too great? I think it would be reasonable to only trigger the optimized method for packed arrays, and pass anything else to HammingDistance, and I would expect that approach to be fast. I guess I should implement it as proof, or counter-proof. – Mr.Wizard Sep 06 '13 at 16:30
@Mr.Wizard : ya it will be good post to learn from, please post your idea. – Pankaj Sejwal Sep 06 '13 at 16:32
@Blackbird Added. I hope you will agree that overhead is insignificant. – Mr.Wizard Sep 06 '13 at 16:59
@yves "blow up in your face" is a tad more idiomatic... :) – cormullion Sep 06 '13 at 18:54
@cormullion ahhh... thanks for scratching that idiosyncratic itch :-) – Yves Klett Sep 06 '13 at 19:22

score 9 · Answer 3 · edited Sep 07 '13 at 11:44

9

You could do this

ab = {1, -1, -1, -1, 1, 1};
ac = {1, -1, -1, 1, 1, 1};

EditDistance[ab, ac]

which would give a result even if the lists had different lengths (or whatever).

The documentation says:

EditDistance[u, v] gives the number of one-element deletions, insertions, and substitutions required to transform u to v.

edited Sep 07 '13 at 11:44

Mr.Wizard

271,378
34
587
1,371

answered Sep 06 '13 at 15:03

Stephen Luttrell

5,044
1
19
18

I'm sorry, I have to down-vote this. EditDistance is not the same thing. Incidentally it is a far more complex measure and will be unusably slow on long vectors. – Mr.Wizard Sep 06 '13 at 17:03
1

The OP said “How I can find the difference between them? Now in our case the result should be 1, since there is one item difference between ab and ac.”, which is such a loose definition of “difference” that EditDistance is a feasible candidate. Though I agree with your comment about complexity. – Stephen Luttrell Sep 07 '13 at 11:40
1

Okay, I have to admit you're right. My impression of the question didn't allow for that but I was wrong. I removed my down-vote (I had to edit to do this). – Mr.Wizard Sep 07 '13 at 11:45

Yves Klett · Answer 4 · 2013-11-05T08:18:39.827

8

A very basic approach:

Count[Equal @@@ Thread[{ab, ac}], False]

1

or perhaps:

Count[MapThread[Equal, {ab, ac}], False]

1

Now if there is only 1 and -1 to watch out for, this will also do (thanks to Aky for pointing out a glaring error):

Plus @@ Abs[ab - ac]/2

edited Nov 05 '13 at 08:18

answered Sep 06 '13 at 12:11

Yves Klett

15,383
5
57
124

2

You don't need Thread in the last one: Tr[ab-ac]/-2 – Mr.Wizard Sep 06 '13 at 12:43
@Mr.Wizard just came to my senses (usually I do not use Thread that much anyway), and Tr would most likely even be faster - benchmark away! – Yves Klett Sep 06 '13 at 12:45
1

It is too short for the answer so maybe you can add Count[ab ac, -1] somewhere for 1/-1 data case. – Kuba Sep 06 '13 at 12:59
1

@Kuba good one! You sure you do not want to answer yourself? – Yves Klett Sep 06 '13 at 13:01
1

@Kuba Short is the best kind of answer. I encourage you to post it. :-) – Mr.Wizard Sep 06 '13 at 13:07
@Mr.Wizard Ok :) – Kuba Sep 06 '13 at 13:09
@Mr.Wizard I´d like to see Tr[ab - ac]/-2 beaten in terms of brevity, but it is lovely, so please add it to your answer. – Yves Klett Sep 06 '13 at 13:19
@YvesKlett Oh crikey, I'm losing my marbles faster than I feared. I didn't remember writing that! :-S – Mr.Wizard Sep 06 '13 at 13:36
2

@Mr.Wizard now you see what I meant by "explode in your face" :-) – Yves Klett Sep 06 '13 at 13:42
@Yves Literally LOL. I just wish it wasn't over my own early senility. – Mr.Wizard Sep 06 '13 at 13:44
Its all weather snippet. Any type of data in list. – Pankaj Sejwal Sep 06 '13 at 16:28

score 7 · Answer 5 · answered Sep 06 '13 at 18:16

These results are specific to the case where the data are 1|-1, and may be specific to v6.

<<Developer`
n = 10^7
PackedArrayQ[ a = RandomInteger[1,n]*2 - 1 ]
PackedArrayQ[ b = RandomInteger[1,n]*2 - 1 ]
(* 10000000
   True
   True *)

AbsoluteTiming[ (Length@a - a.b)/2  ]
(* {0.435134, 4998582} *)

AbsoluteTiming[ Tr@Unitize@Subtract[a,b] ]
(* {0.792662, 4998582} *)

AbsoluteTiming[ Tr@Unitize@BitXor[a,b] ]
(* {0.883002, 4998582} *)

aa = FromPackedArray@a;
bb = FromPackedArray@b;

AbsoluteTiming[ (Length@aa - aa.bb)/2  ]
(* {1.373384, 4998582} *)

AbsoluteTiming[ Tr@Unitize@Subtract[aa,bb] ]
(* {1.366143, 4998582} *)

AbsoluteTiming[ Tr@Unitize@BitXor[aa,bb] ]
(* {2.590419, 4998582} *)

I get the same rankings in v7. I hadn't realized that Subtract was faster than BitXor. — Mr.Wizard, Sep 07 '13 at 11:50
Ah, I see now that it's only faster on this data, not general Integer data. Nevertheless +1 well earned for teaching me something. — Mr.Wizard, Sep 07 '13 at 11:58

score 6 · Answer 6 · answered Sep 06 '13 at 13:11

6

I will abuse the fact that OP hasn't said if it is more general question.

This is my solution fo 1/-1 data case:

Count[ab ac, -1]

answered Sep 06 '13 at 13:11

Kuba

136,707
13
279
740

@YvesKlett Here is another fun one for -1/1: Tr@Log[ab ac]/I/Pi – Mr.Wizard Sep 06 '13 at 13:16
1

@Mr.Wizard If we a priori know the Length then this one is shorter: (n - ab.ac)/2 – Kuba Sep 06 '13 at 13:20
@Mr.Wizard, you could save one character with Tr@Log[-1, ab ac] – Simon Woods Sep 07 '13 at 12:25

How to find the distance of two lists?

6 Answers6

Linked