I would like to set up a function which has to return True if at least two arguments of a given List are equal.
So if I give {1,4,6,2} to the function it has to return False (since none of his arguments are equal) and the same would happen if I gave {{1,2,3},{2,3,4}}, while if I gave {1,2,3,1,4} or {{1,2,3},{2,0,0},{1,2,3},{2,1,2}} it has to return True.
I know this is a very simple problem and i think you can easily tell me a convenient way to achieve that.
As it happens there is a built-in function that already does this: Signature.
If any two elements of
listare the same,Signature[list]gives 0.
dupeQ = 0 === Signature@# &;
I believe this is the "canonical" answer. It is fast on both packed arrays and unpacked lists.
If there are repeated elements in the list, then calling Union[] on it will shorten it so that this element only appears once, so a simple implementation would be to test these lengths:
test[list_] := Length[Union[list]] != Length[list]
If you wanted to know which elements where repeated, you this could be accomplished by using Gather[] to collect identical elements, and picking out which groups have more then one element.
repeats[list_] := Select[Gather[list], Length[#] > 1 &][[1 ;;, 1]]
Note, I'm using Union rather then DeleteDuplicates[] since (as Mr. Wizard corrected me) it is faster. I can't say why except that DeleteDuplicates[] retains the order of elements which may require slightly more bookkeeping. And in this case we don't care about the book keeping. Naturally if you really needed something really speedy, a better solution exists which doesn't search through the entire list, but stops if just a single duplicate is found, Mr. Wizards Answer is just such an function, since Signature exits early if duplicates exist, though it becomes slower if no duplicates are present, it's a trade off.
Union. Compare these timings: a = RandomInteger[10, {100000, 50}]; then
Timing[Union@a;] and
Timing[DeleteDuplicates@a;]
– Mr.Wizard
Jun 13 '12 at 14:29
Union to be slower as it also sorts. It makes me wonder what sorting algorithm it uses.
– rcollyer
Jun 14 '12 at 02:12
duplicatesQ = # != DeleteDuplicates[#] &
Usage:
duplicatesQ[{1, 4, 6, 1}]
(* ===> True *)
duplicatesQ@{1, 4, 6, 2}
(* ==> False *)
duplicatesQ@{{1, 2, 3}, {2, 0, 0}, {1, 2, 3}, {2, 1, 2}}
(* ==> True *)
duplicatesQ /@ {{1, 2, 3}, {2, 0, 0}, {1, 2, 3}, {2, 1, 2}}
(* ==> {False, True, False, True} )
or (to also get the duplicate elements and their count when there are duplicate elements)
dups = (Select[Tally[#], Last[#] > 1 &] /. {} -> "None") &
Usage:
dups /@ {{1, 2, 3}, {2, 0, 0}, {1, 2, 3}, {2, 1, 2}}
(* ==> {"None", {{0, 2}}, "None", {{2, 2}}} *)
dups[{{1, 2, 3}, {2, 0, 0}, {1, 2, 3}, {2, 1, 2}}]
(* ==> {{{1, 2, 3}, 2}} *)
You could also let the pattern matcher do all the work:
DuplicatesQ[l_] := MatchQ[l, {___,x_,___,x_,___}]
Or:
DuplicatesQ[{___,x_,___,x_,___}] := True
DuplicatesQ[_] := False
Since you are looking for duplicates you could adapt any of the methods shown in this answer.
Using the first one for example:
dupeQ =
Module[{f},
f[y_] := (f[y] := Return[True, Module]; y);
Scan[f, #]; False
] &;
This particular one has an advantage on long lists in that it will "short-circuit" on the first duplicate found rather than proceeding through the end of the list as Union or DeleteDuplicates will. Example:
a = Range[1*^7];
a[[{1, 50}]] = 3.14159;
dupeQ[a] // Timing
{0., True}
a != DeleteDuplicates[a] // Timing
{5.944, True}
For this example I stacked the deck in my favor; by replacing elements in an Integer list with Reals I force the list to unpack. In this situation and with an early duplicate (positions 1 and 50) my method is about 65,000 times faster. For a case with a packed array dupeQ is slower than DeleteDuplicates but not as dramatically:
a = Range[1*^7];
a[[50]] = 7;
dupeQ[a] // Timing
{0.359, True}
a != DeleteDuplicates[a] // Timing
{0.031, True}
Following up on this comment of yours:
Anyway I was wondering if you could save information about the argument repeated: I mean from your approach I can't tell which was the argument repeated.
One could return the duplicate value if one is found and False otherwise:
dupe =
Module[{f},
f[y_] := (f[y] := Return[y, Module]; y);
Scan[f, #]; False
] &;
This function has the same advantage of short-circuiting that the first one does.
Return. From the use above, I guess it's something about from which construct you want to return, right? Is that documented somewhere?
– celtschk
Jun 13 '12 at 13:47
Break::nofunc: "The second argument in Break, Continue, or Return is a label that specifies the enclosing function or loop that will be affected. If this label is omitted, the affected function or loop is determined using built-in heuristics."
– Mr.Wizard
Jun 13 '12 at 14:13
Return, Break and Continue. Actually for Break and Continue there is not even a first parameter documented. A quick test shows that for Break the first parameter gives a return value for the construct, just like Return, while for Continue I couldn't see any effect.
– celtschk
Jun 13 '12 at 14:33
A frivolous implementation using patterns:
duplicateQ[list_]:=MemberQ[Tally[list],{_,_?(#>1&)}]
This function uses Tally to arrange the elements of list in bins. For example,
In[2]:= Tally[{1,2,3,1,2}]
Out[2]= {{1,2},{2,2},{3,1}}
Then we look for an element in the output of Tally which looks like {_,n} with $n>1$.
In[3]:= duplicateQ[{{1,2,3},{2,3,4}}]
Out[3]= False
In[4]:= duplicateQ@{{1,2,3},{2,0,0},{1,2,3},{2,1,2}}
Out[4]= True
If we want to keep track of repeated elements, we can use
duplicates[list_] := Cases[Tally[list], {_, _?(# > 1 &)}]
In[6]:= duplicates[{1,4,6,1}]
Out[6]= {{1,2}}
In[7]:= duplicates@{{1,2,3},{2,0,0},{1,2,3},{2,1,2}}
Out[7]= {{{1,2,3},2}}
You could use Gather and then check the length of each group :
Gather[{{1, 2, 3}, {2, 0, 0}, {1, 2, 3}, {2, 1, 2}}]
(* {{{1, 2, 3}, {1, 2, 3}}, {{2, 0, 0}}, {{2, 1, 2}}} *)
Length[#] & /@ Gather[{{1, 2, 3}, {2, 0, 0}, {1, 2, 3}, {2, 1, 2}}]
(* {2, 1, 1} *)
In Version 9, there is a new function DuplicateFreeQ which returns True is there is any duplicate elements in a list. This function is currently not documented, but it is there.
Examples:
snake = {{0, 1}, {0, 0}, {1, 0}, {2, 0}, {3, 0}}
DuplicateFreeQ[snake]
(*True*)
a = {1, 2, 3, 3};
DuplicateFreeQ[a]
(* False *)
a = {1, 2, 3};
DuplicateFreeQ[a]
(* True *)
a = {{1, 1}, {1, 0}, {1, 0}};
DuplicateFreeQ[a]
(*False*)
a = {{1, 1}, {1, 0}, {1, 9}};
DuplicateFreeQ[a]
(*True*)
Update
Answering comment below.
Table[a = Range@n; a[[n/2]] = 1; Signature[a] // Timing // First, {n, 10^Range@8}]
(* {0., 0., 0., 0., 0., 0., 0.015600, 0.156001} *)
Table[a = Range@n; a[[n/2]] = 1; DuplicateFreeQ[a] // Timing // First, {n, 10^Range@8}]
(* {0., 0., 0., 0., 0., 0., 0.046800, 0.483603} *)
Signature in performance, e.g. Table[ a = Range@n; a[[n/2]] = 1; Signature[a] // Timing // First, {n, 10^Range@8} ]? (This uses about 800 MB memory).
– Mr.Wizard
Aug 02 '13 at 18:25
Signature still has value for this application, at least in some cases.
– Mr.Wizard
Aug 02 '13 at 20:30
Using UnsameQ:
MultiSetQ[lst_List]:=Not[UnsameQ@@lst]
Usage:
In[2]:= MultiSetQ[{1, 2, 2, 3}]
Out[2]= True
In[3]:= MultiSetQ[{1, 2, 4, 3}]
Out[3]= False
! UnsameQ @@ lst
– Mr.Wizard
Jun 29 '12 at 06:40
Union faster? I mean, it returns a sorted version of the list, while UnsameQ will stop comparing (I think) when it finds the first pair of equal elements. How can Union be faster??
– a06e
Jun 29 '12 at 21:08
UnsameQ is designed for handling a low number of arguments. Testing shows that Union and DeleteDuplicates have O(n) time complexity, whereas UnsameQ has O(n^2). UnsameQ is actually faster on tests with a limited number of arguments (it is about even with DeleteDuplicates by 45), so I suppose that it was optimized for that kind of use.
– Mr.Wizard
Jul 02 '12 at 16:20
Signaturewhich has been there since v1 but which I've never noticed. Your claim of speed, though, is incorrect. It's only true for lists that actually contain a duplicate. Tryl = RandomInteger[100000, {1000000, 3}];and yourdupeQis almost 10 times slower than kguler'sduplicatesQ. I vote against this being a canonical answer. – Sjoerd C. de Vries Jun 13 '12 at 12:51l = RandomInteger[10, {1000000, 50}];and the timings are totally different.duplicatesQtotally can't handle this and yours is extremely much faster. It takes 5 secs for yours andduplicatesQis still running for a couple of minutes without finishing. Interesting behaviour... – Sjoerd C. de Vries Jun 13 '12 at 13:04RandomInteger[10, {1000000, 50}];? – image_doctor Jun 13 '12 at 14:18test[](defined in my answer) to be faster both on a and b. On my machine the timings are 4 and 12 for Signature based, and 1.4 on both forUnion[]based.Also, it seems like the first time I run Signature on a large dataset, the timing is always lower then repeating the exact same code. If I rerun the tests, both tests on signature give 12 seconds.
– jVincent Jun 13 '12 at 18:02a = Range[1*^7]; a[[{1, 50}]] = 3.14159;– Mr.Wizard Jun 13 '12 at 18:04a = Range[1*^7]; a[[{1, 50}]] = 3.14159;andb = Range[1*^7]; a[[{1, 1*^6}]] = 3.14159;Then on my Machine dupeQ wins on a with 0.063 against 0.984 for test and test wins b with 1.891 vs. 2.344 for dupeQ. To sum up, if there are duplicates then dupeQ wins because it can exit early, and if there are not any duplicates Union wins, most likely because Signatur has to suddenly figure out the actual signature, though oddly still slower if the duplicate is at the end – jVincent Jun 13 '12 at 18:16