How to use Union on list of lists without sorting them first?

Question

If I do

ClearAll[a, d]
lsts = {{a, d}, {a, d}};
Union[lsts]

I get the expected answer

{{a, d}}

but if I do

lsts = {{a, d}, {d, a}};
Union[lsts]

I get

{{a, d}, {d, a}}

Since I am using Union, I thought the order of the lists would not matter. Hence to get around this, now I always add Sort first, like this

lsts = {{a, d}, {d, a}};
Union[Sort /@ lsts]

and now I get expected answer

{{a, d}}

Question: Is this the right way to approach this? or do you recommend a better way?

Answer 1

Sorting of sub-lists seems unavoidable since this is what brings them to a "canonical form" in this problem. If you don't care about the order of your resulting sub-lists, you could used DeleteDuplicates in place of Union though - this should be faster for large lists.

Answer 2

You can provide a custom SameTest to Union where you can take advantage of your knowledge what should qualify as equal, for example:

In[1] := Union[{{a,d},{d,a}}, SameTest -> (Complement[##] === {}&)]
Out[1] = {{a, d}}

Answer 3

It might be that you're slightly misunderstanding what Union does. It finds the union of the elements of the list that is passed to it, but it doesn't dig into lists within that list. So when you write Union[{{a,d},{a,d}}], the function sees a list with two elements, {a,d} (that's element 1) and {a,d} (that's element 2). They are the same, so it removes the duplicate and returns just {a,d}. But when you write Union[{{a,d},{d,a}}], it sees a list with two different elements: {a,d} (that's element 1) and {d,a} (that's element 2). The fact that those two lists contain the same items is irrelevant; they're not equal, according to an ordered element-by-element comparison, so Union has no duplicates to remove.

Now, it seems like what you're trying to do is get all lists which are distinct in terms of their content, irrespective of order - in other words, you're treating the lists as mathematical sets. I think Union[Sort/@lsts] should be a fine way to go, because that's the standard method of comparing sets for equality when you don't have an actual unordered set type. (If Mathematica does, I don't know about it.)

Answer 4

You could do the following:

lists = {{c, b, a}, {c, a, b}};
Union[lists, SameTest -> (Sort[#1] == Sort[#2] &)]

Note that the result is {c,a,b}, which is unsorted. The underlying algorithm can no longer take advantage of a linear comparison of the terms, however. As a result the time complexity is quadratic and will slow down your code considerably for very long lists. Thus, I'd advise against this approach. Ordering the lists first, as you've done, is preferable.

Answer 5

Your problem is reminiscent of the thing one has to do when dealing with noncommutative monomials.

To add to previous answers:

Depending on the typical content of your lists (especially if you have a lot of strictly identical elements), it might be beneficial to apply Union twice, in this way

Union[ Sort/@(Union[ lst ]) ]

or

Union[ Union[ lst], SameTest -> (Equal[Sort[#1],Sort[#2]] &) ]

if you want to retain some of the diversity of the original instead of having everything mapped to a canonically sorted form.

The problem is more complex when you consider more deeply structured lists of course. You might end up with a very costly comparator function.

An object oriented approach would be to define for each object symbol a comparator function that would be called automatically as SameTest when a modified Union is called with arguments of a given head.

Answer 6

To get rid of items that are duplicates under Sort you may use this:

GatherBy[lsts, Sort][[All, 1]]

Afterward you may sort or manipulate that list as you see fit.
Be warned that there is apparently a bug in Mathematica 7 with this specific code.

New in Mathematica 10:

DeleteDuplicatesBy[lsts, Sort]

Answer 7

I am not sure you are doing it right. Union expects a set per argument. As you only give it one argument, you are basically doing the union over one set A, which is incidentally A. What you want to do is Union @@ lsts which is Apply[Union,lsts]