How to remove symmetries from this list?

Question

First of all, I work with modular arithmetics and the elements of the arithmetics are lists themselves.

I recognize two kinds of symmetries - shift and mirroring. We could define shift of a list like this:

mod = {3, 2};
shiftBy[set_, shift_] := Mod[shift + #, mod] & /@ set

I consider two sets to be equivalent if one can be obtained from the other by shifting.

The other symmetry is sign swapping at one or more positions, like swap sign on first member of each element. We could define it like this (return all elements with sign flipped on one of the elements members):

mod = {3, 2};
swapSignAt[set_, position_] := Table[
      element[[position]] = Mod[-element[[position]], mod][[position]]; 
      element, 
      {element, set}]

Example

Let's define some sets:

set1 = {{1, 0}, {1, 0}, {2, 1}};
set2 = {{0, 1}, {0, 1}, {1, 0}};
set3 = {{2, 0}, {2, 0}, {1, 1}};
set4 = {{2, 1}, {2, 1}, {1, 0}};
set5 = {{1, 0}, {2, 0}, {2, 1}};
listOfSets = {set1,set2,set3,set4,set5}

In this case set1, set2, set3, set4 are considered to be equivalent because set2 = shiftBy[set1, {2, 1}], set3 = swapSignAt[set1, 1], set4 = swapSignAt[shiftBy[set1, {0, 1}], 1].

I need such a function that would remove me the equivalent elements:

removeEquivalent[listOfLists]

It should return {set1,set5} (of course the content not the names like set1) or some other set of 2 non-equivalent sets.

How do I construct such a function? I suppose I should create an equivalence test setsEquivQ and use DeleteDuplicates[listOfSets, setsEquivQ], but the symmetries are so complicated I have trouble constructing such test.

Some notes. The value of mod can be list of variable length, in the real code mod is passed to functions. For example the elements could be triples and mod could be {8,9,7}. I also have possibleShifts[mod] available - it seems that such list might be handy when testing the shifting equivalence.

Edit: The element order doesn't matter. We could append //Sort to those symmetry functions to make the sets more standartised.

Edit 2: For the context. What matters for me in the end is Differences @* DeleteDuplicates @ Tuples[set, 2] - differences of all pairs. This result isn't affected by sorting set or shifting the elements. And swapping a sign in a component would only swap a sign in result but does not affect the compositions of the result (if 2 of the differences are the same or not same, it will stay that way when sign of one component is flipped on each). However, I am not directly checking the result as some non-equivalent sets can give the same result. Example.

Answer 1

The idea, I originally learnt from Szabolcs's answer, is to find a canonical form, when possible. Not only is this approach more efficient, I think it is also sounder and helps grasp what is happening.

This is what I suggest in your case to define a canonical form:

1. shift the set using shiftBy so that its first element is {0,0}
2. take the first list of the set which is not {0,0} and apply swapSignAt at position $i$ if and only if it decreases the $i$th element.

The following code returns such a canonical form of a set.

can[set_] := Block[{},
  (* find the shift position which starts by {0,0} by brute force *)
  set0 = shiftBy[set, mod - set[[1]]];

  (* takes the first non-zero list *)      
  list0 = Select[set0, # != {0, 0} &, 1];

  (* swap pos 1 if  first element of list0 would be reduced *)      
  If[swapSignAt[list0, 1][[1, 1]] < list0[[1, 1]], set0 = swapSignAt[set0, 1]];

  (* swap pos 2 if  second element of list0 would be reduced *)
  If[swapSignAt[list0, 2][[1, 2]] < list0[[1, 2]], set0 = swapSignAt[set0, 2]];

  set0
  ]

Then simply use use DeleteDuplicatesBy:

DeleteDuplicatesBy[setlist, can]
(* {{{1, 0}, {1, 0}, {2, 1}}, {{1, 0}, {2, 0}, {2, 1}}} *)

which returns set1 and set5, as expected.

Note The above takes into account the order of the set. If you want to relax this constraint, you should sort the elements of the set (I have not tried; maybe it is more complicated than it seems...).