Replacing duplicates instead of deleting them

Question

My code replaces each repeated element by "X".

ReplaceRepeated[{1, 2, 3, 2, 5, 2, 1, 5, 5, 2, 3, 3, 
    7}, {a1___, y_ /; y != #, a2___, y_, a3___} :> {a1, y, a2, #, 
     a3}] &@"X"
({1, 2, 3, "X", 5, "X", "X", "X", "X", "X", "X", "X", 7})

What is the most elegant code to do this?

kglr · Accepted Answer · 2021-04-25T05:52:59.140

22

ClearAll[replaceDuplicates]
replaceDuplicates[rep_: "X"] := Module[{f}, f[y_] := (f[y] = rep; y); f /@ #] &

Examples:

replaceDuplicates[] @ {1, 2, 3, 2, 5, 2, 1, 5, 5, 2, 3, 3, 7}

{1, 2, 3, "X", 5, "X", "X", "X", "X", "X", "X", "X", 7}

replaceDuplicates[Nothing] @ {1, 2, 3, 2, 5, 2, 1, 5, 5, 2, 3, 3, 7}

{1, 2, 3, 5, 7}

Alternatively:

ClearAll[replaceDuplicates2]
replaceDuplicates2[rep_: "X"] := Module[{y = #}, 
   y[[Join @@ (Rest /@ Values @ PositionIndex[y])]] = rep; y] &

Examples:

replaceDuplicates2[] @ {1, 2, 3, 2, 5, 2, 1, 5, 5, 2, 3, 3, 7}

{1, 2, 3, "X", 5, "X", "X", "X", "X", "X", "X", "X", 7}

replaceDuplicates2[Nothing] @ {1, 2, 3, 2, 5, 2, 1, 5, 5, 2, 3, 3, 7}

{1, 2, 3, 5, 7}

edited Apr 25 '21 at 05:52

answered Apr 24 '21 at 12:26

kglr

394,356
18
477
896

8

Very clever, and ten times faster than what I suggested. Cheers! – Roman Apr 24 '21 at 17:30
I am trying to understand what your function is really doing. – azerbajdzan Apr 24 '21 at 20:03
3

@azerbajdzan the heart of it is this definition f[y_] := (f[y] = rep; y) which will execute the first time it sees a value y and first perform the assignment f[y] = rep and then return y. That is why the values do not get replaced with "X" the first time they are seen. On the next time the value y is seen, because of the assignment f[y] = rep, the function simply returns y. All of this is wrapped in Module so it's cleaned up properly and then the function is mapped over the list of arguments. – b3m2a1 Apr 25 '21 at 02:54
Thank you @Roman. – kglr Apr 25 '21 at 03:32
@azerbajdzan, as b3m2a1 explains, this is a memoization trick (tutorial/FunctionsThatRememberValuesTheyHaveFound): The function f returns a first time it encounters a and redefines itself to return "X" next time it encounters a. – kglr Apr 25 '21 at 03:38
2

+1. One nitpick is that one would need to call Clear[f] (but not ClearAll!) at the end before returning the result, otherwise f will not be garbage-collected and will hang around indefinitely after the call to replaceDuplicates - and if you call it multiple times, then just as many different fs will be left hanging around. – Leonid Shifrin Apr 26 '21 at 11:56
1

Thank you @Leonid. Excellent point. – kglr Apr 26 '21 at 12:04

Roman · Answer 2 · 2021-04-24T12:08:56.187

FoldPairList[If[MemberQ[#1, #2], {"X", #1}, {#2, Append[##]}] &,
             {},
             {1, 2, 3, 2, 5, 2, 1, 5, 5, 2, 3, 3, 7}]

(*    {1, 2, 3, "X", 5, "X", "X", "X", "X", "X", "X", "X", 7}    *)

FoldPairList accumulates a list of "already seen" numbers, starting with the empty list. If the new number is in the "already seen" list, we emit "X" and keep the list unchanged; otherwise, we emit the new number and add it to the "already seen" list (with Append[##] being an abbreviation for Append[#1, #2]).

score 10 · Answer 3 · answered Apr 25 '21 at 02:44

You may use PositionIndex with ReplacePart.

With

x = {1, 2, 3, 2, 5, 2, 1, 5, 5, 2, 3, 3, 7};

and

replaceDups[expr_, rep_ : "X"] := 
 ReplacePart[expr, List /@ Flatten[Rest /@ Values@PositionIndex@expr] -> rep]

Then

replaceDups[x]

{1, 2, 3, "X", 5, "X", "X", "X", "X", "X", "X", "X", 7}

Is also faster than OP solution.

Hope this helps.

score 3 · Answer 4 · answered Apr 26 '21 at 12:09

Not that elegant but rather straight.

A = {1, 2, 3, 2, 5, 2, 1, 5, 5, 2, 3, 3, 7}
ReplacePart[A, {#} & /@ 
   Select[Range[Length[A]], MemberQ[Take[A, # - 1], A[[#]]] &] -> "X"]

Note that

Select[Range[Length[A]], MemberQ[Take[A, # - 1], A[[#]]] &]

is a set of indices those are duplicated.

score 2 · Answer 5 · answered Mar 05 '24 at 21:02

list = {1, 2, 3, 2, 5, 2, 1, 5, 5, 2, 3, 3, 7};

Positions of duplicates

p = Splice @* Rest /@ Values @ PositionIndex[list]

{7, 4, 6, 10, 11, 12, 8, 9}

Using SubsetMap to replace

SubsetMap[Array["X"&,Length @ p]&, list, p]

{1, 2, 3, "X", 5, "X", "X", "X", "X", "X", "X", "X", 7}

score 2 · Answer 6 · answered Mar 05 '24 at 22:27

2

list = {1, 2, 3, 2, 5, 2, 1, 5, 5, 2, 3, 3, 7};

Using MapAt:

p = Outer[List, Catenate[Rest /@ Values@PositionIndex[list]]]
({{7}, {4}, {6}, {10}, {11}, {12}, {8}, {9}})
MapAt["X" &, list, p]
({1, 2, 3, "X", 5, "X", "X", "X", "X", "X", "X", "X", 7})

answered Mar 05 '24 at 22:27

E. Chan-López

23,117
3
21
44

score 1 · Answer 7 · answered Mar 06 '24 at 22:06

Just an alternative relying on Ordering :

list = {1, 2, 3, 2, 5, 2, 1, 5, 5, 2, 3, 3, 7};
Permute[
  KeyValueMap[Splice@ArrayPad[{#1}, {0, #2 - 1}, "X"] &]@KeySort@Counts@#
  , Ordering@#] &@list

{1, 2, 3, "X", 5, "X", "X", "X", "X", "X", "X", "X", 7}

Replacing duplicates instead of deleting them

7 Answers7

Linked