Most of the other answers focus on duplicate positions amongst numbers but what about duplicate positions amongst other types and/or within deeper arrays? Neither of these were originally specified but it can be interesting to consider these alternatives and the order-of-magnitude efficiency improvements achievable from some minimal pre-processing (with some authentic applications). Further, variability in input, not only in terms of type and depth, but also in terms of duplicate-distribution can significantly impact efficiency. Taken together this suggests, perhaps, the need for dedicated DuplicatePositions and DuplicatePositionsBy functions, the case for which is later built. First to the efficiency improvement when finding duplicates amongst lists of reals.
Needs["GeneralUtilities`"]
positionDuplicates[ls_] := GatherBy[Range@Length@ls, ls[[#]] &];
duplicatePositions[ls_] := positionDuplicates[FromDigits /@ ls];
SeedRandom@0;
vectors[n_] := IntegerDigits /@ RandomInteger[n, 5*n];
BenchmarkPlot[{positionDuplicates, duplicatePositions}, vectors[10^#] &, Range@3, "IncludeFits" -> True]
At around 1K items the efficiency difference becomes meaningful (benchmark is defined in a related answer with a green tick indicating identical output for all timings).
fns = {duplicatePositions, positionDuplicates};
n = 10^3;
vectors5k = IntegerDigits /@ RandomInteger[n, 5*n];
benchmark[fns]@vectors5k
By performing the pre-processing that converts vectors into integers, all of GatherBy's optimizations can be bought to bear (namely, reducing the number and cost of pair-wise comparisons that implicitly occur in GatherBy's sorting). Hence efficiently finding duplicate positions ultimately depends on efficient sorting which in turn (often) depends on the underlying objects possessing a natural order. For arbitrary objects this order may need to be user-imposed (thereby motivating a DuplicatePositionsBy function). Note that in the previous example, IntegerDigits produces different sized vectors and for which, apparently, no order has been internally recognized. By fixing the vector size with padding if necessary and this order apparently now does become recognizable with a still discernible but reduced efficiency advantage.
vectors[n_] := IntegerDigits[#, 10, 5] & /@ RandomInteger[n, 5*n];
BenchmarkPlot[{positionDuplicates, duplicatePositions}, vectors[10^#] &, Range@4, "IncludeFits" -> True]
and individually
n = 10^4;
vectors50k = IntegerDigits[#, 10, 5] & /@ RandomInteger[n, 5*n];
benchmark[fns]@vectors50k
A similar efficiency edge, possibly from more efficient pairwise-comparisons arises from using PositionIndex such as for strings.
duplicatePositions[M_] := Values@PositionIndex@M;
n = 10^5;
strings100K = ToString & /@ RandomInteger[n, 5*n];
benchmark[fns]@strings100K
The distribution of duplicates can also directly impact on a method's efficiency. Note how in the OP's example, a random selection amongst n numbers was performed 5n times to ensure that a normalish distribution of duplicates is generated. The following shows how varying this 5 factor affects the duplicate distribution.
Manipulate[Histogram[Length /@ duplicatePositions@RandomInteger[n, n*r // Round], PlotRange -> {{0, 100}, Automatic}], {{n, 1000}, 1, 101}, {{r, 30}, .1, 100, 1}]
Increasing $r$ increases the chance of duplicates eventually leading to a uniform distribution. Decreasing $r$, on the other hand, decreases the chance of duplicates eventually leading to a distribution of singleton sets. For the latter, it turns out there is another implementation that seems to perform unreasonably well for sparse vectors.
duplicatePositionsNew[ls_] := SplitBy[Ordering@ls, ls[[#]]&]//SortBy[First];
SeedRandom@0;
vectors[n_] := RandomInteger[1, {n, n}];
fns = {duplicatePositions, duplicatePositionsNew, positionDuplicates};
BenchmarkPlot[fns, vectors[10^#] &, Range@4, "IncludeFits" -> True]
If it is known in advance that such sparsity is almost guaranteed, then by sacrificing absolute certainty one gains the efficiency of simply returning singleton sets. If one suspects sparse input but still wants to maintain certainty for rare duplicates then this "ordering" implementation may be the desired function.
n = 10^4;
sparseVectors10k = RandomInteger[1, {n, n}];
benchmark[fns]@sparseVectors10k
All these case-based efficiencies suggest a "superfunction"--DuplicatePositions-- that maintains demonstrated efficiency advantages either automatically when efficiently detectable or, when it is not, via a method option and/or defining an order by preprocessing in "DuplicatePositionsBy". Note that such a superfunction defaults to Szabolcs' efficient positionDuplicates for numbers and for good measure includes Carl Woll's eeking out of extra speed via a "UseGatherByLocalMap" option setting. An initial implementation might look something like:
Options[DuplicatePositions] = {Method -> Automatic};
DuplicatePositions[ls_, OptionsPattern[]] :=
With[{method = OptionValue[Method]},
Switch[method,
"UseGatherBy", GatherBy[Range@Length@ls, ls[[#]] &],
"UsePositionIndex", Values@PositionIndex@ls,
"UseOrdering", SplitBy[Ordering@ls, ls[[#]] &] // SortBy[First],
"UseGatherByLocalMap", Module[{func}, func /: Map[func, _] := ls;
GatherBy[Range@Length@ls, func]],
Automatic, Which[
ArrayQ[ls, 1, NumericQ],
DuplicatePositions[ls, "Method" -> "UseGatherBy" ],
ArrayQ[ls, 2, NumericQ], DuplicatePositionsBy[ls, FromDigits],
MatchQ[{{_?IntegerQ ..} ..}]@ls,
DuplicatePositionsBy[ls, FromDigits],
True, DuplicatePositions[ls, Method -> "UsePositionIndex" ]
]]];
DuplicatePositionsBy[ls_, fn_, opts : OptionsPattern[]] :=
DuplicatePositions[fn /@ ls, opts];
Putting it all together
n = 10^3;
normal := RandomInteger[n, 5*n];
vectors5k := IntegerDigits[#, 10, 6] & /@ normal;
vectorsRagged5k := IntegerDigits /@ normal;
strings5k := ToString /@ normal;
benchmark[{
DuplicatePositions, positionDuplicates}] /@
Unevaluated@{vectors5k, vectorsRagged5k, strings5k}
n = 10^4;
normal50k := RandomInteger[n, 5*n];
benchmark[{
DuplicatePositions,
DuplicatePositions[#, Method -> "UseGatherBy"] &,
DuplicatePositions[#, Method -> "UseGatherByLocalMap"] &,
positionDuplicates}] /@ Unevaluated@{normal50k}
n = 10^3;
sparseVectors1k := RandomInteger[1, {n, n}];
benchmark[{
DuplicatePositions,
DuplicatePositions[#, Method -> "UseOrdering"] &,
positionDuplicates}] /@ Unevaluated@{sparseVectors1k}
While it might seem that finding duplicate positions in arbitrary-depth arrays are sparse corner cases, this is only from the perspective of random generation and the challenge of searching within an "interesting" normal distribution. Duplicates are however, frequently injected into high-dimensional structures by satisfying specified symmetries. In fact, such symmetrization defines much of WL's rich tensor framework. DuplicatePositions therefore possesses a natural applicability to symbolic tensors produced from SymmetrizedArray, SparseArrays etc. (with itself potentially returning new, structured objects) with further relations to SymmetrizedDependentComponents, and DeleteDuplicates. For this application one might imagine a 4-argument function along the lines of:
DuplicatePositions[expr, test, comp, levspec]
(note that for arbitrary-depth arrays the very notion of a duplicate becomes level-dependent. In a list of binary vectors it was implicitly assumed that the duplicates of interest were at the top level [the positions of binary vectors] and not at the bottom level [the positions of 0's and 1's] which seem to have greater interest for deeper arrays).
seennecessary?Last@Reap[MapIndexed[Sow[#2, #1] &, list]]– Szabolcs Mar 14 '13 at 19:58Select[result, Length[#] > 1&]or similar. – Szabolcs Mar 14 '13 at 20:05