10

Here is my simple example, and in this case function DeleteDuplicates does not work as expected.

I want to FindRoot of my function $\chi[\nu]$, and since function $\chi$ is very sensitive to initial guess I decided to generate many initial conditions and leave only those solutions which are distinct. For this purpose I want to apply DeleteDuplicates on the resulting list of solution.

Here is the definition of my function:

χ[ν_] := 2*PolyGamma[1] - PolyGamma[1/2 + I*ν] - PolyGamma[1/2 - I*ν]

Here I generate many solution according to many random initial guesses

m = Table[v /. FindRoot[χ[v] == -1.2 - 0.2*I, {v, RandomComplex[]}], {i, 1, 10}]

And finally, I want to leave only distinct solutions:

DeleteDuplicates[m]

Unfortunately, the operation DeleteDuplicates[m] does not change the list m, although there are many identical values.

Namely:

DeleteDuplicates[m]

{1.06423 + 0.0968739 I, 1.06423 + 0.0968739 I, 
 1.06423 + 0.0968739 I, 1.06423 + 0.0968739 I, 0.0250407 + 1.00352 I, 
 1.06423 + 0.0968739 I, 1.06423 + 0.0968739 I, 1.06423 + 0.0968739 I, 
 0.0250407 + 1.00352 I, 1.06423 + 0.0968739 I}

I'm puzzled.

Any help or suggestions are very welcome!

Thanks!

Mr.Wizard
  • 271,378
  • 34
  • 587
  • 1,371
Arnold Klein
  • 315
  • 2
  • 8

2 Answers2

12

You would do well to understand the difference between tools that are intended for structural operations and those that are intended for mathematical operations. DeleteDuplicates is of the former, generally speaking. As such it is comparing the exact FullForm of the objects, or at least something close (caveat).

As b.gatessucks recommends in a comment you can use a mathematical comparison function for the equivalence test of DeleteDuplicates, e.g.:

 DeleteDuplicates[m, Abs[#1 - #2] < 10^-12 &]

{1.06423 + 0.0968739 I, 0.0250407 + 1.00352 I}

Incidentally you could also use Union, but the syntax is a bit different. Note the ( ).

 Union[m, SameTest -> (Abs[#1 - #2] < 10^-12 &)]

{0.0250407 + 1.00352 I, 1.06423 + 0.0968739 I}

Using InputForm to show all of the digits of your expression you can see that they are not structurally identical in the (approximate) way that Mathematica "sees" them:

m // InputForm

{1.0642275928442373 + 0.09687392021742822*I, 1.0642275928442366 + 0.09687392021742817*I, 
 1.0642275928442366 + 0.09687392021742797*I, 1.064227592844237 + 0.09687392021742822*I, 
 1.0642275928442373 + 0.09687392021742852*I, 1.0642275928442366 + 0.09687392021742793*I, 
 1.0642275928442368 + 0.09687392021742801*I, 0.025040728196256346 + 1.0035162552538588*I, 
 1.0642275928442377 + 0.0968739202174282*I, 1.0642275928442375 + 0.0968739202174283*I}

Performance

Yves reminded me to mention something about the performance of using a custom comparison function in DeleteDuplicates or Union as I did above. For long lists this is always considerably slower than using the default method. I gave an example with timings in How to represent a list as a cycle.

To apply that method here we could Round the numbers beforehand:

Round[m, 10^-12] // DeleteDuplicates // N

{1.06423 + 0.0968739 I, 0.0250407 + 1.00352 I}

I added // N to convert back to machine precision, but the values will not be precisely the same. This probably doesn't matter if you consider numbers this close to be duplicates, but should you want the unchanged numbers you could use GatherBy and get performance not far distant.

First /@ GatherBy[m, Round[#, 10^-6] &]

Version 10.0 introduced DeleteDuplicatesBy which works similarly to the GatherBy method; it has the following syntax:

DeleteDuplicatesBy[m, Round[#, 10^-6] &]

However it may not perform as well as GatherBy; see:

Community
  • 1
Mr.Wizard
  • 271,378
  • 34
  • 587
  • 1,371
  • Thanks for the answer, I see. It's like a comparison with epsilon of two floating point numbers. I assumed, that MATHEMATICA already does it for me. – Arnold Klein Mar 20 '13 at 10:08
  • Great, I missed InputForm. Without it it looks the same. – Arnold Klein Mar 20 '13 at 10:09
  • @Andrew On some mathematical operations it may, but as I stated DeleteDuplicates is generally a structural tool. You will run into a similar problem in many cases. For example, replacing elements in a list, because Replace and ReplaceAll are also structural tools: {0, 0., 0.00000} /. 0 -> 1 -- you need a mathematical tool like Equal: {0, 0., 0.00000} /. x_ /; x == 0 -> 1 – Mr.Wizard Mar 20 '13 at 10:12
  • 1
    @Andrew FullForm was a bit needlessly verbose here, but in general you should look at the FullForm when trying to understand how Mathematica will treat an expression using structural tools. There are many cases where not using it will leave you quite befuddled as it can be very different from what is shown in standard output notation. – Mr.Wizard Mar 20 '13 at 10:13
  • Perhaps a few words on possible performance caveats? – Yves Klett Mar 20 '13 at 10:47
  • @Yves you mean the slow-down caused by using the second argument of DeleteDuplicates? I didn't think it was appropriate unless we were comparing this to some other method that works. But then, perhaps I should provide another method... – Mr.Wizard Mar 20 '13 at 10:55
  • yes, I meant that. Sometimes it may be more effective to pre-process the list and then do a pure DeleteDuplicates - although your version is also what I´d go with in any case. – Yves Klett Mar 20 '13 at 11:42
  • 1
    Admonish - lovely acoustics but far too harsh. I thought of it as "gently encouraging" at most... – Yves Klett Mar 21 '13 at 14:13
  • @Yves Perhaps a harsher connotation than I intended; I was thinking "3. to urge to a duty; remind" – Mr.Wizard Mar 21 '13 at 23:14
  • OH No please leave "admonished" in. It really is a lovely word (and I like the ambiguity) – Yves Klett Mar 22 '13 at 08:19
  • How could this be done if instead of just one variable, FindRoot is solving for two or more variables? I.e., the output of FindRoot would be for example {{x-> 0.1,y-> 0.2, z->0.3}, {x-> 0.1,y-> 0.2, z->0.3}, ...} – AJHC Mar 24 '19 at 11:35
  • 1
    @AJHC Converting to plain lists using {x, y, z} /. solutions seems like a good way to start. For example with m2 = {{x -> 0.1, y -> 0.2, z -> 0.3}, {x -> 0.1, y -> 0.2, z -> 0.3}, {x -> 0.17, y -> 0.22, z -> 0.314}} then DeleteDuplicates[{x, y, z} /. m2, AllTrue[Abs[# - #2], # < 10^-5 &] &] or DeleteDuplicatesBy[{x, y, z} /. m2, Round[#, 10^-5] &] for the two fundamentally different methods described in my answer. If you need to convert back to a list of rules then something like Thread[{x, y, z} -> #] & /@ {{0.1, 0.2, 0.3}, {0.17, 0.22, 0.314}} – Mr.Wizard Mar 25 '19 at 12:05
1

Note also that one can use Equal[##] & but not Equal:

DeleteDuplicates[m, Equal[##] &]
(*  {1.06423 + 0.0968739 I, 0.0250407 + 1.00352 I}  *)

DeleteDuplicates[m, Equal] // Length
(*  10  *)

This should work the same on most or all well-behaved FindRoot results.

As already noted, using something like Equal[##] & causes the performance to degrade, significantly on very long lists, but using Equal does not cause the same problem. However, Equal does not tolerate difference, unlike Equal[##] &:

DeleteDuplicates[{1., 1. + $MachineEpsilon}, Equal]
    DeleteDuplicates[{1., 1. + 64 $MachineEpsilon}, Equal[##] &]
(*
  {1., 1.}
  {1.}
*)

Basically DeleteDuplicates[.., Equal] is the same as DeleteDuplicates[].

Michael E2
  • 235,386
  • 17
  • 334
  • 747