5

Array => ["cross", "base", "cross", "dunk"]

I want to delete all repeated values without sorting.

Command: jq '. | unique' file.json

Output: ["base", "cross", "dunk"]

Expected Output: ["cross", "base", "dunk"]

Stéphane Chazelas
  • 544,893
decipher
  • 302

2 Answers2

7
map({key:.,value:1})|from_entries|keys_unsorted

as an alternative to unique seems to work.

map turns the array into:

[{"key":"cross","value":1},{"key":"base","value":1},{"key":"cross","value":1},{"key":"dunk","value":1}]

from_entries turns that into:

{"cross":1,"base":1,"dunk":1}

Removing the duplicates at that point since two elements of an object can't have the same key.

And keys_unsorted returns the keys of that object, in the original order.

A shorter/simpler alternative working on the same principle:

map({(.):1})|add|keys_unsorted

Or you can use a more generic programming language such as perl:

perl -l -0777 -MJSON::PP -MList::Util=uniq -pe '
  $_ = encode_json[uniq @{decode_json $_}]' array.json

-p is the sed mode, pull each record of array.json into $_ for the eexpression to work on.

-0777 sets the record separator to something impossible, meaning there's only one record being the full file (see also -gobble in newer versions of perl).

decode_json $_ decodes that record and returns an array reference, @{...} makes that a list passed to uniq, [...] makes the result another array reference passed to encode_json, the resulting $_ is printed (followed by a newline with -l).

There are a number of JSON plugins for perl. JSON::PP (PP for pure perl) is part of the core of perl, so should always be available. JSON::XS would be more efficient and also comes with a json_xs which is handy to use in shell scripts:

json_xs -e 'use List::Util "uniq"; $_ = [uniq @$_]' < array.json

If you're already familiar with perl, that means you don't need to learn a new language like that of jqs and can reuse what you know to process other formats like XML, YAML, etc.

Stéphane Chazelas
  • 544,893
  • I'd love to know some good reference for jq (with many exemples, ained at people that already handle scripting langages so quite short and to the point, or maybe cookbook style?) if you (or anyone reading this) have some. I had one that "opened" awk to me initially, but haven't found it for jq yet – Olivier Dulac Mar 05 '23 at 13:52
  • 1
    @OlivierDulac, I use man jq for reference. For examples, you can look at answers here. Look out for Kusalanda's for instance. – Stéphane Chazelas Mar 05 '23 at 17:15
  • 1
    @OlivierDulac The online manual is the same as the manual you get with man, but is conveniently searchable and possibly more readable than the ordinary text manual. It is also easy to point to separate sections in it. See e.g. the reduce() function: https://stedolan.github.io/jq/manual/#Reduce – Kusalananda Mar 05 '23 at 17:45
  • This solution picks the "last value in input array", while the @Kusalananda answer picks the "first value in input array" – laconbass May 08 '23 at 12:15
4

Using reduce() to build the resulting array of unique elements from the input:

jq 'reduce .[] as $a ([]; if IN(.[]; $a) then . else . += [$a] end)' file.json

This essentially iterates over the array in file.json, adding each element to a new array unless it's already present in that array.

Testing:

$ jq . file.json
[
  "cross",
  "base",
  "cross",
  "dunk"
]

$ jq 'reduce .[] as $a ([]; if IN(.[]; $a) then . else . += [$a] end)' file.json
[
  "cross",
  "base",
  "dunk"
]
Kusalananda
  • 333,661