Is there a built-in equivalent to Python's enumerate?

Question

Python's built-in function enumerate takes an iterable over $(a_0, a_1, \dots )$ as argument and returns an iterable over the sequence of pairs $((0, a_0), (1, a_1), \dots)$. For example:

>>> for p in enumerate(('a', 'b', 'c', 'd')):
...     print p
... 
(0, 'a')
(1, 'b')
(2, 'c')
(3, 'd')

Furthermore, the value returned by the enumerate function is actually a "generator" object, which means that it generates the $(i, a_i)$ pairs lazily, as it iterates over them. (This is particularly important, of course, when the iteration is over a very large number of items. In fact, enumerate accepts potentially "infinite" arguments.)

Now, given some arbitrary List X, the expression

Transpose[{Range[Length[X]], X}]

will produce a similar list of pairs, but I'd like to know if Mathematica has a built-in analogue of Python's enumerate (hopefully with lazy evaluation as well).

Answer 1

Streaming` module - general, and the case at hand

Starting with V10.1, there is an undocumented support for certain lazy operations in Mathematica. However, the primary goal of Streaming` is to support out of core computations reasonably efficiently, and lazy operations are only the secondary goal.

Example: lazy infinite lists and an analog of enumerate

Here is an example.

Load the Streaming` module:

Needs["Streaming`"]

Define an infinite lazy list of integers:

integers = LazyRange[Infinity];

Form an (infinite) lazy list of primes:

primes = Select[integers, PrimeQ];

Enumerate this list (lazily):

enumerated = MapIndexed[{#2[[1]], #1} &, primes]

Extract some elements:

Take[enumerated,{10000,20000}]//Normal//Short

(*
 {{10000,104729},{10001,104743},{10002,104759},<<9996>>,{19999,224729},{20000,224737}}
*)

Example: traversing a large list, and saving memory

Consider a following example: we have a huge list of matrices, whose elements are only 0 or 1, which we must traverse, for example we want to select only those of them which satisfy a certain criteria.

In-memory version

To be specific, consider this code on a fresh kernel:

Quit
(tuplesMem= Tuples@Table[Tuples[{0,1},11],{i,1,2}])//ByteCount//AbsoluteTiming

(*  {0.381172,738197664} *)

We now select the matrices, which have exactly 3 non-zero elements:

Select[tuplesMem,Total[Flatten[#]]==3&]//Short//AbsoluteTiming

(* {13.9526,{{{0,0,0,0,0,0,0,0,0,0,0},{0,0,0,0,0,0,0,0,1,1,1}},<<1538>>,{{1,1,1,0,0,0,0,0,0,0,0},<<1>>}}}
*)

We can inspect how much memory was required to carry out this operation:

MaxMemoryUsed[]

(* 2008377104 *)

and see that it was about 2Gb of RAM.

Lazy / out-of-core version

Now, let us try to use the out-of-core machinery that Streaming` provides. Here is some preparatory code (we'll need to quit the kernel to have a clean experiment):

Quit

Needs["Streaming`"];
Streaming`PackageScope`$LazyListCachingDirectory = $StreamingCacheBase 
  = FileNameJoin[{$TemporaryDirectory, "Streaming", "Cache"}];
    If[!DirectoryQ[$StreamingCacheBase],CreateDirectory[$StreamingCacheBase]];

(formatting is not ideal due to a bug in SE formatter for code involving $ sign). We will also need to load the code for a lazy version of Tuples, which is not part of Streaming yet:

Import["https://gist.githubusercontent.com/lshifr/56c6fcfe7cafcd73bdf8/raw/LazyTuples.m"]

Now we are ready to test things. So we do:

(lazyTuples = LazyTuples[Table[Tuples[{0, 1}, 11], {i, 1, 2}], 
 "ChunkSize" -> 100000]); // AbsoluteTiming

(* {0.410596, Null} *)

which defines a lazy list of tuples. Now we can try using Select:

(sel = Select[lazyTuples, Total[Flatten[#]] == 3 &]); // AbsoluteTiming

(* {0.00379, Null} *)

which takes almost no time, since Select is lazy by default, on a lazy list. We can inspect that by this time, we still don't use any HDD memory, and the RAM usage has been pretty modest yet:

MaxMemoryUsed[]
Total[FileByteCount /@ FileNames["*.mx", {$StreamingCacheBase}]]

(* 
  41693800

  0
*)

Now, the real work in this approach happens when we request data from the list:

Normal[sel]//Short//AbsoluteTiming

(* {38.6308,{{{0,0,0,0,0,0,0,0,0,0,0},{0,0,0,0,0,0,0,0,1,1,1}},<<1538>>,{{1,1,1,0,0,0,0,0,0,0,0},<<1>>}}} *)

We see that it took about 3 times as much time to get the result in this approach, compared to the previous in-memory approach. Let us now see at memory use:

MaxMemoryUsed[]
Total[FileByteCount /@ FileNames["*.mx", {$StreamingCacheBase}]]

(*
   112128792

   738209516
*)

What we see is a much (almost 20 times) more modest RAM use, but a substantial use of HDD space, where the chunks of the LazyList were saved.

Garbage collection issues

If we now destroy our 2 lazy lists:

LazyListDestroy /@ {sel, lazyTuples}

(* {Streaming`Common`ID[{3642634309, 1}], Streaming`Common`ID[{3642634221, 0}]} *)

those files will be automatically deleted by Streaming garbage collector:

Total[FileByteCount /@ FileNames["*.mx", {$StreamingCacheBase}]]

(* 0 *)

There is a way to make sure that those lists will be destroyed automatically, in case if they are only needed for this particular computation - with the help of LazyListBlock:

LazyListBlock[
  Normal @ Select[
     LazyTuples[Table[Tuples[{0,1},11],{i,1,2}],"ChunkSize"->100000],
     Total[Flatten[#]]==3&
  ]
]//Short//AbsoluteTiming

(* {35.9029,{{{0,0,0,0,0,0,0,0,0,0,0},{0,0,0,0,0,0,0,0,1,1,1}},<<1538>>,{{1,1,1,0,0,0,0,0,0,0,0},<<1>>}}} *)

and in this case, there are no files left on disk after the code has finished:

Total[FileByteCount /@ FileNames["*.mx", {$StreamingCacheBase}]]

(* 0 *)

Notes

This answer should not be considered as any kind of tutorial on this functionality, but just as an illustration. Also, there is no guarantee, that this functionality will remain in future versions and / or have the same syntax in the future. It may also suffer from efficiency problems, to a smaller or greater extent depending on the task, since it has been implemented in top-level Mathematica.

Note by the way, that technically the lists constructed above are not fully lazy. What really happens there is that data is divided into chunks, and a given operation (Map or whatever) is applied to the entire chunk at the same time. The chunk size can be controlled, but the laziness is only there on the coarse - grained level (per chunk) - this was done to keep the performance reasonable. One can, in principle, in most implemented lazy functions, set chunk size to be one element, but that would very seriously degrade the performance.

Answer 2

Here are some ways this could be done:

list = CharacterRange["a", "g"];
Thread[{Range[Length@list], list}]
Transpose[{Range@Length@list, list}]
Table[{j, list[[j]]}, {j, Length@list}];
MapIndexed[{#2[[1]], #1} &, list]
Inner[List, Range@Length@list, list, List]

You could re-index by using Range[0, Length@list-1]

or

i = 1;
{i++, #} & /@ list

or more ridiculously,

j = 1;
Fold[Append[#1, {j++, #2}] &, {}, list]

Partition[Riffle[Range@Length@list, list], 2]

k = 1;
list /. x_String :> {k++, x}