How to determine how much of a table is generated?

Question

If I'm generating a table from a slow function, like this:

foo = Table[SuperSlowExpression, {1000}]

is there a way to determine after submission (so it is too late to instrument the call) to find out how many values already have been generated (so that when it takes longer than expected I can find out whether to wait a bit longer or to abort the calculation)?

So is there still a way to get at the information? Note that there is no iteration variable in the call which I could read.

Optimally, I'd also want to get the already calculated values, of course.

Edit:

Since this seems to have been consistently overlooked:

This is about a calculation which is already running. It is no longer possible to make changes to the call of Table or to the expression inside.

Answer 1

Yes, there is. You can use Monitor

Monitor[
  Table[Pause[1]; i, {i, 10}],
  i
]

Some explanations

Table uses dynamic scoping to localize its variable (i in this case), just like Block. This means that while Table is evaluating, i has a "global" value that one can inspect even from outside the Table if we interrupt the evaluation.

On way of interrupting the evaluation is to request i's value on the pre-emptive link (i.e. using Dynamic expressions) We could just put Dynamic[i] somewhere and it would count up just like Manipulate while the Table is evaluating.

Another way of interrupting the evaluation is through entering a Dialog: You can use the Evaluation -> Interrupt evaluation... menu item, choose Enter subsession. You will get a prompt where you can inspect the kernel state (including the value of i, Stack[], etc.). Evaluate Return[] to resume he evaluation.

I have been using both of these techniques extensively to monitor very lengthy evaluations. The second one even seems to work for parallel evaluations, and least in those cases where I tried it.

Iteration variable and already calculated values

If there is no iteration variable in the Table, just insert one, and use Table[..., {i, 1000}]. Then we can use Monitor or Dynamic[i].

It is not possible to get the already calculated values. If you want to do this (it would often be desirable to be able to!), you need to implement your own version of Table. I asked about how to implement an interruptible parallel Table or Map in this question. The solution I got does not allow inspecting the so far calculated results, but it lets me interrupt the calculation, and continue from where I stopped later.

A table function that allows monitoring partial results

As an example, I provide a custom table function that allows monitoring the partial results.

Try evaluating

Dynamic[table`Results[]]
table[Pause[1]; i^2, {i, 10}]

``tableResults[] could be used in Monitor too.

I am using a linked list to collect the results for better performance than what AppendTo would give. I cannot use Sow/Reap here because they don't allow inspecting the partial results (at least I am not aware of this being possible).

Note: If you use nested tables, only the inner one can be monitored, but all of them will work correctly.

---

The code:

ClearAll[table, table`result, table`Results]
SetAttributes[table, HoldAll]

table`result = {};

table[expr_, iterator__] :=
 Block[{table`result, table`elem},
  table`result = {};
  Do[
   table`result = {table`result, table`elem[expr]},
   iterator
   ];
  table`Results[]
  ]

table`Results[] := Block[{table`res, table`elem},
  table`res = Flatten[table`result];
  table`elem = Identity;
  table`res
  ]

Answer 2

Given your current situation, there is another option that might help if you have set "Enable notebook history tracking" in Preferences > Advanced:

Then, you go to Cell > Notebook history and navigate to your currently evaluating cell and look at the time stamp when you last edited it. Chances are that you edited it just prior to evaluating (note that if you open an old notebook and execute the cell right away, this might not work, because the last edit timestamp will not be what you want it to be).

Now find the current time from your system clock. Assuming you know how long it takes for one evaluation of SuperSlowExpression, you can now simply do:

$$\mathrm{approx\ progress=\frac{current\ time - last\ edited\ time}{time\ for\ one\ evaluation}}$$

This will give you a rough estimate of the progress — enough to make a decision whether to wait a bit longer or to give up.

Remember, this should be a last ditch option! There are many ifs and buts here, but seeing as all the other options require an iterator and you don't have one (and your Table is running), it's worth trying...

Answer 3

Similar, with a progress bar:

tEnd = 10;
ProgressIndicator[Dynamic[i], {0, tEnd}]
Table[Pause[1]; i, {i, tEnd}]

Answer 4

Suppose you're evaluating something slightly different:

Table[Pause[0.1](*SuperSlowExpression*), {i, 1000}]

You can create a cell containing i, and use Evaluation > Evaluate in Subsession to see the current value of i without interrupting the main evaluation. In the case where you don't have an iterator variable to inspect, then the question is whether there is some state or property in your SuperSlowExpression that could be evaluated to get an idea of how much progress has been made.

Answer 5

If you're using a function that doesn't use iterators such as Scan or Fold you could consider using PrintTemporary, for example

f[x_] := (Pause[2]; Sin[x]);
res = {};
Module[{i = 0, temp},
  Scan[(NotebookDelete[temp];
    temp = PrintTemporary[++i];
    AppendTo[res, f[#]]) &, Range[20]]]