10

I would like to send a set of Mathematica expressions (i.e. not simple types that are easy to represent and temporarily store in C) in a List using the MathLink C API. I do not know how many there will be before I compute all of them, but I need their count to be able to call MLPutFunction.

How can I conveniently send a List without needing to know the number of elements beforehand?

One idea I had was to put them on a loopback link while counting them, then transfer the contents of the loopback link at once. But I was hoping for a simpler solution.

Szabolcs
  • 234,956
  • 30
  • 623
  • 1,263
  • 2
    I have a solution for it, but I don't recall all of the details. Essentially, you start with an open List, and then each member is added as Sequence[dat,a] where a is the next data point, i.e. you make a nested list. Then you can close it by adding Sequence[] as your last point, collapsing the whole structure. Alternatively, you could use a loop-link. I'll write it up, if I get the chance. – rcollyer Jan 31 '13 at 18:36
  • 1
    Instead of Sequence,which will lead to a quadratic complexity because of the run-time splicing, I would use some inert head and build a linked list, Flatten-ing this at the end. See my comments under Arnoud's answer. – Leonid Shifrin Jan 31 '13 at 19:12
  • Could also use the Internal`Bag, as shown in Leonid's answer to the question on SO linked by Rolf? – Oleksandr R. Jan 31 '13 at 19:25
  • @OleksandrR. But how will the bag help for transferring data via MathLink? – Leonid Shifrin Jan 31 '13 at 19:27
  • @LeonidShifrin could send each element of the list one-at-a-time wrapped in StuffBag and then unpack them later on. Whether it's acceptably efficient to send each item individually would depend on the application, I suppose. – Oleksandr R. Jan 31 '13 at 19:31
  • @OleksandrR. Thanks, I see. I did not consider such solution. If we are open to this kind of solutions as well, then there are many more options, although the bag is probably the best. – Leonid Shifrin Jan 31 '13 at 19:44

2 Answers2

9

I consider the loopback link solution to this problem to be simple and ideal. I appreciate the cleverness of the LinkedList/Bag/Sequence types of solutions that have been proposed in other answers, but they seem to require a little too much thought for my taste.

This type of problem is one of the practical applications that motivated the creation of loopback links.

int err;
MLINK loop = MLLoopbackOpen(stdenv, &err);
int count = 0;
while (test) {
    ....
    MLPutInteger(loop, i);
    count++;
}
MLPutFunction(stdlink, "List", count);
MLTransferToEndOfLoopbackLink(stdlink, loop);
MLClose(loop);
Todd Gayley
  • 4,198
  • 20
  • 19
  • Does the loopback link interface provide a method for counting the elements already put on the link? – rcollyer Feb 01 '13 at 15:04
  • 1
    @rcollyer But why would one have to need it? Such a counter is easy to set up, if necessary, during putting pieces onto the link, seems to me. – Leonid Shifrin Feb 01 '13 at 15:32
  • @LeonidShifrin Either, I give each function I pass the link to access to a global variable (the counter), or I pass in the counter. In the first case, I break encapsulation. In the second, it increases the work I need to do to use it. By providing such a function, I can simplify how I use the link. – rcollyer Feb 01 '13 at 16:48
  • @rcollyer Ok, good point. In addition to what you suggested, one could create a data structure incorporating the link and the counter (a struct), and pass that one. This will however require one to also define new functions for putting various types on the link, which would increment the counter, so this is still some work. – Leonid Shifrin Feb 01 '13 at 16:59
  • @LeonidShifrin yes, you could define it that way. It is for things like this that c++ is superior: automatic type detection so that you can overload function declarations. From the user perspective, no more MLPutInteger, MLPutShortInteger, MLPutLongInteger, etc., only MLPut. The libs creators still need to make each one, or use a template, so not necessarily less work on their part. But, much simpler to use. I did a little work trying to set up such a c++ interface, but time constraints interfered. – rcollyer Feb 01 '13 at 19:50
  • @rcollyer I generally dislike C++ (while I very much like C), but don't have much experience with it, so my words here don't weigh much. I can well imagine cases where it is appropriate however, perhaps including your suggestion. – Leonid Shifrin Feb 01 '13 at 19:57
  • @LeonidShifrin I am in the opposite boat: I like c++, but am not in general a fan of c, also because I have not had much exposure to it. – rcollyer Feb 01 '13 at 20:43
  • Could you please take a look at this? Only someone who knows the internals of the kernel and MathLink would be able to answer this question (is it safe and portable what I propose or not). – Szabolcs Feb 04 '13 at 22:33
6

See: http://reference.wolfram.com/mathematica/tutorial/HandlingListsArraysAndOtherExpressions.html

From that documentation page:

In order to call functions like MLPutFunction(), you need to know the length of the list you want to send. But by creating a sequence of nested Sequence objects, you can avoid having to know the length of your whole list in advance.

However, based on Leonid's comments below it is (much) more efficient to use an inert head like LinkedList so the example below is updated to use that.

This sets up the List around your result:

MLPutFunction(stdlink, "List", 1);
while( condition ) {
 /* generate an element */

Create the next level LinkedList object.

 MLPutFunction(stdlink, "LinkedList", 2);

Put the element.

 MLPutInteger32(stdlink,  i );
}

This closes off your last LinkedList object.

MLPutFunction(stdlink, "LinkedList", 0);

Finally the kernel will need to flatten this expression:

Flatten[ result, Infinity, LinkedList ]
Arnoud Buzing
  • 9,801
  • 2
  • 49
  • 58
  • 2
    Sorry to bring the bad news, but splicing Sequence will make this a quadratic complexity solution, in the size of the list. It is unfortunate that this example made it into the docs. If the list is short and the complexity is not an issue, this is an acceptable solution, of course. – Leonid Shifrin Jan 31 '13 at 19:05
  • It is easy to cure this solution by using some inert head instead of Sequence, say LinkedList, and make a linked list out of this. Then use Flatten[result, Infinity, LinkedList] at the end. This will have linear complexity. So, +1, assuming this modification to the answer. – Leonid Shifrin Jan 31 '13 at 19:11
  • But isn't there also a limit of the order of 10^4 if I remember correctly? See also here – Rolf Mertig Jan 31 '13 at 19:17
  • @RolfMertig There shouldn't be a limit, at least not so small one. I tried building linked lists of sizes 10^6 or so, and it worked fine, at least if you just build them and flatten them. – Leonid Shifrin Jan 31 '13 at 19:21
  • 1
    @LeonidShifrin why doesn't that exhibit the same complexity as Sequence? – rcollyer Jan 31 '13 at 19:30
  • 1
    @rcollyer To understand this, you have to see why Sequence -based method has the quadratic complexity. Imagine that you have something like Sequence[1,Sequence[2,Sequence[3, Sequence[4]]]]. First, the inner Sequence gets spliced. We get Sequence[1,Sequence[2,Sequence[3,4]]]. And now, the evaluation starts over! But now, the inner Sequence has two elements, so they both will have to be copied during splicing. When you do this n times, each successive time you have to copy i elements. And basically, you copy an array, since Sequence is array-based. – Leonid Shifrin Jan 31 '13 at 19:40
  • @rcollyer Now, for some generic inert head, this is not the case. There are no non-trivial evaluations going on, and building of such an expression is fast because when you have head[1,head[2,head[3]]], on every level you only have two pointers - to the head element and the tail. For the same reason, Flatten-ing is fast. Basically, Flatten performs only one evaluation without copying arrays, while Sequence-splicing is n sub-evaluations, each of which copies the previously formed sequence. – Leonid Shifrin Jan 31 '13 at 19:42
  • @rcollyer Here is some benchmarking code: toLinkedList[l_List] := Fold[ll[#2, #1] &, ll[], Reverse[l]]; and AbsoluteTiming[{toLinkedList[Range[#*1000]] /. ll -> Sequence};] & /@ (2^Range[0, 3]). This versus AbsoluteTiming[List @@ Flatten[toLinkedList[Range[#*1000]], Infinity, ll];] & /@ (2^Range[0, 3]). – Leonid Shifrin Jan 31 '13 at 19:52
  • @LeonidShifrin interesting benchmarks. Initially, I thought ReplaceAll was inflating the benchmark, so I "fixed" it by doing: Block[{ll, llist, count = #},llist = toLinkedList[Range[count*1000]];ll = Sequence; AbsoluteTiming[{llist};]] & /@ (2^Range[0, 3]). The results are fascinating: it is twice as slow as the original. – rcollyer Jan 31 '13 at 20:04
  • @rcollyer I guess, the slowdown is because ReplaceAll, according to its name, replaced all occurrences of ll with Sequence at once, and quite fast. OTOH, using Block, you added an extra evaluation step in substituting ll with Sequence, for each sub-evaluation (Sequence splicing) - thus a factor of 2. – Leonid Shifrin Jan 31 '13 at 20:10