Compile function dynamically

Question

To optimize a brute force algorithm on graphs I want to compile dynamically a list of functions for a list of graphs.
(The representation I'm using for a graph is a list of pairs of integers: integers are nodes, pairs are edges.)

To create one function I need:

the graph representation
a displacement list.

displacement is constant between all the functions of graphs with $n$ nodes, but since my program operates on various $n$ and also because I don't know displacement in advance and I will actually evaluate it, I'd like to pass displacement too as a parameter for my "pure compilator function".

Here's an example of one single graph and its relative compiled function

displacement = {1, 2, 4, 8, 13, 21, 31};
graph = {{1, 2}, {2, 3}, {2, 4}};
tergetFun = Compile[{{a, _Integer, 1}}, With[{p = {1, 2, 4, 8, 13, 21, 31}}, {p[[a[[1]]]] + p[[a[[2]]]], p[[a[[2]]]] + p[[a[[3]]]], p[[a[[2]]]] + p[[a[[4]]]]}]];

It's not hard to dynamically generate the expression for the function

(p[[a[[#1]]]] + p[[a[[#2]]]]) & @@@ graph
Part::partd: Part specification a[[1]] is longer than depth of object.
Part::pkspec1: The expression a[[1]] cannot be used as a part specification.
Part::partd: Part specification a[[2]] is longer than depth of object.
(* the entire list of errors *)
(* {p[[a[[1]]]] + p[[a[[2]]]], p[[a[[2]]]] + p[[a[[3]]]], p[[a[[2]]]] + p[[a[[4]]]]} *)

Done in this way Mathematica attempts to evaluate Part, failing. Nonetheless the output is exactly the one I need.
I've tried any kind of evaluation control but this most direct and wrong(?) way of doing it is the only one actually working when I use it as argument in Compile

Compile[{{a, _Integer, 1}}, With[{p = #1}, #2]] & @@ {#1, ((p[[a[[#1]]]] + p[[a[[#2]]]]) & @@@ #2)} & @@ {displacement, graph}
(* list of errors *)
(* targetFun *)

Should I use Quiet or some form of evaluation control actually works?

Henrik Schumacher · Accepted Answer · 2022-02-16T18:37:05.833

5

Just use Compile`GetElement or Indexed instead of Part:

part = Compile`GetElement;
With[{p = #1}, Compile[{{a, _Integer, 1}},
       #2
       ]
      ] & @@ {#1, ((
         part[p, part[a, #1]] +part[p, part[a, #2]]
         ) & @@@ #2)
     } & @@ {displacement, graph}

But I do not see any reason why you want to recompile the function that many times when you can simply use this:

cf = With[{part = Compile`GetElement},
   Compile[{{a, _Integer, 1}, {p, _Integer, 1}, {edges, _Integer, 2}},
    Table[
     part[p, part[a, part[edges, k, 1]]] + part[p, part[a, part[edges, k, 2]]],
     {k, 1, Length[edges]}
     ],
    CompilationTarget -> "C",
    RuntimeAttributes -> {Listable},
    Parallelization -> True,
    RuntimeOptions -> "Speed"
    ]
   ];

and call it by cf[a, displacement, graph]. If you have a list of many a, then you can call this in a listable and parallelized way as follows:

displacement = {1, 2, 4, 8, 13, 21, 31};
graph = {{1, 2}, {2, 3}, {2, 4}};
n = 10;
alist = Permutations@Range@n;
result = cf[alist, displacement, graph];

This maybe copies displacement and graph to every thread, but it does certainly not make Length[alist] copies of them.

edited Feb 16 '22 at 18:37

answered Feb 16 '22 at 17:49

Henrik Schumacher

106,770
7
179
309

Thanks! Where I could have found documentations of Compile\GetElement? By the way I recompile the function many times because I use it in a brute force search where I map it on aPermutation@Range@nlist (So the function is listable and parallelizable)... Passing to the function this many time a constantdisplacementand a constantgraph` wouldn't be detrimental to speed? (Also I don't know how to make this general version listable) – Domenico Modica Feb 16 '22 at 18:20
Compiling an array is certainly more expensive than passing the array as an argument... – Henrik Schumacher Feb 16 '22 at 18:23
1

And no, Compile`GetElement is undocumented. You have to look around on this site to find more details about it. – Henrik Schumacher Feb 16 '22 at 18:28
1

You're right on your second point, I had give up on making a CompilationTarget -> "C" for each graph I had, because looking at the improvement it seemed it wasn't worth the extra time of compilation. Now this meta-function is clearly faster than one specific non- CompilationTarget -> "C" function and at keeps pace with a specific CompilationTarget -> "C" one, thanks – Domenico Modica Feb 16 '22 at 18:45
1

Have also a look at this: https://stackoverflow.com/questions/7918806/finding-n-th-permutation-without-computing-others Generating the permutations one at the time might save you shoving around a lot of memory. – Henrik Schumacher Feb 16 '22 at 18:48
Yes I had that question already saved, eventually I'll do that way and maybe write the program directly in C... But due to some graph symmetries, many times I can shrink my "searchspace" by some factorial factors. And, on a first thought, this would mean adding some branching on top of the extra code for of the i-th permutation. A problem that now I've avoided generating directly the shrunk list. Maybe for big $n$ I'll divide the space in batches, but for now it's a miracle if I solve $n=10$ in less than a day of computation! :D – Domenico Modica Feb 16 '22 at 19:24

Compile function dynamically

1 Answers1