7

Major deviation from my original post

I realize that the long code I originally had was very discouraging to debug and did not attract any answers. So I have decided to revamp this question significantly (I hope this is acceptable). Please visit the edit history for older versions of this question.

Game Plan

I hypothesize that there are multiple errors in my script the is causing the kernel to crash. But since I can not identify them all at this time, I plan to identify one problem at a time, get solution, implement it in my code and repeat the process of identifying the next problem and posting it on this page, till all the issues are resolved.

Problem 1. The following code compiles fine: 

a=Compile[{{p1,_Real,0},{p2,_Real,0}},Max[0.,Min[p1,10.]-p2],RuntimeAttributes->{Listable}];

b=Compile[{{p1,_Real,0},{p2,_Real,0},{p3,_Real,0},{p4,_Real,0}},
  Max[0.,Min[(20.-p1) 5.-p2-p3,10.]-p4],RuntimeAttributes->{Listable}];

ab=Compile[{{p1,_Real,0},{p2,_Real,0},{p3,_Real,0},{p4,_Real,0},{p5,_Real,0}},
   Min[a[p1,p5],b[p2,p3,p4,p5]],RuntimeAttributes->{Listable},
   CompilationOptions->{"InlineExternalDefinitions"->True}];

f=Compile[{},Module[{k0=Table[0.,{10}]}, ab[k0,k0,k0,k0,k0]],CompilationTarget->"C",
  CompilationOptions->{"InlineExternalDefinitions"->True}];

But when I execute f[], the kernel crashes. I am guessing, it is to do with mixing Scalars with Vectors in the a and b functions. But that is what I thought Listable takes care of. Any thoughts?

brama
  • 898
  • 5
  • 18
  • 2
    As soon as you revert to MainEvaluate compilation is of dubious use in any case, correct? Perhaps useful: Compiling more functions that don't call MainEvaluate./q/24595/131 – Yves Klett Jun 11 '14 at 19:10
  • 2
    Here are two suggestions. (1) Add CompilationOptions -> {"InlineExternalDefinitions" -> True} (2) Define f2[x1_Integer, x2_Integer, x3_Integer] := f[x1, x2, x3] and use f2 in the call to NMinimize. Not sure if these will help but they should at least keep the compiled code from calling on the main evaluator. – Daniel Lichtblau Jun 11 '14 at 19:16
  • @DanielLichtblau, Please look at edit 1. I actually implemented your suggestion 1. Implementing suggestion 2 takes it longer, even though same number of iterations are taken to converge. – brama Jun 11 '14 at 19:38
  • Interesting, the link provided by Yves Klett has a rogue full-stop. http://mathematica.stackexchange.com/questions/24595/compiling-more-functions-that-dont-call-mainevaluate – dr.blochwave Jun 11 '14 at 21:01
  • I get 20 seconds with my two suggestions, and it's still running after 2-3 minutes if I omit the second one. – Daniel Lichtblau Jun 11 '14 at 21:31
  • You state: "When I include CompilationTarget->"C" it does not give any errors, but NMinimize does not work." What actually happened? – Daniel Lichtblau Jun 12 '14 at 14:34
  • @DanielLichtblau Sorry for the delayed response. It is interesting how we both are incurring different results. Implementing your f2[x1_Integer, x2_Integer, x3_Integer] := f[x1, x2, x3] with f2 in NMinimize makes it slower on my i5-3210 2.5GHz processor with 6 GB ram computer. without it, and only the code I shown above, it runs quicker. I am confused. With CompilationTarget->"C", the program executes, but does not produce any output (even though I have Print in my EvaluationMonitor). You can try it too!! – brama Jun 12 '14 at 15:16
  • What I was asking was whether you are sure it executes to completion. In my Linux version it seemingly (and quickly) "does nothing". But in actual fact it crashes the kernel. – Daniel Lichtblau Jun 12 '14 at 15:33
  • Yes. The Compile executes to completion. I use print outside compile and it prints it. However, it does not execute NMinimize properly and terminates without iterating even once. The program does not crash on my windows machine. – brama Jun 12 '14 at 15:40
  • Also, a quick question: I have 38 instances of CopyTensor in the above code. Does it mean it holds memory till I close MMA and garbage collection does not kick in? Could this be the reason why the physical memory keeps increasing continuously as I run my program? – brama Jun 12 '14 at 15:52
  • You are doing a considerable amount of array creation, in a While loop, so yes, that could account for the memory issues. – Daniel Lichtblau Jun 12 '14 at 17:25
  • @MichaelE2. Did you use CompilationTarget->“C”? Without it, it works, but when I use it in the compile, the kernel is crashing with NMinimize. See @Dan's comments above. He also observed same behavior. – brama Jun 19 '14 at 01:31
  • @MichaelE2. That is too weird...I can not get it to work with CompilationTarget -> "C". Just wanted to make sure: Did you use the above code "as is" or made any improvements? @Dan can you please check if this works for you? – brama Jun 19 '14 at 04:27
  • @MichaelE2. Surprising!! Since we use same version of MMA. I am not sure if it is to do with my Windows 8 and your Mac. Let's see if someone else can validate our results. – brama Jun 19 '14 at 05:07
  • Same problem with TDM-GCC-4.8.1, Mathematica 8.0.4, Vista 32bit. Well, in fact it's not the first time I found CompilationTarget -> "C" made trouble, but I haven't yet managed to reproduce this kind of problem with a simple enough example for so long. – xzczd Jul 03 '14 at 08:23
  • @xzczd I have an even simpler case above. check it out. – brama Jul 10 '14 at 08:31

1 Answers1

8

Here's a workaround. Take it as evidence that compilation to C and Listable do not always get along.

a = Compile[{{p1, _Real, 0}, {p2, _Real, 0}}, 
   Max[0., Min[p1, 10.] - p2]
   (*,RuntimeAttributes -> Listable}*)];

b = Compile[{{p1, _Real, 0}, {p2, _Real, 0}, {p3, _Real, 0}, {p4, _Real, 0}}, 
   Max[0., Min[(20. - p1) 5. - p2 - p3, 10.] - p4]
   (*, RuntimeAttributes -> {Listable}*)];

ab = Compile[{{p1, _Real, 0}, {p2, _Real, 0}, {p3, _Real, 0},
              {p4, _Real, 0}, {p5, _Real, 0}}, 
      Min[a[p1, p5], b[p2, p3, p4, p5]],
      RuntimeAttributes -> {Listable},
      CompilationOptions -> {
        "InlineExternalDefinitions" -> True,
        "InlineCompiledFunctions" -> True (* necessary *)}
     ];

f = Compile[{}, 
   Module[{k0 = Table[0., {10}]}, ab[k0, k0, k0, k0, k0]], 
   CompilationTarget -> "C", 
   CompilationOptions -> {"InlineExternalDefinitions" -> True}];

The Listable attribute for a and b are not needed (in this simplified example) since ab is Listable. If you get rid of the attribute, the compiled functions can be inlined. If you don't set "InlineCompiledFunctions" -> True, the kernel crashes on me. You might want to look at

Needs["CompiledFunctionTools`"];
CompilePrint[f]

I take it that the CompiledFunctionCall in f actually is a call-back to the Virtual Machine (WVM née/né MVM), and not compiled to C. It may be that an issue in the communication between the C program and the VM leads to the crash. I do not know the answer to why the crash occurs.

Michael E2
  • 235,386
  • 17
  • 334
  • 747
  • This is insane. when I implemented your suggestion to remove Listable and inlined the upper layer functions in my code, the whole problem is resolved. – brama Jul 10 '14 at 19:34
  • However, it will be interesting to get an insight into why the kernel crashed with the original formulation. – brama Jul 11 '14 at 08:29
  • @brama Yes, it would. It might take someone with knowledge of how the internals work. – Michael E2 Jul 11 '14 at 12:45