Executing local functions remotely

Question

The following

Attributes[remoteExecute] = {HoldAllComplete};
remoteExecute[expr_] := 
 Module[{compressed, 
   ubuntuBox = RemoteConnect[ip, "username", "password"]}, 
  compressed = Compress[Unevaluated[expr]]; 
  ToExpression[
   RemoteRunProcess[ubuntuBox, 
    TemplateApply[
     "wolframscript -format InputForm -code 'Uncompress[\"``\"]'", 
     compressed], "StandardOutput"]]]

is able to execute functions remotely, so long as your function isn't locally defined. So for example if ~/only_on_ubuntu_box is a file that exists only on your remote machine, this function will fail for:

How can one get around this limitation?

The last statement should return True, since the file is on the remote machine and the local function checks whether a file exists. — George, Aug 24 '19 at 14:49
Sorry, I misread. Have you considered setting up a remote kernel instead? Those functions automatically find all the local functions needed to execute the function that you want to execute on the remote kernel and send those definitions along. — C. E., Aug 24 '19 at 14:55
Can the remote kernel be set up with wolframscript? I don't have a license on the remote machine. — George, Aug 24 '19 at 14:57
If you have wolframscript then I assume that you have Wolfram Engine, and I think that Wolfram Engine can be used as a remote kernel even though I haven't tried it. — C. E., Aug 24 '19 at 14:59
It's not very clear from Wolfram documentation (which is unusual; Wolfram documentation is usually excellent) how to do set up a remote kernel over SSH (both machines running Linux) and then forcing commands to execute on the remote kernel. — George, Aug 24 '19 at 15:37
I don't have any experience with this, unfortunately, but this answer and this package seem related. — C. E., Aug 24 '19 at 17:07
Try something like FileExistsQ[File@ExpandFileName["file"]] since you are remotely executing a command try expanding "~" for whatever "/host/user/full-path-to-file"; it seems that FileExistQ does not read all $LocalVariables of the remote host. Also something like TableForm[ ReadList["!ls -la ", String], TableSpacing -> {4, 4}, TableAlignments -> Left, TableHeadings -> {Automatic, None}] // TraditionalForm could work. This could also work remoteFileExistQ[file_String] := ExpandFileName[First@ReadList["!ls -a -1 | grep " ~~ file, String]] in a non-linux remote host using dir. — Schopenhauer, Sep 05 '19 at 18:56
Why not just write your helper functions into a package, put that on the remote, and load that...? Seems a lot easier than managing all this state yourself. — b3m2a1, Sep 05 '19 at 23:54

score 6 · Accepted Answer · answered Sep 07 '19 at 18:52

If I understand correctly, you have some functions defined locally that are not defined on the remote machine. When you remote evaluate your expression, which includes these functions, the remote evaluation doesn't work properly. So, you want your remote evaluation function to transfer all of the necessary local functions to the remote machine, and then evaluate your expression.

The good news is that this is exactly what CloudEvaluate does, although the remote machine in this case is the WolframCloud and not your remote machine. However, the machinery created to enable CloudEvaluate to work is already present in Wolfram Language. The function that does the heavy lifting for CloudEvaluate is Language`ExtendedFullDefinition. A very simple version using this function would be:

SetAttributes[compressWithDefinitions, HoldFirst];
compressWithDefinitions[expr_] := With[
    {def = Language`ExtendedFullDefinition[expr]},

    Compress @ Unevaluated[
        Language`ExtendedFullDefinition[] = def;
        expr
    ]
]

The compressed string created by this function first loads all necessary definitions needed to evaluate expr, and then evaluates expr.

I will simulate your remote usage by using LocalSubmit, which sends an evaluation to a separate kernel. Here is a function only defined in the local kernel:

g[x_] := x^2

Here are two compress strings that return the downvalues of g when uncompressed:

str1 = Compress @ Unevaluated[DownValues[g]];
str2 = compressWithDefinitions @ DownValues[g];

Now, if we use LocalSubmit with str1 we find that, as expected, the remote kernel has no downvalues for g:

Clear[dv];
With[{s = str1},
    LocalSubmit[
        Uncompress[s],
        HandlerFunctions-><|"TaskFinished"->((dv=#EvaluationResult)&)|>,
        HandlerFunctionsKeys->"EvaluationResult"
    ]
];
Pause[1];
dv

{}

On the other hand, when using str2, the local kernel does have the transferred downvalues for g:

Clear[dv];
With[{s = str2},
    LocalSubmit[
        Uncompress[s],
        HandlerFunctions-><|"TaskFinished"->((dv=#EvaluationResult)&)|>,
        HandlerFunctionsKeys->"EvaluationResult"
    ]
];
Pause[1];
dv

{HoldPattern[g[x_]] :> x^2}

So, I think you should be able to use compressWithDefinitions to achieve your goals.

Note that there are some additional subtleties that can arise if you try to use package functions, where you need to make sure that the compressed string includes contexts when necessary.

This worked on first try. Can't thank you enough (being able to use Wolfram on my remote machines from a Wolfram Desktop instance is an amazing superpower for stitching utility scripts together). — George, Sep 10 '19 at 15:49

Executing local functions remotely

1 Answers1

Linked