Syntax highlighting for your own functions

Question

Mathematica has a useful feature that for functions and special constructs getting passed local variables (for example Minimize or Block), the affected variables are shown in a different colour. Now when writing your own functions of this type (like the NOptimize function found in this answer), Mathematica of course doesn't know the meaning of the arguments and doesn't change the colours accordingly.

Is there a way to tell Mathematica that an argument is a set of variables to affect other arguments?

To make this post self-contained, here a small (admittedly silly) example:

SetAttributes[SequenceVars,HoldAll]
SequenceVars[vars_List,expr_]:=Block[vars,vars=Range@Length@vars;expr]
(* add syntax highlighting somehow *)

SequenceVars[{a,b,c},foo[a,b,c,d]]
(* here a, b and c, but not d, should appear in a different colour while typing *)

Answer 1

What you want is SyntaxInformation. With this, you can use every highlighting which already exists for things like Table, Solve, etc. for your own functions. You can specify the pattern of the arguments. With this you get the typical red commas if you use too many parameters. Or you can highlight locally used variables inside the function-arguments:

SetAttributes[SequenceVars, HoldAll]
SyntaxInformation[SequenceVars] = {"ArgumentsPattern" -> {_, _}, 
   "LocalVariables" -> {"Solve", {1, 1}}};
SequenceVars[vars_List, expr_] := 
 Block[vars, vars = Range@Length@vars; expr]

It looks like this in the front-end:

The usage of the "LocalVariables" highlighting is as follows. First, you choose the general type of highlighting. For instance "Solve" provides simple x or lists like {a,b} while "Plot" provides highlighting for the first element in e.g. {a,1,2}.

You can use the following settings as templates

{"Table", "Solve", "Integrate", "Limit", "Plot", "Manipulate"}

Additionally, you need to specify at which places of your function call the local variable specifications can appear. Say you want a function where in the first argument is some expression and then can follow arbitrary many lists of local variables. The definition for that would look like

SyntaxInformation[f] = {"ArgumentPattern" -> {_, __},
  "LocalVariables" -> {"Solve", {2, Infinity}}}

This looks then like

and should make clear how "LocalVariables" has to be used.

Update

Since this function seems to be of unexpected interest, I should add one more important thing: SyntaxInformation can highlight wrong options. If you use OptionsPattern in your "ArgumentsPattern" option, all non-existent options will be highlighted in red

Options[f] = {a -> 1, b -> True};
f[x_] = Identity;
SyntaxInformation[f] = {"ArgumentsPattern" -> {_, OptionsPattern[]}};

Now using f with right/wrong options gives

Answer 2

This answer is focussed at more experienced users, to provide a way for them to find out more information. I do not discuss how anything works.

Information in this answer corresponds to version 10.3

Despite the fact that PrintDefinitions@SyntaxInformation gives nothing, we can still see how the function works, by doing

<< GeneralUtilites`
PrintDefinitions@System`Utilities`GetSystemSyntaxInformation

System`Utilities`GetSystemSyntaxInformation is pretty much equivalent to SyntaxInformation.

From the definition of System`Utilities`GetSystemSyntaxInformation, we see that the full list of "options of SyntaxInformation" is

(*sio is short for SyntaxInformation options*)
sio = {"ArgumentsPattern", "OptionNames", "LocalVariables", "ColorEqualSigns"}

Of these options "OptionNames" is undocumented (see this answer for an example of use, credit to jkuczm)

The following procedure gives a list of basic template names

informationFile = 
  ToFileName[{$InstallationDirectory, "SystemFiles", "Kernel", 
    "TextResources", $Language}, "FunctionInformation.m"];
(*sife is short for System Information File Expression*)
sife = If[FileType[informationFile] === File, 
   Get[informationFile], {}];
templatesWithNames=
Select[
sife[[1,2]]
,
Length@#>3&&#[[4]]=!= None&
][[All,{1,4}]];
DeleteDuplicates@templatesWithNames[[All,2,1]]

{"Manipulate", "Solve", "Plot", "Table", "D", "Integrate", "Limit",
 "SumSign", "IntegralSign"}

Of these, {"D", "IntegralSign", "SumSign"} do not appear in the docs.

I had forgotten about the "Lexical" modifier, but that is not undocumented, see the details section of the docs). Here are some examples of templates that involve it and also one that does not.

manipulateTemplates = 
 Select[templatesWithNames, #[[2, 1]] == "Manipulate" &]

{"Animate",{"Manipulate",{2,∞},"Lexical"}}
{"ControllerManipulate",{"Manipulate",{2,∞},"Lexical"}}
{"Manipulate",{"Manipulate",{2,∞},"Lexical"}}
{"RepeatingElement",{"Manipulate",{2}}}

Also this is weird

"ArgumentsPattern" /. SyntaxInformation@EmbedCode

{_, Optional["/Volumes/Jenkins/workspace/Documentation.Usage.English.release/scratch"], 
_., OptionsPattern[]}

Answer 3

I don't know if this is possible for the Input - style cells, but it is certainly possible for the Program-style cells with the syntax highlighter generator I exposed in a recent answer. The work in that direction is underway, but I don't have a complete package at the moment. In fact, apart from code highlighting for languages like C, Java, etc, a strong motivation for that project for me was to create an extensible syntax highlighter for Mathematica based on it. I hope to have more to say on this soon.

Note that, in general, an implementation of the syntax highlighting of the type you mentioned requires context / semantic analysis of the program - lexical analysis is not enough.