It's my first experience with OSL, so I can't say if this code has no bugs, but at first look it works as it should. I used source from Secrop's comment and adapted just as he advised.
surface Mask(
closure color Surface = 0,
output closure color Closure = 0
) {
float visible = 0.0;
float myMaterialIndex = 0.0;
float hitMaterialIndex = 0.0;
vector VectorTrace = vector(0.0, 0.0, 0.0);
int Hit = 0;
getattribute("material:index", myMaterialIndex);
if(backfacing()) {
VectorTrace -= I;
} else {
VectorTrace += I;
}
int DoTrace = trace(P, VectorTrace);
if(DoTrace) {
int HitTrace = getmessage("trace", "hit", Hit);
if(Hit != 0) {
HitTrace = getmessage("trace", "material:index", hitMaterialIndex);
}
visible = myMaterialIndex == hitMaterialIndex;
}
if(visible != 0) {
Closure = Surface;
} else {
Closure = transparent();
}
}
The setup consists of setting equal pass index for both objects materials and plugging script's node right before last node.


Be aware that any objects, placed behind one with according material index, will block view at your masked object. It sounds obvious, but not really, when it comes to transparent objects too, even those that doesn't contain typical shader output but only volume output.
Ye, it doesn't return boolean, as author wanted, but at least it
solves that tricky problem, when you forced to use multiple render
layers and compositor...)