Why your function doesn't work
Your checkBoolValues does not hold any of its arguments, so when you pass to it, inside BSplineCurveInterpolate, for example op local symbol, it gets evaluated. Inside body of checkBoolValues you really use value of this symbol not the symbol itself.
This behavior was discussed many times on this site, in particular in Attempting to make an assignment to the argument of a function answer to What are the most common pitfalls awaiting new users? question.
That's why in first evaluation of:
BSplineCurveInterpolate[
{{1, 2}, {3, 4}},
OriginPoints -> aaa,
ControlPoints -> bbb,
SplineClosed -> True, TangentVector -> False]
in which you get desired messages and output, you also get undesired side effect. False values that, as I think, you intended to assign to local symbols op and cp, are really assigned to global aaa and bbb symbols that where passed as options.
That's why, in second usage of BSplineCurveInterpolate with aaa and bbb as option values, there are no errors, related to them, reported, because those symbols evaluate to False which is an acceptable value.
Overriding OptionValue
If you want to make sure that values of certain BSplineCurveInterpolate options are tested wherever they are accessed, you can override OptionValue by setting appropriate UpValues on BSplineCurveInterpolate.
Let's define a helper function that overrides OptionValue for given function and for options matching given pattern. Overriden OptionValue will extract value of given option, test it using given testFunc function, and if test returns False it'll return result of given defaultFunc called with option name and option value as arguments.
ClearAll[$useOverriddenOptionValue, setOptionValueTest]
$useOverriddenOptionValue = True;
SetAttributes[setOptionValueTest, HoldFirst]
setOptionValueTest[f_Symbol, optNamePatt_, testFunc_, defaultFunc_] := (
f /: OptionValue[f, opts_ | PatternSequence[], optNames_List] /;
$useOverriddenOptionValue :=
OptionValue[f, opts, #] & /@ optNames;
f /: ov : OptionValue[f, opts_ | PatternSequence[], optName : optNamePatt] /;
$useOverriddenOptionValue :=
If[TrueQ@testFunc[#], #, defaultFunc[optName, #]] &@
Block[{$useOverriddenOptionValue = False}, ov]
)
Now let's define options for BSplineCurveInterpolate, and set appropriate tests for their values.
ClearAll[BSplineCurveInterpolate]
Options[BSplineCurveInterpolate] = {
OriginPoints -> False, ControlPoints -> False,
SplineClosed -> False, TangentVector -> False
};
setOptionValueTest[BSplineCurveInterpolate,
OriginPoints | ControlPoints | TangentVector | SplineClosed,
MatchQ[True | False],
(Message[BSplineCurveInterpolate::opttf, #1, #2]; False) &
]
Now whole option value testing will be performed in ordinary call of OptionValue, so we just use it without any additional code (except when using fourth argument of OptionValue since then returned value can be arbitrary and defined test function might be inappropriate):
BSplineCurveInterpolate[pts : {{_, _} ..} | {{_, _, _} ..}, OptionsPattern[]] :=
Module[{n, op, cp, ss, tv, sc},
{op, cp, tv, sc} = OptionValue[
{OriginPoints, ControlPoints, TangentVector, SplineClosed}
];
(*return the values of options*)
{op, cp, tv, sc}
]
Now BSplineCurveInterpolate gives desired results:
Off[General::stop]
ClearAll[aaa, bbb, Trtttue]
BSplineCurveInterpolate[{{1, 2}, {3, 4}}, OriginPoints -> aaa,
ControlPoints -> bbb, SplineClosed -> True, TangentVector -> False]
BSplineCurveInterpolate[{{1, 2}, {3, 4}}, OriginPoints -> aaa,
ControlPoints -> bbb, SplineClosed -> Trtttue, TangentVector -> False]
BSplineCurveInterpolate[{{1, 2}, {3, 4}}, OriginPoints -> aaa,
ControlPoints -> bbb, SplineClosed -> Trtttue, TangentVector -> 222]
On[General::stop]

(Old answer) Refactoring checkBoolValues
Instead of playing with holding arguments and assigning False to local symbols inside checkBoolValues, I'd use a function that simply returns list of desired option values and prints messages when appropriate.
We start with simple helper function checking option value for True or False, returning value if it's correct. Printing message and returning False if value is incorrect. I'm using built-in General::opttf instead of a custom message.
ClearAll[checkBoolOption]
checkBoolOption[_][_, val : True | False] := val
checkBoolOption[func_][name_, val_] := (
Message[func::opttf, name, val];
False
)
Now a function that works like second and third documented usage of built-in OptionValue, but, in addition to returning option values, it checks whether they are booleans.
ClearAll[boolOptionValue]
boolOptionValue[func_, opts:{OptionsPattern[]}:{}, optName : _Symbol | _String] :=
checkBoolOption[func][optName, OptionValue[func, opts, optName]]
boolOptionValue[func_, opts:{OptionsPattern[]}:{}, optNames:{(_Symbol | _String)...}] :=
MapThread[
checkBoolOption[func],
{optNames, OptionValue[func, opts, optNames]}
]
Example usage in BSplineCurveInterpolate:
ClearAll[BSplineCurveInterpolate]
Options[BSplineCurveInterpolate] = {
OriginPoints -> False, ControlPoints -> False,
SplineClosed -> False, TangentVector -> False
};
BSplineCurveInterpolate[pts : {{_, _} ..} | {{_, _, _} ..}, opts:OptionsPattern[]] :=
Module[{n, op, cp, ss, tv, sc},
(*check and assign values of options*)
{op, cp, tv, sc} =
boolOptionValue[BSplineCurveInterpolate, {opts},
{OriginPoints, ControlPoints, TangentVector, SplineClosed}
];
(*return the values of options*)
{op, cp, tv, sc}
]
Now BSplineCurveInterpolate gives desired results:
Off[General::stop]
ClearAll[aaa, bbb, Trtttue]
BSplineCurveInterpolate[{{1, 2}, {3, 4}}, OriginPoints -> aaa,
ControlPoints -> bbb, SplineClosed -> True, TangentVector -> False]
BSplineCurveInterpolate[{{1, 2}, {3, 4}}, OriginPoints -> aaa,
ControlPoints -> bbb, SplineClosed -> Trtttue, TangentVector -> False]
BSplineCurveInterpolate[{{1, 2}, {3, 4}}, OriginPoints -> aaa,
ControlPoints -> bbb, SplineClosed -> Trtttue, TangentVector -> 222]
On[General::stop]

StringForm::sfrmessages since you only pass one message argument toBSplineCurveInterpolate::invboolwhile it expects 2. AlsoFalsevalues are really assigned toaaaandbbbsymbols and not localopandcpsymbols. You've surely seen Attempting to make an assignment to the argument of a function? It applies to yourcheckBoolValuesand to pure function used inside it inScan. – jkuczm Nov 29 '15 at 09:02General::opttf. – jkuczm Nov 29 '15 at 09:12Message[]will look for the defintion offunc::invbolrather thanBSplineCurveInterpolate::invbol? – xyz Nov 29 '15 at 09:26