Let's look at why your attempt failed.
f[x_, y_ /; x < 10 y] := Mod[x, y]
Although the FrontEnd syntax highlighter is sometimes wrong it is helpful in this case:

Notice that the x on the left-hand side is not colored as a pattern name. Indeed this is why it does not work as you desire. Observe the Trace:
f[5, 10] // Trace

You can see that this attempts comparision to a literal x rather than the value 5. This can be a useful behavior for global conditions, e.g.:
x = 70;
f[5, 10] // Trace

As to why this happens: the values of patterns are only substituted on the right-hand side of operators such as =, :=, :> and pivotally /;. Therefore we want both patterns x_ and y_ on the LHS of /; so that they will be substituted in its RHS.
ClearAll[f]
f[x_, y_] /; x < 10 y := Mod[x, y]
f[5, 10] // Trace

Here is the TreeForm of the definition in case the parsing is not apparent:
TreeForm @ Unevaluated @ Unevaluated[
f[x_, y_] /; x < 10 y := Mod[x, y]
]

See these questions for further guidance on testing arguments:
Placement of Condition /; expressions
Using a PatternTest versus a Condition for pattern matching
f[x_, y_] /; x < 10 y := (* stuff *)– J. M.'s missing motivation Nov 09 '12 at 12:28ConditionandPatternTestversusCondition– Mr.Wizard Nov 10 '12 at 08:20