Can I use Postfix with a condition

Question

I have a nested list Data set with mostly numbers, there are a few elements that occur as strings that I want to convert to a number. By postfixing ToExpression I get the desired result, however it also evaluates strings that I use as blanks, they are always the same string, can I exclude them from the ToExpression?

So I have data like this but much larger

Dataset1 = {{1,2,3,4} 3,"4",5}, {1,2,3,4} 3,"#N/A",5}}

Dataset1 = Dataset1//ToExpression

So this will turn "4" into 4 but it will also make "#N/A" into #N÷A

I've tried postfixing this, but that doesn't work:

ConvertStringNumbers[x_] := If[ x != "#N/A", ToExpression[x], x]

Dataset1 = Dataset1//ConvertStringNumbers