This states truecrypt uses a hash-function like SHA-1 and keystrengthen it by a factor of ~2000 times. So if your attacker knew it was a four digit number; it would take your attacker under 10000 tries to crack, which means they would need about 2000×10000=20 million sha1 hashes. A GPU can generate sha1 hashes at about a billion per second; so your solution would not survive the most basic password cracking attempt.
If security of your data is a requirement, I'd recommend that instead of a four-digit number (13 bits of entropy: 213.3 ≅ 10000); using a four-word (or longer) random diceware passphrase (~52 bits: 252 ≅ (65)4 ≅ 4×1015). Instead of surviving for ~1/50th of a second, your passphrase would survive for about ~200 years of a dedicated GPU attack. This is in the realm of breakable in months by very large organizations that are very interested in getting your data (think NSA with a server farm of thousands of GPUs dedicated to cracking your encryption), but protected against all lesser brute-force threats (and adding a couple more words to the passphrase would protect against very large organizations). (EDIT: There are other methods that could be used; e.g., covert keyloggers to steal your passphrase).
As the preface to Applied Cryptography puts it:
There are two kinds of cryptography in this world: cryptography that will stop your kid sister from reading your files, and cryptography that will stop major governments from reading your files.
Your current solution prevents against the "kid sister" attacks of the world, but would not be much more difficult to upgrade to the "major government" kind if desired.