From Merriam Webster:
the smallest quantity of infective material that regularly produces infection
(emphasis mine)
The word "minimum" here is not meant as a strict minimum it is meant as a statistical bound, the absolute probability for which depends on the types of tests done and their inherent accuracy/inaccuracy.
Reasons that this "minimum" is greater than 1 could include:
Probability that a given particle actually finds a suitable host cell, enters the cell, and replicates; failures can occur at many stages along that chain
Rate of immune system reaction/response. At low exposure levels, the immune system may detect and clear the infection at a rate faster than it can spread, such that the infection stays asymptomatic. Some vaccines use weakened or low doses of pathogens specifically to evoke an immune response. The intent is that the weakened pathogen will not spread fast enough to overcome the immune response and will not produce a full disease.
Clearance of the infectious agent before it reaches symptomatic levels. This probably applies more to bacteria but I don't see why it couldn't also apply to viruses. I think the best example is exposure to bacteria that cause gastrointestinal infections like E.coli or Salmonella. Small numbers may grow and multiply in the gut, but they grow slowly if they aren't present in a high enough concentration and they may pass through the gut before causing any noticeable symptoms.
The minimum infective dose can also vary from individual to individual or due to environmental factors (for example, see this paper).
Some additional references:
Leggett, H. C., Cornwallis, C. K., & West, S. A. (2012). Mechanisms of pathogenesis, infective dose and virulence in human parasites. PLoS pathogens, 8(2), e1002512.
Ward, R. L., Akin, E. W., & D'Alessio, D. J. (1984). Minimum infective dose of animal viruses. Critical Reviews in Environmental Control, 14(4), 297-310.
Yezli, S., & Otter, J. A. (2011). Minimum infective dose of the major human respiratory and enteric viruses transmitted through food and the environment. Food and Environmental Virology, 3(1), 1-30.