Deleting the app bundle (Mathematica.app) is sufficient to remove all components of Mathematica itself, including the Spotlight indexer and the QuickLook plugin. You can check that both of these are located in Mathematica.app/Contents/Library.
To remove WolframScript, which is installed separately from Mathematica, remove WolframScript.app as well as /usr/local/bin/wolframscript (which is just a symlink to an executable to within WolframScript.app). Note that WolframScript does not come with any Spotlight indexers.
To remove not only software components, but also settings and caches, as well as Mathematica packages, look at this support article. Doing this is not necessary if all you want is to remove the Spotlight indexer. To make the answer complete, the locations to remove are ~/Library/Mathematica/, ~/Library/Wolfram/ and ~/Library/Caches/Wolfram/. These may be different for Wolfram One, which I am not familiar with.