This post, shrinking tables for presentations, gives a solution, using \small or \tiny after the \begin{table}. How do I make it effect for all tables, globally?
I also checked this post, What are all the possible first arguments to \setbeamerfont?, and there is no option like table for \setbeamerfont.