The Problem
The reason you get the floating [h] is that your redefinition does not accept an argument. Normally, you'd specify an option argument using something like
\renewenvironment{table}[1][]
Now, in fact, this is not how the standard classes handle the optional argument in this case. article.cls defines the environment with no optional argument at all:
\newenvironment{table}
{\@float{table}}
{\end@float}
So why does it work? It works because of the way \@float is defined in latex.ltx:
\def\@float#1{%
\@ifnextchar[%
{\@xfloat{#1}}%
{\edef\reserved@a{\noexpand\@xfloat{#1}[\csname fps@#1\endcsname]}%
\reserved@a}}
\@float takes the {table} as the first, required, argument, and then scans ahead to see if the next token is a [ and, if so, it processes the contents of the [...] as an argument specifying the placement options for the float. This relies on the [ being the next thing in the input stream. If you change the definition to do something else first:
\renewenvironment{table}
{\@float{table} \fontfamily{phv}\selectfont}
{\end@float}
then the next token is not [, so LaTeX assumes there is no optional argument in this case and processes the rest as the contents of the environment.
If you want the optional argument just disposed of, you can throw it away
\renewenvironment{table}[1][]
However, if you want the optional argument to be processed as usual, then it must be passed to \@float{table} so it must come before the font specification.
A Solution
You could redefine the environment to handle the argument. However, I would recommend patching the environment rather than overwriting it. This requires etoolbox but the package is very widely used and many documents will load it anyway. I would also recommend not hard-coding the font family but using the document's default sans-serif family.
Here's one way to do it, though I'm not sure that this is the best approach.
\makeatletter
\newcommand*\my@starttable[1][]{%
\@float{table}[#1]\sffamily
}
\patchcmd{\table}{\@float{table}}{\my@starttable}{\PackageInfo{mysty}{Table environment patched successfully.}}{\PackageWarning{mysty}{Could not patch table environment.}}
\makeatother
mysty should be replaced with the name of your package, of course, when this goes into the .sty file.

Complete code:
\documentclass{article}
\usepackage{setspace}
\usepackage{booktabs}
\usepackage{etoolbox}
\usepackage[format=plain,
labelformat=simple,
font={small,sf,bf},
indention=0cm,
labelsep=period,
justification=centering,
singlelinecheck=true,
tableposition=top,
figureposition=bottom]{caption}
\makeatletter
\newcommand*\my@starttable[1][]{%
\@float{table}[#1]\sffamily
}
\patchcmd{\table}{\@float{table}}{\my@starttable}{\PackageInfo{mysty}{Table environment patched successfully.}}{\PackageWarning{mysty}{Could not patch table environment.}}
\makeatother
\begin{document}
Blah blah. Should be a serif font.
\begin{table}[h]
\caption{Some caption, which should be bold.}
\label{tab:xxx}\centering
\begin{tabular}{ccc}
test1 & test2 & test3 \\
\toprule
test1 & test2 & test3 \\
test1 & test2 & test3 \\
test1 & test2 & test3 \\
\bottomrule
\end{tabular}
\end{table}
Blah blah. Should be a serif font.
\end{document}