You could try using the recently-released newpxtext and newpxmath packages. In the MWE below, issuing the command \useosf after the math-related packages are loaded enables the use of regular- and bold-weight oldstyle numerals in text mode, while lining numerals will be used in math mode. Small caps are available in bold (as well as regular-weight, of course).

Update April 2018: The newpxtext and newpxmath packages have evolved quite a bit over the past five years. One can still enable oldstyle numerals and small-caps letters by running \useosf. According to the packages' user guide, though, the preferred way is to specify the option osf while loading the newpxtext package.
\documentclass{article}
\usepackage{newpxtext,newpxmath}
\useosf % old-style figures in text, not in math
\linespread{1.05} % Palatino needs a bit more leading than CM
\begin{document}
0123456789, {\bfseries 0123456789}, {\itshape 0123456789}, {\bfseries\itshape 0123456789}
\textsc{Once upon a time, \dots}
{\bfseries \scshape Once upon a time, \dots}
Some math expressions: $1+1=2$, $\int_0^1 \mathrm{d}x = 1$
\end{document}
newpx[text,math]packages should be performed automatically when you run the package update scripts. – Mico Apr 22 '13 at 17:01File 'newpxtext.sty' not found.– Herr K. Apr 22 '13 at 17:16newpx, right? But I couldn't, and the closest package name I found wasnewtx. However, installingnewtxdoesn't seem to change much. Also, I think there might be a typo in the Readme file of thenewpx[text,math]package, where it "confuses" betweennewpxandnewtxin its instruction of installation for MikTeX. – Herr K. Apr 22 '13 at 17:34newpxandnewtxare relatives (both good!), but not the same. For modes: If you back in the past once did a update in user mode you must also do this every time in future! – Speravir Apr 22 '13 at 17:38newpxtextandnewpxmathpackages -- having been released less than a month ago -- will fall into the new-and-not-yet-installed category. – Mico Apr 22 '13 at 17:47newpx, what then contains both mentioned sub-packages. – Speravir Apr 22 '13 at 18:13