When BibTex sees a sequence like {\"O} or {\O}, for sorting purposes these special characters are reduced to O. The task is therefore to trick BibTex into thinking that the first example looks like Oe, while the second should look like ZZZZZO. For that, we introduce two macros, \donothing and \printsecond.
{\donothing{text}} will show nothing in the final bibliography, but "text" will be taken into account for sorting, so {\"O}{\donothing{e}} will print as "Ö", and sorted as "Oe".
{\printsecond{foo}{bar}} will print "bar", and be sorted as "foobar".
Example:
\documentclass{article}
\usepackage{filecontents}
\begin{filecontents}{SM.bib}
@misc{a, author={{\printsecond{ZZZZZ}{\O}}stersund, A}, title={a}, year=1000}
@misc{b, author={{\"O\donothing{e}}zkarag{\"o}z, {\.I}nci}, title={b}, year=1001}
@misc{c, author={Oezl, A}, title={c}, year=1002}
@misc{d, author={Oezj, A}, title={d}, year=1003}
@misc{e, author={Ofa, A}, title={e}, year=1004}
@misc{f, author={Zzz, A}, title={f}, year=1005}
\end{filecontents}
\begin{document}
\nocite{*}
\providecommand*{\donothing}[1]{}
\providecommand*{\printsecond}[2]{#2}
\bibliographystyle{plain} % works
% \bibliographystyle{alpha} % works ...
% \bibliographystyle{apalike} % works
\bibliography{SM}
\end{document}
Using the plain style, the sorting keys for these entries are: zzzzzostersund1000, oezkaragoz1001, oezl1002, oezj1003, ofa1004, zzz1005, which gives the correct order:

Using the alpha style, the entries are sorted by label, which consists of three letters of the author name, and the last two digits of the year for single author references. The sorting keys are therefore: zzzzzost1000, oezk1001, oez1002, oez1003, ofa1004, zzz1005. You can see here that the special characters are treated as one character by BibTeX, even when several plain characters are extracted when generating the sorting key. The output is therefore:

and indeed "Oez03" < "Özk01", as oez1003 < oezk1001, since 1 < k. If you would like it the other way round, the Özk01 has to be shortened to Öz01, which can be achieved in the database by changing the second entry to
@misc{b, author={{\"O}{\donothing{e}}zkarag{\"o}z, {\.I}nci}, title={b}, year=1001}
(Note how the "invisible" e is now a special character of its own, and counted as such, as it jumped out of the first special character.)

{\"O}zkarag{\"o}zshould be sorted (the Ö is followed by z)? Also, don't use braces around single authors, other than those needed for theauthorfield. To get{\O}last, tryauthor = {\noopsort{Ozzz}{\O}stersund, A.}. – mafp Feb 06 '13 at 17:00babel? If so, with which language, or languages? It would be helpful if you posted a full MWE (minimum working example) that generates the problem behavior you're trying to fix, rather than just a code snippet. – Mico Feb 06 '13 at 17:09