This can be quite easily be done with biblatex.
Just load biblatex with style=authoryear and firstinits=true (the latter is - you guessed it, to display only initials), and add the following to your preamble.
For some of the following patches you will need the awesome xpatch.
\DeclareNameAlias{sortname}{last-first}
\DeclareFieldFormat[incollection,article]{title}{\mkbibemph{#1}}
\DeclareFieldFormat[incollection,article]{citetitle}{\mkbibemph{#1}}
\renewbibmacro{in:}{%
\ifentrytype{article}{}{%
\printtext{\bibstring{in}\intitlepunct}}}
\DefineBibliographyStrings{german}{%
bibliography = {Literaturverzeichnis},
references = {Literaturverzeichnis},
}
\AtBeginBibliography{%
\renewcommand*{\mkbibnamelast}[1]{\textsc{#1}}%
\renewcommand*{\labelnamepunct}{\\}%
\renewbibmacro*{title}{%
\ifboolexpr{
test {\iffieldundef{title}}
and
test {\iffieldundef{subtitle}}
}
{}
{\printtext[title]{%
\printfield[titlecase]{title}%
\setunit{\subtitlepunct}%
\printfield[titlecase]{subtitle}}%
\newunit}%
\printfield{titleaddon}%
\printunit{\newunitpunct\\}}%
}
\newbibmacro*{ineditor+others}{%
\ifboolexpr{
test \ifuseeditor
and
not test {\ifnameundef{editor}}
}
{\printnames{editor}%
\setunit{\addspace}%
\usebibmacro{editor+othersstrg}%
\clearname{editor}}
{}}
\xpatchbibdriver{incollection}
{\usebibmacro{maintitle+booktitle}%
\newunit\newblock
\usebibmacro{byeditor+others}%
}
{\usebibmacro{ineditor+others}%
\setunit{\addcolon\space}\newblock
\usebibmacro{maintitle+booktitle}%
\newunit\newblock
\usebibmacro{byeditor+others}%
}%
{\typeout{replaced \usebibmacro{byeditor+others} by \usebibmacro{ineditor+others} in BibliographyDriver{incollection}}}
{\typeout{failed to replace \usebibmacro{byeditor+others} by \usebibmacro{ineditor+others} in BibliographyDriver{incollection}}}%
\xpatchbibdriver{inbook}
{\usebibmacro{bybookauthor}%
\setunit{\labelnamepunct}\newblock
\usebibmacro{maintitle+booktitle}%
\newunit\newblock
\usebibmacro{byeditor+others}%
}
{\usebibmacro{ineditor+others}%
\setunit{\addcolon\space}\newblock
\usebibmacro{maintitle+booktitle}%
\newunit\newblock
\usebibmacro{byeditor+others}%
}%
{\typeout{replaced \usebibmacro{byeditor+others} by \usebibmacro{ineditor+others} in BibliographyDriver{inbook}}}
{\typeout{failed to replace \usebibmacro{byeditor+others} by \usebibmacro{ineditor+others} in BibliographyDriver{inbook}}}%
\DeclareFieldFormat{editortype}{\mkbibparens{#1}}
\xpatchbibmacro{editor}%
{\setunit{\addcomma\space}%
}
{\setunit{\addspace}%
}
{\typeout{patched bibmacro{editor}}}
{\typeout{failed to patch bibmacro{editor}. Note that this is not necessarily bad if bibmacro{bbx:editor} was succesfully patched}}
% bibmacro{bbx:editor} is specially defined by authortitle and friends for some internal fun
\xpatchbibmacro{bbx:editor}%
{\setunit{\addcomma\space}%
}
{\setunit{\addspace}%
}
{\typeout{patched bibmacro{bbx:editor}}}
{\typeout{failed to patch bibmacro{bbx:editor}, note that this is not necessarily bad if bibmacro{editor} was succesfully patched}}
\renewrobustcmd*{\bibinitperiod}{}
\renewrobustcmd*{\bibinitdelim}{}
\renewrobustcmd*{\bibinithyphendelim}{}
\renewcommand*{\revsdnamepunct}{}
This makes sure names in the bibliography are "last, first" and there are no quotation marks for @incollection. Note though that this change is quite inconsistent, as normally the "main work" is emphasised (that would be the @collection) and the specific part (the @incollection) is in quotation marks.
Also the heading (for German documents) is changed to "Literaturverzeichnis".
The \AtBeginBibliography{...} portion makes sure that only in the bibliography names are small caps, there is a new line after the author and year and after the title as well.
We then define a new macro (this is necessary so we do not get any dashes) for the editor of "in ..."-works.
The bibliography drivers for @incollection and @inbook are patched to include the editor before the title.
Finally, we make sure the editor string ("eds."/"Hrsg." etc.) is printed in parentheses without any preceding comma.
The last chunk of code formats names and initials. There are no periods after the initial, neither are there spaces between two initials. Hyphenated names are treated like non-hyphenated ones and there is no comma between the last name and the initials of the first name
The MWE
\documentclass[ngerman]{scrartcl}
\usepackage[utf8]{inputenc}
\usepackage{babel}
\usepackage{csquotes}
\usepackage[backend=biber, style=authoryear, maxnames=999, maxcitenames=3, firstinits=true]{biblatex}
\usepackage{xpatch}
\usepackage{hyperref}
\addbibresource{biblatex-examples.bib}
\DeclareNameAlias{sortname}{last-first}
\DeclareFieldFormat[incollection,article]{title}{\mkbibemph{#1}}
\DeclareFieldFormat[incollection,article]{citetitle}{\mkbibemph{#1}}
\renewbibmacro{in:}{%
\ifentrytype{article}{}{%
\printtext{\bibstring{in}\intitlepunct}}}
\DefineBibliographyStrings{german}{%
bibliography = {Literaturverzeichnis},
references = {Literaturverzeichnis},
}
\AtBeginBibliography{%
\renewcommand*{\mkbibnamelast}[1]{\textsc{#1}}%
\renewcommand*{\labelnamepunct}{\\}%
\renewbibmacro*{title}{%
\ifboolexpr{
test {\iffieldundef{title}}
and
test {\iffieldundef{subtitle}}
}
{}
{\printtext[title]{%
\printfield[titlecase]{title}%
\setunit{\subtitlepunct}%
\printfield[titlecase]{subtitle}}%
\newunit}%
\printfield{titleaddon}%
\printunit{\newunitpunct\\}}%
}
\newbibmacro*{ineditor+others}{%
\ifboolexpr{
test \ifuseeditor
and
not test {\ifnameundef{editor}}
}
{\printnames{editor}%
\setunit{\addspace}%
\usebibmacro{editor+othersstrg}%
\clearname{editor}}
{}}
\xpatchbibdriver{incollection}
{\usebibmacro{maintitle+booktitle}%
\newunit\newblock
\usebibmacro{byeditor+others}%
}
{\usebibmacro{ineditor+others}%
\setunit{\addcolon\space}\newblock
\usebibmacro{maintitle+booktitle}%
\newunit\newblock
\usebibmacro{byeditor+others}%
}%
{\typeout{replaced \usebibmacro{byeditor+others} by \usebibmacro{ineditor+others} in BibliographyDriver{incollection}}}
{\typeout{failed to replace \usebibmacro{byeditor+others} by \usebibmacro{ineditor+others} in BibliographyDriver{incollection}}}%
\xpatchbibdriver{inbook}
{\usebibmacro{bybookauthor}%
\setunit{\labelnamepunct}\newblock
\usebibmacro{maintitle+booktitle}%
\newunit\newblock
\usebibmacro{byeditor+others}%
}
{\usebibmacro{ineditor+others}%
\setunit{\addcolon\space}\newblock
\usebibmacro{maintitle+booktitle}%
\newunit\newblock
\usebibmacro{byeditor+others}%
}%
{\typeout{replaced \usebibmacro{byeditor+others} by \usebibmacro{ineditor+others} in BibliographyDriver{inbook}}}
{\typeout{failed to replace \usebibmacro{byeditor+others} by \usebibmacro{ineditor+others} in BibliographyDriver{inbook}}}%
\DeclareFieldFormat{editortype}{\mkbibparens{#1}}
\xpatchbibmacro{editor}%
{\setunit{\addcomma\space}%
}
{\setunit{\addspace}%
}
{\typeout{patched bibmacro{editor}}}
{\typeout{failed to patch bibmacro{editor}. Note that this is not necessarily bad if bibmacro{bbx:editor} was succesfully patched}}
% bibmacro{bbx:editor} is specially defined by authortitle and friends for some internal fun
\xpatchbibmacro{bbx:editor}%
{\setunit{\addcomma\space}%
}
{\setunit{\addspace}%
}
{\typeout{patched bibmacro{bbx:editor}}}
{\typeout{failed to patch bibmacro{bbx:editor}, note that this is not necessarily bad if bibmacro{editor} was succesfully patched}}
\renewrobustcmd*{\bibinitperiod}{}
\renewrobustcmd*{\bibinitdelim}{}
\renewrobustcmd*{\bibinithyphendelim}{}
\renewcommand*{\revsdnamepunct}{}
\begin{document}
\nocite{murray,brandt,hyman,wilde,westfahl:space,aristotle:anima,herrmann,baez/article}
\printbibliography
\end{document}
gives

Feel free to comment, in case your requirements specify things differently.
natbibor are you OK withbiblatexas well? Unfortunately, the information you provide is quite limited. What about@articleentries,@online, etc. pp.? ISBN, URL ...? – moewe Sep 27 '13 at 14:42biblatex(it might be quite daunting to begin with, but is actually quite easy to use What to do to switch to biblatex? and biblatex for idiots). I'm still not sure what format for@articles you like, these normally do not havepublisherandedition, butjournal,number,issueetc. – moewe Sep 27 '13 at 15:04@books. That might end up looking quite inconsistent. – moewe Sep 27 '13 at 15:10@book) or ... – moewe Sep 27 '13 at 15:16BibTeX(andnatbib), help is close at hand: Run the makebst utility (vialatex makebstfrom the command line) and follow the detailed prompts. The output of the program will be a custom-made.bstfile that should meet all of your requirements. – Mico Sep 28 '13 at 10:44