I not sure about what without manually calculating the coordinates of R means.
May be something like
\node[outer sep=0pt,circle, draw,inner sep=1.5pt, label={[fill=white]right:$R$}] (R)
at ($(P)!1cm*sqrt(5)!-90:(Q)$){};
In any case, TikZ draws everything in the same order as code is written. Therefore, as you draw
\draw[green!20!white] (Q) -- (P) -- (R) -- (Q) -- cycle;
after
\coordinate (R) at ($(P)!1cm*sqrt(5)!-90:(Q)$);
\node[outer sep=0pt,circle, fill,inner sep=1.5pt, label={[fill=white]right:$R$}] at(R) {};
the triangle is drawn over R. But why over R and not over P and Q? Because R, as you can see from previous lines, is a not drawn coordinate (you draw the dot with un unnamed node over R) while P and Q has been defined as nodes with a certain size. And
\draw[green!20!white] (Q) -- (P) -- (R) -- (Q) -- cycle;
draws lines between node borders, borders which don't only exist in case of R because it's a point.
See what happens with following code. Nodes P and Q are only drawn and remember that circle around R is an unnamed node which is not referenced later on.
\documentclass[10pt]{amsart}
\usepackage{tikz}
\usetikzlibrary{calc}
\begin{document}
\begin{tikzpicture}
\node[outer sep=0pt,circle, draw ,inner sep=1.5pt,label={[fill=white]left:$P$}] (P) at (-1,-1) {};
\node[outer sep=0pt,circle, draw,inner sep=1.5pt, label={[fill=white]right:$Q$}] (Q) at (2,1) {};
\coordinate (R) at ($(P)!1cm*sqrt(5)!-90:(Q)$);
\node[outer sep=0pt,circle, draw,inner sep=1.5pt, label={[fill=white]right:$R$}] at(R) {};
\draw[red, fill=green] (Q) -- (P) -- (R) --(Q)-- cycle;
\end{tikzpicture}
\end{document}

While
\documentclass[10pt]{amsart}
\usepackage{tikz}
\usetikzlibrary{calc}
\begin{document}
\begin{tikzpicture}
\node[outer sep=0pt,circle, draw ,inner sep=1.5pt,label={[fill=white]left:$P$}] (P) at (-1,-1) {};
\node[outer sep=0pt,circle, draw,inner sep=1.5pt, label={[fill=white]right:$Q$}] (Q) at (2,1) {};
\draw[green!20!white] (P) -- (Q);
%\coordinate (R) at ($(P)!1cm*sqrt(5)!-90:(Q)$);
\node[outer sep=0pt,circle, draw,inner sep=1.5pt, label={[fill=white]right:$R$}] (R) at ($(P)!1cm*sqrt(5)!-90:(Q)$){};
\draw[red, fill=green] (Q) -- (P) -- (R) --(Q)-- cycle;
\end{tikzpicture}
\end{document}
produces

As you can see from previous figure, in this case is not possible to close an area to be filled because R has a real size and the path is disjoint (is it correct?).
How to solve both problems: filling the triangle and doing it below nodes?
With backgrounds tikzlibrary and closing the path.
Following code shows a possible solution. Grid and axes are drawn on the background layer which is declared by backgrounds library. And the white triangle is also drawn on background layer, but over grid because everything is drawn in order.
\begin{scope}[on background layer]
...
\end{scope}
But triangle vertex and corner are drawn on foreground layer.
The triangle can be filled referencing nodes centers instead of only their names
\draw[green!20!white, fill=white] (Q.center) -- (P.center) -- (R.center) -- cycle;
The complete code:
\documentclass[tikz,10pt,border=2mm]{standalone}
\usepackage{tikz}
\usetikzlibrary{calc,backgrounds}
\begin{document}
\begin{tikzpicture}[dot/.style={circle, fill, inner sep=1.5pt, outer sep=0pt}]
\begin{scope}[on background layer]
\draw[yellow, line width=0.1pt] (-1.75,-3.25) grid[xstep=0.5, ystep=0.5] (2.75,1.75);
\draw[draw=gray!30,latex-latex] (0,1.75) +(0,0.25cm) node[above right] {$y$} -- (0,-3.25) -- +(0,-0.25cm);
\draw[draw=gray!30,latex-latex] (-1.75,0) +(-0.25cm,0) -- (2.75,0) -- +(0.25cm,0) node[below right] {$x$};
\end{scope}
\coordinate[dot, label={[fill=white]left:$P$}] (P) at (-1,-1) {};
\coordinate[dot, label={[fill=white]right:$Q$}] (Q) at (2,1) {};
\coordinate[dot, label={[fill=white]below right:$R$}] (R) at ($(P)!1cm*sqrt(5)!-90:(Q)$) {};
%
%%\draw[green!20!white] (P) -- (Q);
%
%%\coordinate (R) at ($(P)!1cm*sqrt(5)!-90:(Q)$);
%
\begin{scope}[on background layer]
\draw[green!20!white, fill=white] (Q.center) -- (P.center) -- (R.center) -- cycle;
\end{scope}
%
\coordinate (a) at ($ (P)!5mm! -45:(Q) $);
\draw[green!20!white] ($(P)!(a)!(Q)$)--(a)--($(P)!(a)!(R)$);
\end{tikzpicture}
\end{document}

fill=whitedeclaration? (I will also refer to section 90 of the manual.) – Adelyn Apr 23 '15 at 15:56fill=whiteoption. – Alenanno Apr 23 '15 at 16:00fill, but sincefill=whiteis specified later, you can safely delete the former. :) I have fixed my answer accordingly. – Alenanno Apr 23 '15 at 16:17.style={}. Where can I find a description of the syntax of this command? What does the declarationminimum size=4ptspecify for drawing the vertices of the triangle? – user74973 Apr 23 '15 at 16:20minimum size=4ptis the size of the black circles. If you write10ptor5cm, they'll get bigger and so on. I can't find anything specific about Tikzset, except that I know it used to be Tikzstyle (which is still supported but I don't think it's used anymore). You can see this question that mentions a bit of history. But the syntax is quite easy to edit for other uses. – Alenanno Apr 23 '15 at 16:26