The examples in the documentation work fine, but your example is wrong. If you wish to use special characters within the tree, you need to use the low-level commands instead of the parsetree environment - not in addition to it.
Here's what the documentation says:
The parsetree environment is simply a wrapper that: (a) makes these special characters 'active' (i.e. special), and (b) calls the commands \ptbegtree \ptendtree at the beginning and end of the environment respectively. If we use these latter commands directly, then these special characters retain their normal meanings, and can appear in node lables [sic.]. However, we must now use the underlying parsetree commands to draw the tree ...
The point is that, within the parsetree environment, the category codes are changed. So the special characters are short-hands for the lower-level commands.
If you need to use those characters in the nodes of the tree, then you need the category codes to be their usual selves.
One solution would be to create a mechanism to temporarily change the codes back. However, that is not the solution supported by the package.
What the package supports instead is direct use of the lower-level commands, so that the category codes remain their usual selves. Obviously, this means you cannot use the special characters as short-hands. But you can use them within the nodes.
\documentclass{article}
\usepackage{parsetree}
\begin{document}
\ptbegtree
\ptbeg\ptnode{ROOT}
\ptbeg\ptleaf{((parentheses))}
\ptend
\ptend
\ptendtree
\ptbegtree
\ptbeg \ptnode{(VP)}
\ptbeg \ptnode{(V)} \ptleaf{(`saw)} \ptend
\ptbeg \ptnode{NP} \ptleaf{Sam's~~~toy.} \ptend
\ptend
\ptendtree
\end{document}
Alternative Solutions
A modern package such as forest or qtree or tikz-qtree would definitely make life easier.
Here are four examples from above and/or the documentation for parsetree redone using forest. This makes it possible to use common styling and to specify the trees very concisely using bracket notation.
For a brief introduction to forest and an explanation of the brackets syntax, see the second part of my answer to an earlier question.
\documentclass[border=10pt,tikz,multi]{standalone}
\usepackage{forest,array}
Load array as we want it for trees whose nodes are multiline maths.
Define two standard styles: my forest for primarily-text-in-nodes trees and my maths forest for primarily-maths-in-nodes trees. You can always switch within a node, of course, but this saves repeating the same switches for every node as a routine.
\forestset{
my forest/.style={
for tree={
parent anchor=south,
child anchor=north,
Branches start from a common point below the parent and end top centre of the child.
align=center
Align node contents centrally.
}
},
my maths forest/.style={
for tree={
parent anchor=south,
child anchor=north,
align={@{}>{$}c<{$}@{}}
As before, but the content of the nodes is in maths mode by default.
}
}
}
\begin{document}
From the question:
\begin{forest}
my forest
[ROOT
[\itshape(parentheses)]
]
\end{forest}
Here's an example with a triangular roof:
\begin{forest}
my forest,
where n children=0{font=\itshape}{}
All terminal nodes should be italicised.
[S
[NP
[we]
]
[VP
[V
[gave]
]
[NP
[them]
]
[NP
[a toy, triangle]
]
]
]
\end{forest}
The example involving the special characters:
\begin{forest}
my forest
[(VP)
[(V)
[\textit{(`saw)}]
]
[NP
[\textit{Sam's toy.}]
]
]
\end{forest}
The more complex example involving special characters from the documentation:
\begin{forest}
my maths forest
[{S\\see' (s, k)\\\lambda y see' ( y,k ) (s)}
[NP\\s
[Sam\\s
]
]
[{VP\\\lambda y see' ( y,k )\\\lambda x \lambda y see' ( y,x ) (k)}
[{V\\\lambda x \lambda y see' ( y,x ) }
[{saw\\\lambda x \lambda y see ( y,x )}
]
]
[NP\\k
[Kim\\k
]
]
]
]
\end{forest}
\end{document}
Complete code:
\documentclass[border=10pt,tikz,multi]{standalone}
\usepackage{forest,array}
\forestset{
my forest/.style={
for tree={
parent anchor=south,
child anchor=north,
align=center
}
},
my maths forest/.style={
for tree={
parent anchor=south,
child anchor=north,
align={@{}>{$}c<{$}@{}}
}
}
}
\begin{document}
\begin{forest}
my forest
[ROOT
[\itshape(parentheses)]
]
\end{forest}
\begin{forest}
my forest,
where n children=0{font=\itshape}{}
[S
[NP
[we]
]
[VP
[V
[gave]
]
[NP
[them]
]
[NP
[a toy, triangle]
]
]
]
\end{forest}
\begin{forest}
my forest
[(VP)
[(V)
[\textit{(`saw)}]
]
[NP
[\textit{Sam's toy.}]
]
]
\end{forest}
\begin{forest}
my maths forest
[{S\\see' (s, k)\\\lambda y see' ( y,k ) (s)}
[NP\\s
[Sam\\s
]
]
[{VP\\\lambda y see' ( y,k )\\\lambda x \lambda y see' ( y,x ) (k)}
[{V\\\lambda x \lambda y see' ( y,x ) }
[{saw\\\lambda x \lambda y see ( y,x )}
]
]
[NP\\k
[Kim\\k
]
]
]
]
\end{forest}
\end{document}
