As a LaTeX3 solution is acceptable, I'd use xparse. The internals of xparse can deal correctly with nested optional argument brackets or similar, and so \defpoint(sin(45);cos(45)){A} is not an issue.
You don't say if \defpoint needs to be expandable. Assuming that it does not, a solution which meets the criteria is
\documentclass{article}
\usepackage{xparse}
\ExplSyntaxOn
\NewDocumentCommand \defpoint
{ > { \SplitArgument { 2 } { ; } } D ( ) { 0 ; 0 } m }
{ \defpoint_aux:nnnn #1 {#2} }
\cs_new_protected:Npn \defpoint_aux:nnnn #1#2#3#4
{
\IfNoValueTF {#3}
{ 2D~co-ordinates~-~(#1;#2),~mandatory~argument~-~'#4' \\ }
{ 3D~co-ordinates~-~(#1;#2;#3),~mandatory~argument~-~'#4' \\ }
}
\ExplSyntaxOff
\begin{document}
\noindent
\defpoint(1;2){A}
\defpoint(12;13){B}
\defpoint(sin(45);cos(45)){A}
\defpoint(1;2;-5){A}
\end{document}
I need an internal function (\defpoint_aux:nnnn) here to allow me to deal with the variable number of co-ordinates. What happens is that \SplitArgument will divide up the first argument at a maximum of two ; tokens, and will always produce three <balanced text>. These are picked up by the auxiliary function as #1, #2 and #3, and so we can test for 2D versus 3D by seeing if the third argument is the special \NoValue marker.
I've not covered it above, but you could also test for whether the argument in parentheses is given at all and if it contains only one argument (i.e. no ;), again using \IfNoValueTF tests.
For an expandable approach, you need a little more work, and an up-to-date copy of xparse (nested optional arguments did not work expandability until I examined it for this question).
\documentclass{article}
\usepackage{xparse}
\ExplSyntaxOn
\DeclareExpandableDocumentCommand \defpoint
{ D ( ) { 0 ; 0 } m }
{ \defpoint_aux:nn {#1} {#2} }
\cs_new:Npn \defpoint_aux:nn #1#2
{ \defpoint_aux:nw {#2} #1 ; \q_nil ; \q_stop }
\cs_new:Npn \defpoint_aux:nw #1#2 ; #3 ; #4 ; #5 \q_stop
{
\quark_if_nil:nTF {#4}
{ \defpoint_aux:nnnn {#2} {#3} { \NoValue } {#1} }
{ \defpoint_aux:nnnn {#2} {#3} {#4} {#1} }
}
\cs_new:Npn \defpoint_aux:nnnn #1#2#3#4
{
\IfNoValueTF {#3}
{ 2D~co-ordinates~-~(#1;#2),~mandatory~argument~-~'#4' \\ }
{ 3D~co-ordinates~-~(#1;#2;#3),~mandatory~argument~-~'#4' \\ }
}
\ExplSyntaxOff
\begin{document}
\noindent
\defpoint(1;2){A}
\defpoint(12;13){B}
\defpoint(sin(45);cos(45)){A}
\defpoint(1;2;-5){A}
\end{document}
Much the same idea in the internals as others have suggested, except I'm using a pre-build test for a 'quark' (special marker). Again, I've not done a complete job on testing the input here, so for example an empty optional argument will cause problems.
