The issue here is that shortening paths is done very late in the path's construction, so things like positioning nodes doesn't notice the shortening. I suspect this is because the shortening of shorten >=15pt is done by the same mechanism as when an arrow tip is added and generally this shouldn't affect the positioning of things along the path.
So one way to achieve this is to force the shortening to happen earlier. Ideally, one would do this before the positioning stuff was calculated, but I don't know where that is done and I happen to know an alternative way to specify points on a path after the path has been defined, so I'm using that instead.
Life is made slightly more complicated by your use of edge, since that creates a separate path that is drawn inside its own scope so we need to use a few globals to get things out of those scopes.
Here's two ways to achieve your goal. In the first, we save the shortened the path and then use the spath3 library to place the circles at its ends. In the second, we save the original path and then use the spath3 library to shorten it - this has the advantage that the shortening is genuinely along its path (the green lines in each picture are the original unshortened paths).
In both cases, we use the spath3 TikZ library to then access coordinates at positions along the shortened path, using the syntax (spath cs:<path name> <position>).
\documentclass{article}
%\url{https://tex.stackexchange.com/q/652865/86}
\usepackage{tikz}
\usetikzlibrary{spath3,intersections}
\makeatletter
\tikzset{
shorten path early/.code={
\tikz@addmode{%
\pgf@prepare@end@of@path
\pgf@prepare@start@of@path
\pgfsetshortenstart{0pt}%
\pgfsetshortenend{0pt}%
}%
},
shorten then name path/.style={
shorten path early,
spath/save global=#1
}
}
\makeatother
\begin{document}
\begin{tikzpicture}
% Version 1: save the shortened path
\begin{scope}
\coordinate (A) at (3.5,3.5);
\coordinate (B) at (5.1,2.);
\draw[color=red,fill=red] (A) circle[radius=0.15cm];
\draw[color=red,fill=red] (B) circle[radius=0.15cm];
\draw[ultra thick, green] (A) edge[out=-20,in=140] (B);
\draw (A) edge[out=-20,in=140,shorten >=15pt, shorten <=15pt, shorten then name path=short] coordinate[pos=0] (C) coordinate[pos=1] (D) (B);
\draw[black,fill=black] (C) circle[radius=0.05cm];
\draw[black,fill=black] (D) circle[radius=0.05cm];
\fill[magenta] (spath cs:short 0) circle[radius=0.05cm];
\fill[magenta] (spath cs:short 1) circle[radius=0.05cm];
\end{scope}
% Version 2: save the path and shorten it afterwards
\begin{scope}[xshift=3cm]
\coordinate (A) at (3.5,3.5);
\coordinate (B) at (5.1,2.);
\draw[color=red,fill=red] (A) circle[radius=0.15cm];
\draw[color=red,fill=red] (B) circle[radius=0.15cm];
\draw (A) edge[ultra thick, green, out=-20,in=140, spath/save global=short] coordinate[pos=0] (C) coordinate[pos=1] (D) (B);
\tikzset{
spath/shorten at both ends={short}{15pt}
}
\draw[spath/use=short];
\draw[black,fill=black] (C) circle[radius=0.05cm];
\draw[black,fill=black] (D) circle[radius=0.05cm];
\fill[magenta] (spath cs:short 0) circle[radius=0.05cm];
\fill[magenta] (spath cs:short 1) circle[radius=0.05cm];
\end{scope}
\end{tikzpicture}
\end{document}

On reflection, I think that the spath3 library should apply the shortening before it saves the path. I've implemented it in the development version (on github). With that version, the following works:
\begin{tikzpicture}
\coordinate (A) at (3.5,3.5);
\coordinate (B) at (5.1,2.);
\draw[color=red,fill=red] (A) circle[radius=0.15cm];
\draw[color=red,fill=red] (B) circle[radius=0.15cm];
\draw (A) edge[ultra thick, green, out=-20,in=140,shorten >=15pt, shorten <=15pt, spath/save global=short] coordinate[pos=0] (C) coordinate[pos=1] (D) (B);
\draw[black,fill=black] (C) circle[radius=0.05cm];
\draw[black,fill=black] (D) circle[radius=0.05cm];
\fill[magenta] (spath cs:short 0) circle[radius=0.05cm];
\fill[magenta] (spath cs:short 1) circle[radius=0.05cm];
\end{tikzpicture}
poskey) is done by TikZ. Thus, the positioning uses only the start, end and control points to calculate the positions of the nodes while PGF draws something else. And then, your path could be combined of multiple segments, theshortening is only done to the very first and the very last part which doesn't make it easy to catch all possibilities in TikZ. – Qrrbrbirlbel Aug 03 '22 at 16:22