Let $\alpha, \beta, \gamma$ be the complex roots of the polynomial $P_3(x)=ax^3+bx^2+cx+d$.
Is there any known formula for calculating $\alpha^2 \beta+\beta^2 \gamma+ \gamma^2\alpha \; , \; \alpha \beta^2+\beta \gamma^2+\gamma\alpha^2$ (in terms of $a,b,c,d$)?
If no, can someone obtain it?
Let $u=\alpha^2\beta+\beta^2\gamma+\gamma^2\alpha$ and $v=\alpha\beta^2+\beta\gamma^2+\gamma\alpha^2$. We have the following relations between the elementary symmetric polynomials $e_1$, $e_2$ and $e_3$ of $\alpha$, $\beta$ and $\gamma$ and the coefficients of $P_3$: $$ e_1 = \alpha+\beta+\gamma = -\frac{b}{a} \\ e_2 = \alpha\beta+\beta\gamma+\gamma\alpha = +\frac{c}{a} \\ e_3 = \alpha\beta\gamma = -\frac{d}{a} $$ We find \begin{eqnarray*} p & := u+v = & e_1e_2-3e_3 \\ q & := uv = & e_1^3e_3-6e_1e_2e_3+e_2^3+9e_3^2 \end{eqnarray*} From this, you can obtain $u$ and $v$ solving $t^2-pt+q=0$.
Let
$$\cases{E(\alpha,\beta,\gamma):=\alpha^2 \beta+\beta^2 \gamma+ \gamma^2\alpha\\F(\alpha,\beta,\gamma):= \alpha \beta^2+\beta \gamma^2+\gamma\alpha^2}$$
Your problem, strictly speaking, has no solution (but we will see hereafter that we can somewhat "bypass" this issue):
Here is a counterexample: Consider $P_3(x)=x(x^2-1)$ with roots $\{0,1,-1\}$.
Expression $E$ can take two different values depending on the order in which the roots are taken : $$E(\alpha,\beta,\gamma)=\begin{cases}-1 \ & \text{if} \ \ \alpha=1,\beta=-1,\gamma=0\\ \ \ \ 1 \ & \text{if} \ \ \alpha=-1,\beta=1,\gamma=0\end{cases}.$$
As said in the other answers, the fundamental reason is that neither $E$ nor $F$ is symmetrical in $\alpha, \beta, \gamma$, thus not invariant for certain permutations of the roots.
Of course, if you take $E+F$, you get a symmetrical polynomial in $\alpha, \beta, \gamma.$
But a slightly deeper analysis will be rewarding. In fact, expressions $E$ and $F$ are conjugate in a sense that will become clear, once we have seen the two following examples:
taking $P_3(x)=x^3+x^2-x$, value $E(\alpha,\beta,\gamma)=\Phi\approx1.618$ is taken 3 times, and value $1-\Phi\approx-0.618$ is taken as well 3 times ($\Phi$ is the golden ratio).
taking $P_3(x)=x^3+x^2+x$, (with some complex roots), value $-\frac12+i\frac{\sqrt{3}}{2}$ is taken 3 times, and conjugate value (with the meaning of "conjugation in $\mathbb{C}$"): $-\frac12-i\frac{\sqrt{3}}{2}$ is taken 3 times.
Let us prove this "conjugation" by inspecting the set $S_3$ of $3!=6$ permutations of the roots $\alpha,\beta,\gamma$:
$$\binom{\alpha \ \beta \ \gamma}{\alpha \ \beta \ \gamma}: \ \ E(\alpha,\beta,\gamma)=E(\alpha,\beta,\gamma)$$
$$\binom{\alpha \ \beta \ \gamma}{\beta \ \gamma \ \alpha}: \ \ E(\beta,\gamma,\alpha)=E(\alpha,\beta,\gamma)$$
$$\binom{\alpha \ \beta \ \gamma}{\gamma \ \alpha \ \beta}: \ \ E(\gamma,\alpha,\beta)=E(\alpha,\beta,\gamma)$$
$$\binom{\alpha \ \beta \ \gamma}{\beta \ \alpha \ \gamma}: \ \ E(\beta,\alpha,\gamma)=F(\alpha,\beta,\gamma)$$
$$\binom{\alpha \ \beta \ \gamma}{\beta \ \gamma \ \alpha}: \ \ E(\beta,\gamma ,\alpha)=F(\alpha,\beta,\gamma)$$
$$\binom{\alpha \ \beta \ \gamma}{\gamma\ \beta \ \alpha}: \ \ E(\alpha,\beta,\gamma)=F(\alpha,\beta,\gamma)$$
We have thus proved that, once a permutation is done on the roots, the value taken by $E$ is either unchanged (3 cases) or is the value of $F$ (the 3 other cases). As said by @CCorn in his answer, it is the mathematical object $\{E,F\}$ which is invariant under the action of $S_3$.
Remark: The first 3 cases correspond to a circular permutation: it is evident that $E$ has been built so that it is invariant by this type of permutation.
Addendum:
This is a little window on the origin of the so-called "Galois theory".
Have a look at the very pedagogical presentation here where it is explained how the ideas of Lagrange for solving the quartic equation where based on polynomials (called resolvents) that, under permutation, could take only a very small number of values. See also (https://en.wikipedia.org/wiki/Quartic_function). Another reference (Galois theory: splitting field of cubic as a vector space).
No. $a,b,c,d$ are symmetric polynomials in $\alpha,\beta,\gamma$ and thus invariant under every permutation of $(\alpha,\beta,\gamma)$. The expressions you want are invariant under some permutations, but not all, and therefore cannot be expressed in terms of only $a,b,c,d$.
Their sum is symmetric however, as @DougM's answer indicates.
Update: Their product too, and this results in a quadratic equation for the two expressions, as @ReinhardMeier's answer shows. Thus, the individual values of your desired expressions (call them $u$ and $v$) are not determined uniquely from $a,b,c,d$, but the set $\{u,v\}$ is.
$\frac {b}{a} = -\alpha - \beta - \gamma\\ \frac {c}{a} = \alpha \beta + \beta\gamma + \gamma\alpha\\ \frac {d}{a} = -\alpha \beta\gamma\\ -\frac {b}{a}\frac {c}{a} + 3\frac {d}{a} = (\alpha^2 \beta + \beta^2\gamma + \gamma^2\alpha)+(\alpha \beta^2 + \beta\gamma^2 + \gamma\alpha^2)$
I don't know if I can do much more that that.
There can be no such formula. The coefficients $a$, $b$, $c$ and $d$ are symmetric functions of the roots $\alpha$, $\beta$ and $\gamma$. Your polynomials $\alpha^2 \beta+\beta^2 \gamma+ \gamma^2\alpha$ and $\alpha \beta^2+\beta \gamma^2+\gamma\alpha^2$ are not symmetric functions, so they cannot be expressed as functions of the coefficients (since any such function would be symmetric).
