For the limit itself, André Nicolas gave the good hint.
You can do a bit more using Taylor $$A=2n\left(\sqrt{n^6+5n^2}-n^3\right)=2 n^4\left(\sqrt{1+\frac5{n^4}}-1\right)$$ Now, us Taylor for $\sqrt{1+x}=1+\frac x2-\frac {x^2}8 +\cdots$ and replace $x$ by $\frac 5{n^4}$. You then arrive to $$A=2n^4\big(1+\frac{5}{2 n^4}-\frac{25}{8 n^8}+\cdots-1\Big)=5-\frac{25}{4 n^4}+\cdots$$ which shows the limit and how it is approached.
For illustration purposes, try with $n=10$; the original expression evaluates $\approx 4.999375156$ while the approximation formula gives $4.999375000$. For $n=100$, the corresponding results would be $\approx 4.99999993750000156$ and $4.99999993750000000$.
http://math.stackexchange.com/questions/1200544/finding-the-limit-lim-x-rightarrow-infty-sqrt3x1-sqrt3x, http://math.stackexchange.com/questions/1125334/help-with-lim-x-rightarrow-infty-x2-sqrtx4-x2-1 – lab bhattacharjee May 27 '15 at 05:22