You will find a description of several of the layouts implemented in Mathematica in the following documentation page:
It cites the relevant papers so check out the References section at the end.
SpringElectricalEmbedding
Based on a physical model, it puts a spring-like force (i.e. $V \sim (\text{distance} - d_0)$ potential) between connected vertices and an electrostatic-like repulsion ($V \sim \frac{1}{\text{distance}^\alpha}$) between all vertex pairs. Then it tries to find a state with minimal potential energy.
This is probably similar to the Fruchterman-Reingold algorithm.
SpringEmbedding
Also based on a physical model. Connected vertices have tigher springs, unconnected ones have looser springs.
This is probably similar to the Kamada-Kawai algorithm.
HighDimensionalEmbedding
Like "SpringElectricalEmbedding", but in higher dimensions. Then the result is projected down to 2D or 3D.
LayeredDigraphEmbedding
This uses the Sugiyama algorithm. It is suitable for directed acylic graphs, and will route the edges as well as placing the vertices.
BalloonEmbedding, RadialEmbedding, LayeredEmbedding
These are meant for trees. Starting from a root, nodes are laid out on successive "layers". It is described in the tutorial above. The difference between them is how the layers are positioned: parallel lines, concentric circles, etc.
Also google for the Reingold-Tilford algorithm.
Tutte embedding
From Wikipedia:
In graph drawing and geometric graph theory, a Tutte embedding or barycentric embedding of a simple 3-vertex-connected planar graph is a crossing-free straight-line embedding with the properties that the outer face is a convex polygon and that each interior vertex is at the average (or barycenter) of its neighbors' positions. If the outer polygon is fixed, this condition on the interior vertices determines their position uniquely as the solution to a system of linear equations.
Only works for 3-vertex-connected planar graphs.
Spectral embedding
You'll find a lot by googling, but I'm not sure which particular variety it uses.
The basic idea is to take the two eigenvectors belonging to the largest eigenvalues of some graph matrix (adjacency matrix, Laplacian, etc.), then use these as vertex coordinates in 2D.
HierarchicalEdgeBundling
It likely works similar to what I described here:
See Y Jia, M Garland, JC Hart: Hierarchial edge bundles for general graphs.
You may want to look at the papers of Yifan Hu:
I believe he worked on Mathematica's graph drawing framework.
The following paper has a detailed description of the force-directed methods used in Mathematica:
There is a lot more to Mathematica's graph drawing than these basic algorithms. I find that Mathematica is very good at automatically producing a pleasing layout for many different kinds of graphs. It's usually better at this than other tools, which tend to require more manual parameter tuning to produce a pleasing or useful layout. I assume Mathematica chooses parameter values in a smart way based on the type of graph it is given.
One trick Mathematica uses is that graphs are first broken into connected components. Each component is laid out separately, then the result is rotated so that it will fit in a wide rectangle nicely. All these rectangles are then packed together using the selected "PackingLayout".
To get access to other layout types, or other variations on the same basic methods described above, check out IGraph/M, the igraph interface for Mathematica.