What is the chain of reasoning (beginning, of course, from observations about the universe) that leads one to predict that Minkowski space provides an accurate description of space-time in the framework of special relativity?
That is, why should flat space-time have the Minkowski metric $$\eta = \left(\eta_{\alpha\beta}\right) = \left( \begin{matrix} 1 & 0 & 0 & 0 \\ 0 & -1 & 0 & 0 \\ 0 & 0 & -1 & 0 \\ 0 & 0 & 0 & -1 \end{matrix}\right)\,?$$
I think there are a few related, yet separate, questions that must be addressed here.
1) How do we know the Minkowski metric is a flat metric?
There are several ways to go about this question. Perhaps the most rigorous is to calculate the Riemann tensor, as this approach gives you quantities that are independent of the coordinate system. OP, in your question as written now, you've written the metric with respect to cartesian coordinates. A Minkowski metric is still Minkowski even when written in other coordinate systems, and it is still flat.
2) Why do we expect that the universe should be modeled with a flat metric?
This relies upon both mathematical logic as well as physical observations. Even when we expect the universe to be curved in some way, any curved manifold will look flat at a small enough length scale. Once we've already decided we live on some arbitrary curved, differentiable manifold, the flat space argument is pure mathematics--math tells us this is the case, but it doesn't tell us what length scales we must limit ourselves to to make it so. That has to be determined through observation. Nevertheless, relying upon a flat space model gives us a good jumping off point to eventually build up to GR, so it's usually useful.
3) Why do we pick Minkowski space, then, as the model? What physical considerations drive this choice of model?
Differential geometry will tell you that there are only three flat models of geometries: Minkowski space, Galilean space, and Euclidean space. EAch of these can be parameterized with different coordinate systems, but these are the only flat models available to us that meet certain broad criteria. In particular, we restrict ourselves to one timelike dimension, which may or may not have a different signature from spatial dimensions. In Minkowski space, the timelike dimension has the opposite sign in its metric component. In Galilgen space, the timelike dimension is null and has zero for its metric component. In Euclidean space, it has the same sign as spatial dimensions.
Here's where the physics comes in: we know that the speed of light appears invariant in all frames of reference. We associate different frames of reference with rotation-like transformations on each of these spaces that mixes time and space. In Minkowski space, the rotation-like transformation is Lorentz boosting. In Galilean space, the transformation is a Galilean transformation. In Euclidean space, the operation is just rotation. These operations are unique to each kind of spacetime model, so there is no ambiguity.
To measure the same speed of light, one eventually concludes that light follows a trajectory that is invariant under the rotation-like transformation. That rules out Euclidean space, as there are no vectors in the plane of rotation that aren't transformed. It also rules out Galilean space, as there is only one vector that doesn't change under a Galilean transformation--the direction of time--and that direction has no speed with respect to the observer in any Galilean frame of reference.
So the only option is Minkowski space, which admits two vectors light can follow in any given plane, both of which involve trajectories that are measured as having the same coordinate speed regardless of reference frame.
Minkowski space is the only model of flat spacetime that allows us associate light with specific trajectories in the spacetime and in ways that correspond to our physical observations.
I think the key observation is Lorentz symmetry.
It was realized by the end of the 19th/beginning of the 20th century that the Maxwell equations adhere to a symmetry other than the Galilean symmetry of Newtonian mechanics. This is Lorentz symmetry.
Many researchers tried reconciling the difference between those symmetries by introducing an aether into Maxwell's picture of electromagnetism. In the end though, relativity was realized to be true.
In (special) relativity one expands the framework of Newtonian mechanics to Lorentz symmetry rather than mere Galilean symmetry. But taking this symmetry face-value, the metric of spacetime should also be the metric that is left invariant by Lorentz transformations and that is the Minkowski metric:
$$ \Lambda^T \eta \Lambda = \eta $$
for any Lorentz transformation $\Lambda$ and the Minkowski metric $\eta$. In today's textbooks the reasoning is reversed, and Lorentz symmetry is defined by the above equation rather than the symmetry of Maxwell's equations.
Minkowski space provides an accurate description of flat space-time simply because it was derived from the equation for light traveling in Euclidean space. There is nothing unusual about the metric - Minkowski metric is just a way of presenting the good old Euclidean space. And as in Special Relativity there is no gravitation (acceleration) to curve this space-time, so it remains flat.
If you take a look at Einstein's "Relativity: The Special and General Theory", you will find in the Appendix I (just before equation (10)) that in order to derive the metric, Einstein simply started off with the Pythagorean Theorem in 3D, which he put like this:
$$r = \sqrt{x^2 + y^2 + z^2} = ct$$
So what he proposed (as well as Minkowski himself) is to show the vector of light traveling in a three dimensional space.
He then squared both sides and moved the $x^2 + y^2 + z^2$ to the right to obtain the equation $ct^2 - x^2 - y^2 - z^2 = 0$ (and later $0$ was replaced with $1$, which was to assure symmetry). Therefore we have the signature [$+, -, -, -$], which corresponds to the matrix you presented in your question.
So going back to our first equation we can see that it represents the movement of light in flat (Euclidean) space-time by definition. The basic equation that was used to derive the matrix refers to Euclidean space, and therefore the space-time it represents must be flat. And flat space-time is the framework of Special Relativity which pertains to inertial frames, i.e. non-accelerated ones (as opposed to General Relativity where gravity - i.e. acceleration - is the source of curvature).
