The size of the universe is given by a scale factor, normally written as $a(t)$, that is a function of time and is calculated by solving Einstein's equation for an isotropic homogeneous universe. Once we've calculated $a(t)$ we can differentiate it wrt time to get $\dot{a}(t)$ and use this to calculate the recession velocities.
The scale factor is a function of time, but as we look farther away we are also looking back in time so we can measure $\dot{a}(t)$ over a range of times and not just at our present time. Then we can compare experiment with the results of our calculation.
Now, when we calculate $a(t)$ and $\dot{a}(t)$ there are some adjustable parameters that we need to feed in by hand. One is the average density of matter, and another is the cosmological constant, $\Lambda$, i.e. dark energy. If we set the cosmological constant to zero and do the calculation then our calculated values for $\dot{a}(t)$ do not match what we see by looking at type 1a supernovae. In order to make the calculation fit the experimental data we need to use a non-zero value for $\Lambda$. This in turn means that $\dot{a}(t)$ is increasing at late times, which means the expansion is accelerating.
The point of all this is to answer your question:
Why do we infer this, and not that expansion was faster in the past (when the light from those galaxies was generated)
The point is that we aren't randomly making up the rate of expansion. We get it from General Relativity. The way $a(t)$ and $\dot{a}(t)$ change is dictated by GR and our supplied values for the density and $\Lambda$. The expansion can't have been faster in the past than we calculate unless there is something fundamentally wrong with either GR or our assumptions about the early universe.