Disclaimer: As pointed out by jensen paull in the comments, there is a problem with this answer. I'm keeping the original version in here because I find it to be pedagogical in terms of understanding that Ampère's Law is only useful in a few situations, but I'll address the flaws in the following section.
While Ampère's Law holds in any magnetostatic situation and can be used if you are assuming the wire to be part of a circuit, it is not always useful. When you compute the field using Ampère's Law, you are assuming implicitly that the field does not depend on the position along the wire (otherwise, you wouldn't be able to compute the necessary integrals that go in using Ampère's Law to compute the field). As a consequence, you are throwing away the dependence on the wire's length. Not surprisingly, your result obtained with this trick is precisely the result for an infinite wire ($L \to \infty$, as you noticed), which is indeed the magnetic field one would obtain for the wire in the case where the field is independent of the position along the wire (i.e., it is the case in which you have cylindrical symmetry, rather than only axial symmetry).
In short, Ampère's Law is always valid (in magnetostatics), but it is not always useful when it comes to computing the magnetic field. It will only be helpful in situations with a lot of symmetry, which this problem lacks. Griffiths' Introduction to Electrodynamics, Sec. 5.3., lists the possible symmetries in which one can employ this trick as being
- infinite straight lines;
- infinite planes;
- infinite solenoids,
- toroids.
Erratum: as pointed out by jensen paull in the comments, one does not really need to assume cylindrical symmetry to perform the integrals in Ampère's Law, as I previously stated. The problem is actually more subtle and deeper. This section is my way of rephrasing the point made in the comments.
As pointed out by the OP, the finite wire only makes sense as a part of circuit. This is due to the fact that a finite wire with constant current will fail to satisfy charge conservation: one needs charge coming from nowhere and going to nowhere at the extremities of the wire in order to keep the current constant. However, consider Ampère's Law in differential form. It is given by
$$\nabla\times\mathbf{B} = \mu_0 \mathbf{J},$$
and since the divergence of a curl is always zero, one has a consequence
$$\nabla\cdot\mathbf{J} = 0.$$
Hence, conservation of charge is an integrability condition for Ampère's Law: if the current is not divergenceless, it is impossible to find a magnetic field that respects Ampère's Law.
This explains why one can't use Ampère's Law for the finite wire: it indeed doesn't hold. Maxwell's equations will certainly hold if you consider the entire circuit, but to consider the piece of wire alone is to consider an unphysical situation, which ends up "breaking" the equations.
The Biot–Savart Law, on the other hand, reads
$$\mathbf{B}(\mathbf{x}) = \frac{\mu_0}{4\pi} \int \frac{\mathbf{J}(\mathbf{x}') \times (\mathbf{x} - \mathbf{x}')}{\|\mathbf{x} - \mathbf{x}'\|^3} \mathrm{d}^3{x'}.$$
If we compute the curl of this expression, we'll find it doesn't equal $\mu_0 \mathbf{J}$, but rather has an additional term depending on $\nabla\cdot\mathbf{J}$. If I didn't make any mistakes, it reads
$$\nabla\times\mathbf{B} = \mu_0 \mathbf{J} - \frac{\mu_0}{4\pi} \int \frac{[\nabla'\cdot\mathbf{J}(\mathbf{x}')] (\mathbf{x} - \mathbf{x}')}{\|\mathbf{x} - \mathbf{x}'\|^3} \mathrm{d}^3{x'}.$$
Hence, for divergenceless currents, the Biot–Savart law will yield a solution to Ampère's Law. For more general currents, it yields something else but adding up the pieces of a circuit will cancel out the differences and lead you to a correct result.
However, Using Ampere's Law in line-integral form, the field should be:
\begin{align}B=\frac{\mu_0}{2\pi r}\end{align}
As shown in the diagram, the field would always be tangential to the Amperian loop.
