I don't have much authority to answer here, but this is my best guess.
Both affine and linear spaces inherit axiom structures. If you talk about a linear subspace, it is, itself, a linear space in its own right. Similarly, if you take an affine subspace of an affine space, it is an affine subspace in its own right. As it happens, you can also take an affine subspace of a linear space in a natural way, which is a weaker notion, but still falls into an axiomatic framework.
A convex set does not. It has its own algebraic structure, specifically, a line segment between two points must be contained in the set, but such a structure relies on the affine structure of the space it's contained in. It's not so easy to consider convex sets as algebraic spaces in their own right; I've certainly never seen axioms for them. It's difficult to describe a line segment without affine structure, and it's even harder to talk about being "in the set" when you disregard everything out of the set!
My guess is, of course, a little spurious at best, since vector spaces rely on ground fields, and affine spaces rely on an entire vector space. But, I do think that, in some way, convexity is even less independent than linear or affine spaces.
It's also worth comparing to topology (which I'm guessing you might have picked up this enthusiasm for the word "space"). In topology, every subset can be naturally turned into a topological space in its own right. This is just not true in linear algebra.