Okay, so you're talking about the domain of a set-theoretic relation, not a function, so that clears up a lot of initial confusion.
So, the domain of a relation $R$ on a set $A$ (i.e. $R \subseteq A \times A$) is the collection of all lefthand components of its members (which are ordered pairs).
Here, then, $\text{domain}(C)= [-1,1]$. This is easy to see for two reasons:
- Graphed a relation in $\mathbb{R}^2$, the circle itself only encompasses the $x$ values in $[-1,1]$
- Algebraically, if $|x|>1$, then $x^2 > 1$. Since $y^2 \ge 0$, there is no $y$ for which such an $x$ satisfies $(x,y) \in C$.
This essentially matches up with your ideas.
I say the closed interval $[-1,1]$ as opposed to the open $(-1,1)$, however, since $(\pm 1,0) \in C$ though.
Addendum:
I'm adding this after reading Ethan Bolker's answer and the text in question (Discrete Mathematics with Applications by Susanna Epp).
It seems that the definition of domain used in this case is not the usual one I see.
The definition I tend to see is this: let $R$ be a relation from $A$ to $B$, so that $R \subseteq A \times B$. Then
$$\text{domain}(R) := \{ a \in A \mid \text{there is a $b \in B$ such that $(a,b) \in R$\}}$$
Epp's text uses the simpler $\text{domain}(R) := A$, which can be seen in this definition (bottom of page $14$):

It somewhat makes me think of the distinction between a function and a partial function.