How can these statistical arguments be made logically rigorous?

Question

Suppose an urn contains unknown but non-random numbers of red and green marbles, and I take a random sample of a known and non-random size. Observing the numbers of red and green marbles in the sample, I need to hazard my best guess as to the proportion (not the total number) of red marbles in the urn.

The number of distinct marbles I observe is likely to be larger if the sample is taken without replacement than with replacement. Therefore, there should be less uncertainty in my estimate if I sample without replacement.

THEREFORE the standard deviation of the random number of red marbles in my sample is smaller if I sample without replacement than with.

Clearly the above is not a logically rigorous argument. (Of course, one can establish the same result by easy standard arguments, but that will be off-topic here.)

Now suppose I take a sample of $n$ independent measurements $X_1,\dots,X_n$ from a population that is normally distributed with unknown mean $\mu$ and with variance $\sigma^2$. Let $\overline{X} = (X_1+\cdots+X_n)/n$ and $S^2 = \sum_{i=1}^n (X_i - \overline{X})^2/(n-1)$ be respectively the sample mean and sample variance. Then $$ \frac{\overline{X} - \mu}{\sigma/\sqrt{n}} $$ is normally distributed with known mean and variance 0 and 1 respectivly, so the endpoints of a confidence interval for $\mu$ are $$ \overline{X} \pm A \frac{\sigma}{\sqrt{n}} $$ where $A$ is a suitable percentage point of the normal distribution (provided (unrealistically) that $\sigma$ is known).

And $$ \frac{\overline{X} - \mu}{S/\sqrt{n}} $$ has a Student's t-distribution with $n-1$ degrees of freedom, so $$ \overline{X} \pm B_n \frac{S}{\sqrt{n}} $$ where $B_n$ is the corresponding percentage point of Student's distribution, are the endpoints of a corresponding confidence interval if (more realistically) $\sigma$ is unknown.

There is more uncertainty if $\sigma$ is unknown than if $\sigma$ is known, THEREFORE $B_n > A$.

If $\sigma$ is unknown, then the amount of uncertainty decreases as the sample size $n$ increases, THEREFORE $B_n$ decreases as $n$ increases. (Of course, the fact that $B_2 > B_3 > B_4 > \cdots > A$ can be proved by standard arguments involving fiddling with integrals.)

The logical leaps corresponding to the THEREFOREs above have a seeming intellectual compellingness about them. It would be extraordinarily paradoxical if the conclusions following them failed to hold despite the reasoning preceding them. But these are not logically rigorous arguments.

So my question is whether there could be some theorem of logic that says things like those following the THEREFOREs are true when things like those preceding them are true.

Answer 1

Let's recall how all these confidence intervals work in general. You have some real parameters $t,s$ that define the distribution. You have two unbiased estimators $T$ and $S$ for these parameters that depend on the (vector) observation $X$. You want to create the confidence interval for $t$ from $T$ and known $s$ in one case and from $T$ and $S$ in the other case. We will also assume that the distribution of $T$ is symmetric around $t$ for all $s$ with the density decreasing in both directions, and the distribution of $T-t$ depends on $s$ but not on $t$. The first case is then simple: we take $T$ and find the least $a=a(s)$ such that $P(|T-t|>a(s))<\delta$ and say that $[T-a(s),T+a(s)]$ is the $\delta$-confidence interval. What we mean by that is that regardless of what the actual value of $t$ is, the probability that this interval won't cover it is less than $\delta$. Let's also assume that $s>0$ and $a(s)$ is increasing in $s$ Now, let us ask whether it is possible that $[T-a(S),T+a(S)]$ is a $\delta'$-confidence interval with $\delta'<\delta$ (which, in your terminology, would mean that determining $s$ from observation beats knowing it)?

The answer is, of course "Yes": despite $S$ is an unbiased estimator, it is perfectly possible that $S$ overestimates more often than it underestimates and $a$ is more sensitive to overestimates than to underestimates. To create a particular example, take the uniform distribution on the interval $[t-s^2,t+s^2]$ and the observation $X_1,X_2,X_3$. Take $T=X_1$ (I know, this is a bit stupid, but how do you know whether your estimators are not equally stupid in general?). Then $a(s)=(1-\delta)s^2$. Now, take $S=c\sqrt{|X_2-X_3|}$ where $c$ is chosen so that $S$ is unbiased. It remains to compute the probability that the true value is in the interval. $S$ is independent of $T$ and is $ys$ with probability $p(y)dy$ with some positive continuous $p$ on $[0,c\sqrt{2})$. So, we get $\int p(y)\min(1,(1-\delta)y^2)dy=(1-\delta) \int p(y)y^2dy>(1-\delta)$ because $\int p(y)ydy=1$ already ($S$ is unbiased), provided that $1-\delta$ is small enough (which is ridiculous too, but how do you tell the difference?). So, your $B_n$ may be larger than $A$ not for some deep probabilistic reason, but for the mundane reason that $\sqrt s$ is concave, not convex like $s^2$ in my example.

Now, this doesn't show that we can do better with less information. It merely shows that the proposed way to compare effectiveness is fundamentally flawed. But how else can you compare the confidence intervals?

Answer 2

$\newcommand\si{\sigma}$There is no THEREFORE in general. Indeed, suppose we observe the values of the pair $(X,S)$ of independent random variables, where $P(X=\mu+\si)=P(X=\mu-\si)=1/4$, $P(X=\mu)=1/2$, $S:=\sqrt{S^2}$, $P(S^2=\si^2/2)=P(S^2=3\si^2/2)=1/2$, $\mu$ is an unknown real parameter, and $\si$ is a possibly unknown positive real parameter. If $\si$ is known, our confidence interval (CI) (for $\mu$) is symmetric about $X$ and based on the pivot $\frac{X-\mu}\si$ (by definition, a pivot is any random variable whose distribution does not depend on the parameters). If $\si$ is unknown, our CI is again symmetric about $X$ but now based on the pivot $\frac{X-\mu}S$. This setting is quite similar to your normal-sampling setting, with my pair $(X,S)$ in place of your pair $(\bar X,S)$. In particular, I also have $ES^2=\si^2$. (If desired, one may arbitrarily closely approximate the discrete distributions of $X$ and $S$ by continuous ones.)

Now, the CI with $\si$ known will be always better than the CI with $\si$ unknown if and only if $U:=\big(\frac{X-\mu}\si\big)^2$ is stochastically less than $V:=\big(\frac{X-\mu}S\big)^2$, that is, if $$P(U\ge u)\overset{\text{(?)}}\le P(V\ge u) \tag{1}$$ for all real $u$. However, $$P(U\ge1)=\tfrac12>\tfrac12\times\tfrac12 \\ =P\big((X-\mu)^2=\si^2\big)\,P(S^2=\si^2/2) \\ =P\big((X-\mu)^2=\si^2,S^2=\si^2/2\big) \\ =P(V\ge1).$$ So, (1) fails to hold for $u=1$.

However, $U$ is better than $V$ on an average: $$EV=E\Big(U\frac{\si^2}{S^2}\Big)=EU\,E\Big(\frac{\si^2}{S^2}\Big) >EU\,\frac{\si^2}{ES^2}=EU, $$ where the inequality is an instance of Jensen's inequality.