Geometric intuition on $\langle x, A^\top y\rangle = \langle y, Ax\rangle$

Question

I am looking for a geometric intuition on $\langle x, A^\top y\rangle = \langle y, Ax\rangle$. This can be proven algebraically by disassembling the expression into basic sums and product and reordering a few terms. But that does not offer any insight.

Semantically this equality states that the scaled projection (dot product) of $x$ with a linear combination of the rows of $A$ weighted by $y$ is equal to the scaled projection of $y$ onto linear combination of the columns of $A$ weighted by $x$. But I fail to understand this in an intuitive geometric way. Do you know of any insightful interpretation?

If this equality has a name, that would also be useful. Right now I cannot really research it without a name.

I came across the equation here: Eigenvectors of real symmetric matrices are orthogonal

$ \langle x, A^\top y\rangle = x^\top (A^\top y) = ( x^\top (A^\top y))^\top =(A^\top y)^\top x^\top= ( y^\top A) x= y^\top A x=y^\top (A x)=\langle y,Ax\rangle$ so this is the definition of $\top$ and associativity. The geometric intuition is that for any linear map $A$ there is $B$ such that $ \langle y, Ax\rangle= \langle x,By\rangle$ and $B$ is given by $\top$. — reuns, Jun 11 '19 at 22:52
I'm not sure there is a geometric intuition behind that. Given a non-degenerate bilinear form $b$ on a vector space $V$, it isbpossible to associate, to each endomorphism $A:V\rightarrow V$, another operator $A^t:V\rightarrow V$, via $$b(X,A^tY) =b(AX,Y),$$ but I cannot see a geometric interpretation of this assignment. — Dog_69, Jun 11 '19 at 22:56
@Hyperplane I know that video. Can you point out the connection to my question? I don't see it. — user3578468, Jun 11 '19 at 23:03
How does $Ax$ mean “$x$ expressed in terms of the columns of $A$?” Indeed, it’s quite the opposite. — amd, Jun 11 '19 at 23:31
@amd Because it is a linear combination of the columns of $A$, where the $i$th column is weighted by the $i$th entry of $x$. So while $x_i$ in standard basis means $x_i$ times the standard basis vector of the $i$th dimension, $(A x)_i$ means $x_i$ times the $i$th column vector of $A$. — user3578468, Jun 11 '19 at 23:38
You should use the singular value decomposition of $A$ as $A=P\Sigma Q$ where $P$ and $Q$ are orthogonal (rotations) and $\Sigma$ is diagonal. — user52817, Jun 12 '19 at 00:33
@user52817 How exactly do you mean that? But actually that would be circular logic to me, as I want to understand the equality in my question for better understanding eigenvalue decomposition, which is again part of SVD. If I use SVD o better understand my question I am basically begging the question. — user3578468, Jun 12 '19 at 00:46
But that’s not “$x$ expressed in terms of the columns of $A$,” which would be some other linear combination of the $A_i$ that equals $x$ (whose coefficients are in fact $A^{-1}x$). — amd, Jun 12 '19 at 00:57
+1 Great question. Geometric understanding is way more important than proving via algebraic manipulation. — JavaMan, Jun 12 '19 at 02:36

user52817 · Answer 1 · 2019-06-12T02:55:23.637

Use the singular value decomposition $A=P\Sigma Q$ where $P$ and $Q$ are orthogonal, and $\Sigma$ is diagonal. Regard $A$ as acting on $V$ and $A^T$ as acting on the isometric dual space $V^*$. The SVDs of $A$ and $A^T$ make it clear that the "geometry" of the action of one is the same as the action of its transpose partner on the dual. The equality $\langle x,A^Ty\rangle=\langle y,Ax\rangle$ simply unravels this via the inner product, which of course provides the isometry between the two space.

CopyPasteIt · Answer 2 · 2019-06-12T15:37:59.610

The OP is interested in the geometric intuition behind

$$\tag 0 \langle x, A^\top y\rangle = \langle y, Ax\rangle$$

and asked for the name of this property.

To truly appreciate the beauty of $\text{(1)}$ you have to pass into the realm of abstract linear algebra. To do that, one can begin by ruminating on the following snippet taken from this historical summary,

The first modern and more precise definition of a vector space was introduced by Peano in 1888; by 1900, a theory of linear transformations of finite-dimensional vector spaces had emerged. Linear algebra took its modern form in the first half of the twentieth century, when many ideas and methods of previous centuries were generalized as abstract algebra.

(emphasis mine)

I tried to find the name of the property described in $\text{(0)}$ and have come to the conclusion that whenever you see it, you must simply 'drop your jaw' in amazement. If you want to use it, simply precede the use by writing

$$\quad \text{Since ...}$$

OK, I have to leave the OP with something better than the above. If you want to give $\text{(0)}$ a name, call it Theorem AIP and refer to this link.

Below I copy the theorem and the proof. As an exercise, the OP can restate all the definitions/theorems/proofs by replacing the complex numbers with the real numbers, where for $x \in \Bbb R$, $\,\bar x = x$. While working on this, the OP can attempt to attach geometric intuitions to the ideas and concepts as they are analyzed.

Theorem AIP: Adjoint and Inner Product. Suppose that $A$ is an $m \times n$ matrix and $\vec x \in {\Bbb C}^n$, $\vec y \in {\Bbb C}^m$. Then

$$\tag 1 \langle A \vec x, \vec y\rangle = \langle \vec x, A^{*} \vec y\rangle$$

Proof

I examined the proof of Theorem AIP and found myself wondering why, intuitively,

$$\tag 2 (AB)^t = B^t A^t$$

and found this stack link

Why, intuitively, is the order reversed when taking the transpose of the product?

That lead me to write up the following answer, where the result is proved and due to our intuition, it comes as no surprise.

Geometric intuition on $\langle x, A^\top y\rangle = \langle y, Ax\rangle$

2 Answers2