Swapping the factors

Idea. $AB$ and $BA$ can have different ranks. Rank 4 for $BA$ is impossible, and an explicit pair reaches 3.

Step 1: not 4. If $BA$ has rank 4, it is invertible, so $A$ and $B$ are both invertible. Then $AB$ is invertible too, with rank 4, not 2.

Step 2: an example with 3. Let $A$ shift the basis down: $A{e}_1=0$, $A{e}_2=e_1$, $A{e}_3=e_2$, $A{e}_4=e_3$. Let $B$ keep the first three coordinates and kill the fourth: $B=\mathrm{diag}(1,1,1,0)$.

• $AB$: $e_1\mapsto 0$, $e_2\mapsto e_1$, $e_3\mapsto e_2$, $e_4\mapsto 0$. Rank 2.
• $BA$: $e_1\mapsto 0$, $e_2\mapsto e_1$, $e_3\mapsto e_2$, $e_4\mapsto B{e}_3=e_3$. Rank 3.

Answer. The largest possible rank of $BA$ is 3. In fact every value from 0 to 3 can occur.

Why they differ. $AB$ and $BA$ have the same nonzero eigenvalues, but not necessarily the same rank: here both are nilpotent, and the order of the factors changes which basis vector gets lost.