R² from pure noise
You have observations of a target and of candidate predictors. All of them are independent standard normal noise: no predictor has anything to do with . You run the least-squares regression of on the predictors with an intercept.
What is the expected value of the in-sample ?
Fractions, decimals with at least 3 significant digits, and expressions such as sqrt(2)/2, e^-1 or pi/4.
After removing the means, lives in a space of dimension . The fitted values are the projection of onto the subspace spanned by the centred predictors.
is independent of that subspace and its centred version points in a uniformly random direction. What fraction of its squared length falls in a -dimensional subspace, on average?
h
s Shown after a correct answer. Revealing it first marks the problem as seen, not solved.
Idea. is the share of the (centred) variation of that lies in the span of the predictors. With pure noise, points in a random direction, so on average each dimension captures the same share.
Step 1: geometry. Centre everything. The centred lives in the -dimensional space of vectors with zero mean. The centred predictors span a subspace of dimension (almost surely), and where is the orthogonal projection onto .
Step 2: symmetry. has i.i.d. normal entries and is independent of , so its centred version is spherically symmetric in the -dimensional space. Write it in an orthonormal basis whose first vectors span : by symmetry, each of the squared coordinates has the same expected share of . So
The lesson. With useless predictors you expect an of about , and with you get exactly. This is why the adjusted subtracts this amount, and why in-sample fit says little when the number of features is not small compared with the number of observations.