产生虚假相关性的子空间内部

between cholesterol and blood pressure is measured in 10 patients. The observed correlation is 0.62.

Is this enough evidence that a relationship exists?

Studies that measure 20,000 gene expression levels in 10 mice routinely apply multiple-testing corrections and stringent correlation thresholds. Researchers know that the more correlations they examine, the more likely they are to observe large values purely by chance, even when the genes are completely unrelated.

With small studies, researchers often assume they are safer from spurious correlations.他们是吗？

The distribution of a sample correlation is the same regardless of the total number of variables measured. It depends primarily on the number of subjects. High-dimensional datasets do not create spurious correlations—they simply make them easier to encounter.

This article answers the question: what correlation values should we expect when the variables are independent and the true population correlation is zero? It also explains why such correlations occur.

Using the geometry of Pearson’s correlation coefficient, the article visualizes the effects of centering and normalization. It then discusses the role of rotational invariance. Together, these ideas build intuition for the exact distribution of Pearson’s correlation coefficient and its asymptotic normal approximation.

Finally, the article offers practical guidance on how to use this intuition when reading and conducting research.

The code to reproduce all figures and experiments is available in the accompanying notebook.

The Gaussian Experiment

Imagine measuring d variables on n individuals.

To study correlations that arise purely by chance, consider a simulated dataset whose columns are independent Gaussian random vectors of dimension n. The resulting feature smatrix, where rows correspond to subjects and columns to variables, is

with columns satisfying

What is the sample correlation between two vectors?

The sample (Pearson) correlation between columns k and l is

where