A (Formal) Probabilistic Interpretation of Linear Regression

Linear regression is a canonical problem often introduced at the start of courses in machine intelligence. (Ng, 2000) justifies the model by endowing the data with certain probabilistic assumptions, from which the least-squares cost function is derived. However, the justifications provided are at times handwavy, leaving the reader grasping at straws. Here, we explicitly state the assumptions needed to derive least-squares and provide a formal justification of its derivation. Our work is inspired by these discussions.

Necessary Assumptions

We begin by detailing the assumptions required to derive the linear regression model. Let our dataset $\mathcal{D}$ consist of input-target pairs $(x_i, y_i)$. Let us assume that:

1. The target variables $y_i$ and inputs $x_i$ originate from random variables $Y_i, X_i$ that have a common density $f_{X_i, Y_i}$. The variables $Z_i = (X_i, Y_i)$ are independent.

2. The target variables and input variables are related via the equation \(y_i = \theta^T x_i + \epsilon_i\) where the error terms $\epsilon_i$ capture random noise or unmodeled effects.

3. The error terms $\epsilon_i \sim \mathcal{N}(0, \sigma^2)$ are independently and identically distributed.

These assumptions are quite reasonable: in English, we’re simply requiring that the datapoints $(x_i, y_i)$ are drawn independently from the data distribution $f_{X, Y}$, and that each target variable is related to its input variable by a linear transformation and a component representing random noise.

Representing $\ f_{X_i, Y_i}$

Our next step is to derive the likelihod of our dataset under the aforementioned assumptions. To do so, we must represent the joint distribution of each target-variable pair, $f_{X_i, Y_i}$. We’ll approach this problem by first representing the joint distribution of each input-error pair, $f_{\epsilon_i, X_i}$, and applying the change-of-variables formula.

By assumption 3, $\epsilon_i \perp x_i$. We can therefore write the joint distribution of each input-error pair as

\[f_{\epsilon_i, X_i}(\epsilon, x) = f_{\epsilon_i}(\epsilon) f_{X_i}(x)\]

Furthermore, by assumption 2, there exists a linear relationship between $y_i$ and $x_i$. This relationship allows us to define a transformation $\phi : (\epsilon_i, X_i) \to (Y_i, X_i)$ such that

\[\begin{align} \phi(\epsilon_i, x_i) &= (\theta^Tx_i + \epsilon_i, x_i) \\ \phi^{-1}(y_i, x_i) &= (y_i - \theta^T x_i, x_i) \end{align}\]

We are now ready to apply the change-of-variables formula from $f_{\epsilon_i, X_i}$ to $f_{Y_i, X_i}$. Specifically, for an invertible mapping $\phi : \mathbf{R}^n \to \mathbf{R}^n$ between random variables $A_1 \dots A_n$ and $B_1 \dots B_n$ such that $\mathbf{B} = \phi(\mathbf{A})$ and $\mathbf{A} = \phi^{-1}(\mathbf{B})$, we have that

\[p_{B_1 \dots B_n}(B_1 \dots B_n) = p_{A_1 \dots A_n} (\phi^{-1} (B_1 \dots B_n)) \left| \text{det} \left( \frac{\partial \phi^{-1} (A_1 \dots A_n)}{\partial A_1 \dots A_n} \right) \right|\]

In our case, $B_1 = Y_i$, $B_2 = X_i$, $A_1 = \epsilon_i$, $A_2 = X_i$. We first compute

\[\partial \phi^{-1} = \begin{bmatrix} 1 & - \theta^T \\ 0 & 1 \end{bmatrix}\]

which has determinant 1. We therefore have

\[f_{Y_i, X_i}(y_i, x_i) = f_{\epsilon_i, X_i} (y_i - \theta^T x_i, x_i) = f_{\epsilon_i} (y_i - \theta^T x_i) f_{X_i}(x_i)\]

again due to $\epsilon_i \perp x_i$.

Deriving the Likelihood

Since linear regression is a discriminative model, we do not model the prior density of the input variables $f_{X_i}$ and focus our efforts solely on maximizing the conditional likelihood $f_{Y_i \mid X_i}$ across our dataset. We can write the conditional as

\[f_{Y_i \mid X_i}(y_i \mid x_i) = \frac{f_{Y_i, X_i} (y_i, x_i)}{f_{X_i}(x_i)} = f_{\epsilon_i} (y_i - \theta^T x_i)\]

By assumption 1, each $(X_i, Y_i)$ is independent, and so we have

\[f_{X \mid Y} = \prod_{(x_i, y_i) \in \mathcal{D}} f_{\epsilon_i}(y_i - \theta^T x_i)\]

Explicitly, this is

\[f_{X \mid Y} = \prod_{(x_i, y_i) \in \mathcal{D}} \frac{1}{\sqrt{2 \pi \sigma^2}} \exp \left( - \frac{(y_i - \theta^T x_i)^2}{2 \sigma^2} \right)\]

Maximizing $f_{X \mid Y}$ with respect to $\theta$ is now a simple exercise in calculus; one typically maximizes $\log f_{X \mid Y}$ as a proxy to transform the product into a sum. After some calculus as in (Ng, 2000), we conclude that maximizing the log-likelihood is equivalent to minimizing

\[\sum_{(x_i, y_i) \in \mathcal{D}} (y_i - \theta^T x_i)^2\]

which is the canonical least-squares cost function for linear regression, as desired.