Let us assume that our design matrix is given by unit (identity) matrix, that is a square diagonal matrix with ones only along the diagonal. In this case we have an equal number of rows and columns n=p .
Our model approximation is just \tilde{\boldsymbol{y}}=\boldsymbol{\beta} and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term 1/n )
C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2,and minimizing we have that
\hat{\beta}_i^{\mathrm{OLS}} = y_i.