Lasso case

For Lasso we need now, keeping a constraint on \( \vert\beta_0\vert+\vert\beta_1\vert=1 \), to take the derivative of the absolute values of \( \beta_0 \) and \( \beta_1 \). This gives us the following derivatives of the cost function

$$ C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\vert\beta_0\vert+\vert\beta_1\vert), $$ $$ \frac{\partial C(\boldsymbol{\beta})}{\partial \beta_0}=-4(4-2\beta_0)+\lambda\mathrm{sgn}(\beta_0)=0, $$

and

$$ \frac{\partial C(\boldsymbol{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}(\beta_1)=0. $$

We have now four cases to solve besides the trivial cases \( \beta_0 \) and/or \( \beta_1 \) are zero, namely

  1. \( \beta_0 > 0 \) and \( \beta_1 > 0 \),
  2. \( \beta_0 > 0 \) and \( \beta_1 < 0 \),
  3. \( \beta_0 < 0 \) and \( \beta_1 > 0 \),
  4. \( \beta_0 < 0 \) and \( \beta_1 < 0 \).