Example of gradient descent applications

We will use linear regression as a case study for the gradient descent methods. Linear regression is a great test case for the gradient descent methods discussed in the lectures since it has several desirable properties such as:

An analytical solution.
The gradient can be computed analytically.
The cost function is convex which guarantees that gradient descent converges for small enough learning rates

We have the following problem

x = 2*np.random.rand(m,1)
y = 4+3*x+np.random.randn(m,1)

with $ x_i \in [0,1] $ is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution $ \cal {N}(0,1) $. The linear regression model is given by

$$ h_\beta(x) = \boldsymbol{y} = \beta_0 + \beta_1 x, $$

such that

$$ \boldsymbol{y}_i = \beta_0 + \beta_1 x_i. $$