Boosting is a way of fitting an additive expansion in a set of elementary basis functions like for example some simple polynomials. Assume for example that we have a function
f_M(x) = \sum_{i=1}^M \beta_m b(x;\gamma_m),where \beta_m are the expansion parameters to be determined in a minimization process and b(x;\gamma_m) are some simple functions of the multivariable parameter x which is characterized by the parameters \gamma_m .
As an example, consider the Sigmoid function we used in logistic regression. In that case, we can translate the function b(x;\gamma_m) into the Sigmoid function
\sigma(t) = \frac{1}{1+\exp{(-t)}},where t=\gamma_0+\gamma_1 x and the parameters \gamma_0 and \gamma_1 were determined by the Logistic Regression fitting algorithm.
As another example, consider the cost function we defined for linear regression
C(\boldsymbol{y},\boldsymbol{f}) = \frac{1}{n} \sum_{i=0}^{n-1}(y_i-f(x_i))^2.In this case the function f(x) was replaced by the design matrix \boldsymbol{X} and the unknown linear regression parameters \boldsymbol{\beta} , that is \boldsymbol{f}=\boldsymbol{X}\boldsymbol{\beta} . In linear regression we can simply invert a matrix and obtain the parameters \beta by
\boldsymbol{\beta}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.In iterative fitting or additive modeling, we minimize the cost function with respect to the parameters \beta_m and \gamma_m .