Below are my personal thoughts and reflections on Andrew Ng’s Stanford Machine Learning lecture on Generalized Linear Models (GLMs), including the key ideas I found interesting and how I understood them.
There is something that feels almost like magic in the mathematics of GLMs.
By changing only the underlying assumed probability distribution of the hypothesis function, and expressing it in the form of an exponential family distribution, we can derive several well-known machine learning algorithms:
- Ordinary Least Squares, using the Gaussian distribution
- Logistic Regression, using the Bernoulli distribution
- Softmax Regression, using the Multinomial distribution
This is a beautiful mathematical abstraction. We can essentially plug in a different probability distribution and derive a corresponding model. For example, the Gamma or Poisson distributions.
What makes this even more interesting is that these machine learning algorithms are quite different from one another. Each solves a different type of problem: regression, binary classification, or multiclass classification.
And yet, they can all be understood through the same underlying mathematical framework.
Isn’t that awesome?
The GLMs can be derived from the probabilistic interpretation of these machine learning algorithms. Once we have that probabilistic foundation, typical gradient descent or gradient ascent can be used to minimize the loss function J(x, y, θ) or maximize the log-likelihood, log(L(θ)), when fitting a GLM to a dataset.
There is another aspect I find particularly fascinating.
From the probabilistic interpretation of these regression and classification algorithms, the logistic function emerges naturally from the mathematics. It isn’t simply an arbitrary function that happens to work well. The probabilistic formulation helps answer the deeper “why?” behind its use.
And that is what makes GLMs so beautiful to me.
Different algorithms. Different probability distributions. Different problems.
Yet one elegant mathematical framework connects them all.
This is magic.
Efstathios Chatzikyriakidis 