Here, we described some of the most commonly used probability distributions in Machine Learning.

Get the source code for this lesson

Uniform Distribution

In Uniform Distribution, the probabilities of different values are identical in a particular interval and zero, anywhere else. Hence, the Probability Density Function (PDF) can be defined as below:

$$f(x)=left{begin{matrix}frac{1}{b-a} a<x<b\ 0 x in elsewhereend{matrix}right$$

The uniform distribution.

Gaussian Distribution

The most commonly used distribution for continuous random variables is Gaussian distribution. It is also called Normal distribution as is defined as follows:

$$f(x)=frac{1}{sigmasqrt{2pi}}e^{-frac{1}{2}(frac{x-mu}{sigma})^2}$$

The parameter $mu$ is the expectation of the distribution and $sigma$ is the standard deviation, which is the square root of the variance of the distribution $sigma^2$.

Notation: A normal distribution with mean $mu$ and variance of $sigma^2$, is denoted as $mathcal{N}(mu,sigma^2)$.

Two Gaussian distributions with the same mean $mu$. The red distribution has a larger standard deviation, and hence, it is wider (fatter). The blue distribution, is thinner and has a higher maximum. The maximum always happens at $x=mu$.

NOTE: The parameter $mu$, which is the mean of gaussian distribution, equals to the median or mode of the Gaussian distribution as well.

On the importance of Gaussian Distribution

In Machine Learning (and many other science branches), we often observe the employment of Gaussian distribution for the approximation of the distribution of phenomena. But why?

Natural Phenomena

The normal distribution can approximate many natural phenomena. The height of the population is a case of a Gaussian distribution which is the result of the combination of many other genetic and environmental factors.

Mathematical Point of View: Central Limit Theorem

In probability theory, the Central Limit Theorem (CLT) verifies that, in some circumstances, when independent random variables are combined, their normalized sum leads to a normal distribution regardless of the distribution of the original variables. This theorem is a fundamental notion in probability theory because it indicates normal distributions can be a statistical representation of many other distributions indirectly.

Then, the central limit theorem states the following:

Central Limit Theorem

Let's have $k$ independent and identically distributed (i.i.d.) random variables ${X_1,X_2,ldots,X_k}$, any of which with mean $mu$ and variance $sigma^2$. Now, we create the following random variable:

$$mathcal{V}_k=frac{X_1,X_2,ldots,X_k}{k}$$

Then, as $k rightarrow infty$, the random variables $sqrt{k}(mathcal{V}_k-mu)$ converge to the normal distribution $mathcal{N}(0,sigma^2)$.

$$sqrt{k}(mathcal{V}_k-mu) sim mathcal{N}(0,sigma^2)$$

Math Friendly!

Doing math with Gaussian distributions is much easier that the majority of other distributions since:

Assume we have multiple Gaussian random variables ${X_1,X_2,ldots,X_k}$. Doing operations such as adding and multiplying does not change the type of distribution. For example, The variable $mathcal{V}= X_1 + X_2 * X_3 + X_4$ has also Gaussian distribution.
The parameters that define Gaussian distributions ($mu$ and $sigma^2$) are independent of each other.

Some Continuous Probability Distributions