When we study the characteristics of a random variable $X$ and its associated probability distribution $P_X(x)$, one useful measure is the expected value $E[X]$. However, the expected value does not give any information about the variations or how the values are distributed. For example, the below variables have the same expected value (why?) but are very different in terms of variations and values:

$$begin{cases}p_X(x)=frac{1}{2} & text{ if } x= -1\ p_X(x)=frac{1}{2} & text{ if } x= 1\p_X(x)=0 & text{ if } elseend{cases} , begin{cases}p_Y(y)=frac{1}{2} & text{ if } y= -1000\ p_Y(y)=frac{1}{2} & text{ if } y= 1000\p_Y(y)=0 & text{ if } elseend{cases}$$

To explore more about a random variable $X$ and its distribution $P_X(x)$, the variance is another useful measure that is frequently employed in probability theory. One wise approach is to investigate the variation around the mean $E[X]$. A possible approach is to measure $E[|X-E[X]|]$. However, the absolute value function $|X-E[X]|$ is not differentiable and not mathematically very suitable. Variance offers a more reasonable approach as it deals with the square root of $|X-E[X]|$ as below:

Variance

If $X$ is a random variable with mean $E[X]$, then the variance of $X$ defined as below:

$$ Var(X)=E[(X-E[X])^2] $$

Notation

Variance is denoted with different notations such as Var(X) and $sigma^2$. The most common one is $sigma^2$.

We can easily prove the the variance has the equivalent formula below:

$$ Var(X)=E[X^2] - (E[X])^2 $$

Proof:

begin{equation*}

begin{align}

Var(X)=&E[(X-E[X])^2]=E[(X-mu)^2]\=&sum (X^2-2Xmu+mu^2)P_X(x)\

=&sum X^2P_X(x)-2musum XP_X(x)+mu^2sum P_X(x)\=&E[X^2] - 2mu^2+mu^2\

=&E[X^2]-mu^2 equiv E[X^2]-(E[X])^2 blacktriangle

end{align}

end{equation*}

Above, we have $mu=E[X]$.

Example

Assume in flipping a coin, heads equal to -1 and tails equals to 1. So the random variable $X$ takes either -1 or 1 with equal probability. What is the variance of $X$?

Solution: $X$ can be expressed a below:

$$X = begin{cases}1, with probability frac{1}{2} & text{ if } Event= H\ -1, with probability frac{1}{2} & text{ if } Event= Tend{cases}$$

Let's calculate $E[X^2]$ first:

$$E[X^2] = sum x^2 P_X(x)=(-1)^2 frac{1}{2}+(1)^2 frac{1}{2}=1$$

It's simple to realize that $E[X]=0$. So, we have variance as below:

$$Var(X)=E[X^2] - (E[X])^2 = 1 - 0 = 1$$

Standard Deviation

The square root of the variance $simga^2$ is called the standard deviation, and is denoted it by $simga$.

Standard deviation has also different notations. SD(X) is a common notation. In Machine Learning, however, the $simga$ notation is more common.

NOTE: The variance is the squared of the deviation from the mean of $X$ and, therefore, may not the best measurement. Due to this fact, the standard deviation is more commonly used as it reflects the original measure.

Variance of a Discrete Variables