In this post, we are going to walk through the Essential Definitions in Probability Theory. Understanding these concepts is critical to comprehend further advanced concepts in probability theory. Here, you will learn:
- The key concepts such as random variables and conditional probabilities.
- You gain knowledge about how these notions are related.
Random Variables
The random variable is one of the essential definitions in probability theory. In probability theory, the outcomes of a random phenomenon determine the random variable values. In other words, a random variable is a variable that its values are determined with a random event. We should be able to measure a random variable that provides the capability to assign probabilities to its possible values. The domain of a random variable is the sample space. For example, in the case of having a dice, only six possible outcomes are considered, as {1,2,3,4,5,6}.
By using more precise mathematics notation, a random variable is a measurable function defined as
, which is from all possible space
to some event
. Let’s have an illustrative example. Assume we would like to roll a dice, and the measurable event space is all numbers less than 4. Here, we have
, the random variable
as the outcome of rolling the dice, and we define the event as
.

Conditional probability
One of the essential definitions in probability theory is the conditional probability. This is because a lot of events depends on other precedent events or available partial information. Recognizing and calculating this dependency can lead to a more precise probability estimation. As below figure, how many layers should you wear, depends on the weather!!


Let’s have an example. Assume we toss a dice and then flip a fair coin. What is the probability of getting number one (Event ) in tossing the dice and getting heads (Event
) after flipping the coin? Clearly
. So, the probability of getting number one (in tossing the dice) and getting heads (in flipping the coin) is as below:
Let’s take a look at a conditional situation. Assume we want to calculate the probability of having heads (in flipping the coin) if we get one (in tossing the dice)? This is a conditional statement. Basically, is conditioned on happening
:
The mathematical formulation of the conditional probability of two events is as below:
Bayes’ Rule
In order to explain the Bayes’ rule, let’s start with something simple as a special case. Assume we have two events and
. Do you agree with the following statement?
NOTE: and
events are mutually exclusive. Think why?
Now, let’s calculate the probability of event :
(1)
Above, we used the conditional probability rules to expand the event intersections. The above equation asserts that the probability of event is a weighted average of the conditional probability of
given that
has occurred and the conditional probability of
given that
has not happened. This is a very useful formula as a lot of times, directly calculating the probability of an event such as
may not be easy or even possible. This rule conditions the likelihood of an event on different events.
The above special case can be extended to the below more general rule called the Bayes’ rule.
The concept of independence
We previously discussed conditional probabilities. You learn the concept of probability of an event dependent to another event
as
. Intuitive, we can infer that if
and
are independent, then
does not depend on
, and it simply equals to
. We previously investigated the formulation as well. Technically, being independent is mutual, i.e., when
, then
and vice verse. We can conclude that when
and
are independent, then
. This formulation leads to the following definition:
Conclusion
In a previous post, you learned about what is probability and some mathematical background. In this post, you acquire knowledge about the fundamentals of probability theory and its key concepts. What you learned so far, aimed to prepare you to utilize probability notions and further strengthen your background for more advanced probabilistic concepts in Machine Learning. Do you have any questions or suggestions? Feel free to comment and share your point of view.