Basic Linear Algebra Definitions that You Hear Every Day

Do you want to start learning Machine Learning and keep hearing buzzwords related to Linear Algebra? Vector, matrix, tensor? Those are basic Linear Algebra Definitions.

Perhaps you want to learn Linear Algebra for Machine Learning and looking for a place to hit first?

In one of the previous articles, we discussed the importance of Linear Algebra in Machine Learning. Here, we introduce some of the most commonly used Linear Algebra definitions.

This article aims to accomplish the following:

Provide the basic definitions in Linear Algebra that we hear every day while dealing with Machine Learning.
Describe the connection between those basic concepts.
Demonstration of the commonly used notations.

If you already know Linear Algebra and/or familiar with basic definitions, you may still find some aspects of this article useful, such as the representation of “the commonly used notations.”

Let’s get started!

Before You Move On

You may find the following resource helpful to better understand the concept of this article:

The Remarkable Importance of Linear Algebra in Machine Learning: This article talks about why you should care about Linear Algebra if you want to master Machine Learning.

Scalar and Vector

By using mathematical definition, a scalar is “an element of a field, which is used to define a vector space, usually the field of real numbers.” Simply speaking, a scalar is just a number. We usually denote a scalar with a lower case symbol, such as $a$ .

It is worth to have a comparison between scalars and vectors. You are familiar with measures such as the velocity of a moving car and the direction of moving, which have both quantity and direction. Such elements are called vector as opposed to scalars, which have magnitude only. We denote vectors with bold symbols such as $\mathbf{a}$ .

Linear Algebra Definitions — The red arrow v is the vector and a and b are scalars denoting the magnitudes of v in horizontal and vertical directions.

Subscripts usually denote the elements of the vector. For example the second element of the vector $\mathbf{v}$ is denoted as $v_2$ . In general, we show a k-dimensional vector as an array of elements:

$\mathbf{v} = \begin{bmatrix} v_1\\ v_2\\ …\\ v_{k-1}\\ v_k \end{bmatrix}$

NOTATION: We denote a k-dimensional vector as $\mathbf{v} \in \mathbb{R}^k$ .

Matrix

In mathematics, a matrix is a rectangular array of elements, such as numbers organized in rows and columns. Usually, the matrices are denoted with bold uppercase, such as $\mathbf{M}$ . An example of a matrix is below:

$\mathbf{A} = \begin{bmatrix} 1 & -1 &3 \\ 4& 11 & -0.2 \end{bmatrix}$

We denote the matrix elements with subscripts. In the matrix $\mathbf{M}$ , the element in $i^{th}$ row and $j^{th}$ column is denoted as $\mathbf{M}_{i,j}$ . So you can easily say for the example above, $\mathbf{M}_{1,2}=-1$ and $\mathbf{M}_{2,3}=-0.2$ .

Another important notation is when we desire to refer a row or a column as a whole, i.e., all elements in that particular row/column. In this case, we use “:” sign in the subscript. For example, assume we want to refer to all elements of $i^{th}$ row for matrix $\mathbf{M}$ . We denote that row with $\mathbf{M}_{i,:}$ . For matrix $\mathbf{A}$ , $\mathbf{A}_{1,:}$ is as below:

$\mathbf{A}_{1,:} = \begin{bmatrix} 1 & -1 &3 \end{bmatrix}$

NOTE: We usually represent a vector using one column and multiple rows of elements. Henceforth, we can call a vector of size $k$ as a $k \times 1$ matrix. In general, we can informally say vectors are special kind of matrices which are 1-dimensional.

NOTATION: We denote a matrix as $\mathbf{M} \in \mathbb{R}^{m \times n}$ in which $m$ and $n$ are the number of rows and columns, respectively.

Tensor

We usually represent data in machine learning, numerically. Tensors are used for such numerical representation. A tensor is simply a container that holds data in N-dimensional space. A Tensor can host and represent multiple matrices. An example is shown below:

$\begin{bmatrix} \begin{bmatrix} 10 & 3 &3.1 \\ 6 & 12 & -0.2 \end{bmatrix} & \begin{bmatrix} 1.0 & -1 &3 \\ 4& 11 & -2 \end{bmatrix}\\ \begin{bmatrix} 1 & -2 &7 \\ 4.& 1.1 & 81.1 \end{bmatrix}& \begin{bmatrix} 1.1 & -2 &-2 \\ 4& 4 & -5 \end{bmatrix} \end{bmatrix}$

You can consider a simple 3D tensor as the data cube depicted below:

In this tensor, each color represents a matrix and the concatenation of the matrices, form the tensor.

Conclusion

In this post, we described some of the most commonly used Linear Algebra definitions in Machine Learning. Scalar, vector, matrix, and tensor are what you hear the most. It was crucial to address them before we proceed to explain other concepts in Linear Algebra.

P.S. Please share with me your thoughts by commenting below. I might be wrong in what I say, and I love to know when I am wrong. Furthermore, your questions might be my questions, as well. It’s always good to become better even if being the best is impossible in our belief system. So let’s help each other to become better.

Basic Linear Algebra Definitions that You Hear Every Day

Scalar and Vector

Matrix

Tensor

Conclusion

Leave a Comment Cancel Reply