Matrix Eigendecomposition, its Importance, and the Applications

This tutorial is dedicated to explaining the concept of matrix decomposition, its definition, and the process. You will learn how you can decompose a matrix to its constituent elements.

What is going to be the benefit of decomposing a matrix? What does that mean? When we decompose anything, we break it into its constituent elements. Assume we are going to disintegrate a tool (a car or a watch!). Such action helps us to understand the core particles and their tasks. Furthermore, it helps to have a better understanding of how that specific tool works and its characteristics! Assume that the tool is a matrix which we would like to decompose. There are different approaches to decompose a matrix. However, perhaps the most commonly used one is matrix eigendecomposition which is decomposing a matrix using its eigenvectors and eigenvalues.

In this tutorial, you will learn:

The definition of eigendecomposition
The concepts of eigenvectors and eigenvalues
The benefits of decomposing a matrix
The important properties associated with matrix decomposition
How to do it in Python and Numpy

Before You Move On

You may find the following resources helpful to better understand the concept of this article:

Python Tutorials – A FREE Video Course: You will become familiar with Python and its syntax.
Numpy – An Introduction to a Great Package for Linear Algebra: Numpy is one of the best scientific computing packages for Linear Algebra! You always need it for Machine Learning as you always need Linear Algebra for Machine Learning!
The Remarkable Importance of Linear Algebra in Machine Learning: This article talks about why you should care about Linear Algebra if you want to master Machine Learning.
Basic Linear Algebra Definitions that You Hear Every Day: Covers the primary and most frequently used Linear Algebra definitions in Machine Learning.
Matrix Operations In Practice: Everything you need to know about matrices and their operations.

The Definition of Matrix Eigendecomposition

In this section, I am going to show you the definition of eigendecomposition and the subsequent concepts necessary to understand it.

Eigenvector and Eigenvalue

Before we move on, we should know the definition of eigenvector and eigenvalue. The definition of eigenvector and eigenvalue are somehow connected.

Definition: Assuming we have the square matrix of $\mathbf{A} \in \mathbb{R}^{N \times N}$ . The nonzero vector $\mathbf{v} \in \mathbb{R}^{N \times 1}$ is an eigenvector and scalar $\lambda$ is its associated eigenvalue if we have:

$\mathbf{A}\mathbf{v} = \lambda \mathbf{v}$

From the above definition, it is clear than if $\mathbf{v}$ is an eigenvector, any vector $\alpha\mathbf{v}, \alpha \in \mathbb{R}$ is also an eigenvector with the same eigenvalue $\lambda$ . Therefore, if we have one eigenvector, then we have infinite ones!

Due to that, it is customary to only work with eigenvectors that have unit norm. It is simple to construct an eigenvector with the unit norm. Assume $\mathbf{v}$ is our eigenvector. Then, the following vector is also an eigenvector with the unit norm:

$\mathbf{w}=\alpha \mathbf{v} = \frac{1}{||\mathbf{v}||}\mathbf{v}$

where $||\mathbf{v}||$ is the norm of vector $\mathbf{v}$ . We usually consider the euclidean norm.

The Process

Here, I want to explain how we decompose a matrix to its constituent elements and we call it the eigendecomposition of a matrix.

Matrix Eigendecomposition: Assuming we have the square matrix of $\mathbf{A} \in \mathbb{R}^{N \times N}$ which has N linear independent eigenvectors $\mathbf{v}^{i}, i \in {1,\hdots,N}$ . Then, we can factorize matrix $\mathbf{A}$ as below:

$\mathbf{A} = \mathbf{V} \Lambda \mathbf{V}^{-1}$

where $\mathbf{V} \in \mathbb{R}^{N \times N}$ is the square matrix whose $j^{th}$ column is the eigenvector $\mathbf{v}^{j}$ of $\mathbf{A}$ , and $\Lambda$ is the diagonal matrix whose diagonal elements are the corresponding eigenvalues, $\Lambda_{jj}=\lambda_{j}$ .

Above, we basically concatenate eigenvectors $\mathbf{v}^{i}$ to form the $\mathbf{V}$ matrix as below:

$\begin{bmatrix} | & | & \hdots & | \\ | & | & \hdots & | \\ \mathbf{v}^1 & \mathbf{v}^2 & \hdots & \mathbf{v}^N \\ | & | & \hdots & | \\ | & | & \hdots & | \end{bmatrix}$

Discussion on Matrix Eigendecomposition

Note that we can only factorize diagonalizable matrices as above. But the question is what is a diagonalizable matrix?

Diagnolizable Matrix: Assuming we have the square matrix of $\mathbf{A} \in \mathbb{R}^{N \times N}$ . It is called diagonalizable or nondefective if there exists an invertible matrix $\mathbf{H}$ such that $\mathbf{H}\mathbf{A} \mathbf{H}^{-1}$ is a diagonal matrix.

I previously mentioned a matrix is invertible if it is non-singular! Now, let’s have a more precise definition of a matrix being singular or non-singular.

Singular Matrix: Assume we have the square matrix of $\mathbf{A} \in \mathbb{R}^{N \times N}$ . It is called singular if and only if any of the eigenvalues ( $\lambda$ ) are zero.

Under some circumstances, we can calculate the matrix inverse using the decomposition.

Matrix Inverse: Assume we have the square matrix $\mathbf{A} \in \mathbb{R}^{N \times N}$ , it can be eigendecomposed and it is nonsingular. Therefore, we can calculate its inverse as below:

$\mathbf{A}^{-1} = \mathbf{V}\mathbf{\Lambda}^{-1} \mathbf{V}^{-1}$

Since $\Lambda$ is diagonal, it inverse $\Lambda^{-1}$ is also diagnoal and we can calculate it as:

${\Lambda}^{-1}_{jj}=\frac{1}{\lambda_{j}}$

One Special Matrix Type and its Decomposition

We are interested to investigate a special kind of matrix: Real symmetric matrix. A real symmetric matrix is basically a symmetric matrix in which all elements belong to the space of real numbers $\mathbb{R}$ .

For the real symmetric matrix of $\mathbf{A} \in \mathbb{R}^{N \times N}$ , the eigenvalues are real numbers and we can choose eigenvectors in a way that they are orthogonal to each other. Then, we can factorize matrix $\mathbf{A}$ as below:

$\mathbf{A} = \mathbf{Q} \Lambda \mathbf{Q}^{T}$

where $\mathbf{Q}$ is an orthogonal matrix whose columns are the eigenvectors of $\mathbf{A}$ , and $\Lambda$ is a diagonal matrix whose diagonal elements are the corresponding eigenvalues, ${\Lambda}_{jj}=\lambda_{j}$ .

Useful Properties

So far, I explained the concepts and how we can decompose a matrix. Let’s see how we can leverage it. Assuming $\mathbf{A} \in \mathbb{R}^{N \times N}$ :

The determinant of the matrix $\mathbf{A}$ equals the product of its eigenvalues.
The trace of the matrix $\mathbf{A}$ equals the summation of its eigenvalues.
If the eigenvalues of $\mathbf{A}$ are $\lambda_{i}$ , and $\mathbf{A}$ is non-singular, then the eigenvalues of $\mathbf{A}^{-1}$ are simply $\frac{1}{\lambda_{i}}$ .
The eigenvectors of $\mathbf{A}^{-1}$ are the same as the eigenvectors of $\mathbf{A}$ .

Do it in Python and Numpy

Now, let’s do some practical work. I want to use Python and Numpy to compute eigenvalues and eigenvectors.

# Import Numpy library
import numpy as np
# Define random 4x4 matrix using np.array
# Ref: https://docs.scipy.org/doc/numpy-1.15.1/reference/generated/numpy.random.randint.html
N=4
A = np.random.randint(10, size=(N, N))
print('A:\n',A)
# Eigendecomposition
eigenvalues,eigenvectors= np.linalg.eig(A)
# Show values
print('Eigenvalues:', eigenvalues)
print('Eigenvectors:', eigenvectors)
# Create the diagonal matrix of \Lambda
Lambda = np.diag(eigenvalues)
# Create V, the matrix of eigenvectors
V = eigenvectors
# Check and Confirm the decomposition
A_ = np.matmul(np.matmul(V,Lambda),np.linalg.inv(V))
print('Computing A with decomposed elements:\n', A_)

Run the above code to see the results. Pretty simple. Right? In the above code, line 24 aims to confirm if by using the decomposed elements $\mathbf{A} = \mathbf{V} \Lambda \mathbf{V}^{-1}$ we can reconstruct $\mathbf{A}$ . What is your observation?

Conclusion

In this tutorial, you learned about decomposing a matrix to its constituent elements using its eigenvectors and eigenvalues. If I be honest with you, you may rarely need this concept in coding Machine Learning projects, BUT it does not mean it is NOT important! On the contrary, matrix decomposition is one of the most critical concepts in Linear Algebra, which is essential when you desire to dig into a Machine Learning problem. My goal is to cover whatever you may encounter in Machine Learning. If you want to know the applicable Linear Algebra in Machine Learning, trust me, you need to know matrix decomposition. Do you have any questions? Do you see any ambiguous concept here? Anything missing or wrong? Feel free to comment and ask as it will help me, yourself, and the others to learn better.