This tutorial is dedicated to explaining the concept of matrix decomposition, its definition, and the process. You will learn how you can decompose a matrix to its constituent elements.
What is going to be the benefit of decomposing a matrix? What does that mean? When we decompose anything, we break it into its constituent elements. Assume we are going to disintegrate a tool (a car or a watch!). Such action helps us to understand the core particles and their tasks. Furthermore, it helps to have a better understanding of how that specific tool works and its characteristics! Assume that the tool is a matrix which we would like to decompose. There are different approaches to decompose a matrix. However, perhaps the most commonly used one is matrix eigendecomposition which is decomposing a matrix using its eigenvectors and eigenvalues.
In this tutorial, you will learn:
- The definition of eigendecomposition
- The concepts of eigenvectors and eigenvalues
- The benefits of decomposing a matrix
- The important properties associated with matrix decomposition
- How to do it in Python and Numpy
Before You Move On
You may find the following resources helpful to better understand the concept of this article:
- Python Tutorials – A FREE Video Course: You will become familiar with Python and its syntax.
- Numpy – An Introduction to a Great Package for Linear Algebra: Numpy is one of the best scientific computing packages for Linear Algebra! You always need it for Machine Learning as you always need Linear Algebra for Machine Learning!
- The Remarkable Importance of Linear Algebra in Machine Learning: This article talks about why you should care about Linear Algebra if you want to master Machine Learning.
- Basic Linear Algebra Definitions that You Hear Every Day: Covers the primary and most frequently used Linear Algebra definitions in Machine Learning.
- Matrix Operations In Practice: Everything you need to know about matrices and their operations.
The Definition of Matrix Eigendecomposition
In this section, I am going to show you the definition of eigendecomposition and the subsequent concepts necessary to understand it.
Eigenvector and Eigenvalue
Before we move on, we should know the definition of eigenvector and eigenvalue. The definition of eigenvector and eigenvalue are somehow connected.
Definition: Assuming we have the square matrix of . The nonzero vector is an eigenvector and scalar is its associated eigenvalue if we have:
From the above definition, it is clear than if is an eigenvector, any vector is also an eigenvector with the same eigenvalue . Therefore, if we have one eigenvector, then we have infinite ones!
Due to that, it is customary to only work with eigenvectors that have unit norm. It is simple to construct an eigenvector with the unit norm. Assume is our eigenvector. Then, the following vector is also an eigenvector with the unit norm:
where is the norm of vector . We usually consider the euclidean norm.
Here, I want to explain how we decompose a matrix to its constituent elements and we call it the eigendecomposition of a matrix.
Matrix Eigendecomposition: Assuming we have the square matrix of which has N linear independent eigenvectors . Then, we can factorize matrix as below:
where is the square matrix whose column is the eigenvector of , and is the diagonal matrix whose diagonal elements are the corresponding eigenvalues, .
Above, we basically concatenate eigenvectors to form the matrix as below:
Discussion on Matrix Eigendecomposition
Note that we can only factorize diagonalizable matrices as above. But the question is what is a diagonalizable matrix?
Diagnolizable Matrix: Assuming we have the square matrix of . It is called diagonalizable or nondefective if there exists an invertible matrix such that is a diagonal matrix.
I previously mentioned a matrix is invertible if it is non-singular! Now, let’s have a more precise definition of a matrix being singular or non-singular.
Singular Matrix: Assume we have the square matrix of . It is called singular if and only if any of the eigenvalues () are zero.
Under some circumstances, we can calculate the matrix inverse using the decomposition.
Matrix Inverse: Assume we have the square matrix , it can be eigendecomposed and it is nonsingular. Therefore, we can calculate its inverse as below:
Since is diagonal, it inverse is also diagnoal and we can calculate it as:
One Special Matrix Type and its Decomposition
We are interested to investigate a special kind of matrix: Real symmetric matrix. A real symmetric matrix is basically a symmetric matrix in which all elements belong to the space of real numbers .
For the real symmetric matrix of , the eigenvalues are real numbers and we can choose eigenvectors in a way that they are orthogonal to each other. Then, we can factorize matrix as below:
where is an orthogonal matrix whose columns are the eigenvectors of , and is a diagonal matrix whose diagonal elements are the corresponding eigenvalues, .
So far, I explained the concepts and how we can decompose a matrix. Let’s see how we can leverage it. Assuming :
- The determinant of the matrix equals the product of its eigenvalues.
- The trace of the matrix equals the summation of its eigenvalues.
- If the eigenvalues of are , and is non-singular, then the eigenvalues of are simply .
- The eigenvectors of are the same as the eigenvectors of .
Do it in Python and Numpy
Now, let’s do some practical work. I want to use Python and Numpy to compute eigenvalues and eigenvectors.
# Import Numpy library import numpy as np # Define random 4x4 matrix using np.array # Ref: https://docs.scipy.org/doc/numpy-1.15.1/reference/generated/numpy.random.randint.html N=4 A = np.random.randint(10, size=(N, N)) print('A:\n',A) # Eigendecomposition eigenvalues,eigenvectors= np.linalg.eig(A) # Show values print('Eigenvalues:', eigenvalues) print('Eigenvectors:', eigenvectors) # Create the diagonal matrix of \Lambda Lambda = np.diag(eigenvalues) # Create V, the matrix of eigenvectors V = eigenvectors # Check and Confirm the decomposition A_ = np.matmul(np.matmul(V,Lambda),np.linalg.inv(V)) print('Computing A with decomposed elements:\n', A_)
Run the above code to see the results. Pretty simple. Right? In the above code, line 24 aims to confirm if by using the decomposed elements we can reconstruct . What is your observation?
In this tutorial, you learned about decomposing a matrix to its constituent elements using its eigenvectors and eigenvalues. If I be honest with you, you may rarely need this concept in coding Machine Learning projects, BUT it does not mean it is NOT important! On the contrary, matrix decomposition is one of the most critical concepts in Linear Algebra, which is essential when you desire to dig into a Machine Learning problem. My goal is to cover whatever you may encounter in Machine Learning. If you want to know the applicable Linear Algebra in Machine Learning, trust me, you need to know matrix decomposition. Do you have any questions? Do you see any ambiguous concept here? Anything missing or wrong? Feel free to comment and ask as it will help me, yourself, and the others to learn better.