Autoencoders in Deep Learning: Components, Types and Applications

Autoencoder is a special kind of neural network in which the output is nearly same as that of the input. It is an unsupervised deep learning algorithm. We can consider an autoencoder as a data compression algorithm which performs dimensionality reduction for better visualization.

Example: Let take an example of a password. You create an account on a website, your password is encrypted and stored in the database. Now, when you try to login to that website, your encrypted password is fetched, decrypted and matched with the password your provided.

Components of an Autoencoder 

Autoencoders consists of 4 main parts:

1. Encoder: It is the layer in which the model learns how to reduce the input dimensions and compress the input data into an encoded representation. This is the part of the network that compresses the input into a latent space representation.

2. Bottleneck / Code: It is the layer that contains the compressed representation of the input data. This is the lowest possible dimensions of the input data. It decides which aspects of the data are relevant and which aspects can be thrown away.

3. Decoder: It is the layer in which the model learns how to reconstruct the data from the encoded representation to be as close to the original input as possible. The decoded image is a lossy reconstruction of the original image.

4. Reconstruction Loss: This is the method that measures how well the decoder is performing and how close the output is to the original input.

Properties of an Autoencoder

1. Unsupervised: Autoencoders are considered as an unsupervised learning technique as these don't need explicit labels to train on.

2. Data-specific: Autoencoders are only able to compress and decompress the data similar to what they have been trained on. For example, an autoencoder which has been trained on human faces, would not perform well with the images of buildings.

3. Lossy: The output of the autoencoder will not be exactly the same as the input, it will be a close but degraded representation.

How does an Autoencoder work?

Autoencoders compress the input into a latent-space representation and then reconstruct the output from this representation. We calculate the loss by comparing the input and output. This difference between the input and output is called reconstruction loss. Main objective of autoencoder is to minimize this reconstruction loss so that the output is similar to the input. To reduce this reconstruction loss, we back propagate through the network and update the weights using gradient descent algorithm. 

Autoencoder should have generalization capabilities: As a general rule of thumb, our autoencoder should be sensitive enough to recreate the original observation but insensitive enough to the training data such that the model learns a generalization. In other words, autoencoders should have some generalization capabilities. Mainly all types of autoencoders like undercomplete, sparse, convolutional and denoising autoencoders use some mechanism to have generalization capabilities.

How to increase generalization capabilities of an autoencoders?

1. Keep the code layer small so that there is more compression of data. More is the data compression, more is the generalization.

2. Limit the number of nodes in the hidden layers of the network (undercomplete autoencoders).

3. Use L1 and L2 regularization (sparse autoencoders)

4. Add random noise to the inputs and let the autoencoder recover the original noise-free data (denoising autoencoder)

Types of an Autoencoder

1. Undercomplete autoencoder: In this type of autoencoder, we limit the number of nodes present in the hidden layers of the network. In this way, it also limits the amount of information that can flow through the network which makes our model to learn only the most important attributes of the input data. 

By limiting the number of nodes in the hidden layers, we can make sure that our model does not memorize the training data and have some generalization capabilities. 

For regularization and generalization, we don't use any regularization penalty to train our model, we just limit the number of nodes in the hidden layers.

2. Sparse autoencoder: Instead of limiting the number of nodes in the hidden layers like undercomplete autoencoders, we introduce regularization techniques to regularize or penalize activations instead of weights in our loss function so that it activates only a small number of neurons in a given hidden layer. 

Individual nodes of a trained model which activate are data-dependent, different inputs will result in activations of different nodes through the network. For regularization, we can use L1 regularization or KL-divergence regularization techniques.

3. Denoising autoencoder:  Another approach towards developing a generalizable model is to slightly corrupt the input data (add some random noise) but still maintain the uncorrupted data as our target output. With this approach, our model isn't able to simply develop a mapping which memorizes the training data because our input and target output are no longer the same. In this way, we train the autoencoder to reconstruct the input from a corrupted version of it.

4. Convolutional autoencoder: It uses convoluted layers to compress images with the help of kernels (filter). Then it uses max pooling to further down-sample the image. To understand fully about CNN, you can visit my this post on CNN.

Applications of an Autoencoder

1. Signal Denoising: Denoising or noise reduction is the process of removing noise from a signal. The signal can be an image, audio or a scanned document.

2. Dimensionality Reduction for Visualization: Lesser the dimension, better the visualization. Autoencoders outperform PCA in this regard as autoencoders work really well with non-linear data while PCA is only meant for linear data.

3. Anomalies and outliers detection: Autoencoders learn to generalize the patterns. So, if anything is out of pattern, it can detect easily. For an anomaly, reconstruction loss is very high as compared to the regular data.

4. Image coloring: It is also used for image coloring.

Autoencoder Hyper-parameters

1. Code Size: It represents number of nodes in the middle layer. Smaller size results in more compression.

2. Number of Layers: An autoencoder can be as deep as we wanted to be. We can have as many layers both in encoder and decoder.

3. Number of nodes per layer: Number of nodes per layer decreases with each subsequent layer in the encoder and increases back in the decoder.

4. Loss Function: You can set any loss function like mean square error, binary cross entropy etc. If input values are in the range of 0 to 1, we typically use cross entropy otherwise we can use the mean square error. 

Loss Function is usually composed of two parts: 

1. Reconstruction Loss: measures how much is the difference between the original data and reconstructed data.

2. Regularization Penalty: adds some penalty so that the model learns generalization.

Restricted Boltzmann Machines (RBM)

Restricted Boltzmann Machines are shallow and two-layer (input and hidden) neural networks. There is no output layer unlike Autoencoders. 

The nodes or neurons are connected to each other across the layers, but no two nodes of the same layer are linked. Due to this restriction, it is called Restricted Boltzmann Machines instead of simple Boltzmann Machines. 

It is probabilistic, unsupervised, generative deep learning algorithm. RBM’s objective is to find the joint probability distribution that maximizes the log-likelihood function.

RBM’s are used for:

1. Dimensionality reduction
2. Collaborative filtering for recommender systems
3. Feature learning
4. Topic modelling
5. Helps improve efficiency of Supervised learning

For more details on RBM, please go through this and this article.

How to find Correlation Score and plot Correlation Heatmap using Seaborn Library in Python?

Lets try to find out the correlation among the variables in a dataset. Correlated variables don't provide any useful information to the model.We should remove correlated variables from the dataset for better accuracy and performance. 

We will analyze the correlation among the variables through correlation heatmap using seaborn library in Python. corr method is used to find out the correlation. Then we will also find the correlation score of the variables with respect to target variable.

Consider Ames Housing dataset. 

Step 1: Load the required libraries
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

Step 2: Load the dataset
dataset = pd.read_csv("C:/datasets/train.csv")

Step 3: Separate numeric and categorical variables
numeric_data = dataset.select_dtypes(include=[np.number])
categorical_data = dataset.select_dtypes(exclude=[np.number])

Step 4: Remove the Id column
del numeric_data['Id']

Step 5: Draw Correlation Heatmap
corr = numeric_data.corr()
plt.figure(figsize=(10, 10))
sns.heatmap(corr)

Notice the last row of this map. We can see the correlation of all the variables against SalePrice. As you can see, some variables seem to be strongly correlated with the target variable. 

Step 6: Get Correlation Score
print (corr['SalePrice'].sort_values(ascending=False)[:10]) #top 10 correlations
print (corr['SalePrice'].sort_values(ascending=False)[-5:]) #least 5 correlations

Here we see that the OverallQual feature is 79% correlated with the target variable. Overallqual feature refers to the overall material and quality of the materials of the completed house. Well, this make sense as well. People usually consider these parameters for their dream house. 

In addition, GrLivArea is 70% correlated with the target variable. GrLivArea refers to the living area (in sq ft.) above ground. The following variables show people also care about if the house has a garage, the area of that garage, the size of the basement area, etc.

What is Factor Analysis? What is the difference between Exploratory Factor Analysis and Confirmatory Factor Analysis?

Factor Analysis is a statistical techniques used for dimensionality reduction in machine learning. Factor Analysis is used to reduce a large number of variables into fewer numbers of factors (variables) based on the correlation among the variables. It tries to capture maximum variance in the data with minimum number of variables. 

You can find more details about dimensionality reduction in my following articles:

Why is Dimensionality Reduction required?
Feature Selection and Feature Extraction Techniques
Difference between Covariance and Correlation
What is Multicollinearity?

Types of Factor Analysis

There are mainly two types of Factor Analysis:

1. Exploratory Factor Analysis (EFA)
2. Confirmatory Factor Analysis (CFA)

1. Exploratory Factor Analysis: It assumes that any indicator or variable may be associated with any factor. This is the most common factor analysis used by researchers and it is not based on any prior theory. Best example of Exploratory Factor Analysis is PCA (Principal Component Analysis). 

Advantages and Disadvantages of PCA
PCA vs t-SNE

2. Confirmatory factor analysis (CFA): It is used to determine the factor and factor loading of measured variables, and to confirm what is expected on the basic or pre-established theory. CFA assumes that each factor is associated with a specified subset of measured variables.

What is Multicollinearity? What is Structural and Data Multicollinearity?

Multicollinearity is a situation in which two or more predictor (independent variables) in a model are highly correlated. 

For example, you have two explanatory variables – ‘time spent on treadmill in minutes’ and ‘calories burnt’. These variables are highly correlated as the more time you spend running on a treadmill, the more calories you will burn. Hence, there is no point in storing both as just one of them does what you require. 

Generally, if the correlation between the two independent variables is high (>= 0.8), then we drop one independent variable, otherwise it may lead to multicollinearity problem. If the degree of multicollinearity between the independent variables is high enough, it can cause problems when you fit the model and interpret the results.

Collinearity is a linear association between two explanatory variables. Two variables are perfectly collinear if there is an exact linear relationship between them.

Types of Multicollinearity

1. Structural Multicollinearity: This type of multicollinearity occurs when we create a variable based on the another variable while creating the model. For example, there is a variable say "x", and you create another variable based on the "x" say "y" where y=cx (c is any constant). In this case, both "x" and "y" are correlated variables.

2. Data Multicollinearity: This type of multicollinearity is present in the data itself. So, we need to identify it during the data wrangling process.

How to remove correlated variables?

Following techniques are used to handle multicollinearity problem in a dataset:

1. PCA (Principal Component Analysis)
2. SVD (Singular value Decomposition)

Related: Covariance vs Correlation

Data Wrangling Techniques: Steps involved in Data Wrangling Process

Data Wrangling is the first step we take while creating a Machine Learning model. This is the main step in which we prepare data for a Machine Learning algorithm. This step is very crucial and takes up to 60 to 80 percent of time. 

In Data Wrangling, we convert the raw data into a suitable format which we can input to any Machine Learning algorithm. Data Wrangling and Data Preprocessing terms are used interchangeably. Data Wrangling is an art. You should have a lot of patience while making your data fit for Machine Learning algorithm. 

Lets see what are the various steps one should take while Data Wrangling?

1. Drop unnecessary columns

1A. Drop the columns which contain IDs, Names etc. 

For example, in Titanic dataset, we can easily drop Passenger Id, Passenger Name and Ticket Number columns which are not required for any kind of prediction. Read more...

1B. Drop the columns which contain a lot of null or missing values

The columns which contain around 75% of missing values should be dropped from the dataset. For example, in Titanic dataset, cabin column contains 687 null values out of 891 observations (77% missing values). So, it makes sense to drop this column from the dataset. Read more... 

Visualize missing values using Bar Plot

1C. Drop the columns which have low variance 

You can drop a variable with zero or low variance because the variables with low variance will not affect the target variable. If all the values in a variable are approximately same, then you can easily drop this variable. 

For example, if almost all the values in a numerical variable contain 1, then you can drop this variable.

2. Remove rows containing null values

If there are around 10-15% observations which contain null values, we can consider removing those observations. Read more...

3. Remove noise

Noise a is data that is meaningless, distorted and corrupted. Noise includes invalid values, outliers and skewed values in the dataset. We need to remove this noise before supplying this dataset to an algorithm. Domain knowledge plays an important role in identifying and removing the noisy data.

3A. Replace invalid values

Many times there are invalid values present in the dataset. For example, in Pima Indian Diabetes dataset, there are zero values for Blood Pressure, Glucose, Insulin etc. which is invalid. So, we need to replace these values with some meaningful values. Domain knowledge plays a crucial role in identifying the invalid values. Read more...

3B. Remove outliers

It is very important to remove outliers from the dataset as these outliers adversely affect the accuracy of the algorithms.

What are outliers? How to remove them?

3C. Log Transform Skewed Variables

We should check distribution for all the variables in the dataset and if it is skewed, we should use log transformation to make it normal distributed.

What is Skewnesss? How to visualize it with Histogram and how to remove it?
How to visualize skewness of numeric variables by plotting histograms?
Log Transforming the Skewed Data to get Normal Distribution

4. Impute missing values

Step 2 is usually not recommended as you may lose significant data. So, better try imputing the missing values with some meaningful values.

For numeric columns, you can impute the missing values with mean, median or mode. Read more...

Implementation of Imputer in Python

For categorical columns, you can impute the missing values by introducing a new category or with the category which is most frequently used. Read more...

5. Transform non-numeric variables to numeric variables

There are numeric and non-numeric variables in the dataset. We need to handle these differently. 

How to separate numeric and categorical variables?

5A. Transform categorical variables to dummy variables

To transform categorical variables to dummy variables, we can use LabelEncoder, OneHotEncoder and get_dummies methods present in Scikit Learn and Pandas library in Python.

5B. Transform date variables to numeric variables

By default, dates are treated as string values. We should convert it to numeric one.

How to convert dates into numbers in the dataset?

6. Feature Engineering

Feature Engineering involves Binning, Scaling (Normalization and Standardization), Dimensionality Reduction etc. We need to standardize and normalize all the features in the dataset before running any algorithm on the dataset. Standardization and Normalization are the feature scaling techniques which bring down all the values on the same scale and range. Features should be numeric in nature.

Binning Technique
Importance of Feature Scaling
Standardization vs Normalization
Implement Normalization in Python
Which algorithms require scaling and which not?

7. Dimensionality Reduction

Dimensionality reduction is required to remove the correlated variables and maximize the performance of the model. Basic techniques used for dimensionality reduction are:

• PCA (Principal Component Analysis)
• SVD (Singular Vector Decomposition)
• LDA (Linear Discriminant Analysis)
• MDS (Mulit-dimension Scaling)
• t-SNE (t-Distributed Stochastic Neighbor Embedding)
• ICA (Independent Component Analysis)

Please go through my previous posts on dimensionality reduction to understand the need of this step.

Multicollinearity
Covariance vs Correlation
Visualize correlation score using Heatmap
Feature Selection and Feature Extraction
Need of Dimensionlity Reduction
Factor Analysis
PCA, t-SNE, PCA vs t-SNE 
Implement PCA in Python

8. Splitting the dataset into training and testing data

We should not use the entire dataset to train a model. We should keep aside around 20% of data to test the accuracy of the model. So, usually we maintain a ratio of 80:20 between training and testing datasets.

What is t-SNE? How does it work using t-Distribution?

t-SNE stands for t-Distributed Stochastic Neighbor Embedding. It is a non-linear dimensionality reduction algorithm. t-SNE uses normal distribution (in higher dimension) and t-Distribution (in lower dimension) to reduce the dimensions of the dataset. We will see it in detail. 

As per documentation:

“t-Distributed stochastic neighbor embedding (t-SNE) minimizes the divergence between two distributions: a distribution that measures pairwise similarities of the input objects and a distribution that measures pairwise similarities of the corresponding low-dimensional points in the embedding”.

t-SNE is a technique to convert high-dimensional data into lower dimensional data while keeping the relative similarity of the data points as close to the original (in high dimensional space) as possible.

Lets see how does t-SNE work? I will just illustrate a high level understanding of the algorithm (because even I don't understand the complex mathematics behind it :)).

Higher Dimension

Step 1: Picks a data point (say P1), calculates its Euclidean distance from a neighboring data point (say P2) and converts this distance into the conditional probability using normal distribution. 

Conditional probability represents the similarity between the pairs of data points.

Step 2: Again it calculates the distance of P1 from other neighboring point (say P3) and does the same as in Step 1.

It keeps doing same thing for all the data points. In this way t-SNE keeps computing pairwise conditional probabilities for each data point using normal distribution in higher dimension.

To summarize, t-SNE measures the similarity between each and every pair of data points. Similar data points will have more value of similarity and the different data points will have less value. Then it converts that similarity distance to the conditional probability according to the normal distribution. It also creates a similarity matrix (say S1).

Lower Dimension

Step 3: t-SNE arranges all of the data points randomly on the required lower dimension (say two dimensional space).

Step 4: It again does the same calculations for all the data points in the lower dimension as it did for all the data points in the higher dimension in Step 1 and Step 2. The only difference is that it uses t-Distribution in this case instead of normal distribution. That is why, it is called t-SNE instead of simple SNE.

It also creates a similarity matrix in lower dimension (say S2).

Now t-SNE compares the similarity matrix S1 and S2 and tries to minimize the difference such that all the pairs have a similar probability distribution (both in higher and lower dimension). Gradient Descent is used with Kullback Leibler Divergence between the two distributions as a cost function. To measure the minimization of sum of difference of conditional probability, SNE minimizes the sum of Kullback-Leibler divergences of overall data points using a Gradient Descent method.

Difference between normal distribution and t-Distribution

t-Distribution is a lot like a normal distribution. The only difference is that t-distribution is not as tall as normal distribution in middle but its tails are taller at the ends. 

Why is t-Distribution used instead of normal distribution in lower dimension because without it the clusters would all clump up in the middle and will be harder to visualize.

Related:

Advantages and Disadvantages of t-SNE over PCA (PCA vs t-SNE)
Advantages and Disadvantages of Principal Component Analysis
Dimensionality Reduction: Feature Selection and Feature Extraction
Why is Dimensionality Reduction required in Machine Learning?

Advantages and Disadvantages of t-SNE over PCA (PCA vs t-SNE)

Both PCA (Principal Component Analysis) and t-SNE (t-Distributed Stochastic Neighbor Embedding) are the dimensionality reduction techniques in Machine Learning and efficient tools for data exploration and visualization. In this article, we will compare both PCA and t-SNE. We will see the advantages and disadvantages / limitations of t-SNE over PCA.

Advantages of t-SNE

1. Handles Non Linear Data Efficiently: PCA is a linear algorithm. It creates Principal Components which are the linear combinations of the existing features. So, it is not able to interpret complex polynomial relationships between features. So, if the relationship between the variables is nonlinear, it performs poorly. On the other hand, t-SNE works well on non-linear data. It is a very effective non-linear dimensionality reduction algorithm. 

PCA tries to place dissimilar data points far apart in a lower dimension representation. But in order to represent high dimension data on low dimension, non-linear manifold, it is important that similar datapoints must be represented close together, which is not what PCA does. This is done efficiently by t-SNE. So, it can efficiently capture the structure of trickier manifolds in the dataset.

2. Preserves Local and Global Structure: t-SNE is capable to preserve local and global structure of the data. This means, roughly, that points which are close to one another in the high-dimensional dataset, will tend to be close to one another in the low dimension. On the other hand, PCA finds new dimensions that explain most of the variance in the data. So, it cares relatively little about local neighbors unlike t-SNE.

Disadvantages of t-SNE

1. Computationally Complex: t-SNE involves a lot of calculations and computations because it computes pairwise conditional probabilities for each data point and tries to minimize the sum of the difference of the probabilities in higher and lower dimensions.

“Since t-SNE scales quadratically in the number of objects N, its applicability is limited to data sets with only a few thousand input objects; beyond that, learning becomes too slow to be practical (and the memory requirements become too large)”.

t-SNE has a quadratic time and space complexity in the number of data points. This makes it particularly slow, computationally quite heavy and resource draining while applying it to datasets comprising of more than 10,000 observations. 

Use both PCA and t-SNE: Solution of the above problem is to use both PCA and t-SNE in conjunction. So, if you have thousands of features in a dataset, don't use t-SNE for dimensionality reduction in the first step. First use PCA to reduce the dimensions to a reasonable number of features and then run t-SNE to further reduce the dimensionality.

2. Non-deterministic: Sometimes different runs with same hyper parameters may produce different results. So, you won't get exactly the same output each time you run it, though the results are likely to be similar.

3. Requires Hyperparameter Tuning: t-SNE involves hyperparameters to be tuned unlike PCA (does not have any hyperparameter). Handing hyperparameters incorrectly may lead to unwanted results.

4. Noisy Patterns: Patterns may be found in random noise as well, so multiple runs of the algorithm with different sets of hyperparameter must be checked before deciding if a pattern exists in the data.

Related:

What is t-SNE? How does it work using t-Distribution?
Advantages and Disadvantages of Principal Component Analysis

Dimensionality Reduction: Feature Selection and Feature Extraction
Why is Dimensionality Reduction required in Machine Learning?

Implement PCA in Python using Scikit Learn Library

We calculate Principal Components on a dataset using the PCA() class in the Scikit-Learn library. 

While creating the PCA() class, we can pass following parameters in the constructor:

1. Number of Principal Components we need to consider OR
2. Amount of Variance we need to retain

If we don't pass any parameter in the constructor, all the Principal Components (which are usually equal to number of features in the dataset) are used to create a model. We should refrain from this approach as this will not serve the purpose of PCA.

We will see it in detail in this article. We will use bank note authentication dataset and implement Random Forest to identify the authenticity of the currency. For theory on PCA, you can go through this article.

You can download the dataset from here and my Jupyter notebook implementing PCA from here.

Step 1: Import the required Python libraries like pandas, numpy and sklearn

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import confusion_matrix, accuracy_score, classification_report

Step 2: Load and examine the dataset

names = ['Variance', 'Skewness', 'Curtosis', 'Entropy', 'Class']
dataset = pd.read_csv('bank_note_authentication.csv', names=names)
dataset.shape
dataset.head()

Step 3: Mention X and Y axis

X = dataset.drop('Class', axis=1)  
y = dataset['Class']    

X contains the list of attributes
Y contains the list of labels

Step 4: Split the dataset into training and testing dataset

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.20, random_state=0) 

Step 5: Feature scaling

standardScaler = StandardScaler()  
X_train = standardScaler.fit_transform(X_train)  
X_test = standardScaler.transform(X_test)

Step 6.1: Apply PCA (Method 1)

pca = PCA()  
X_train = pca.fit_transform(X_train)  
X_test = pca.transform(X_test)  

explained_variance = pca.explained_variance_ratio_
print(explained_variance)

Output:
[0.54578721 0.31931922 0.09136961 0.04352396]

Note: Here I have not passed any parameter to the PCA class constructor. So, it will return 4 Principal Components (equal to the number of features in our dataset). 

explained_variance_ratio_ returns the variance delivered by each Principal Component.

From the output, you can see that first PC accounts for 54% of variance and second PC accounts for around 32% of variance. So, both Principal Components account for around 86% of the variance in the dataset. So, instead of using all the four features, we can use only two or three principal component to build our model. 

This example does not highlight the great importance of PCA as we have only 4 features in our dataset. But in real world, we can easily have 40 features or 40K features or more. In these scenarios, PCA does a fantastic job. In these cases, instead of using 40K features, we will need to just use some hundreds of Principal Components which will drastically increase the performance of our model.

Step 6.2: Apply PCA (Method 2)

pca = PCA(n_components=2)
X_train = pca.fit_transform(X_train)  
X_test = pca.transform(X_test)

PC_DataFrame = pd.DataFrame(data = X_train, columns = ['PC1', 'PC2'])
print(PC_DataFrame)

Note: From the step 6.1, it is clear that if we use only two Principal Components, we can still cover 86% of the variance. So, I passed number of components as 2 in the constructor. 

Step 6.3: Apply PCA (Method 3)

pca = PCA(0.86) 
X_train = pca.fit_transform(X_train)  
X_test = pca.transform(X_test)
print(pca.components_)

Note: This is the another way of doing PCA on the dataset. If I want to retain 86% of variance in my dataset and don't want to bother about the number of Principal Components, I can use this approach.

components_ returns the number of Principal Components considered to achieve the variance of 86%. In this case, number of Principal Components are 2.

So, we can use either 6.2 or 6.3 approach to implement PCA.

The number of Principal Components to retain in a feature set depends on several conditions such as storage capacity, training time, performance etc. In some datasets, where all the features are contributing equally to the overall variance, all the principal components are crucial to the predictions and none can be ignored. A general rule of thumb is to take number of Principal Components that contribute to significant variance and ignore those with diminishing variance returns. 

Step 7: Create and fit the model

model = RandomForestClassifier(max_depth=2, random_state=0)  
model.fit(X_train, y_train)

Step 8: Predict from the model

y_pred = model.predict(X_test) 

The y_pred is a numpy array that contains all the predicted values for the input values in the X_test.

Lets see the difference between the actual and predicted values.

df=pd.DataFrame({'Actual':y_test, 'Predicted':y_pred})  
df 

Step 9: Check the accuracy

confusionMatrix = confusion_matrix(y_test, y_pred)
accuracyScore = accuracy_score(y_test, y_pred)
classificationReport = classification_report(y_test, y_pred)
print(confusionMatrix)
print(accuracyScore * 100)
print(classificationReport)

Advantages and Disadvantages of Principal Component Analysis in Machine Learning

Principal Component Analysis (PCA) is a statistical techniques used to reduce the dimensionality of the data (reduce the number of features in the dataset) by selecting the most important features that capture maximum information about the dataset. 

The features are selected on the basis of variance that they cause in the output. Original features of the dataset are converted to the Principal Components which are the linear combinations of the existing features. The feature that causes highest variance is the first Principal Component. The feature that is responsible for second highest variance is considered the second Principal Component, and so on. 

In simple words, Principal Component Analysis is a method of extracting important features (in the form of components) from a large set of features available in a dataset. 

PCA finds the directions of maximum variance in high-dimensional data and project it onto a smaller dimensional subspace while retaining most of the information. By projecting our data into a smaller space, we’re reducing the dimensionality of our feature space.

Following are some of the advantages and disadvantages of Principal Component Analysis:

Advantages of Principal Component Analysis

1. Removes Correlated Features: In a real world scenario, this is very common that you get thousands of features in your dataset. You cannot run your algorithm on all the features as it will reduce the performance of your algorithm and it will not be easy to visualize that many features in any kind of graph. So, you MUST reduce the number of features in your dataset. 

You need to find out the correlation among the features (correlated variables). Finding correlation manually in thousands of features is nearly impossible, frustrating and time-consuming. PCA does this for you efficiently.

After implementing the PCA on your dataset, all the Principal Components are independent of one another. There is no correlation among them.

2. Improves Algorithm Performance: With so many features, the performance of your algorithm will drastically degrade. PCA is a very common way to speed up your Machine Learning algorithm by getting rid of correlated variables which don't contribute in any decision making. The training time of the algorithms reduces significantly with less number of features.

So, if the input dimensions are too high, then using PCA to speed up the algorithm is a reasonable choice. 

3. Reduces Overfitting: Overfitting mainly occurs when there are too many variables in the dataset. So, PCA helps in overcoming the overfitting issue by reducing the number of features.

4. Improves Visualization: It is very hard to visualize and understand the data in high dimensions. PCA transforms a high dimensional data to low dimensional data (2 dimension) so that it can be visualized easily. 

We can use 2D Scree Plot to see which Principal Components result in high variance and have more impact as compared to other Principal Components. 

Even the simplest IRIS dataset is 4 dimensional which is hard to visualize. We can use PCA to reduce it to 2 dimension for better visualization.  

Consider a situation where we have 50 features (p = 50). There can be p(p-1)/2 scatter plots i.e. 1225 plots possible to analyze the variable relationships. It would be a tedious job to perform exploratory analysis on this data. That is why, we have to use PCA to get rid of this problem.

Disadvantages of Principal Component Analysis

1. Independent variables become less interpretable: After implementing PCA on the dataset, your original features will turn into Principal Components. Principal Components are the linear combination of your original features. Principal Components are not as readable and interpretable as original features.

2. Data standardization is must before PCA: You must standardize your data before implementing PCA, otherwise PCA will not be able to find the optimal Principal Components. 

For instance, if a feature set has data expressed in units of Kilograms, Light years, or Millions, the variance scale is huge in the training set. If PCA is applied on such a feature set, the resultant loadings for features with high variance will also be large. Hence, principal components will be biased towards features with high variance, leading to false results.

Also, for standardization, all the categorical features are required to be converted into numerical features before PCA can be applied.

PCA is affected by scale, so you need to scale the features in your data before applying PCA. Use StandardScaler from Scikit Learn to standardize the dataset features onto unit scale (mean = 0 and standard deviation = 1) which is a requirement for the optimal performance of many Machine Learning algorithms.

3. Information Loss: Although Principal Components try to cover maximum variance among the features in a dataset, if we don't select the number of Principal Components with care, it may miss some information as compared to the original list of features.

Related:

Advantages and Disadvantages of t-SNE over PCA (PCA vs t-SNE)
What is t-SNE? How does it work using t-Distribution?
Dimensionality Reduction: Feature Selection and Feature Extraction
Why is Dimensionality Reduction required in Machine Learning?