What is Skewnesss? How to visualize it with Histogram and how to remove it?

Skewness is a measure of the asymmetry in a variable. It can be positive (right skewed), negative (left skewed), and zero. Ideally there should be zero skewness in a variable. Larger the skewness, greater the number of outliers in a variable.

How to remove skewness from variables? 

Our aim should be to have near zero skewness in our variables in the dataset. Taking log of the skewed variable helps a lot in decreasing the skewness. So, lets see how to do that?

Consider a Load Prediction dataset. We will analyze skewness of LoanAmount variable.

Step 1: Import the required libraries

import pandas as pd
import numpy as np
import matplotlib as plt
%matplotlib inline
import seaborn as sns

Step 2: Load the dataset

dataset = pd.read_csv("C:/train_loan_prediction.csv")

Step 3: Draw histogram of LoanAmount variable with 20 bins

dataset['LoanAmount'].hist(bins=20)

Step 4: Create a new variable by taking log of LoanAmount variable

dataset['LoanAmount_Log'] = np.log(dataset['LoanAmount'])

Step 5: Draw histogram of newly created variable

dataset['LoanAmount_Log'].hist(bins=20)

We can see that distribution of the values in the LoanAmount_Log variable is normal and symmetrical and skewness is near to zero. In this way, you should check skewness of all the variables and remove it.

Related: Log Transforming the Skewed Data to get Normal Distribution

Data Exploration: Univariate, Bivariate and Multivariate Analysis

Data Exploration is used to get insights from data. A good data exploration strategy is a key to solve any complicated problem in the world of Machine Learning. 

We can't determine everything by just looking at the data. We need to dig deeper. This step helps us understand the nature of variables (skewed, missing, zero variance feature) so that they can be treated properly. It involves creating charts, graphs (univariate and bivariate analysis), and cross-tables to understand the behavior of features.

A good data exploration strategy comprises the following:

Univariate Analysis - It is used to visualize one variable in one plot. Examples: histogram, density plot, etc.

Bivariate Analysis - It is used to visualize two variables (x and y axis) in one plot. Examples: bar chart, line chart, area chart, etc.

Multivariate Analysis - As the name suggests, it is used to visualize more than two variables at once. Examples: stacked bar chart, dodged bar chart, etc.

Cross Tables -They are used to compare the behavior of two categorical variables (used in pivot tables as well).

Related: Data Exploration using Pandas Library in Python

What is Boxplot? How is it used to find outliers in a dataset?

A Boxplot is a graph that indicates how the values in the dataset are spread out. Boxplots are used to visualize the distribution of the data based on following parameters:

1. minimum
2. first quartile (Q1)
3. median
4. third quartile (Q3)
5. maximum

Below is the detail of the above parameters:

median (Q2/50th Percentile): the middle value of the dataset.

first quartile (Q1/25th Percentile): the middle number between the smallest number (not the “minimum”) and the median of the dataset.

third quartile (Q3/75th Percentile): the middle value between the median and the highest value (not the “maximum”) of the dataset.

interquartile range (IQR): 25th to the 75th percentile.

whiskers (shown in blue)
outliers (shown as green circles)

maximum: Q3 + 1.5*IQR
minimum: Q1 -1.5*IQR

Advantages of Barplots

1. Used to find out skewness of variables.
2. Used to find out outliers in a variable.
3. Used to find out if the data is symmetrical or not? How tightly the data is grouped?

How to visualize outliers in categorical variables using boxplots?

Outliers must be removed from a dataset. In my last post, we saw how to visualize outliers in numeric variables? In this post, we will use barplots to visualize the outliers in the categorical variables. 

Consider Ames Housing dataset. 

Step 1: Load the required libraries
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

Step 2: Load the dataset
dataset = pd.read_csv("C:/datasets/train.csv")

Step 3: Create barplots for all the categorical variables

First separate out all the categorical variables from the dataset and then draw barplot for each variable.

def boxplot(x,y,**kwargs):
            sns.boxplot(x=x,y=y)
            x = plt.xticks(rotation=90)

cat_vars = [f for f in dataset.columns if dataset.dtypes[f] == 'object']
p = pd.melt(dataset, id_vars='SalePrice', value_vars=cat_vars)
g = sns.FacetGrid (p, col='variable', col_wrap=2, sharex=False, sharey=False, size=5)
g = g.map(boxplot, 'value','SalePrice')
g

It will draw 43 plots representing outliers in each variable. You need to clearly examine each graph and try to remove the outliers from it. 

Related: What are Outliers? How to find and remove outliers using JointPlot in Seaborn Library?

How to visualize skewness of numeric variables by plotting histograms?

It is utmost important to remove skewness of variables before applying any Machine Learning algorithm. Skewed variables have outliers which must to be removed otherwise the accuracy of the model is adversely affected. 

Lets plot distribution plot for each numeric variable and examine its skewness. 

Consider Ames Housing dataset. 

Step 1: Load the required libraries
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

Step 2: Load the dataset
dataset = pd.read_csv("C:/datasets/train.csv")

Step 3: Create histogram for all the numeric variables

First separate out all the numeric variables from the dataset. Remove the Id column and then draw the distribution plot.

num_vars = [f for f in dataset.columns if dataset.dtypes[f] != 'object']
num_vars.remove('Id')
nd = pd.melt(dataset, value_vars = num_vars)
n1 = sns.FacetGrid (nd, col='variable', col_wrap=4, sharex=False, sharey = False)
n1 = n1.map(sns.distplot, 'value')
n1

It will draw 37 plots representing skewness of each variable. You need to clearly examine each graph and try to remove the outliers from it. One of the way to remove skewness of variable is log transformation. I have written a detailed article on log transformation in my this post.

Related: What are Outliers? How to find and remove outliers using JointPlot in Seaborn Library?