Difference between Decision Tree and Random Forest in Machine Learning

Random Forest is a collection of Decision Trees. Decision Tree makes its final decision based on the output of one tree but Random Forest combines the output of a large number of small trees while making its final prediction. Following is the detailed list of differences between Decision Tree and Random Forest:

1. Random Forest is an Ensemble Learning (Bagging) Technique unlike Decision Tree: In Decision Tree, only one tree is grown using all the features and observations. But in case of Random Forest, features and observations are spitted into multiple parts and a lot of small trees (instead of one big tree) are grown based on the spitted data. So, instead of one full tree like Decision Tree, Random Forest uses multiple trees. Larger the number of trees, better is the accuracy and generalization capability. But at some point, increasing the number of trees does not contribute to the accuracy, so one should stop growing trees at that point. 

2. Random Forest uses voting system unlike Decision Tree: All the trees grown in Random Forest are called weak learners. Each weak learner casts a vote as per its prediction. The class which gets maximum votes is considered as the final output of the prediction. You can think of it like a democracy system. On the other hand, there is no voting system in Decision Tree. Only one tree predicts the outcome. No democracy at all!! 

3. Random Forest rarely overfits unlike Decision Tree: Decision Tree is very much prone to overfitting as there is only one tree which is responsible for predicting the outcome. If there is a lot of noise in the dataset, it will start considering the noise while creating the model and will lead to very low bias (or no bias at all). Due to this, it will show a lot of variance in the final predictions in real world data. This scenario is called overfitting. In Random Forest, noise has very little role in spoiling the model as there are so many trees in it and noise cannot affect all the trees.

4. Random Forest reduces variance instead of bias: Random forest reduces variance part of the error rather than bias part, so on a given training dataset, Decision Tree may be more accurate than a Random Forest. But on an unexpected validation dataset, Random Forest always wins in terms of accuracy.

5. Performance: The downside of Random Forest is that it can be slow if you have a single process but it can be parallelized.

6. Decision Tree is easier to understand and interpret: Decision Tree is simple and easy to interpret. You know what variable and what value of that variable is used to split the data and predict the outcome. On the other hand, Random Forest is like a Black Box. You can specify the number of trees you want in your forest (n_estimators) and also you can specify maximum number of features to be used in each tree. But you cannot control the randomness, you cannot control which feature is part of which tree in the forest, you cannot control which data point is part of which tree. 

Advantages and Disadvantages of Linear Regression in Machine Learning

Linear Regression is a supervised machine learning algorithm which is very easy to learn and implement. Following are the advantages and disadvantage of Linear Regression:

Advantages of Linear Regression

1. Linear Regression performs well when the dataset is linearly separable. We can use it to find the nature of the relationship among the variables.

2. Linear Regression is easier to implement, interpret and very efficient to train. 

3. Linear Regression is prone to over-fitting but it can be easily avoided using some dimensionality reduction techniques, regularization (L1 and L2) techniques and cross-validation.

Disadvantages of Linear Regression

1. Main limitation of Linear Regression is the assumption of linearity between the dependent variable and the independent variables. In the real world, the data is rarely linearly separable. It assumes that there is a straight-line relationship between the dependent and independent variables which is incorrect many times.

2. Prone to noise and overfitting: If the number of observations are lesser than the number of features, Linear Regression should not be used, otherwise it may lead to overfit because is starts considering noise in this scenario while building the model.

3. Prone to outliers: Linear regression is very sensitive to outliers (anomalies). So, outliers should be analyzed and removed before applying Linear Regression to the dataset.

4. Prone to multicollinearity: Before applying Linear regression, multicollinearity should be removed (using dimensionality reduction techniques) because it assumes that there is no relationship among independent variables.

In summary, Linear Regression is great tool to analyze the relationships among the variables but it isn’t recommended for most practical applications because it over-simplifies real world problems by assuming linear relationship among the variables

Injection at Home Sevice in Gurgaon

Implement XGBoost with K Fold Cross Validation in Python using Scikit Learn Library

In this post, we will implement XGBoost with K Fold Cross Validation technique using Scikit Learn library. We will use cv() method which is present under xgboost in Scikit Learn library. You need to pass nfold parameter to cv() method which represents the number of cross validations you want to run on your dataset. 

Before going through this implementation, I highly recommend you to have a look at normal implementation of XGBoost in my previous post.

You can download my Jupyter notebook implementing XGBoost using Cross Validation from here.

Step 1: Import the required Python libraries like pandas and sklearn

import pandas as pd
from sklearn.datasets import load_boston
import xgboost as xgb

Step 2: Load and examine the dataset (Data Exploration)

dataset = load_boston()
dataset.keys()
dataset.data
dataset.target
dataset.data[0:5]
dataset.target[0:5]
dataset.data.shape
dataset.target.shape
dataset.feature_names
print(dataset.DESCR)

#convert the loaded dataset from scikit learn library to pandas library

data = pd.DataFrame(dataset.data)
data.columns = dataset.feature_names
data.head()
data['PRICE'] = dataset.target
data.head()
data.info()
data.describe() 

Please note that "describe()" is used to display the statistical values of the data like mean and standard deviation.

Step 3: Mention X and Y axis

X, y = data.iloc[:,:-1], data.iloc[:,-1]

X contains the list of attributes
Y contains the list of labels

Step 4: Convert dataset into DMatrix

Lets convert our dataset into an optimized data structure called DMatrix that XGBoost supports and delivers high performance and efficiency gains. 

data_dmatrix = xgb.DMatrix(data=X, label=y)

Step 5: Create the model

Lets create a hyper-parameter dictionary params which holds all the hyper-parameters and their values as key-value pairs. We will exclude n_estimators from the hyper-parameter dictionary because we will use num_boost_rounds instead.

params = {'objective':'reg:linear",'colsample_bytree':0.3,'learning_rate':0.1,
                'max_depth':5, 'alpha':10}

cv_results = xgb.cv(dtrain=data_dmatrix, params=params, nfold=10,                     num_boost_round=50, early_stopping_rounds=10, metrics='rmse", as_pandas=True, seed=0)  

Explanation of above statement:

nfold = 10: we want to run 10 Fold Cross Validation.
metrics = rmse: we want to use root mean square error to check the accuracy.
num_boost_round = 50: number of trees you want to build (analogous to n_estimators)
early_stopping_rounds = 10: finishes training of the model early if the hold-out metric ("rmse" in our case) does not improve for a given number of rounds.
as_pandas: returns the results in a pandas data frame. So, cv_results is a pandas data frame in the above statement.
seed: for reproducibility of results.

Step 6: Examine the results

cv_results.head()
print((cv_results["test-rmse-mean"]).tail())

I get the RMSE for the price prediction around 3.67 per 1000$. You can reach an even lower RMSE for a different set of hyper-parameters. You may consider applying techniques like Grid Search, Random Search and Bayesian Optimization to reach the optimal set of hyper-parameters.

Related:
Implement XGBoost in Python using Scikit Learn Library
Difference between GBM and XGBoost
Advantages of XGBoost Algorithm

Advantages of XGBoost Algorithm in Machine Learning

XGBoost is an efficient and easy to use algorithm which delivers high performance and accuracy as compared to other algorithms. XGBoost is also known as regularized version of GBM. Let see some of the advantages of XGBoost algorithm:

1. Regularization: XGBoost has in-built L1 (Lasso Regression) and L2 (Ridge Regression) regularization which prevents the model from overfitting. That is why, XGBoost is also called regularized form of GBM (Gradient Boosting Machine).

While using Scikit Learn libarary, we pass two hyper-parameters (alpha and lambda) to XGBoost related to regularization. alpha is used for L1 regularization and lambda is used for L2 regularization.

2. Parallel Processing: XGBoost utilizes the power of parallel processing and that is why it is much faster than GBM. It uses multiple CPU cores to execute the model.

While using Scikit Learn libarary, nthread hyper-parameter is used for parallel processing. nthread represents number of CPU cores to be used. If you want to use all the available cores, don't mention any value for nthread and the algorithm will detect automatically.

3. Handling Missing Values: XGBoost has an in-built capability to handle missing values. When XGBoost encounters a missing value at a node, it tries both the left and right hand split and learns the way leading to higher loss for each node. It then does the same when working on the testing data.

4. Cross Validation: XGBoost allows user to run a cross-validation at each iteration of the boosting process and thus it is easy to get the exact optimum number of boosting iterations in a single run. This is unlike GBM where we have to run a grid-search and only a limited values can be tested.

5. Effective Tree Pruning: A GBM would stop splitting a node when it encounters a negative loss in the split. Thus it is more of a greedy algorithm. XGBoost on the other hand make splits upto the max_depth specified and then start pruning the tree backwards and remove splits beyond which there is no positive gain.

For example: There may be a situation where split of negative loss say -4 may be followed by a split of positive loss +13. GBM would stop as it encounters -4. But XGBoost will go deeper and it will see a combined effect of +9 of the split and keep both.

Related: Difference between GBM and XGBoost

Implement XGBoost in Python using Scikit Learn Library in Machine Learning

XGBoost is an implementation of Gradient Boosting Machine. XGBoost is an optimized and regularized version of GBM. In this post, we will try to build a model using XGBRegressor to predict the prices using Boston dataset. To know more about XGBoost and GBM, please consider visiting this post.

You can download my Jupyter notebook implementing XGBoost from here.

Step 1: Import the required Python libraries like pandas, numpy and sklearn

import pandas as pd
import numpy as np
from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split
from xgboost import XGBRegressor
from sklearn.metrics import mean_absolute_error, mean_squared_error  

Step 2: Load and examine the dataset (Data Exploration)

dataset = load_boston()
dataset.keys()
dataset.data
dataset.target
dataset.data[0:5]
dataset.target[0:5]
dataset.data.shape
dataset.target.shape
dataset.feature_names
print(dataset.DESCR)

#convert the loaded dataset from scikit learn library to pandas library

data = pd.DataFrame(dataset.data)
data.columns = dataset.feature_names
data.head()
data['PRICE'] = dataset.target
data.head()
data.info()
data.describe() 

Please note that "describe()" is used to display the statistical values of the data like mean and standard deviation.

Step 3: Mention X and Y axis

X, y = data.iloc[:,:-1], data.iloc[:,-1]

X contains the list of attributes
Y contains the list of labels

Step 4: Split the dataset into training and testing dataset

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.20, random_state=0) 

Step 5: Create and fit the model

model = XGBRegressor(objective='reg:linear', colsample_bytree=0.3, learning_rate=0.1, max_depth=5, alpha=10, n_estimators=10)

model.fit(X_train, y_train)  

I will write a separate post explaining above hyperparameters of XGBoost algorithm.

Step 6: Predict from the model

y_pred = model.predict(X_test)

The y_pred is a numpy array that contains all the predicted values for the input values in the X_test.

Lets see the difference between the actual and predicted values.

df=pd.DataFrame({'Actual':y_test, 'Predicted':y_pred})  
df 

Step 7: Check the accuracy

meanAbsoluteError = mean_absolute_error(y_test, y_pred)
meanSquaredError = mean_squared_error(y_test, y_pred)
rootMeanSquaredError = np.sqrt(meanSquaredError)
print('Mean Absolute Error:', meanAbsoluteError)  
print('Mean Squared Error:', meanSquaredError)  
print('Root Mean Squared Error:', rootMeanSquaredError)

Implement AdaBoost in Python using Scikit Learn Library in Machine Learning

We are going to implement Adaboost algorithm in Python using Scikit Learn library. This is an ensemble learning technique and we will use AdaBoostClassifier to solve IRIS dataset problem. 

You can download my Jupyter notebook implementing Adaboost from here.

Step 1: Import the required Python libraries

from sklearn import datasets
from sklearn.model_selection import train_test_split
from sklearn.ensemble import AdaBoostClassifier
from sklearn.metrics import confusion_matrix, accuracy_score, classification_report

Step 2: Load and examine the dataset

dataset = datasets.load_iris()
dataset.feature_names
dataset.target_names
dataset.data.shape
dataset.target.shape
dataset.data[0:5]
dataset.target[0:5]

Step 3: Mention X and Y axis

X = dataset.data
y = dataset.target

Step 4: Split the dataset into training and testing dataset

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1)

Step 5: Create and fit the model

model = AdaBoostClassifier(n_estimators=50, learning_rate=1)
model.fit(X_train, y_train)

Step 6: Predict from the model

y_pred = model.predict(X_test)

Step 7: Check the accuracy

confusionMatrix = confusion_matrix(y_test, y_pred)
accuracyScore = accuracy_score(y_test, y_pred)
classificationReport = classification_report(y_test, y_pred)
print(confusionMatrix)
print(accuracyScore * 100)
print(classificationReport)

To learn more about Adaboost, you can refer my below posts:

Difference between Random Forest and AdaBoost in Machine Learning
Difference between AdaBoost and Gradient Boosting Machine (GBM)

Difference between AdaBoost and Gradient Boosting Machine (GBM)

AdaBoost stands for Adaptive Boosting. So, basically, we will see the differences between Adaptive Boosting and Gradient Boosting. The basic idea of boosting (an ensemble learning technique) is to combine several weak learners into a stronger one. The general idea of boosting algorithms is to try predictors sequentially, where each subsequent model attempts to fix the errors of its predecessor.

As per documentation:

In Adaboost, ‘shortcomings’ are identified by high-weight data points.

In Gradient Boosting, ‘shortcomings’ (of existing weak learners) are identified by gradients.

Lets elaborate the above statements:

Adaboost is more about ‘voting weights’ and Gradient boosting is more about ‘adding gradient optimization’. 

Adaboost increases the accuracy by giving more weightage to the target which is misclassified by the model. At each iteration, Adaptive boosting algorithm changes the sample distribution by modifying the weights attached to each of the instances. It increases the weights of the wrongly predicted instances and decreases the ones of the correctly predicted instances.

Gradient boosting calculates the gradient (derivative) of the Loss Function with respect to the prediction (instead of the features). Gradient boosting increases the accuracy by minimizing the Loss Function (error which is difference of actual and predicted value) and having this loss as target for the next iteration.

Gradient boosting algorithm builds first weak learner and calculates the Loss Function. It then builds a second learner to predict the loss after the first step. The step continues for third learner and then for fourth learner and so on until a certain threshold is reached.

Related Articles:

Difference between Random Forest and AdaBoost in Machine Learning

Difference between GBM (Gradient Boosting Machine) and XGBoost (Extreme Gradient Boosting)

Difference between Random Forest and AdaBoost in Machine Learning

Both Random Forest and Adaboost (Adaptive Boosting) are ensemble learning techniques. Lets discuss some of the differences between Random Forest and Adaboost.

1. Random Forest is based on bagging technique while Adaboost is based on boosting technique. See the difference between bagging and boosting here.

2. In Random Forest, certain number of full sized trees are grown on different subsets of the training dataset. Adaboost uses stumps (decision tree with only one split). So, Adaboost is basically a forest of stumps. These stumps are called weak learners. These weak learners have high bias and low variance. 

3. Each tree in the Random Forest is made up using all the features in the dataset while stumps use one feature at a time.

4. In Random Forest, each decision tree is made independent of each other. So, the order in which trees are made is not important. While in Adaboost, order of stumps do matter. The error that first stump makes, influence how the second stump is made and so on. Each stump is made by taking the previous stump's mistakes into account. It takes the errors from the first round of predictions, and passes the errors as a new target to the second stump. The second stump will model the error from the first stump, record the new errors and pass that as a target to the third stump. And so forth. Essentially, it focuses on modelling errors from previous stumps.

5. Random Forest uses parallel ensembling while Adaboost uses sequential ensembling. Random Forest runs trees in parallel, thus making it possible to parallelize jobs on a multiprocessor machine. Adaboost instead uses a sequential approach. 

6. Each tree in the Random Forest has equal amount of say in the final decision while in Adaboost different stumps have different amount of say in the final decision. The stump which makes less error in the prediction, has high amount of say as compared to the stump which makes more errors.

7. Random Forest aims to decrease variance not bias while Adaboost aims to decrease bias not variance.

8. There are rare chances of Random Forest to overfit while there are good chances of Adaboost to overfit.

Related: Difference between GBM (Gradient Boosting Machine) and XGBoost (Extreme Gradient Boosting)

Implement PCA in Python using Scikit Learn Library

We calculate Principal Components on a dataset using the PCA() class in the Scikit-Learn library. 

While creating the PCA() class, we can pass following parameters in the constructor:

1. Number of Principal Components we need to consider OR
2. Amount of Variance we need to retain

If we don't pass any parameter in the constructor, all the Principal Components (which are usually equal to number of features in the dataset) are used to create a model. We should refrain from this approach as this will not serve the purpose of PCA.

We will see it in detail in this article. We will use bank note authentication dataset and implement Random Forest to identify the authenticity of the currency. For theory on PCA, you can go through this article.

You can download the dataset from here and my Jupyter notebook implementing PCA from here.

Step 1: Import the required Python libraries like pandas, numpy and sklearn

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import confusion_matrix, accuracy_score, classification_report

Step 2: Load and examine the dataset

names = ['Variance', 'Skewness', 'Curtosis', 'Entropy', 'Class']
dataset = pd.read_csv('bank_note_authentication.csv', names=names)
dataset.shape
dataset.head()

Step 3: Mention X and Y axis

X = dataset.drop('Class', axis=1)  
y = dataset['Class']    

X contains the list of attributes
Y contains the list of labels

Step 4: Split the dataset into training and testing dataset

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.20, random_state=0) 

Step 5: Feature scaling

standardScaler = StandardScaler()  
X_train = standardScaler.fit_transform(X_train)  
X_test = standardScaler.transform(X_test)

Step 6.1: Apply PCA (Method 1)

pca = PCA()  
X_train = pca.fit_transform(X_train)  
X_test = pca.transform(X_test)  

explained_variance = pca.explained_variance_ratio_
print(explained_variance)

Output:
[0.54578721 0.31931922 0.09136961 0.04352396]

Note: Here I have not passed any parameter to the PCA class constructor. So, it will return 4 Principal Components (equal to the number of features in our dataset). 

explained_variance_ratio_ returns the variance delivered by each Principal Component.

From the output, you can see that first PC accounts for 54% of variance and second PC accounts for around 32% of variance. So, both Principal Components account for around 86% of the variance in the dataset. So, instead of using all the four features, we can use only two or three principal component to build our model. 

This example does not highlight the great importance of PCA as we have only 4 features in our dataset. But in real world, we can easily have 40 features or 40K features or more. In these scenarios, PCA does a fantastic job. In these cases, instead of using 40K features, we will need to just use some hundreds of Principal Components which will drastically increase the performance of our model.

Step 6.2: Apply PCA (Method 2)

pca = PCA(n_components=2)
X_train = pca.fit_transform(X_train)  
X_test = pca.transform(X_test)

PC_DataFrame = pd.DataFrame(data = X_train, columns = ['PC1', 'PC2'])
print(PC_DataFrame)

Note: From the step 6.1, it is clear that if we use only two Principal Components, we can still cover 86% of the variance. So, I passed number of components as 2 in the constructor. 

Step 6.3: Apply PCA (Method 3)

pca = PCA(0.86) 
X_train = pca.fit_transform(X_train)  
X_test = pca.transform(X_test)
print(pca.components_)

Note: This is the another way of doing PCA on the dataset. If I want to retain 86% of variance in my dataset and don't want to bother about the number of Principal Components, I can use this approach.

components_ returns the number of Principal Components considered to achieve the variance of 86%. In this case, number of Principal Components are 2.

So, we can use either 6.2 or 6.3 approach to implement PCA.

The number of Principal Components to retain in a feature set depends on several conditions such as storage capacity, training time, performance etc. In some datasets, where all the features are contributing equally to the overall variance, all the principal components are crucial to the predictions and none can be ignored. A general rule of thumb is to take number of Principal Components that contribute to significant variance and ignore those with diminishing variance returns. 

Step 7: Create and fit the model

model = RandomForestClassifier(max_depth=2, random_state=0)  
model.fit(X_train, y_train)

Step 8: Predict from the model

y_pred = model.predict(X_test) 

The y_pred is a numpy array that contains all the predicted values for the input values in the X_test.

Lets see the difference between the actual and predicted values.

df=pd.DataFrame({'Actual':y_test, 'Predicted':y_pred})  
df 

Step 9: Check the accuracy

confusionMatrix = confusion_matrix(y_test, y_pred)
accuracyScore = accuracy_score(y_test, y_pred)
classificationReport = classification_report(y_test, y_pred)
print(confusionMatrix)
print(accuracyScore * 100)
print(classificationReport)

Difference between Linear Regression and Logistic Regression

Linear Regression and Logistic Regression have a lot of similarities and differences. Linear Regression is used to solve regression problems while Logistic Regression is used to solve classification problems. Below is the comparison of both the algorithms:

1. Definition

Linear Regression: Predicts a continuous dependent variable based on the values of independent variables.
Logistic Regression: Predicts a categorical / discrete dependent variable based on the values of independent variables.

2. Curve

Linear Regression: Straight Line. Equation of Line: y = mx + c
Logistic Regression: Sigmoid Curve (S Curve): Equation of S Curve: log(y/(1-y)) = mx + c

3. Output

Linear Regression: Predicts a continuous variable (example: What will be the GDP of the country this year?)
Logistic Regression: Predicts a categorical / discrete variable (example: Whether it will rain today or not?)

4. Accuracy Measurement Techniques

Linear Regression: Mean Absolute Error, Mean Square Error, Root Mean Square Error, R Square Method, Adjusted R Square Method
Logistic Regression: Confusion Matrix, Classification Report, Accuracy Score, F1 Score, Precision, Recall, ROC (Receiver Operating Characteristic), AUC (Area Under the ROC Curve)

Advantages and Disadvantages of Logistic Regression in Machine Learning

Logistic Regression Model is a generalized form of Linear Regression Model. It is a very good Discrimination Tool. Following are the advantages and disadvantage of Logistic Regression:

Advantages of Logistic Regression

1. Logistic Regression performs well when the dataset is linearly separable.

2. Logistic regression is less prone to over-fitting but it can overfit in high dimensional datasets. You should consider Regularization (L1 and L2) techniques to avoid over-fitting in these scenarios.

3. Logistic Regression not only gives a measure of how relevant a predictor (coefficient size) is, but also its direction of association (positive or negative).

4. Logistic regression is easier to implement, interpret and very efficient to train. 

Disadvantages of Logistic Regression

1. Main limitation of Logistic Regression is the assumption of linearity between the dependent variable and the independent variables. In the real world, the data is rarely linearly separable. Most of the time data would be a jumbled mess.

2. If the number of observations are lesser than the number of features, Logistic Regression should not be used, otherwise it may lead to overfit.

3. Logistic Regression can only be used to predict discrete functions. Therefore, the dependent variable of Logistic Regression is restricted to the discrete number set. This restriction itself is problematic, as it is prohibitive to the prediction of continuous data.

Advantages and Disadvantages of Naive Bayes in Machine Learning

Bayes theorem is an extension of the conditional probability. While using Bayes theorem, you use one conditional probability to calculate another one. 

Bayes theorem is represented with the following expression:

P(A|B) = P(B|A) * P(A) / P(B)

Here, we calculate the probability of event A, given that event B has occurred. The right hand side of the equation consists of the probability of event B, given that event A has occurred, multiplied by the ratio of probability of event A to Probability of event B.

Advantages of Naive Bayes

1. When assumption of independent predictors holds true, a Naive Bayes classifier performs better as compared to other models.

2. Naive Bayes requires a small amount of training data to estimate the test data. So, the training period is less.

3. Naive Bayes is also easy to implement.

Disadvantages of Naive Bayes

1. Main imitation of Naive Bayes is the assumption of independent predictors. Naive Bayes implicitly assumes that all the attributes are mutually independent. In real life, it is almost impossible that we get a set of predictors which are completely independent.

2. If categorical variable has a category in test data set, which was not observed in training data set, then model will assign a 0 (zero) probability and will be unable to make a prediction. This is often known as Zero Frequency. To solve this, we can use the smoothing technique. One of the simplest smoothing techniques is called Laplace estimation.

Advantages and Disadvantages of SVM (Support Vector Machine) in Machine Learning

SVM (Support Vector Machine) classifies the data using hyperplane which acts like a decision boundary between different classes. Extreme data points from each class are called Support Vectors. SVM tries to find the best and optimal hyperplane which has maximum margin from each Support Vector. 

Kernel functions / tricks are used to classify the non-linear data. It transforms non-linear data into linear data and then draws a hyperplane. 

Below are the advantages and disadvantages of SVM:

Advantages of Support Vector Machine (SVM)

1. Regularization capabilities: SVM has L2 Regularization feature. So, it has good generalization capabilities which prevent it from over-fitting.

2. Handles non-linear data efficiently: SVM can efficiently handle non-linear data using Kernel trick.

3. Solves both Classification and Regression problems: SVM can be used to solve both classification and regression problems. SVM is used for classification problems while SVR (Support Vector Regression) is used for regression problems.

4. Stability: A small change to the data does not greatly affect the hyperplane and hence the SVM. So the SVM model is stable.

Disadvantages of Support Vector Machine (SVM)

1. Choosing an appropriate Kernel function is difficult: Choosing an appropriate Kernel function (to handle the non-linear data) is not an easy task. It could be tricky and complex. In case of using a high dimension Kernel, you might generate too many support vectors which reduce the training speed drastically. 

2. Extensive memory requirement: Algorithmic complexity and memory requirements of SVM are very high. You need a lot of memory since you have to store all the support vectors in the memory and this number grows abruptly with the training dataset size.

3. Requires Feature Scaling: One must do feature scaling of variables before applying SVM.

4. Long training time: SVM takes a long training time on large datasets.

5. Difficult to interpret: SVM model is difficult to understand and interpret by human beings unlike Decision Trees.

Advantages and Disadvantages of Cross Validation in Machine Learning

Cross Validation in Machine Learning is a great technique to deal with overfitting problem in various algorithms. Instead of training our model on one training dataset, we train our model on many datasets. Below are some of the advantages and disadvantages of Cross Validation in Machine Learning:

Advantages of Cross Validation

1. Reduces Overfitting: In Cross Validation, we split the dataset into multiple folds and train the algorithm on different folds. This prevents our model from overfitting the training dataset. So, in this way, the model attains the generalization capabilities which is a good sign of a robust algorithm.

Note: Chances of overfitting are less if the dataset is large. So, Cross Validation may not be required at all in the situation where we have sufficient data available.

2. Hyperparameter Tuning: Cross Validation helps in finding the optimal value of hyperparameters to increase the efficiency of the algorithm.

More details...

Disadvantages of Cross Validation

1. Increases Training Time: Cross Validation drastically increases the training time. Earlier you had to train your model only on one training set, but with Cross Validation you have to train your model on multiple training sets. 

For example, if you go with 5 Fold Cross Validation, you need to do 5 rounds of training each on different 4/5 of available data. And this is for only one choice of hyperparameters. If you have multiple choice of parameters, then the training period will shoot too high.

2. Needs Expensive Computation: Cross Validation is computationally very expensive in terms of processing power required.

What are Hyperparameters in Machine Learning Algorithms? How is Cross Validation used for Hyperparameter Tuning?

Hyperparameters are the parameters which we pass to the Machine Learning algorithms to maximize their performance and accuracy. 

For example, we need to pass the optimal value of K in the KNN algorithm so that it delivers good accuracy as well as does not underfit / overfit. Our model should have a good generalization capability. So, choosing the optimal value of the hyperparameter is very crucial in the Machine Learning algorithms.

Related: How to choose optimal value of K in KNN Algorithm? 

Examples of Hyperparameters

1. K in KNN (Number of nearest neighbors in KNN algorithm)

2. K in K-Means Clustering (Number of clusters in K-Means Clustering algorithm)

3. Depth of a Decision Tree

4. Number of Leaf Nodes in a Decision Tree

5. Number of Trees in a Random Forest

6. Step Size and Learning Rate in Gradient Descent (or Stochastic Gradient Descent)

7. Regularization Penalty (Lambda) in Ridge and Lasso Regression

Hyperparameters are also called "meta-parameters" and "free parameters".

Hyperparameters and Cross Validation

Selecting the optimal value of hyperparameters is usually not an easy task. These parameters are usually chosen using cross validation. These values remain fixed once chosen throughout the training of the model. 

In a K Fold Cross Validation, we initialize hyper-parameters to some value and and then train our model K times, every time using different Test Folds. Note down the average performance of the model over all the test folds and repeat the whole process for another set of hyper-parameters. Then we choose the set of hyper-parameters that corresponds to the best performance during cross-validation. 

As you can see, the computation cost of this process heavily depends on the number of hyper-parameter sets that need to be considered. 

Finding good hyper-parameters allows to avoid or at least reduce overfitting, but keep in mind that hyper-parameters can also overfit the data.

How to choose optimal value of K in KNN Algorithm?

There is no straightforward method to calculate the value of K in KNN. You have to play around with different values to choose the optimal value of K. Choosing a right value of K is a process called Hyperparameter Tuning.

The value of optimum K totally depends on the dataset that you are using. The best value of K for KNN is highly data-dependent. In different scenarios, the optimum K may vary. It is more or less hit and trail method.

You need to maintain a balance while choosing the value of K in KNN. K should not be too small or too large. 

A small value of K means that noise will have a higher influence on the result. 

Larger the value of K, higher is the accuracy. If K is too large, you are under-fitting your model. In this case, the error will go up again. So, at the same time you also need to prevent your model from under-fitting. Your model should retain generalization capabilities otherwise there are fair chances that your model may perform well in the training data but drastically fail in the real data. Larger K will also increase the computational expense of the algorithm.

There is no one proper method of estimation of K value in KNN. No method is the rule of thumb but you should try considering following suggestions:

1. Square Root Method: Take square root of the number of samples in the training dataset.

2. Cross Validation Method: We should also use cross validation to find out the optimal value of K in KNN. Start with K=1, run cross validation (5 to 10 fold),  measure the accuracy and keep repeating till the results become consistent. 

K=1, 2, 3... As K increases, the error usually goes down, then stabilizes, and then raises again. Pick the optimum K at the beginning of the stable zone. This is also called Elbow Method.

3. Domain Knowledge also plays a vital role while choosing the optimum value of K.

4. K should be an odd number.

I would suggest to try a mix of all the above points to reach any conclusion. 

Which Machine Learning Algorithms require Feature Scaling (Standardization and Normalization) and which not?

Feature Scaling (Standardization and Normalization) is one of the important steps while preparing the data. Whether to use feature scaling or not depends upon the algorithm you are using. Some algorithms require feature scaling and some don't. Lets see it in detail.

Algorithms which require Feature Scaling (Standardization and Normalization)

Any machine learning algorithm that computes the distance between the data points needs Feature Scaling (Standardization and Normalization). This includes all curve based algorithms. 

Example:

1. KNN (K Nearest Neigbors)
2. SVM (Support Vector Machine)
3. Logistic Regression
4. K-Means Clustering

Algorithms that are used for matrix factorization, decomposition and dimensionality reduction also require feature scaling.

Example:

1. PCA (Principal Component Analysis)
2. SVD (Singular Value Decomposition)

Why is Feature Scaling required? 

Consider a dataset which contains the age and salary of the employee. Now age is a 2 digit number while salary would be of 5 to 6 digit. If we don't scale both the features (age and salary), salary will adversely affect the accuracy of the algorithm (if the algorithm is distance based). So, if we don't scale the features, then large scale features will dominate the small scale features due to which algorithm will produce wrong predictions.

Algorithms which don't require Feature Scaling (Standardization and Normalization)

The algorithms which rely on rules like tree based algorithms don't require Feature Scaling (Standardization and Normalization).

Example:

1. CART (Classification and Regression Trees)
2. Random Forests
3. Gradient Boosted Decision Trees

Algorithms that rely on distributions of the variables also don't need feature scaling.

Example: 

1. Naive Bayes 

Related Article: Why to standardize and transform the features in the dataset before applying Machine Learning algorithms?