Implement a Linear Classification Model using TensorFlow Estimator

Lets see how can we perform linear classification using TensorFlow library in Python. We will use LinearClassifier function from TensorFlow Estimator. We will use California Census Data and try to predict what class of income (>50k or <=50k) people belong to. You can download this dataset from here. This dataset has 32561 observations and 15 features. You can also download my Jupyter notebook containing below code from here. So, lets get started.

Step 1: Import required libraries

import pandas as pd
from sklearn.model_selection import train_test_split
import tensorflow as tf
from sklearn.metrics import classification_report, confusion_matrix, accuracy_score

Step 2: Load and explore the dataset

dataset = pd.read_csv('adult.csv')
dataset.head()
dataset.size
dataset.shape
dataset.columns
dataset.dtypes
dataset.describe()

Step 3: Drop fnlwgt column

We are not going to use this column as it does not seem to contribute any relevant information in our prediction. So, better drop it.

dataset.drop('fnlwgt', axis=1, inplace=True)

Step 4: Convert label into 0 and 1

dataset['income'].unique()

Output: array(['<=50K', '>50K'], dtype=object)

It means, we have only two string labels. Lets convert these into numeric labels (0 and 1).

def label_fix(label):
    if label == '<=50K':
        return 0
    else:     
        return 1

dataset['income'] = dataset['income'].apply(label_fix)
dataset.head()
dataset['income'].unique()
dataset['income'].value_counts()

Step 5: Split dataset into training and testing set

X = dataset.drop('income', axis=1)
y = dataset['income']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)

Step 6: Create Feature Columns

All the independent variables need to be converted into a proper type of tensor. The estimator needs to have a list of features to train the model. Hence, the column's data requires to be converted into a tensor. 

We need to create feature columns for our numeric and categorical data. Feature columns act as the intermediaries between raw data and TensorFlow Estimators.

Convert numeric columns into feature columns.

tf.feature_column.numeric_column: Use this to convert numeric column into feature columns.

Convert categorical columns into feature columns.

tf.feature_column.categorical_column_with_hash_bucket: Use this if you don’t know the set of possible values for a categorical column in advance and there are too many of them.

tf.feature_column.categorical_column_with_vocabulary_list: Use this if you know the set of all possible feature values of a column and there are only a few of them

So, lets convert our all the columns into feature columns as discussed above.

workclass = tf.feature_column.categorical_column_with_hash_bucket('workclass', hash_bucket_size=1000)

education = tf.feature_column.categorical_column_with_hash_bucket('education', hash_bucket_size=1000)

marital_status = tf.feature_column.categorical_column_with_hash_bucket('marital_status', hash_bucket_size=1000)

occupation = tf.feature_column.categorical_column_with_hash_bucket('occupation', hash_bucket_size=1000)

relationship = tf.feature_column.categorical_column_with_hash_bucket('relationship', hash_bucket_size=1000)

race = tf.feature_column.categorical_column_with_hash_bucket('race', hash_bucket_size=1000)

sex = tf.feature_column.categorical_column_with_vocabulary_list('sex', ['Female', 'Male'])

native_country = tf.feature_column.categorical_column_with_hash_bucket('native_country', hash_bucket_size=1000)

age = tf.feature_column.numeric_column('age')

education_num = tf.feature_column.numeric_column('education_num')

capital_gain = tf.feature_column.numeric_column('capital_gain')

capital_loss = tf.feature_column.numeric_column('capital_loss')

hours_per_week = tf.feature_column.numeric_column('hours_per_week')

feature_columns = [workclass, education, marital_status, occupation, relationship, race, sex, native_country, age, education_num, capital_gain, capital_loss, hours_per_week]

Step 7: Create Input Function

We now create an input function that would feed Pandas DataFrame into our classifier model. It requires you to specify the features, labels and batch size. It also has a special argument called shuffle,which allows the model to read the records in a random order, thereby improving model performance. You can also specify number of epochs you want to use.

input_fn = tf.estimator.inputs.pandas_input_fn(x=X_train, y=y_train, batch_size=128, num_epochs=None, shuffle=True)

I have set the batch size of 128 and None for number of epochs. By default number of epochs is 1.

Step 8: Create a model using feature columns and input function

model = tf.estimator.LinearClassifier(feature_columns = feature_columns)
model.train(input_fn = input_fn, steps=1000)

Let the optimizer perform 1000 steps.

Step 9: Make predictions

pred_fn = tf.estimator.inputs.pandas_input_fn(x=X_test, batch_size=len(X_test), shuffle=False)
predictions = list(model.predict(input_fn = pred_fn))
predictions[0]
final_preds = []
for pred in predictions:
    final_preds.append(pred['class_ids'][0])
final_preds[:10]
df=pd.DataFrame({'Actual':y_test, 'Predicted':final_preds})  
df 

Step 10: Check accuracy

print(classification_report(y_test, final_preds))
print(confusion_matrix(y_test, final_preds))
print(accuracy_score(y_test, final_preds))

We got around 82.5% accuracy. You can play around with hyper-parameters like number of epochs, number of steps, batch size etc. to improve the accuracy.

Implement a simple Gradient Descent Optimizer in TensorFlow

Lets implement a simple Gradient Descent Optimizer in TensorFlow for a linear model. We will use GradientDescentOptimizer function present in TensorFlow. We will use Gradient Descent optimizer to find out optimal values of weight and bias so that the loss is minimized. You can download my Jupyter notebook containing below code on Gradient Descent from here.

Step 1: Import TensorFlow library

import tensorflow as tf

Step 2: Declare all the variables and placeholders

W = tf.Variable([0.3], tf.float32)
b = tf.Variable([-0.3], tf.float32)

We have initialized the weight and bias with a random number 0.3 and -0.3 respectively. Task of Gradient Descent Optimizer is to find out an optimal value for both of these variables so that our loss is minimum. Lets see how it happens in next steps?

x = tf.placeholder(tf.float32)
y = tf.placeholder(tf.float32)

Step 3: Create a linear model

linear_model = W * x + b

Step 4: Create a loss function

squared_delta = tf.square(linear_model - y)
loss = tf.reduce_sum(squared_delta)

1. We are using sum of squared errors as a loss function.

2. "linear_model - y" expression is used to compute the error. "linear_model" contains the predicted values and "y" contains the actual values.

3. "square" function is used to square all the errors. 

4. "reduce_sum" function is used to sum all the squared errors.

Step 5: Create a Gradient Descent Optimizer

optimizer = tf.train.GradientDescentOptimizer(0.01)
train = optimizer.minimize(loss)

We are passing 0.01 as a learning rate.

Step 6: Initialize all the variables

init = tf.global_variables_initializer()

Step 7: Create a session and run the graph

session = tf.Session()
session.run(init)
print(session.run(loss, {x:[1,2,3,4], y:[0,-1,-2,-3]}))

Output: 23.66

So, our loss is 23.66 which is quite high. It means the initial values of weight and bias which we took as 0.3 and -0.3 in step 2 are not optimal values. We need to take help of Gradient Descent to optimize our weight and bias.

In next step, we will use Gradient Descent Optimizer (with 1000 iterations and learning rate of 0.01) and try to minimize this loss.

for _ in range(1000):
    session.run(train, {x:[1,2,3,4], y:[0,-1,-2,-3]})
print(session.run([W,b]))
session.close()

Output: [array([-0.9999969], dtype=float32), array([0.9999908], dtype=float32)]

Now we get W as -0.9999969 (approx -1) and b as 0.9999908 (approx 1). So, final conclusion is that optimized value of W is -1 and b is 1. If we initialize W and b as 1 and -1 in our step 2, we will get zero loss. So, our Gradient Descent Optimizer has done a pretty decent job for us.

Constants, Placeholders, Variables and Sessions in TensorFlow

In this article on TensorFlow, we will see how to build and run a graph taking simple examples of constants, placeholders and variables. We will also learn something about sessions and feed dictionary. In order to learn some theory about TensorFlow, you can look at my this post. You can download my Jupyter notebook containing following code from here.

First thing first, lets import the TensorFlow library.

import tensorflow as tf

Constants

We have declared three nodes (a, b and c). Operations being performed at node "a" and "b" are to declare constant values. Node "c" is performing multiplication operation. 

a = tf.constant(5.0)

b = tf.constant(6.0)

c = a * b

Till now, we have just built a graph. We need to run it. In order to run a graph in TensorFlow, we need to create a session. Following is a way to create a session and run the graph.

session = tf.Session()

result = session.run(c)

print(result)

session.close()

Output: 30.0

Another example

We can also pass the data type to a node.

a = tf.constant(5.0, tf.float32)

b = tf.constant(6.0)

print(a,b)

If we print the nodes before running the graph, we will get following output:

Output: Tensor("Const_2:0", shape=(), dtype=float32) Tensor("Const_3:0", shape=(), dtype=float32)

To print the actual values, we need to create a session and run the graph.

session = tf.Session()

result = session.run([a,b])

print(result)

session.close()

Output: [5.0, 6.0]

Placeholders

We provide values to placeholders while running the graph.

a = tf.placeholder(tf.float32)

b = tf.placeholder(tf.float32)

c = a * b

d = 2 * a

session = tf.Session()

result = session.run(c, {a:[1,3], b:[2,4]})

print(result)

Output: [ 2. 12.]

Feed Dictionary: We can also create a dictionary and input it into a placeholder while running a graph. We use feed_dict parameter for this.

dictionary = {a:[[[1,2,3],[4,5,6],[7,8,9]],[[1,2,3],[4,5,6],[7,8,9]]]}

result = session.run(d, feed_dict=dictionary)

print(result)

session.close()

Output:

[[[ 2.  4.  6.]

  [ 8. 10. 12.]

  [14. 16. 18.]]

 [[ 2.  4.  6.]

  [ 8. 10. 12.]

  [14. 16. 18.]]]

Naming a node

We can also provide a name to a node. This is helpful in visualizing the nodes in Tensor Board.

a = tf.placeholder(tf.float32, name="A")

b = tf.placeholder(tf.float32, name="B")

c = tf.multiply(a, b, name="C")

with tf.Session() as session:

    result = session.run(c, feed_dict={a:[1,2,3], b:[4,5,6]})

    print(result)

Output: [ 4. 10. 18.]

Variables

Lets create some nodes and assign them some operations.

zero = tf.Variable(0)

one = tf.constant(1)

new_value = tf.add(zero, one)

updated_variable = tf.assign(zero, new_value)

We must initialize all the variables while running the graph. So, following line is must.

init = tf.global_variables_initializer()

Now, create a session and run the graph.

session = tf.Session()

session.run(init) 

print(session.run(zero)) 

print(session.run(one)) 

print(session.run(new_value)) 

print(session.run(updated_variable))

Output:

0

1

1

1

Lets run the "updated_variable" node 5 times and observe the results.

for _ in range(5):

    session.run(updated_variable)

    print(session.run(zero))

Output:

2

3

4

5

6

Finally, close the session.

session.close()  

Strings

Below is the illustration of the string concatenation operation in TensorFlow.

hello = tf.constant('hello')

world = tf.constant('world')

hello_world = tf.add(hello, world)

with tf.Session() as session:

    print(session.run(hello_world))

Output: b'helloworld'

TensorFlow: Tensors, Computational Graphs, Nodes, Estimators and TensorBoard

TensorFlow library was developed by the Google Brain Team for complex numeric calculations (like numpy). It relies on a lot of matrix multiplications. Later on, Google started using it as a library for deep learning. First stable version of TensorFlow appeared in 2017. It is an open source library under Apache Open Source license.

TensorFlow allows you to create large-scale neural networks with many layers like CNN, RNN etc. TensorFlow is a computational framework used to build deep learning models. It is an open source library for numerical computation and large scale machine learning.

Tensors and TensorFlow

A tensor is a vector or matrix of n-dimensions that represents all types of data. We can say that a tensor is a collection of feature vectors (i.e. array) of n-dimensions. Tensors can be considered as dynamically sized multi-dimensional array. The shape of the data is the dimensionality of the matrix or array. One dimensional tensor is known as scalar.

“TensorFlow” has been derived from the operations which neural networks perform on the tensors. It’s literally a flow of tensors. Tensor goes inside, flows through various nodes in neural network, and then comes out. That is why, it is called TensorFlow.

TensorFlow is made up of two terms – Tensor and Flow: In TensorFlow, the term tensor refers to the representation of data as multi-dimensional array whereas the term flow refers to the series of operations that one performs on tensors.

Tensor Ranks: 0 (scaler), 1 (vector), 2 (matrix), 3 (3-tensor)......., n (n-tensor)

Why TensorFlow?

1. It provides both C++ and Python APIs.

2. It has faster compilation time as compared to other deep learning libraries like Keras and PyTorch.

3. It supports both CPU and GPU computing devices.

4. Available both on mobile and desktop.

5. TensorFlow runtime is a cross platform library.

Versions of TensorFlow

1. TensorFlow with CPU support only
2. TensorFlow with GPU support

The model can be trained and used on GPUs as well as CPUs.

Google Colaboratory: It is a free Jupyter notebook environment that requires no setup and runs entirely in the cloud. With Colaboratory you can write and execute code, save and share your analyses, and access powerful computing resources, all for free from your browser.

Graph (Computational graph)

Graph is made up of nodes and edges. Series of TensorFlow operations are arranged as nodes in the computational graph. 

Nodes: Each nodes take 0 or more tensors as input and produces a tensor as output. Node carries the mathematical operation and produces an endpoints outputs. 

Properties of a node: unique label (name), dimension (shape), data type (dtype)

Datatypes: float32, float64, int8, int16, int32, int64, uint8, string and bool

Types of nodes: constant, placeholder, Variable, SparseTensor

In case you have not provided the data type explicitly, TensorFlow will infer the type of the constant / variable from the initialized value.

Must initialize a variable in TensorFlow

Constants are initialized when you call tf.constant but variables are not initialized when you call tf.Variable.

To initialize all the variables in a TensorFlow program, you must explicitly call a special operation as shown below:

init = tf.global_variables_initializer()
sess.run(init)

Variables must be initialized before a graph is used for the first time.

TensorFlow variables are in-memory buffers that contain tensors, but unlike normal tensors that are only instantiated when a graph is run and are immediately deleted afterwards, variables survive across multiple executions of a graph.

Edges: Edges explain the input/output relationships between nodes. Edge of the nodes is the tensor, i.e. a way to populate the operation with data. Each operation is called an op node.

Executing a Graph: There are two steps involved while executing a graph:

1. Building a Computational Graph: Just create nodes and assign them operations.

2. Running a Computational Graph: We need to run the computational graph within a session. Session is also called TensorFlow Runtime. Session places the graph operations onto devices, such as CPUs or GPUs, and provides methods to execute them. A session encapsulates the control and state of the TensorFlow runtime i.e. it stores the information about the order in which all the operations will be performed and passes the result of already computed operation to the next operation in the pipeline.

Advantages of Computational Graph

1. Parallel execution: The operations assigned to different nodes of a computational graph can be performed in parallel, thus, providing a better performance in terms of computations.
Nodes and edges can be spread over several clusters of computers (in distributed manner).

2. Portability: The portability of the graph allows to preserve the computations for immediate or later use. The graph can be saved and executed in the future.

Pipeline, Batches, Epochs, Iterations and Estimators

Pipeline and Batches: If you have a dataset of 50 GB, and your computer has only 16 GB of memory, then the machine will crash. In this situation, you need to build a TensorFlow pipeline. The pipeline will load the data in batches. Each batch will be pushed to the pipeline and be ready for the training. It allows you to use parallel computing. It means TensorFlow will train the model across multiple CPUs or GPUs.

Tip: Use Pipeline if you have a large dataset. Use Pandas for less than 10GB data.

Epoch: An epoch defines how many times you want the model to see the data. One epoch is counted when all your data is once forward and backward propagated through the entire neural network.

Lets take an example. Suppose you have a dataset with 1200 observations. You are going to use a pipeline and have created 3 batches containing 400 observations each. Now, it will take 3 iterations (1200 / 400) to completely propagate the data forward and backward through the neural network to complete one epoch.   

Estimators: It is a Tensorflow API used to implement algorithms. We can import following estimators APIs to solve a lot of classification and regression problems. 

tf.estimator.LinearRegressor
tf.estimator.LinearClassifier
tf.estimator.BoostedTreesRegressor
tf.estimator.BoostedTreesClassifier
tf.estimator.DNNClassifier
tf.estimator.DNNLinearCombinedClassifier

Estimators are used for creating computational graphs, initializing variables, training the model and saving checkpoint and logging files for Tensorboard. In order to user estimators we need to create feature columns and input functions. 

Input functions are used for passing input data to the model for training and evaluation. Feature columns are specifications for how the model should interpret the input data. We will see these concepts in detail when we solve a problem using TensorFlow in my future posts.

TensorBoard

TensorBoard enables to monitor graphically and visually what TensorFlow is doing. TensorFlow is based on graph computation; it allows the developer to visualize the construction of the neural network with Tensorboad.