Warning: inet_pton(): Unrecognized address 2a02:598:2:0000:0000 in /home/machinu8/public_html/wp-content/plugins/wp-ulike/inc/general-functions.php on line 1564
Exploring TensorFlow Variables: Initialization - Machine Learning Mindset

Introduction

Defining variables is necessary because of the hold the parameter. Henceforth, exploring TensorFlow variables is necessary. Without having parameters, training, updating, saving, restoring and any other operations cannot be performed. The defined variables in TensorFlow are just tensors with certain shapes and types. The tensors must be initialized with values to become valid. In this tutorial, we are going to explain how to define and initialize variables. The source code is available on the dedicated GitHub repository. To become familiar with implementing basic machine learning algorithms please refer to Learn Logistic Regression Effortlessly Using TensorFlow and Linear Regression using TensorFlow.

Creating variables

For a variable generation, the class of tf.Variable() will be used. When we define a variable, we basically pass a tensor and its value to the graph. Basically, the following will happen:

  • variable tensor that holds a value will be pass to the graph.
  • By using tf.assign, an initializer set initial variable value.

Some arbitrary variables can be defined as follows:

import tensorflow as tf
import xlrd
import matplotlib.pyplot as plt
import os
from sklearn.utils import check_random_state

# Generating artificial data.
n = 50
XX = np.arange(n)
rs = check_random_state(0)
YY = rs.randint(-20, 20, size=(n,)) + 2.0 * XX
data = np.stack([XX,YY], axis=1)

#######################
## Defining flags #####
#######################
tf.app.flags.DEFINE_integer('num_epochs', 50, 'The number of epochs for training the model. Default=50')
# Store all elemnts in FLAG structure!
FLAGS = tf.app.flags.FLAGS

In the above script, line 15 gets the list of all defined variables from the defined graph. The “name” key, define a specific name for each variable on the graph

Initialization

Initializers of the variables must be run before all other operations in the model. For an analogy, we can consider the starter of the car. Instead of running an initializer, variables can be restored too from saved models such as a checkpoint file. Variables can be initialized globally, specifically, or from other variables. We investigate different choices in the subsequent sections.

Initializing Specific Variables

By using tf.variables_initializer, we can explicitly command the TensorFlow to only initialize certain variables. The script is as follows:

# "variable_list_custom" is the list of variables that we want to initialize.
variable_list_custom = [weights, custom_variable]

# The initializer
init_custom_op = tf.variables_initializer(var_list=all_variables_list)

Noted that custom initialization does not mean that we don’t need to initialize other variables! All variables that some operations will be done upon them over the graph, must be initialized or restored from saved variables. This only let us realize how we can initialize specific variables by hand.

Golobal variable initialization

All variables can be initialized at once using the tf.global_variables_initializer(). This op must be run after the model being fully constructed. The script is as below:

# Add an op to initialize the variables.
init_all_op = tf.global_variables_initializer()

# Method-2
init_all_op = tf.variables_initializer(var_list=all_variables_list)

Both the above methods are identical. We only provide the second one to demonstrate that the tf.global_variables_initializer() is nothing but tf.variables_initializer when you yield all the variables as its input argument.

Initialization of a variable using other existing variables

New variables can be initialized using other existing variables’ initial values by taking the values using initialized_value().

# Create another variable with the same value as 'weights'.
WeightsNew = tf.Variable(weights.initialized_value(), name="WeightsNew")

# Now, the variable must be initialized.
init_WeightsNew_op = tf.variables_initializer(var_list=[WeightsNew])

As it can be seen from the above script, the WeightsNew variable is initialized with the values of the weights predefined value.

Running the session

All we did so far was to define the initializers ops and put them on the graph. In order to truly initialize variables, the defined initializers’ ops must be run in the session. The script is as follows:

with tf.Session() as sess:
    # Run the initializer operation.
    sess.run(init_all_op)
    sess.run(init_custom_op)
    sess.run(init_WeightsNew_op)

Each of the initializers has been run separated using a session.

Summary

In this tutorial, we walked through the variable creation and initialization. The global, custom and inherited variable initialization have been investigated. In future posts, we investigate how to save and restore the variables. Restoring a variable eliminate the necessity of its initialization.

0

Amirsina Torfi

Currently, as a CS Ph.D. student, I'm a research assistant at Virginia Tech. My research is mainly about Machine Learning & Deep Learning and their applications in Computer Vision and NLP. I'm interested in developing software packages and open-source projects.

Leave a Reply

avatar
  Subscribe  
Notify of
Tweet
Share
Pin
Share