A wrapper around batch_normalization

Usually I am using Sonnet, however recently an overlook of the document in-lined with source code made me thought there is a potential bug in the implementation. But when I turned to the implementation provided by TensorFlow, there is no better off. Lots of pitfalls here and there.

The following is a wrapper by me to demonstrate a user case of the routine, hope it will be useful. And I believe you know how to save and restore the variables, yes?

Enjoy coding no matter how frustrating.

import numpy as np
import tensorflow as tf
import sonnet as snt

from tensorflow.python.layers import normalization


class MyBatchNorm(object):
    def __init__(self):
        self._bn = normalization.BatchNormalization(axis = 1,
            epsilon = np.finfo(np.float32).eps, momentum = 0.9)

    def __call__(self, inputs, is_training = True, test_local_stats = False):
        outputs = self._bn(inputs, training = is_training)

        self._add_variable(self._bn.moving_mean)
        self._add_variable(self._bn.moving_variance)

        return outputs

    def _add_variable(self, var):
        if var not in tf.get_collection(tf.GraphKeys.MOVING_AVERAGE_VARIABLES):
            tf.add_to_collection(tf.GraphKeys.MOVING_AVERAGE_VARIABLES, var)

t = tf.truncated_normal([2, 4, 4, 2])


bn = MyBatchNorm()
bn2 = MyBatchNorm()

n = bn(t)
n2 = bn2(t)

update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
with tf.control_dependencies(update_ops):
    n = tf.identity(n)


with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())

    n_v, n2_v = sess.run([n, n2])

    print(tf.trainable_variables())
    print(tf.moving_average_variables())

A tricky error regarding multiple-GPU training

A must undergoing step for utilizing multiple GPU to train model is to average gradients computed by different GPUs. A typical error could happen when the gradient is partial available or stop_gradient is called into the graph. The error message is like this:

ValueError: Tried to convert 'input' to a tensor and failed. Error: None values not supported.

If it happens, try to explicitly disable trainable property of the variables.

Dynamic Programming code in TensorFlow

Following is the code implemented in TensorFlow for dynamic programming of example 4.1 from the great book: Reinforcement Learning: An Introduction. As promised at last all the pseudo code will be implemented in TensorFlow.

Enjoy it and welcome further discussion.

import tensorflow as tf

num_iters = 1000
num_states = 16

V = [tf.get_variable("V%d" % i, [], tf.float64, initializer = tf.zeros_initializer()) for i in range(num_states)]

V0 = V[0]
V1 = -0.25 * (1 - V[0] + 1 - V[1] + 1 - V[2] + 1 - V[5])
V2 = -0.25 * (1 - V[1] + 1 - V[2] + 1 - V[3] + 1 - V[6])
V3 = -0.25 * (1 - V[2] + 1 - V[3] + 1 - V[3] + 1 - V[7])
V4 = -0.25 * (1 - V[4] + 1 - V[0] + 1 - V[5] + 1 - V[8])
V5 = -0.25 * (1 - V[4] + 1 - V[1] + 1 - V[6] + 1 - V[9])
V6 = -0.25 * (1 - V[5] + 1 - V[2] + 1 - V[7] + 1 - V[10])
V7 = -0.25 * (1 - V[6] + 1 - V[3] + 1 - V[7] + 1 - V[11])
V8 = -0.25 * (1 - V[8] + 1 - V[4] + 1 - V[9] + 1 - V[12])
V9 = -0.25 * (1 - V[8] + 1 - V[5] + 1 - V[10] + 1 - V[13])
V10 = -0.25 * (1 - V[9] + 1 - V[6] + 1 - V[11] + 1 - V[14])
V11 = -0.25 * (1 - V[10] + 1 - V[7] + 1 - V[11] + 1 - V[15])
V12 = -0.25 * (1 - V[12] + 1 - V[8] + 1 - V[13] + 1 - V[12])
V13 = -0.25 * (1 - V[12] + 1 - V[9] + 1 - V[14] + 1 - V[13])
V14 = -0.25 * (1 - V[13] + 1 - V[10] + 1 - V[15] + 1 - V[14])
V15 = V[15]


delta_lst = []
for i in range(num_states):
    verbose_op = tf.Print(V[i], [tf.round(V[i])], message = "value of V(%d) = " % i)
    delta_lst.append(tf.abs(V[i] - eval("V%d" % i)))
    with tf.control_dependencies([verbose_op]):
        V[i] = tf.assign(V[i], eval("V%d" % i))

delta = tf.reduce_max(delta_lst)

stop_op = tf.cond(tf.less(delta, 0.0001), lambda: True, lambda: False)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())

    for i in range(num_iters):
        if sess.run(stop_op):
            print("\ncurrent iteration {}".format(i))
            break

        for j in range(num_states):
            sess.run(V[j])