How to prepare tfrecords utilizing TensorFlow for training models (II)

The following code snippet is the corresponding code for retrieving tfrecords from the prepared tfrecords files just in the previous post. Hope it's useful for relieving some difficulties for beginners.

'''
@author: Yurui Ming (yrming@gmail.com)
'''
import tensorflow as tf
import os
import skimage.io as io

class TFRecordPumper(object):
    '''
    classdocs
    '''

    def __init__(self, graph = None, sess = None):
        '''
        Constructor
        '''
        if graph == None:
            self._graph = tf.Graph()
        else:
            self._graph = graph
        
        if sess == None:
            self._sess = tf.Session(graph = self._graph)
            self._self_sess = True
        else:
            self._sess = sess
            self._self_sess = False
    
    def __exit__(self):
        if self._coord:
            self._coord.request_stop()
            self._coord.join(self._threads)
        
        if self._self_sess == True:
            self._sess.close()
        
    
    def Pump(self, tfr_dir, tfr_basename, batch_size = 2, features = None, img_shape = None,
             capacity = 10, num_threads = 1, min_after_dequeue = 5):
        '''
        Pump
        pumping out tfrecords
        Args:
            tfr_dir: directory contains tfrecords file
            tfr_basename: basename pattern for collecting tfrecords files
            batch_size: batch number of tfrecords to pump each time
            features: features describing tfrecords
        '''
        
        # assume the most general feature if nono provided
        if features == None:
            features = {'image': tf.FixedLenFeature([], tf.string),
                        'label': tf.FixedLenFeature([1], tf.int64)
                        }
        
        with self._graph.as_default():
            ptn = os.path.join(tfr_dir, tfr_basename + "*.tfrecords")
        
            filenames = tf.train.match_filenames_once(ptn)
            
            tf_record_filename_queue = tf.train.string_input_producer(filenames)
            
            # Notice the different record reader, this one is designed to work with TFRecord files which may
            # have more than one example in them.
            
            tf_record_reader = tf.TFRecordReader()
            _, tf_record_serialized = tf_record_reader.read(tf_record_filename_queue)
            
            # The label and image are stored as bytes but could be stored as int64 or float64 values in 
            # serialized tf.Example protobuf.
            if 'train' in tfr_basename:
                label_key = 'train/label'
                image_key = 'train/image'
            elif 'xval' in tfr_basename:
                label_key = 'xval/label'
                image_key = 'xval/image'
            elif 'test' in tfr_basename:
                label_key = 'test/label'
                image_key = 'test/image'
            else:
                label_key = 'label'
                image_key = 'image'
                
            tf_record_features = tf.parse_single_example(tf_record_serialized,
                                                         features = {
                                                             label_key: tf.FixedLenFeature([], tf.int64),
                                                             image_key: tf.FixedLenFeature([], tf.string),
                                                             })
            
            # Using tf.uint8 because all of the channel information is between 0-255
            tf_record_image = tf.reshape(tf_record_features[image_key], [])
            
            tf_record_image = tf.decode_raw(tf_record_image, tf.uint8)
            
            # Reshape the image to look like the image saved, not required
            if img_shape:
                tf_record_image = tf.reshape(tf_record_image, img_shape)
            
            # Use real values for the height, width and channels of the image because it's required
            # to reshape the input.
            
            tf_record_label = tf_record_features[label_key];
            
            
            images, labels = tf.train.shuffle_batch([tf_record_image, tf_record_label],
                                                    batch_size = batch_size,
                                                    capacity = capacity,
                                                    min_after_dequeue = min_after_dequeue,
                                                    num_threads = num_threads)
            
            init_op = tf.group(tf.global_variables_initializer(), tf.local_variables_initializer())

            with self._sess.as_default():
                self._sess.run(init_op)
                
                self._coord = tf.train.Coordinator()
                self._threads = tf.train.start_queue_runners(coord = self._coord)

                #images, labels = self._sess.run([tf_record_image, tf_record_label])
                
                yield self._sess.run([images, labels])
            
                
if __name__ == '__main__':
    tf_pumper = TFRecordPumper()
    #images, labels = tf_pumper.Pump('', 'train', img_shape = [64, 64, 3])

    images, labels = next(tf_pumper.Pump('', 'xval', img_shape = [64, 64, 3]))
    
    for i in range(images.shape[0]):
        io.imshow(images[i, ...])
        
    io.show() 

How to prepare tfrecords utilizing TensorFlow for training models

The merit of utilizing tfrecords is manifest, since high throughput of feeding can obviously keep the training iteration from starving. A precondition is one should have tfrecords prepared before launching the whole process. The general guidelines could be easily understood however since example codes are scattered here and there, so it's not easy for assembling the snippets to form something actually workable. The following code has such an aim and intention in mind, so hope it's useful for everybody's work concerning deep learning. BTW no hesitate for providing any feedback concerning improvement of the code quality.

'''
@author: Yurui Ming (yrming@gmail.com)
'''
import numpy as np
import tensorflow as tf
import os

class TFRecordGenerator(object):
    '''
    classdocs
    '''
    def __init__(self, params = None):
        '''
        Constructor
        '''
        self._graph = tf.Graph()
    def _int64_feature(self, value):
        return tf.train.Feature(int64_list = tf.train.Int64List(value = [value]))

    def _bytes_feature(self, value):
        return tf.train.Feature(bytes_list = tf.train.BytesList(value = [value]))
    
    def Generate(self, img_dir, img_fmt = None, img_shape = [64, 64], partition = [0.8, 0.1, 0.1], 
                  train_tfrecord_base_name = 'train{}.tfrecords',
                  xval_tfrecord_base_name = 'xval{}.tfrecords',
                  test_tfrecord_base_name = 'test{}.tfrecords', 
                  split_unit = 500):
        '''
        Generate
        Generate TFRecord files
        Three categories of TFRecord files will be generated, namely, training category, cross-validating category and testing category
        Args:
            img_dir: directory containing the images. The label should be decided from the training name
            img_fmt: image encoding standard, e.g., jpeg or png
            partition: portions of percentage of each category, namely, training, cross-validating and testing
            train_tfrecord_base_name: base training tfrecord file name paradigm for generating training tfrecord file name
            xval_tfrecord_base_name: base cross-validating tfrecord file name paradigm for generating cross-validating tfrecord file name
            test_tfrecord_base_name: base testing tfrecord file name paradigm for generating testing tfrecord file name
            split_unit: number of accumulated tfrecords in each tfrecord file 
        '''
        if not img_fmt:
            raise ValueError('Unspecified image format')
                    
        with self._graph.as_default():
            ptn = None
            if 'jpg' in img_fmt:
                ptn = os.path.join(img_dir, '*.jpg')
            if 'png' in img_fmt:
                ptn = os.path.join(img_dir, '*.png')
            if not ptn:
                raise ValueError('Unsupported image format')

            filenames = tf.train.match_filenames_once(ptn)
            filename_queue = tf.train.string_input_producer(filenames)
            image_reader = tf.WholeFileReader()
            image_key, image_file = image_reader.read(filename_queue)

            if 'jpg' in img_fmt:
                image_data = tf.image.decode_jpeg(image_file)
            if 'png' in img_fmt:
                image_data = tf.image.decode_png(image_file)
         
            image_data_shape = tf.shape(image_data)
            
            if img_shape:
                image_data = tf.cond(image_data_shape[0] > image_data_shape[1], \
                                     lambda: tf.image.resize_image_with_crop_or_pad(image_data, image_data_shape[1], image_data_shape[1]),
                                     lambda: tf.image.resize_image_with_crop_or_pad(image_data, image_data_shape[0], image_data_shape[0]))
                
                image_data = tf.image.resize_images(image_data, img_shape)        
            
            image_data = tf.cast(image_data, tf.uint8)
            
            #image_data = tf.image.encode_jpeg(image_data);
                
            init = tf.group(tf.global_variables_initializer(), tf.local_variables_initializer())
            
            with tf.Session() as sess:
                sess.run(init)
                
                coord = tf.train.Coordinator()
                threads = tf.train.start_queue_runners(sess = sess, coord = coord)
                
                num_files = len(sess.run(filenames))
                
                if np.sum(partition) > 1:
                    raise ValueError('Invalid partition')
                
                partition = [v * num_files for v in partition]
                
                # training tfrecord category
                writer = None
                for i in range(int(partition[0])):
                    if not i % split_unit:
                        if writer:
                            writer.close()
                        train_filename = train_tfrecord_base_name.format(i)
                        writer = tf.python_io.TFRecordWriter(train_filename)
                        
                    image_label, image_cont = sess.run([image_key, image_data])
                    
                    if b'cat' in image_label:
                        label = 0
                    elif b'dog' in image_label:
                        label = 1
                    else:
                        raise ValueError('Invalid file name: {}'.format(image_label))
                    
                    feature = {
                        'train/label': self._int64_feature(label),
                        'train/image': self._bytes_feature(image_cont.tobytes())
                        }

                    
                    example = tf.train.Example(features = tf.train.Features(feature = feature))
                    writer.write(example.SerializeToString())
                writer.close()
                
                writer = None
                for i in range(int(partition[1])):
                    if not i % split_unit:
                        if writer:
                            writer.close()
                        xval_filename = xval_tfrecord_base_name.format(i)
                        writer = tf.python_io.TFRecordWriter(xval_filename)

                    image_label, image_cont = sess.run([image_key, image_data])

                    if b'cat' in image_label:
                        label = 0
                    elif b'dog' in image_label:
                        label = 1
                    else:
                        raise ValueError('Invalid file name: {}'.format(image_label))
                    
                    feature = {
                        'xval/label': self._int64_feature(label),
                        'xval/image': self._bytes_feature(image_cont.tobytes())
                        }

                    example = tf.train.Example(features = tf.train.Features(feature = feature))
                    writer.write(example.SerializeToString())
                writer.close()
                
                writer = None
                for i in range(int(partition[2])):
                    if not i % split_unit:
                        if writer:
                            writer.close()
                        test_filename = test_tfrecord_base_name.format(i)
                        writer = tf.python_io.TFRecordWriter(test_filename)

                    image_label, image_cont = sess.run([image_key, image_data])

                    if b'cat' in image_label:
                        label = 0
                    elif b'dog' in image_label:
                        label = 1
                    else:
                        raise ValueError('Invalid file name: {}'.format(image_label))
                    
                    feature = {
                        'test/label': self._int64_feature(label),
                        'test/image': self._bytes_feature(image_cont.tobytes())
                        }

                    example = tf.train.Example(features = tf.train.Features(feature = feature))
                    writer.write(example.SerializeToString())

                
                writer.close()
                writer = None
            
                coord.request_stop()
                coord.join(threads)
            
if __name__ == '__main__':
    tf_generator = TFRecordGenerator()
    tf_generator.Generate('C:\\Users\\MSUser\\Downloads\\mytest', 'jpg')

Derivative of softmax with cross-entropy as loss function

The following are diagram and detailed procedure of how to obtain the derivative of softmax with cross-entropy as loss function. As back-propagation, the sensitivity map into output layer can be combined with the derivative of loss function and the derivative of activation function of output layer. That's the theoretic basis for the beautiful result in this post.

How to add another kernel in Anaconda

The latest release of Anaconda officially has Python 3.6 supported. However When I try to create a virtual environment to install TensorFlow 1.0, it complains no matching version found. So I have to create a virtual environment to install TensorFlow 1.0.

I probably too rush to do things, so I create a virtual environment by issuing:
conda create -n tensorflow

This is the nightmare begins. Later I realise I should clone things from the default root environment by issuing the following command:
conda create -n tensorflow --clone root

But that's another story.

Now activate it to install the proper python version:
conda search python
conda install python=3.5.3

To my surprise, when I start the notebook, only one kernel can be found. So it obviously we cannot expect things go as expected by just install python.

Seems now we need generate the kernel spec file for our current virtual environment
activate tensorflow
ipython kernel install

Generally the spec file will be created in the corresponding virtual environment directory of jupyter. Following the output of installation process to find it and do some modification. The most important one is changing the executable to the proper python.

When I try to start the notebook, it complains ipykernel couldn't be found.

So now let's install another packages:
conda install ipykernel
conda install ipywidgets

Now start the notebook, everything should be fine.

How to extend Python on Windows (Deep Learning Course on Udacity related)

I am recently learning Deep Learning from Udacity.

For 1_notmnist.ipynb, Problem 1: Let's take a peek at some of the data to make sure it looks sensible. Each exemplar should be an image of a character A through J rendered in a different font. Display a sample of the images that we just downloaded. Hint: you can use the package IPython.display.

One alternative way might be the following code:

import os, fnmatch

img_files = []

def all_img_files(img_files, search_path, pattern = '*.png'):

    for path, subdirs, files in os.walk(search_path):

        if files and fnmatch.fnmatch(files[0], pattern):

            img_files.append(os.path.join(path, files[0]))

            break;

                

for folder in train_folders:

    all_img_files(img_files, folder)

        

for folder in test_folders:

    all_img_files(img_files, folder)

      

for img in img_files:

    Image(filename = img)

However I found it's extremely slow, probably due to every sub-directories and files will be gathered on os.walk returning. the break statement has only a little affect on the whole processing time.

So I decide to write some code which genuinely fetches the first png file in each A-J directories respectively. Readers can follow the below linkage for reference on how the work can be down via VC++:

https://msdn.microsoft.com/en-us/library/windows/desktop/aa365200(v=vs.85).aspx

There's plenty of material on how to do that, namely write extension for Python, so the following is just source code without much explanation. I did it with Anaconda python 3.5 with Visual C++ 2015. Other platforms probably need some adjustment:

#include <Python.h>

#include <tchar.h> 

#include <stdio.h>

#include <strsafe.h>

#include <Windows.h>

#include <Shlwapi.h>

#include "deelx.h"

#pragma comment(lib, "python35.lib")

#pragma comment(lib, "User32.lib")

#pragma comment(lib, "Shlwapi.lib")

static PyObject *get_first_matched_file_error;

static PyObject* get_first_matched_file(PyObject* self, PyObject* args)

{

WIN32_FIND_DATA ffd;

TCHAR szDir[MAX_PATH];

HANDLE hFind = INVALID_HANDLE_VALUE;

DWORD dwError = 0;

int wchars_num;

char* directoryA;

wchar_t* directoryW;

char* patternA;

wchar_t* patternW;

if (!PyArg_ParseTuple(args, "sz", &directoryA, &patternA))

return NULL;

wchars_num = MultiByteToWideChar(CP_UTF8, 0, directoryA, -1, NULL, 0);

directoryW = new wchar_t[wchars_num];

MultiByteToWideChar(CP_UTF8, 0, directoryA, -1, directoryW, wchars_num);

if (!PathFileExists(directoryW))

{

PyErr_SetString(get_first_matched_file_error, "Non-existing directory");

delete[] directoryW;

return NULL;

}

// Prepare string for use with FindFile functions.  First, copy the

// string to a buffer, then append '\*' to the directory name.

StringCchCopy(szDir, MAX_PATH, directoryW);

delete[] directoryW;

StringCchCat(szDir, MAX_PATH, TEXT("\\*"));

wchars_num = MultiByteToWideChar(CP_UTF8, 0, patternA, -1, NULL, 0);

patternW = new wchar_t[wchars_num];

MultiByteToWideChar(CP_UTF8, 0, patternA, -1, patternW, wchars_num);

CRegexpT <wchar_t> regexp(patternW);

// Find the first file in the directory.

hFind = FindFirstFile(szDir, &ffd);

if (INVALID_HANDLE_VALUE == hFind)

{

delete[] patternW;

PyErr_SetString(get_first_matched_file_error, "Cannot open directory");

return NULL;

}

PyObject * pyFileName = NULL;

// List all the files in the directory with some info about them.

do

{

if (ffd.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY)

{

continue;

}

else

{

MatchResult result = regexp.Match(ffd.cFileName);

if (result.IsMatched())

{

char* cFileName;

int chars_num;

chars_num = WideCharToMultiByte(CP_UTF8, 0, ffd.cFileName, -1, NULL, 0, NULL, NULL);

cFileName = new char[chars_num];

WideCharToMultiByte(CP_UTF8, 0, ffd.cFileName, -1, cFileName, chars_num, NULL, NULL);

pyFileName = Py_BuildValue("s", cFileName);

delete[] cFileName;

break;

}

}

} while (FindNextFile(hFind, &ffd) != 0);

if (GetLastError() == ERROR_NO_MORE_FILES)

pyFileName = Py_BuildValue("s", "");

FindClose(hFind);

delete[] patternW;

return pyFileName;

}

static PyMethodDef get_first_matched_file_method[] = {

{

"get_first_matched_file",  get_first_matched_file,

METH_VARARGS, "Get the first file given directory and pattern"

},

{NULL, NULL, 0, NULL}        /* Sentinel */

};

static struct PyModuleDef get_first_matched_file_module =

{

PyModuleDef_HEAD_INIT,

"get_first_matched_file", /* name of module */

"Get the first file given directory and pattern",          /* module documentation, may be NULL */

-1,          /* size of per-interpreter state of the module, or -1 if the module keeps state in global variables. */

get_first_matched_file_method

};

PyMODINIT_FUNC PyInit_get_first_matched_file(void)

{

PyObject *m = PyModule_Create(&get_first_matched_file_module);

if (m == NULL)

return NULL;

get_first_matched_file_error = PyErr_NewException("get_first_matched_file.error", NULL, NULL);

Py_INCREF(get_first_matched_file_error);

PyModule_AddObject(m, "error", get_first_matched_file_error);

return m;

}

There's only one dependency, namely deelx.h, referring to the websites below:

http://www.regexlab.com/

https://www.codeproject.com/articles/12595/greta-how-to-use-regular-expressions-in-visual-c-c

A testing script is as follows:

import sys

sys.path.append("C:\\Users\\MS User\\Documents\\Visual Studio 2015\\Projects\\PythonExtensions\\x64\\Release")

import get_first_matched_file

directory = "C:\\tensorflow\\tensorflow\\examples\\udacity\\notMNIST_large\\B"

pattern = "\\.png$"

file = get_first_matched_file.get_first_matched_file(directory, pattern)

print(file)

Enjoy python, enjoy learning from Udacity.

Project setting:

How to change serving directory of Jupyter on Windows

Sometimes it's convenient altering the default directory which Jupyter serving from. For example, I prefer it serving from C:\tensorflow\tensorflow\examples\udacity since I git clone everything there.

First run the following command to generate the configuration file nammed jupyter_notebook_config.py, usually it resides in the .jupyter folder in your home directory:
jupyter notebook --generate-config


Now open the file and search the following line:
#c.NotebookApp.notebook_dir = ''

Uncomment it, put the target directory into the semicolon. Since on Windows platform, so we need to escape the backslash character:
c.NotebookApp.notebook_dir = 'C:\\tensorflow\\tensorflow\\examples\\udacity'

Final result:

How to use Anaconda

Anaconda is a one stop distribution of Python related scientific computing components. Probably its convenience is more obvious on Windows instead of Linux. Following is some summary of how to use conda on Windows platform.

To avoid anything unexpected happened, suggest to start Anaconda by choosing from start menu and launch the Anaconda Prompt.

1. Show all virtual environments created:

conda info --envs

2. Activate specific environment, like root:

activate root

3. Deactive specific environment, like root:

deactivate

There’s no need to append the option value since the command is aware of which environment it's currently in.

However, try not to deactivate the default root environment, since on *nix platform, it will try to remove Anaconda path variable from current shell environment variable. What you need to do is just switch to another virtual environment, and conda is clever enough to deactivate the previous one.

4. Create specified environment (here take “default” as an example) and initially with specified lib to be installed (here take “matplotlib” as an example):

conda create -n default matplotlib 

5. Create specified environment (for instance “default”) by clone another (here root)

conda create -n default --clone root

6. List the packages installed into specified environment (for instance “default”):

conda list -n default

7. Install package (here with option value “tensorflow”) into the current environment:

conda install tensorflow

8. Install package (as an example, “tensorflow”) into the specified environment (here with name root):

conda install -n root tensorflow

9. Search uncommon package in Anaconda website:

anaconda search -t conda package-name 

10. Show detail about found package

anaconda show user/package-name

11. Install specific package from specified channel:

conda install --channel https://conda.anaconda.org/user package-name

12. Install specific package (like tensorflow) with pip (it’s recommended to do in some virtual environment) by virtue of auto-resolving dependency:

pip instll tensorflow