Lesson learnt from twiddling Mandlebrot Fractals with TensorFlow

I don't know it should be called a real lesson since I am recently exposed to TensorFlow. Since I guess my later work involves visualization of massive data, so I prefer things can be done on Windows (Just my preference to the old Windows API). So the first time I read the Mandelbrot example in the book "Get started with TensorFlow", I wondered could I visualize the process in Windows. So I did a slight modification of the example, as follows:

import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt

Y, X = np.mgrid[-1.3:1.3:0.005, -2:1:0.005] 

Z = X + 1j*Y
c = tf.constant(Z.astype(np.complex64))

zs = tf.Variable(c)
ns = tf.Variable(tf.zeros_like(c, tf.float32))

zs_square = tf.pow(zs, 2)
zs_final = tf.add(zs_square, c)

not_diverged = tf.complex_abs(zs_final) < 4

update = tf.group(zs.assign(zs_final), ns.assign_add(tf.cast(not_diverged, tf.float32)), name = "update")
output = tf.identity(ns, name="output")

saver = tf.train.Saver()

sess = tf.Session()
sess.run(tf.initialize_all_variables())

#tf.assign(zs, sess.run(zs))
#tf.assign(ns, sess.run(ns))

tf.train.write_graph(sess.graph_def, "models/", "graph.pb", as_text = True)

saver.save(sess, "models/model.ckpt")

for i in range(200):
    sess.run(update)

plt.imshow(sess.run(ns))
plt.show()

sess.close()


And it got the wonderful Mandelbrot fractal:

In order to appreciate the whole process generating the Mandelbrot fractal, I wonder possible I freeze the model, load it under Windows. For each step, I run the "update" node, then fetch the result via "output" node.

However, it's not applicable since when freezing, the variables will be substituted with constants, and it cannot be later updated again. See the following complain:

I think it's quite understandable, because the intention of TensorFlow is to let the trained model run as quickly as possible, so no surprise that variables are eliminated finally.

If we uncomment the following lines and save the graph as binary:
tf.assign(zs, sess.run(zs))
tf.assign(ns, sess.run(ns))

tf.train.write_graph(sess.graph_def, "models/", "graph.pb", as_text = False)

The fact is it seems the variable didn't get initialized if we explicitly do it in the program:

I am still working on it, but log it here in reminding someone who's applying TensorFlow to the scenario that requires no explicit input, please rethink about it.

Handwritten digits recognition via TensorFlow based on Windows MFC (V) - Result demo

The final finised project named DigitRecognizer developed under Visual Studio 2015, referring to the following video for an demonstration.

However, still a long way to go to master TensorFlow.

Thanks for the guys at Google for developing TensorFlow, however support of bazel on Windows still need more improvements.

Thanks guys at Microsoft for developing Visual Studio, which always cease the pain for development on Windows.

Thanks guys make and still bread Machine Learning, will always need to learn from you and to show my appreciations.

Happy learning, happy coding!

Handwritten digits recognition via TensorFlow based on Windows MFC (IV) - Load trained model

I think two good article have detailed everything, thanks a lot to their efforts:
https://medium.com/jim-fleming/loading-a-tensorflow-graph-with-the-c-api-4caaff88463f#.t78tjznzu, by Jim Fleming; http://jackytung8085.blogspot.kr/2016/06/loading-tensorflow-graph-with-c-api-by.html by Jacky Tung.

So I directly paste the code here for reference:

MnistModel.cc:

#include<Windows.h>

#include <stdio.h>

#include <vector>
#include <string>
#include <sstream>
#include <iostream>
#include <utility>

#include "tensorflow/core/public/session.h"
#include "tensorflow/core/platform/env.h"

#include "MNistComm.h"

using std::vector;
using std::string;
using std::ostringstream;
using std::endl;
using std::pair;

using namespace tensorflow;

void fillErrMsg(MNIST_COMM_ERROR *err, MNIST_ERROR_CODE c, Status& status)
{
    memset(err, 0, sizeof(MNIST_COMM_ERROR));
        
    err->err = c;
        
    ostringstream ost;
    ost << status.ToString() << endl;
        
    snprintf(err->msg, MAX_MSG_SIZ, "%s", ost.str().c_str());
}

// Windows are Unicode supportted, so everything is natively Unicode
int wmain(wchar_t* argc, wchar_t* argv[])
{    
    // Open file mapping object
    MnistShm mnistShm(false);
    if (!mnistShm)
        return MNIST_OPEN_SHM_FAILED;        
   
    MnistEvent mnistEvent(false);
    if (!mnistEvent)
        return MNIST_OPEN_EVT_FAILED;

    Session* session = NULL;
    Status status = NewSession(SessionOptions(), &session);
    if(!status.ok())
    {
        MNIST_COMM_ERROR err;
        fillErrMsg(&err, MNIST_SESSION_CREATION_FAILED, status);
        mnistShm.SetError(reinterpret_cast<char*>(&err));
        
        return MNIST_SESSION_CREATION_FAILED;
    }
        
    char modelPath[MAX_PATH];
    CMnistComm::WChar2Char(modelPath, argv[1], MAX_PATH - 1);
    
    GraphDef graph_def;    
    status = ReadBinaryProto(Env::Default(), modelPath, &graph_def);
    if (!status.ok())
    {        
        MNIST_COMM_ERROR err;
        fillErrMsg(&err, MNIST_MODEL_LOAD_FAILED, status);
        mnistShm.SetError(reinterpret_cast<char*>(&err));

        return MNIST_MODEL_LOAD_FAILED;
    }
    
    status = session->Create(graph_def);
    if (!status.ok()) {

        MNIST_COMM_ERROR err;
        fillErrMsg(&err, MNIST_GRAPH_CREATION_FAILED, status);
        mnistShm.SetError(reinterpret_cast<char*>(&err));

        return MNIST_GRAPH_CREATION_FAILED;
    }
    
    // Setup inputs and outputs:
    Tensor img(DT_FLOAT, TensorShape({1, MNIST_IMG_DIM}));

    MNIST_COMM_EVENT evt;
    
    while (evt = mnistEvent.WaitForEvent(MNIST_EVENT_PROC))
    {        
        auto buf = img.flat<float>().data();
    
        mnistShm.GetImageData(reinterpret_cast<char*>(buf));

        vector<pair<string, Tensor>> inputs = {
            { "input", img}
        };
        
        // The session will initialize the outputs
        vector<Tensor> outputs;
        // Run the session, evaluating our "logits" operation from the graph
        status = session->Run(inputs, {"recognize"}, {}, &outputs);
        if (!status.ok()) {
            MNIST_COMM_ERROR err;
            fillErrMsg(&err, MNIST_MODEL_RUN_FAILED, status);
            mnistShm.SetError(reinterpret_cast<char*>(&err));
            
            return MNIST_MODEL_RUN_FAILED;
        }
        
        auto weights = outputs[0].shaped<float, 1>({10});
        int index = 0;
        int digit = -1;
        
        float min_ = 0.0;
        for (int i = 0; i < 10; i ++, index ++)
        {
            if (weights(i) > min_)
            {
                min_ = weights(i);
                digit = index;
            }
        }
                
        mnistShm.SetImageLabel(reinterpret_cast<char*>(&digit));
        mnistEvent.NotifyReady();
                
    }

    session->Close();
    
    return 0;
}

Handwritten digits recognition via TensorFlow based on Windows MFC (III) - Inter-process Communication framework (II)

MnistComm.h:

#pragma once

#include <Windows.h>
#include <stdio.h>
#include <conio.h>
#include <tchar.h>
#include <stddef.h>

#include <map>
#include <string>

using std::map;
using std::string;

// Data structure definitions
typedef enum tagMNIST_ERROR_CODE
{
MNIST_OK = 0,
MNIST_CREATE_SHM_FAILED = 1,
MNIST_OPEN_SHM_FAILED = 2,
MNIST_CREATE_EVT_FAILED = 3,
MNIST_OPEN_EVT_FAILED = 4,

MNIST_SESSION_CREATION_FAILED = 10,
MNIST_MODEL_LOAD_FAILED = 11,
    MNIST_GRAPH_CREATION_FAILED = 12,
MNIST_MODEL_RUN_FAILED = 13,

} MNIST_ERROR_CODE;

#define MAX_MSG_SIZ 255

typedef struct tagMNIST_COMM_ERROR
{
MNIST_ERROR_CODE err;
char msg[MAX_MSG_SIZ + 1];
} MNIST_COMM_ERROR;


#define MNIST_IMG_HEIGHT 28
#define MNIST_IMG_WIDTH 28
#define MNIST_IMG_DIM (MNIST_IMG_HEIGHT * MNIST_IMG_WIDTH)
#define MNIST_IMG_SIZ (MNIST_IMG_DIM * sizeof(float))
typedef struct tagMNIST_IMG_LABEL
{
float img[MNIST_IMG_HEIGHT][MNIST_IMG_WIDTH];
int label;
} MNIST_IMG_LABEL;

typedef struct tagMNIST_COMM_SHM_LAYOUT
{
MNIST_COMM_ERROR mnist_err;
MNIST_IMG_LABEL mnist_data;
} MNIST_COMM_SHM_LAYOUT;

class MnistShm
{
public:
MnistShm(bool host);
~MnistShm();

bool operator !()
{
return !m_bInitialized;
}

bool GetError(char* pBuf)
{
CopyMemory(pBuf, m_pBuf + offsetof(MNIST_COMM_SHM_LAYOUT, mnist_err), sizeof(MNIST_COMM_ERROR));
return true;
}

bool SetError(char* pBuf)
{
CopyMemory(m_pBuf + offsetof(MNIST_COMM_SHM_LAYOUT, mnist_err), pBuf, sizeof(MNIST_COMM_ERROR));
return true;
}

bool GetImageData(char* pBuf)
{
CopyMemory(pBuf, m_pBuf + offsetof(MNIST_COMM_SHM_LAYOUT, mnist_data), MNIST_IMG_SIZ);
return true;
}

bool SetImageData(char* pBuf)
{
CopyMemory(m_pBuf + offsetof(MNIST_COMM_SHM_LAYOUT, mnist_data), pBuf, MNIST_IMG_SIZ);
return true;
}

bool GetImageLabel(char* pBuf)
{
CopyMemory(pBuf, m_pBuf + offsetof(MNIST_COMM_SHM_LAYOUT, mnist_data) + MNIST_IMG_SIZ, sizeof(int));
return true;
}

bool SetImageLabel(char* pBuf)
{
CopyMemory(m_pBuf + offsetof(MNIST_COMM_SHM_LAYOUT, mnist_data) + MNIST_IMG_SIZ,
pBuf, sizeof(int));
return true;
}


private:
static wchar_t* MnistShmName;

bool m_bInitialized;
HANDLE m_hMapFile;
char* m_pBuf;
};

typedef enum tagMNIST_COMM_EVENT
{
MNIST_EVENT_NULL = -1,
MNIST_EVENT_QUIT = 0,
MNIST_EVENT_PROC = 1,
MNIST_EVENT_REDY = 2,
MNIST_EVENT_COUNT = 3,
} MNIST_COMM_EVENT;

typedef struct tagMNIST_EVENT_NAME
{
MNIST_COMM_EVENT e;
wchar_t* name;
} MNIST_EVENT_NAME;

typedef HANDLE MNIST_EVENT_HANDLE[MNIST_EVENT_COUNT];

class MnistEvent
{
public:
MnistEvent(bool host);
~MnistEvent();

bool operator !()
{
return !m_bInitialized;
}

bool NotifyQuit()
{
return PulseEvent(m_hEvt[MNIST_EVENT_QUIT]);
}

bool NotifyProc()
{
return PulseEvent(m_hEvt[MNIST_EVENT_PROC]);
}

bool NotifyReady()
{
return PulseEvent(m_hEvt[MNIST_EVENT_REDY]);
}

MNIST_COMM_EVENT WaitForEvent(MNIST_COMM_EVENT event)
{
DWORD dwEvent;
MNIST_COMM_EVENT e;

do
{
dwEvent = WaitForMultipleObjects(MNIST_EVENT_COUNT, m_hEvt, FALSE, INFINITE);
e = static_cast<MNIST_COMM_EVENT>(dwEvent - WAIT_OBJECT_0);
if (e == event)
break;
} while (e != MNIST_EVENT_QUIT);

return e;
}

private:
static MNIST_EVENT_NAME MnistEventName[MNIST_EVENT_COUNT];

bool m_bHost;
bool m_bInitialized;
MNIST_EVENT_HANDLE m_hEvt;
};


class CMnistComm
{
public:
CMnistComm();
~CMnistComm();

static bool Char2WChar(char* cs, wchar_t* wcs, int size)
{
return swprintf(wcs, size, L"%S", cs);
}

static bool WChar2Char(char* cs, wchar_t* wcs, int size)
{
return snprintf(cs, size, "%S", wcs);
}

};



MnistComm.cpp:

#include "MnistComm.h"

wchar_t* MnistShm::MnistShmName = _T("MnistSharedMemory");

MnistShm::MnistShm(bool host) : m_bInitialized(false), m_hMapFile(NULL)
{
if (host)
m_hMapFile = CreateFileMapping(INVALID_HANDLE_VALUE, NULL, PAGE_READWRITE,
0, sizeof(MNIST_COMM_SHM_LAYOUT), MnistShmName);
else
m_hMapFile = OpenFileMapping(FILE_MAP_ALL_ACCESS, FALSE, MnistShmName);

if (m_hMapFile == NULL)
{
OutputDebugString(host ? TEXT("Could not create file mapping object") :
TEXT("Could not open file mapping object"));
return;
}

m_pBuf = reinterpret_cast<char*>(MapViewOfFile(m_hMapFile, FILE_MAP_ALL_ACCESS, 0, 0, sizeof(MNIST_COMM_SHM_LAYOUT)));
if (m_pBuf == NULL)
{
OutputDebugString(TEXT("Could not map view of file"));
return;
}

m_bInitialized = true;
}

MnistShm::~MnistShm()
{
if (m_pBuf)
UnmapViewOfFile(m_pBuf);
if (m_hMapFile)
CloseHandle(m_hMapFile);
}

MNIST_EVENT_NAME MnistEvent::MnistEventName[] =
{
{ MNIST_EVENT_QUIT, TEXT("MnistEventQuit") },
{ MNIST_EVENT_PROC, TEXT("MnistEventProc") },
{ MNIST_EVENT_REDY, TEXT("MnistEventRedy") },
};

MnistEvent::MnistEvent(bool host) :
m_bHost(host),
m_bInitialized(false),
m_hEvt{ 0 }
{
if (host)
{
m_hEvt[MNIST_EVENT_QUIT] = CreateEvent(NULL, TRUE, FALSE, MnistEventName[MNIST_EVENT_QUIT].name);
m_hEvt[MNIST_EVENT_PROC] = CreateEvent(NULL, TRUE, FALSE, MnistEventName[MNIST_EVENT_PROC].name);
m_hEvt[MNIST_EVENT_REDY] = CreateEvent(NULL, TRUE, FALSE, MnistEventName[MNIST_EVENT_REDY].name);
}
else
{
m_hEvt[MNIST_EVENT_QUIT] = OpenEvent(EVENT_ALL_ACCESS, FALSE, MnistEventName[MNIST_EVENT_QUIT].name);
m_hEvt[MNIST_EVENT_PROC] = OpenEvent(EVENT_ALL_ACCESS, FALSE, MnistEventName[MNIST_EVENT_PROC].name);
m_hEvt[MNIST_EVENT_REDY] = OpenEvent(EVENT_ALL_ACCESS, FALSE, MnistEventName[MNIST_EVENT_REDY].name);

}

if (m_hEvt[MNIST_EVENT_QUIT] == NULL)
{
OutputDebugString(host ? TEXT("Could not create quit event object") :
TEXT("Could not open quit event object"));
return;
}

if (m_hEvt[MNIST_EVENT_PROC] == NULL)
{
OutputDebugString(host ? TEXT("Could not create processing event object") :
TEXT("Could not open processing event object"));
return;
}

if (m_hEvt[MNIST_EVENT_REDY] == NULL)
{
OutputDebugString(host ? TEXT("Could not create ready event object") :
TEXT("Could not open ready event object"));
return;
}

m_bInitialized = true;
}

MnistEvent::~MnistEvent()
{
if (m_hEvt[MNIST_EVENT_QUIT])
CloseHandle(m_hEvt[MNIST_EVENT_QUIT]);

if (m_hEvt[MNIST_EVENT_PROC])
CloseHandle(m_hEvt[MNIST_EVENT_PROC]);

if (m_hEvt[MNIST_EVENT_REDY])
CloseHandle(m_hEvt[MNIST_EVENT_REDY]);

}


CMnistComm::CMnistComm()
{
}


CMnistComm::~CMnistComm()
{
}

Handwritten digits recognition via TensorFlow based on Windows MFC (III) - Inter-process Communication framework (I)

As I mentioned in the first post, it's planned to have the inference run as a standalone process, so we feed it with image data and obtain the predicted result. We need some Inter-communication framework to do so. (I later realize probably DLL is enough, so I maybe fallback to DLL someday).

All things are done under Windows, please follow me patiently.

First as suggested by official tutorial of bazel and tensorflow, create a folder named tools under root directory like C:\ and install bazel there. Also some dependency like swigwin.


Second git clone tensorflow source code from github.

Third, under the directory C:\tensorflow\tensorflow\cc, create folder mnist. all of our work goes into this folder.

To utilize inter-process communication, I create two files MnistComm.h and MnistComm.cpp, I will paste the content in the next post. I also create another file MnistModel.cc to load the model and run inference.

Fourth, let's create files related to the build process.

1. BUILD file:
# Description:
# TensorFlow is a computational framework, primarily for use in machine
# learning applications.

package(
    default_visibility = ["//visibility:public"],
)

licenses(["notice"])  # Apache 2.0

exports_files(["LICENSE"])

load(":mnist.bzl", "mnist_copts")

cc_library(
    name = "MnistComm",
    srcs = ["MnistComm.cpp"],
    hdrs = ["MnistComm.h"],
    copts = ["/Zc:wchar_t", "/D_UNICODE", "/DUNICODE"],
    linkopts = [],
    deps = [
    ],
)

cc_binary(
    name = "MnistModel",
    srcs = ["MnistModel.cc"],
    copts = mnist_copts(["/Zc:wchar_t", "/D_UNICODE", "/DUNICODE"]),
    deps = [
        ":MnistComm",
        "//tensorflow/core:tensorflow",
    ],
)

2. a tiny extension file named mnist.bzl:

def mnist_copts(fs):

    cflags = fs + ["-DEIGEN_AVOID_STL_ARRAY",

        "-Iexternal/gemmlowp",

        "-Wno-sign-compare",

        "-fno-exceptions"] + \

        select({

            "//tensorflow:windows": [

                "/DLANG_CXX11",

                "/D__VERSION__=\\\"MSVC\\\"",

            ],

            "//conditions:default": ["-pthread"]})

    

    return cflags

Then how to build these targets?

1. Open the msys2 console, set the environment variables:

cd c:/tensorflow

export JAVA_HOME="$(ls -d C:/Program\ Files/Java/jdk* | sort | tail -n 1)"

export BAZEL_SH=c:/tools/msys64/usr/bin/bash.exe

export BAZEL_VS="C:/Program Files (x86)/Microsoft Visual Studio 14.0"

export BAZEL_PYTHON="C:/Program Files/Python35/python.exe"

export PATH=$PATH:/c/tools/swigwin-3.0.10:/c/tools/bazel:/c/Program\ Files/Python35

Adjust these variables properly.

2. Configure and build:

./configure

bazel build -c opt --cpu=x64_windows_msvc --host_cpu=x64_windows_msvc //tensorflow/cc/mnist:MnistComm --verbose_failures

bazel build -c opt --cpu=x64_windows_msvc --host_cpu=x64_windows_msvc //tensorflow/cc/mnist:MnistModel --verbose_failures

Handwritten digits recognition via TensorFlow based on Windows MFC (II) - Train the MNist model

Now let's approach the second episode of this series.
Now we will train the MNist model and later load it to do prediction.

WARNING: The record surrounded by asterisks below is the first try of my work, however, due to poor recording during the process, I don't remember it's workable or not, I just log here for my reference. Please safely neglected it, directly jump to the lines below the second asterisk line


****************
For simplicity, it will directly utilize the existing one, namely fully_connected_feed.py, to obtain such a model.

First clone the tensorflow source code from github.
Since the master branch has some problem with regards running fully_connected_feed.py, so we have to rebase the branch to the stable r0.11 branch:
git checkout r0.11
Then check everything ok:
git status

Second make a copy of the original fully_connected_feed.py, since we have no intention to contaminate the original one:
cp  fully_connected_feed.py fully_connected_feed2.py

Third add an named op under the default graph clause:
    # Create the recognizer
    digit = tf.argmax(tf.nn.softmax(logits), 1, name = 'recognize')

The fourth step is to get rid of the intermediate checkpoint files, so the condition becomes:
      if (step + 1) == FLAGS.max_steps:
        checkpoint_file = os.path.join(FLAGS.train_dir, 'checkpoint')
        saver.save(sess, checkpoint_file, global_step=step)

The fifth step is to save the model:

    tf.train.write_graph(sess.graph_def, 'models/', 'mnist-mlp.pb')

And the last step is to get rid of reluctant things:

python /opt/tensorflow/tensorflow/python/tools/freeze_graph.py  --input_checkpoint=data/checkpoint-1999 --input_graph=models/mnist-mlp.pb --output_graph=models/frozenn-mnist-mlp.pb --output_node_names=recognize

Now we have the model file prepared.

****************


For the model preparation process, I highly referred to the post by Jacky Tung, the address is http://jackytung8085.blogspot.kr/2016/06/loading-tensorflow-graph-with-c-api-by.html.

I git cloned everything of his work on github, done the model based on the existing work. It's quite easy to follow, so I just mention something tricky.

1. First it probably complains the some file doesn't exist, it's due to recursively creating file, so first create a directory named "models" under the mnist.py script directory.
Or speaking alternatively, when creating some file like foo/bar, the "-p" option may be mandatory for some OS.

2. When reducing the file, probably the freeze_graph script will complain the checkpoint file doesn't exist, I guess freeze_graph probably assume cwd is where it resides, so passing the absolute path for model file and checkpoint file

3. It seems when doing inference, tf.argmax can further reduce the work of retrieving result, however, I guess since it doesn't associate with an obvious derivative, so it seems freeze_graph strips it from the file model file, even if you add it in the graph. So probably we have to analyze the result returned by softmax manually.

4. Since freeze_graph is a python script, seems it prefers that graph and checkpoint files are all in text format, if you save them as binaries, probably it will complains decoding fault.

Welcome further discussion.

Keep learning, keep tweaking!