C++ Tensorflow NodeJs Addon

I was playing around with NodeJs and its Addon architecture. It's nice and I thought it would make for a very simple way to provide an Image Classification service using Tensorflow.

Tensorflow

The tensorflow docs have improved a bit with version 1 but things are still a little thin. Particularly on the C++ side. Even so you can get quite far by adapting the example code provided.

You'll want to follow the install from source instructions. You'll need google build tool bazel for this. I found the install process for both seemless (if lengthy) on both OSX and Linux - The windows install is described as experimental and I haven't tried it.

I used WebStorm which has a nodejs express plugin which makes creating projects pretty simple but there are loads of other ways to write the node side.

Start Simple

In this post we'll make a very simple c++ tensorflow lib which does a matrix multiply. Tensorflow is a general maths tool, not only for machine learning, so it's nice to show it's not just about big deep learning architectures.


	float example::MatrixMul()  {
	
	    Scope root = Scope::NewRootScope();
	    
	    auto vectorConstant = Const(root,  {  {2.0, 7.0 } });
	    auto matrixMultipy = MatMul(root, vectorConstant,  {  {7.0 },  {4.0 } });
	    
	    ClientSession session(root);
	    std::vector outputs;
	    session.Run( {matrixMultipy }, &outputs);
	
	    float score = outputs[0].flat()(0);
	    
	    return score;
         }
		
The codes pretty simple and we only return a single float so we don't have to worry about types on the node side for the moment. It should all make sense if you've used tensorflow through python.
If you haven't then look up some python tutorials (I'll put some up here soon too):
The basic thing you need to grasp is that tensorflow works by first creating a computational graph (like a program) so matrixMultipy above does not return a value, it return the node in the computational graph (like a method in a program).
To get an answer you have to run the graph (program), in session.Run we feed in the node we want to get an answer from and a address for where we want the result stored. If we add a main() we can build it as a binary first to see things are working before placing another layer of possible error with our addon code.

Have a look at the source file here example.cc
There's a simple method in there (called Simple) you can call from nodejs to see communication is working without using the tensorflow libraries. That should always work, so if the node call fails you know something is badly wrong...
The h file is here example.h

The build file looks like this


	# Description:
	# TensorFlow C++ example
	
	cc_binary(
	    name = "libexample.so",
	    srcs = ["example.cc", "example.h"],
	    linkshared = 1,
	    deps = [
	        "//tensorflow/cc:cc_ops",
	        "//tensorflow/cc:client_session",
	        "//tensorflow/core:tensorflow",
	    ],
            visibility = ["//visibility:public"],
        )
		
It's pretty simple but I spent time messing around with cc_library which seemed like a good way to go but I was getting unrecognized symbols when my node c++ code tried to connect.
This works so it's good enough for the moment.
Remove the linkshared line and change * name = "libexample.so" * to * name = "example" * if you want to test the output as a binary first (you can just double click on the resulting example file in OSX to see the result)
I've followed the tensorflow tutorials example and put these in $pathToTensorFlow/tensorflow/cc/example/ where $pathToTensorFlow is where you git cloned Tensorflow.
To build it from $pathToTensorFlow run
bazel build tensorflow/cc/example/...
This takes time so don't worry - future builds are much faster.

Node

Let's move to the node side: I created an Express project in Webstorm but do whatever you feel comfortable with. Create an addons folder, I put mine at root level:
At the root level create binding.gyp, this is where you'll build the addon
{
  'targets': [{
    'target_name': 'example',
   'sources': ["addons/hellotf.cc", "addons/example.h"],
        'libraries': [
            '$pathToTensorflow/tensorflow/bazel-bin/tensorflow/cc/example/libexample.so'
        ],
        'include_dirs' : [
          "<!(node -e \"require('nan')\")"
         ]
     }]
}
		
Again replacing the $pathToTensorflow.
target_name is the name you where the code will be build in this case: ../build/Release/example.node
Lets create hellotf.cc in the addons folder you created

        // hello.cc
        #include <node.h>
        #include "example.h"
        
        namespace demo {
        
        using v8::FunctionCallbackInfo;
        using v8::Isolate;
        using v8::Local;
        using v8::Object;
        using v8::String;
        using v8::Value;
        
        using examples::example;
        
        auto simpleExample = example();
        
        void Method(const FunctionCallbackInfo<Value> args) {
            Isolate* isolate = args.GetIsolate();
            
            auto b = simpleExample.MatrixMul();
            
            char buffer [100];
            int cx;
            
            cx = snprintf(buffer, 100, "The answer to ltuae is %e", b);
            
            args.GetReturnValue().Set(String::NewFromUtf8(isolate, buffer));
         }
        
        void init(Local<Object> exports) {
            NODE_SET_METHOD(exports, "hellotf", Method);
            }
            
            NODE_MODULE(addons, init)
            
        }  //

This is a standard addon outline - the init method exports our method as "hellotf" - so from javascript we can call foo.hellotf() after we bring it in using require (see below)
Add the example.h header file to the addons directory as well but remove the tensorflow includes from the top
Now build the addon by going to the root directory. Install node gyp
    npm install -g node-gyp
	
now call
    node-gyp configure
    
to create the build files and finally
    node-gyp build
    
Look at the node addon page if you get lost.
In the routes/index.js (Webstorm will create this if you choose an Express project) paste the following

  const example = require('../build/Release/example.node');
	
  console.log(example.hellotf());
	
  router.get('fs', function(req, res, next) {
	
    res.render('index', { title: 'DeepThought' , text: 
	
    example.hellotf()});
	  
  });
Everything should work now - we just render using a jade (pug) file. Fire up the node server and see if it works.

It didn't?

Did you get something like "Library not loaded /...../libexample.so", Reason: image not found

If the bazel build went well then all that's missing is node knowing where the lib is: The binding.gyp might have the correct root but this isn't necessarily the shared lib address.
On OSX use otool -D on the .so file, it might give you bazel-out/local-opt/bin/.. so if the tensorflow directory is not on the path searched for shared libraries it won't work.
You can fix it in OSX with
sudo install_name_tool -id 
  '$pathToTensorflow/tensorflow/bazel-bin/tensorflow/cc/example/libexample.so' libexample.so

Like before change the $pathToTensorflow.
If you're not in the libexample.so directory you'll need the full path. You'll also need to call node-gyp build again (and change something in the .cc or it'll say nothing has changed and do nothing, I'm guessing there should be a flag to force the build but I didn't find anything).

Getting images across

Let's start from the node side, we'll write a simple template for an upload form and connect it to a handler, I used Busboy but if you have a preference go for it.

Here's the route:

var router = express.Router();

router
  .get('fs', function(req, res, next) {
    res.render('upload', { title: 'Upload' });
  });
and the template:
extends layout

block content
  form(action="/classify", method="post", cnctype="multipart/form-data")
    input(type="text", name="title")
    input(type="file", name="image")
    button(type="submit", value="Upload") Upload File
The handling code is in common.js and we also have a little helper for dealing with the incoming file util.js

If you've used a file uploader it should all look pretty standard - we base64 encode the data so we can send it to the classify page to display.

This time our addon is going to have a method that accepts the buffer and an image type we get from mimetype.
We wait for the addon to return a value but obviously you don't have to block.

  console.log('Done parsing form!');

  var img = new Buffer(buffer, 'base64');
  var imgString = img.toString('base64');
  console.log('Image string to parse: ' + imgString);

  console.time("classify");
  var returnValue = image.classify(img, fileType);
  console.timeEnd("classify");

  console.log('Returned value: ' + returnValue);

  res.render('classify', { 
        title: 'Classify', imageData: imgString, imageJson: returnValue}, 
        function(err, html) {
            res.send(html);
        });
Let's skip across to the new addon now. It's going to look a lot like the other one but with code for handling the Uint8Array we're sending it so we can pass it on to Tensorflow.

You'll find it in classify.cc

The business part is here:

  assert(args[0] -> IsUint8Array());
  assert(args[1] -> IsNumber());

  int imageType = args[1]->NumberValue();

  auto imageArray = args[0].As<Uint8Array>();
  //We want the length to create the string
  auto length = imageArray->Length();

  Nan::TypedArrayContents dataPtr(imageArray);

  std::string charString = std::string(*dataPtr, *dataPtr + length);
There may be a shorter way to create this but I find v8 pretty short on detail. I've use native abstractions for node NAN to help out a bit after getting lost trying to pull stuff out of v8's local wrapper.

Back to tensorflow

Tensorflow is expecting... tensors - what's a tensor? If you're familiar with numpy then it's just an NDArray. For a color image we are going to have 3 channels RGB or whatever ordering (this can matter - it's fine if you train from scratch but if you use a trained model with the channels in the wrong order things break) and a batch dimension - Tensorflow expects everything in batches, rolling things up into big matrices is efficient.

So our tensor will be have size (numberOfImage, channels, numberOfPixels)

numberOfPixels might be 2 dimensions (width * height), futher down the line convolutions can change this again, lots of things can change the tensor shape as we move through the graph.
This is why an NDArray, tensor type representation is one of the first things you should check if a library supports fully, they are useful.

We are sending a string from our Node code so we'll want to decode it on the Tensorflow side. The library has some helper code for this DecodeJpeg, DecodePng, DecodeGif, which expect a Tensorflow Input, luckily a string is a valid input so there's no work to do.

Status classifier::StringToImage(tensorflow::Scope root, tensorflow::Input encodedImage,
                                             int imageType, tensorflow::Output* imageOutput)  {
    const int wantedChannels = 3;
    //TODO put in some peaking code here
    
    tensorflow::Output decodedImage;
    
    if (imageType == 1) {
        auto imageReader = DecodeJpeg(root, encodedImage, DecodeJpeg::Channels(wantedChannels));
        decodedImage = imageReader.image;
     }
    else if (imageType == 2)  {
        auto imageReader = DecodePng(root, encodedImage, DecodePng::Channels(wantedChannels));
        decodedImage = imageReader.image;
     }
    else if (imageType == 3)  {
        auto imageReader = DecodeGif(root, encodedImage);
        decodedImage = imageReader.image;
     }
    
    auto floatCaster = Cast(root.WithOpName("float_caster"), decodedImage, tensorflow::DT_FLOAT);
    // Turn our image into a batch
    auto dimsExpander = ExpandDims(root, floatCaster, 0);
    
    auto resized = ResizeBilinear(
                                  root, dimsExpander,
                                  Const(root.WithOpName("resized"),  {inputHeight, inputWidth }));
    
    // Give us the image squashed into a 1 range
    *imageOutput = Div(root.WithOpName(outputName), Sub(root, resized,  {inputMean }),  {inputStd });

    return Status::OK();
 }
 
This shouldn't need too much explanation. We found the imageType on the node side, we could do some peeking at the headers to verify the value sent but we don't need it here.

The decoded image is a tensor, also a tensorflow::Output, it's a single image so we turn it into a batch tensor of batch size 1 and then we resize it.

CNNs will either need a fixed size input or a way to window over an input with the fixed window size feeding into the network below. So it's good to have this here - for images these are usually square so you can consider cropping and padding though these don't have as much effect on the output as you might naively assume.

You've seen the Scope object before, right back at the start when we did the trivial matrix multiply. It's where we build the graph before we run it in a session. Here's the Session code.

//This is what our service calls - returns negative for failure
int classifier::ReadAndRun(std::string byteString, int encoding, std::string* json) {
    timestamp_t t0 = timer::get_timestamp();
    auto root = tensorflow::Scope::NewRootScope();
    
    tensorflow::Output imageTensor;
    Status imageStatus = StringToImage(root, byteString, encoding, &imageTensor);
    
    std::unique_ptr<ClientSession> inputSession(new ClientSession(root));
    std::vector<Tensor> imageOutputs;
    
    LOG(INFO) << "about to get image";
    Status inputStatus = inputSession -> Run( { },  {imageTensor }, &imageOutputs);
    
    LOG(INFO) << "image session loc: " << &inputSession;
    LOG(INFO) << "image tensor loc: " << &imageOutputs;
    
    if (!inputStatus.ok())  {
        LOG(ERROR) << "Running check failed: " << inputStatus;
        return -1;
     }
    else  {
        LOG(INFO) << "Got Image!";
     }
    timestamp_t t1 = timer::get_timestamp();
    double timeTaken = (t1 - t0);
    LOG(INFO) << "Image adjust time: " << timeTaken;
    
We put some timer code in there and some generous logging to assure ourselves that things aren't running out of control