Tuesday, 1 January, 2019 UTC
Summary
When you talk about Machine Learning and Google’s TensorFlow, most people think of Python and specialized hardware rather than JavaScript and any browser. This article explains what TensorFlow.js can do and why it makes sense to run machine learning in the browser.
TensorFlow.js is a JavaScript library that runs in the browser as well as with Node.js on the server. However, we are interested in this article only for the application in the browser. The interface of TensorFlow.js is strongly based on TensorFlow’s High Level API Keras. Keras code is often only distinguishable from TensorFlow code at second glance. Most of the differences are due to the different language constructs of Python and JavaScript for configuration parameters.
TensorFlow.js: Machine learning with every GPU
TensorFlow.js lets you build machine learning projects from zero. When necessary data is available, then the models can be trained and executed directly in the browser. TensorFlow.js uses the graphics card (GPU) of the computer via the browser API WebGL. You lose a bit of performance because WebGL can only be tricked into executing the matrix multiplication required by TensorFlow.js. But they are necessary because TensorFlow, as a machine learning strategy, mainly supports neural networks. These can be mapped very well by matrix multiplications during training as well as during prediction. Here is the first advantage of TensorFlow.js over TensorFlow: While TensorFlow currently only supports NVIDIA GPU over CUDA, TensorFlow.js works with any graphics card.
Here is the example code to use the High Level API to create a sequential neural network in the browser:
Interact with all browser APIs// create a sequential model
const model = tf.sequential();
// add a fully connected layer with 10 units (neurons)
model.add(tf.layers.dense({units: 10}));
// add a convolutional layer to work on a monochrome 28x28 pixel image with 8
// filter units
model.add(tf.layers.conv2d({
inputShape: [28, 28, 1],
filters: 8
}));
// compile the model like you would do in Keras
// the API speaks for itself
model.compile({
optimizer: 'adam',
loss: 'categoricalCrossentropy',
metrics: ['accuracy']
});
Addressing interfaces on different operating systems and devices can still be a painful experience. Not so when developing a browser-based application. Even access to such complex hardware as a camera or a microphone are anchored in the HTML standard and are supported by all current browsers. The nature of the browser, which is naturally designed for interaction as well, suits you here. Interactive applications with a share of machine learning are now easier than ever.
As an example, we have a simple game, Scavenger Hunt , which of course also runs in the browser of a mobile phone and thus brings the most fun. As shown in Figure 1, in the real world you have to quickly find an object that matches the displayed emoticon. For this purpose, the built-in camera and a trained neural network is used, which can detect the matching objects. Such a model can be used by any JavaScript developer even without machine learning skills.
Try out this simple game, which is built With TensorFlow.js. For this purpose, the built-in camera and a trained neural network is used and can detect the matching objects. Such a model can be used by any JavaScript developer even without machine learning skills.
Emoji Scavenger Hunt – Use your phone’s camera to identify emojis in the real world.
Machine learning without installation on any computer
TensorFlow.js allows you to deploy a model created in advance with TensorFlow. This model may already have been fully or partly trained on strong hardware. In the browser it comes then only to the application or is further trained.
In the next example you can play Pac-Man using images trained in your browser through your webcam.
Webcam Controller – Play Pac-Man using images trained in your browser.
The model is converted by a supplied program and can be loaded asynchronously after being loaded by a line similar to the following:
const model = await tf.loadModel('model.json');
After that, the model is no longer distinguishable from a model created directly in the browser. It is very easy to use for prediction, which in turn runs asynchronously on the GPU:
const example = tf.tensor([[150, 45, 10]]);
const prediction = model.predict(example);
const value = await prediction.data();
In addition to entertainment through games even more useful applications are conceivable here. Gesture navigation or interaction can be helpful to people with disabilities as well as people in special situations. And as I said: without any installation, by simply loading a website.
Another example of such a position detection is the PoseNet. It is already so pre-trained that it can recognize the position of the face, arms and legs even with several people in the picture.
Real-time Human Pose Estimation in the browser. User data does not have to leave the browserimport * as posenet from '@tensorflow-models/posenet';
import * as tf from '@tensorflow/tfjs';
// load the posenet model
const model = await posenet.load();
// get the poses from a video element linked to the camera
const poses = await model.estimateMultiplePoses(video);
// poses contain
// - confidence score
// - x, y positions
Nowadays, where privacy due to GDPR regulation gets more and more of importance, some think twice, whether he wants to have a particular cookie on his computer or whether you want to send a statistic of his user data for improving the user experience to the manufacturer of a software. But what about the reversal? The manufacturer provides a general model of how to use software, and similar to the Pac-Man game, it is adapted to the individual user using a transfer-learning model.
The Conclusions
Machine learning in the browser does not seem to make much sense at first for many developers. But if you take a closer look, there are applications that no other platform can offer:
- Education: You can interact directly with machine-learning concepts and learn by experimenting
- Development: If you already have or want to build a JS application, you can use machine learning models or practice it directly
- Games: Real-Time Position Estimation only via the camera (ie how the people in front of the camera are moving) or Image Recognition can be coupled directly with games. There are very cool examples of what you can do with it. However, the possibilities go well beyond games
- Deployment: You already have a machine learning model and wonder how to get it into production. The browser is a solution. Even finished models can be integrated into the own application without deep knowledge of machine learning
- Interactive visualizations: For interactive clustering or artistic projects
As we see, we still have some drawbacks to the same hardware in terms of performance over TensorFlow. The comparison runs on a 1080 GTX GPU and measures the time for a prediction with the MobileNet, as it is used for the examples here. In this case, TensorFlow runs three to four times faster than TensorFlow.js; however, both values are very low. The WebGPUS standard, which will allow more direct access to the GPU, gives hope for better performance.
Tutorials can be found on the TensorFlow.js website.
Share This:
The post TensorFlow.js: Machine learning in the browser appeared first on FrontNet Blog.