Ep. 20 — TensorFlow.js is awesome. It opens up new paths towards creative ML features

I spent last month building Gently, exploring possibilities opened up by TensorFlow.js, Google’s JavaScript implementation of its popular machine learning framework. The result? Very positive.

It gives developers an easy way to add ML capabilities like object classification inside web apps. It’s particularly useful for running inference on large images or datasets that your user may not want to upload to remote servers (or it would take too long to transfer them.)

To demonstrate what Tensorflow.js could achieve as an example, check out Gently. It’s a chrome extension that analyzes web page on the fly as you browse. If there are images that may make the viewer feel uncomfortable (e.g. spiders, snakes, centipedes), it hides those automatically.

Everything is performed locally for user privacy and best performance.

Below is a screenshot when you browse a page with centipede images (with Gently installed.) You can see the image being replaced by a message on the bottom right.

Behind the scenes, it uses Tensorflow.js / MobileNet v2 model to classify all images as they arrive.

Key observations during my journey?

1. Tensorflow.js, even though it’s fairly new, performs quite well already. Inferences take 20–100ms on modern laptops with WebGL support. It can take advantage of GPUs and that has made a huge difference in responsiveness. (on the flip side, if you don’t have WebGL support on the browser, it’s very slow, by roughly two orders of magnitude.)
2. Examples on its github repo are well-written and has good coverage of different use cases. They are easy to follow, even for occasional JavaScript developers like me.
3. tfjs offers a compelling solution pattern. Inference performed at the edge (closest to the data), zero installation, familiar programming environments in JavaScript/TypeScript and NodeJS.
4. Browser compatibility is good too, on both mobile and desktops.
5. Don’t forget to turn on production mode (tf.enableProdMode()) to optimize for performance after you are done with testing.

Hope this helps!