Skip to content

Creating an AI gesture recognition game with next generation Angular and Tensorflow.js



Ahead of Google I/O in May 2023, the Angular Core Team reached out to This Dot Labs to help them deliver a demo application, showcasing a highly interactive ML integrated application built using Angular with TensorFlow.js in time for the conference.

Leveraging TensorFlow.js' pre-trained hand-detection model, our team rapidly delivered an interactive Snake Game that allows players to control the snake using their hand without having to make contact with a touch screen.

Angular tensorflow snake game start screen

Services Provided

We provided full service application development from concept and design to testing and implementation in time to present a minimal viable product for Google I/O 2023.


Angular LogoAngular
Tensorflow LogoTensorflow

Our team used the Angular 16 beta and Tensorflow.js as the basis of the reactive web application, which benefited from the latter's built-in pre-trained hand-detection model. This allowed us to feed the app data from the web camera stream and provide real time estimates of hand joint positions.

To customize the game's design,our team used SCSS for its component-encapsulated styling. Finally, we used Firebase for deployment, and GitHub Actions for automated deployment on code push.

This Dot Labs is a development consultancy focused on providing staff augmentation, architectural guidance, and consulting to companies.

We help implement and teach modern web best practices with technologies such as React, Angular, Vue, Web Components, GraphQL, Node, and more.


The Angular Core Team is responsible for developing and maintaining the Angular web development framework and led by a group at Google.

Showcasing the Reactive Primitives

Angular 16 introduces a new reactive primitive called signals that was central to the control flow of the application. To showcase how the signals can be used in Angular, our team implemented the Snake game functionality twice. We created one version that utilizes RxJS, and another implementation that uses Angular's new signals API. To make both versions readily available, we built an abstract base Angular service with two concrete implementations, and two game board components that implement the functionality in the chosen method. This enables users to see both versions, so we exposed each implementation on their own route making all versions available to anyone using the app.


Integrating Video Feed Into App

To integrate the video feed from the web camera with the Tensorflow.js hand-detector, our team implemented wrappers around the browser's Media Capture and Streams API features. These implementations allowed us to quickly interact with the input from the player's hand gestures, and have the game respond in real time. This experience presented some challenges on mobile devices with camera access as mobile browsers maintain a different set of permission APIs. Due to our feature encapsulation, we were able to quickly identify the root cause of the issue and fix it for all related systems.

Meeting the Design Threshold In Time for the Event

Because the application would be demoed at Google I/O 2023, it had to meet a minimum design threshold. It had to be clean, attractive, and easy to use. Using only CSS, our team designed a classic monitor and controller interface that included other effects that gave the game a retro video game look and feel. We iterated over the design with the client, and our internal team to make the game's design accessible as some of the gradients and styles selected presented challenges for visibly impaired users. Ensuring the application could be used by the highest number of users was of the utmost importance.


In summary, This Dot Labs collaborated with the Angular Core Team at Google to develop an ML-integrated demo application for Google I/O 2023. They successfully delivered an interactive Snake Game that utilized hand gestures for control, leveraging TensorFlow.js. The application, built with Angular 16, showcased reactive primitives and integrated video feed from the web camera. The team's use of SCSS for styling, Firebase for deployment, and GitHub Actions for automated deployment ensured a seamless development process. The final product met the design threshold, offering a visually appealing and accessible experience for users.


Let's see how This Dot can help you

Are you looking for a development consulting firm that can provide staff augmentation, architectural guidance, and consulting to your company? Look no further than This Dot Labs. Our team of experienced developers and managers can help you implement and teach modern web best practices, and deliver your projects on time and within budget. Contact us today to learn more about our services and how we can help you achieve your development goals.

You might also like


Our clients are building for the future. Explore This Dot Labs' portfolio, and see how we have helped them achieve their diverse technical goals.