Building a Handwriting Recognition System for the New York Times Crossword

This session explores the development of an experimental amd potential future handwriting recognition feature for The New York Times Crossword app, on the New York Times Games Android App.

We'll discuss the transformation of crossword squares into interactive "SketchBoxes" that capture user input, the challenges of determining input completion timing, and the preprocessing steps like downscaling and binarizing user-drawn characters.

The talk will dive into the selection and training of a deep convolutional neural network (Deep-CNN) using augmented datasets to handle diverse handwriting styles, and the integration of the TensorFlow Lite model into the app for on-device inference.

Key Takeaways & Learning Points:

* Learn how to create responsive interfaces that accurately capture and process user handwriting.

* Understand strategies to determine when a user has completed writing a character, balancing responsiveness and accuracy.

* Explore methods for normalizing and preparing diverse handwriting inputs for machine learning models.
Training Deep-CNNs with augmented datasets

* Gain insights into enhancing model accuracy by expanding training data to include varied handwriting samples.
Integrating machine learning models on-device with TensorFlow Lite

* Discover best practices for deploying efficient ML models within mobile applications.

As a Senior Android Engineer on The New York Times Games team, I led the exploration of this handwriting recognition work during MakerWeek 2023. My work involved designing the SketchBox component, implementing input handling mechanisms, preprocessing data, training the Deep-CNN model, and integrating it into the app using TensorFlow Lite. I have about 11 years of experience in Android development.

Keywords: Handwriting recognition, machine learning, Android development, TensorFlow Lite, convolutional neural networks, on-device inference, data preprocessing, user input handling.

This work has been published in Nieman Lab:
https://www.niemanlab.org/2024/01/how-the-new-york-times-is-building-experimental-handwriting-recognition-for-its-crosswords-app/

And the New York Times:
https://open.nytimes.com/experimenting-with-handwriting-recognition-for-new-york-times-crossword-a78e08fec08f