Digital Ink Recognizer

The ML Kit Digital Ink Recognition API offers the capability to identify handwritten text, classify gestures on a digital surface, and recognize sketches. This technology, which is employed in Gboard, Google Translate, and the Quick, Draw! game, supports hundreds of languages.

Key features of digital ink recognition include:

• Allowing users to write on the screen instead of using a virtual keyboard, enabling the input of characters not readily available on their keyboard (e.g., ệ, अ, or 森 for Latin alphabet keyboards).

• Recognizing hand-drawn shapes and emojis (via Auto Draw Model).

This API operates based on the strokes drawn by the user on the screen. Digital ink recognition functions entirely offline and is compatible with both Android and iOS platforms.

Import VoxelBusters.EasyMLKit and VoxelBusters.CoreLibrary namespaces

using VoxelBusters.EasyMLKit;
using VoxelBusters.CoreLibrary;

Create Instance

Create an instance of the DigitalInkRecognizer instance by passing one of the input sources.

private DigitalInkRecognizer CreateDigitalInkRecognizer()
{
    DigitalInkRecognizer recognizer = new DigitalInkRecognizer();   
    return recognizer;
}

Check for Model Availability

For using Digital Ink feature, you need to first check if the model that is internally used is available or not. If it's not available, it needs to be downloaded. The model handling is handled by an utility class called DigitalInkRecognizerModelManager

// Get Model Manager from DigitalInkRecognizer instance
// recognizerInstance = new DigitalInkRecognizer(); 
// ...
private DigitalInkRecognizerModelManager GetModelManager()
{
    DigitalInkRecognizerModelManager modelManager = recognizerInstance.GetModelManager();
    return modelManager;
}

Each model is referred with an identifier. So first create the model identifier

DigitalInkRecognizerModelIdentifier modelIdentifier = DigitalInkRecognizerModelIdentifier.English;

Check if a model identifier exists with model manager

private bool IsModelAvailable(DigitalInkRecognizerModelIdentifier modelIdentifier)
{
    DigitalInkRecognizerModelManager modelManager = GetModelManager();
    bool isAvailable = modelManager.IsModelAvailable(modelIdentifier);

    Log("Model available : " + isAvailable);
    return isAvailable;
}

If a model identifier is not available, just download it with model manager

private void DownloadModel(DigitalInkRecognizerModelIdentifier modelIdentifier)
{
    DigitalInkRecognizerModelManager modelManager = GetModelManager();
    modelManager.DownloadModel(modelIdentifier, (Error error) =>
    {
        if (error == null)
        {
            Log("Successfully downloaded model");
        }
        else
        {
            Log("Failed downloading model model : " + error);
        }
    });
}

Prepare

Once a required model is available, you can call Prepare. For preparing, you need to pass on an Drawing Input Source on which you add the strokes and DigitalInkRecognizerOptions. A callback is triggered when prepare is complete.

private IDrawingInputSource GetInputSource()
{
    m_inputSource = new DrawingInputSource(m_drawingAreaHandler.GetAreaWidth(), m_drawingAreaHandler.GetAreaHeight());
    foreach(DrawingStroke each in m_drawingStrokes)
    {
        m_inputSource.AddStroke(each);
    }
    return m_inputSource;
}


private DigitalInkRecognizerOptions CreateDigitalInkRecognizerOptions(IDrawingInputSource inputSource)
{
    DigitalInkRecognizerOptions.Builder builder = new DigitalInkRecognizerOptions.Builder();
    builder.SetModelIdentifier(modelIdentifier);
    builder.SetWidth(inputSource.GetWidth());
    builder.SetHeight(inputSource.GetHeight());
    builder.SetPreContext(null);
    return builder.Build();
}

private void Prepare()
{
    IDrawingInputSource inputSource = GetInputSource();
    DigitalInkRecognizerOptions options = CreateDigitalInkRecognizerOptions(inputSource);
    
    Debug.Log("Starting prepare...");
    scanner.Prepare(inputSource, options, OnPrepareComplete);
}

private void OnPrepareComplete(DigitalInkRecognizer recognizer, Error error)
{
        Debug.Log("Prepare complete..." + error);
        if (error == null)
        {
            Debug.Log("Prepare completed successfully!");
        }
        else
        {
            Debug.Log("Failed preparing Digital Ink Recognizer : " + error.Description);
        }
}

Process

Once prepare is complete, you can start processing which gives the result in a callback.

private void OnPrepareComplete(DigitalInkRecognizer recognizer, Error error)
{
        Debug.Log("Prepare complete..." + error);
        if (error == null)
        {
            Debug.Log("Prepare completed successfully!");
            recognizer.Process(OnProcessUpdate);
        }
        else
        {
            Debug.Log("Failed preparing Digital Ink Recognizer : " + error.Description);
        }
}

private void OnProcessUpdate(DigitalInkRecognizer scanner, DigitalInkRecognizerResult result)
{
    if (!result.HasError())
    {
        Debug.Log(string.Format("Recognized values : {0}", result.Values.Count), false);

        foreach (DigitalInkRecognizedValue each in result.Values)
        {
           Debug.Log(string.Format("Recognized Text : {0}", each.Text));
        }
    }
    else
    {
        Debug.Log("Digital Ink Recognizer failed processing : " + result.Error.Description);
    }
});

Close

Close the scanner once you are done processing.

recognizerInstance.Close(null);//Or pass a callback to know once its complete