Object Detector and Tracker

With ML Kit's on-device Object Detection and Tracking API, you can detect and track objects in an image or live camera feed.

Optionally, you can classify detected objects, either by using the coarse classifier built into the API.

Because object detection and tracking happens on the device, it works well as the front end of the visual search pipeline.

• Fast object detection and tracking Detect objects and get their locations in the image. Track objects across successive image frames.

• Optimized on-device model The object detection and tracking model is optimized for mobile devices and intended for use in real-time applications, even on lower-end devices.

• Prominent object detection Automatically determine the most prominent object in an image.

• Coarse classification Classify objects into broad categories, which you can use to filter out objects you're not interested in. The following categories are supported: home goods, fashion goods, food, plants, and places.

• Classification with a custom model (Coming Soon) Use your own custom image classification model to identify or filter specific object categories. Make your custom model perform better by leaving out background of the image.

Import VoxelBusters.EasyMLKit and VoxelBusters.CoreLibrary namespaces

using VoxelBusters.EasyMLKit;
using VoxelBusters.CoreLibrary;

Create Instance

Create an instance of the ObjectDetectorAndTracker instance by passing one of the input sources.

private ObjectDetectorAndTracker CreateObjectDetectorAndTracker()
{
    ObjectDetectorAndTracker detector = new ObjectDetectorAndTracker();
    return detector;
}

Prepare

For preparing, you need to pass on ObjectDetectorAndTrackerOptions and a callback to know when prepare is complete.

private IInputSource CreateImageInputSource(Texture2D texture)
{
    return new ImageInputSource(texture);
}

private IInputSource CreateLiveCameraInputSource()
{
    IInputSource inputSource = new LiveCameraInputSource()
    {
        EnableFlash = false,
        IsFrontFacing = false
    };

    return inputSource;
}

private ObjectDetectorAndTrackerOptions CreateObjectDetectorAndTrackerOptions(IInputSource inputSource)
{
    ObjectDetectorAndTrackerOptions.Builder builder = new ObjectDetectorAndTrackerOptions.Builder(inputSource);
    builder.EnableClassification(true);
    builder.EnableMultipleObjectDetection(true);
    builder.SetClassificationConfidenceThreshold(0.5f);
    builder.SetCustomModelPath(null);
    return builder.Build();
}

private void Prepare()
{
    IInputSource inputSource = CreateImageInputSource(YOUR_TEXTURE_HERE);
    ObjectDetectorAndTrackerOptions options = CreateObjectDetectorAndTrackerOptions();
    Debug.Log("Starting prepare...");
    detector.Prepare(inputSource, options, OnPrepareComplete);
}

private void OnPrepareComplete(ObjectDetectorAndTracker detector, Error error)
{
        Debug.Log("Prepare complete..." + error);
        if (error == null)
        {
            Debug.Log("Prepare completed successfully!");
        }
        else
        {
            Debug.Log("Failed preparing Object Detector and Scanner : " + error.Description);
        }
}

Process

Once prepare is complete, you can start processing which gives the result in a callback.

private void OnPrepareComplete(ObjectDetectorAndTracker detector, Error error)
{
        Debug.Log("Prepare complete..." + error);
        if (error == null)
        {
            Debug.Log("Prepare completed successfully!");
            detector.Process(OnProcessUpdate);
        }
        else
        {
            Debug.Log("Failed preparing Object detector and tracker : " + error.Description);
        }
}

private void OnProcessUpdate(ObjectDetectorAndTracker detector, ObjectDetectorAndTrackerResult result)
{
    if (!result.HasError())
    {
        Debug.Log(string.Format("Received {0} detected objects", result.DetectedObjects.Count));

        foreach (DetectedObject each in result.DetectedObjects)
        {
            Debug.Log(string.Format("Tracking Id : {0}, Labels : {1}, Bounding Box : {2}", each.TrackingId, string.Join(",", each.Labels), each.BoundingBox));
        }
        
        if(result.DetectedObjects.Count > 0)
        {
            detector.Close(null);
        }
    }
    else
    {
        Debug.Log("Object detector failed processing : " + result.Error.Description, false);
    }
});

Close

Close the detector once you are done processing.

detector.Close(null);//Or pass a callback to know once its complete