Unity Performance: Enhancing Speed With Kinect Azure And OpenCV

Unity Optimization : Improving Performance With Kinect Azure And OpenCV

Computer vision | Electronic | IOT

For a recent project involving the detection of balls thrown against a wall, I wanted to use the new Kinect Azure depth camera. As the project was intended to be integrated into a game, I wanted to code everything on Unity, also using OpenCV Sharp for image analysis.

The beginning of trouble

I started by prototyping my detection pipeline:

Retrieving the colour and depth images.
Transformation of the depth image into a point cloud.
Change of coordinate system between colour camera and depth.
Change of coordinate system between the colour camera and the projection screen coordinate system (previously calculated by projecting and analysing a chessboard).
Reproduction of the point cloud into a depth image (2D).
Image analysis to detect ‘blobs’, which are my balls touching the screen.

From the very first stages I started to notice some surprising slowdowns, which were generally corrected by changing the way I called the Kinect functions: if I had to perform an operation on each pixel, calling the Kinect DLL functions on Unity for each pixel was very time-consuming. If I passed a complete array of pixels, the operation was much faster (by a factor of 10 to 50).

The example below shows a very concrete example, with the pointCloud variable of type Image which comes from the Kinect’s K4ADotNet DLL. If the values are stored in variables before executing the loops, the execution time is much faster.

private Mat ConvertPointCloudImageToMat(Image pointCloud)
{



    for (int i = 0; i < pointCloud.HeightPixels; i++)
    {
        for (int j = 0; j < pointCloud.WidthPixels; j++)
        {
             /// DO NOTHING
        }
    }
}

Execution time : 26ms

private Mat ConvertPointCloudImageToMat(Image pointCloud)
{
    int height = pointCloud.HeightPixels;
    int width = pointCloud.WidthPixels;

    for (int i = 0; i < height ; i++)
    {
        for (int j = 0; j < width ; j++)
        {
             /// DO NOTHING
        }
    }
}

Execution time : 0.1ms

The further I got, the more I noticed a similar problem with OpenCV, when multiplying pixels on the Matrix.

I finally realised that the problem was with Unity and its DLL management. Calls to DLL functions are very costly because of significant overhead. So you have to limit the number of calls as much as possible by trying to ‘package’ the data to be processed so that it can be sent in a single call.

What can you do?

If you have a lot of processing to do with the Kinect and even OpenCV, it is preferable to directly create your own DLL in which the most sensitive functions will be coded. For my part, I’m going to do it in C++ because OpenCV and the Kinect SDK are available in this language (and not in C#).

If there is little processing, I’d advise you to analyse the execution times of your functions using StopWatch for example, and to change the call format. Favour arrays of pixels and image by image rather than pixel by pixel. Also remember to use the very fast Marshall.Copy function for Intr Arrays.

Below is another example of optimising the use of OpenCV’s Mat. Load all the data into a double [ ] and do a Mat.put on the complete array. The Matrixes must have the right dimensions beforehand.

private Mat ConvertPointCloudImageToMat(Image pointCloud)
{
    int indx = 0;
    int height = pointCloud.HeightPixels;
    int width = pointCloud.WidthPixels;
    double[] data = new double[3];

    for (int i = 0; i < height; i++)
    {
        for (int j = 0; j < width; j++)
        {
            indx = j * 3 + i * width * 3;
            data[0] = pointCloudBuffer[indx] * 0.001f;
            data[1] = pointCloudBuffer[indx + 1] * 0.001f;
            data[2] = pointCloudBuffer[indx + 2] * 0.001f;
            pointCloudMat.put(i, j, data);
        }
    }
    return pointCloudMat;
}

Execution time : 80ms

private Mat ConvertPointCloudImageToMat(Image pointCloud)
        {
            int indx = 0;
            int height = pointCloud.HeightPixels;
            int width = pointCloud.WidthPixels;
            double[] data = new double[3* height* width];

            for (int i = 0; i < pointCloudBuffer.Length; i+=3)
            {
                    data[i] = pointCloudBuffer[i] * 0.001f;
                    data[i+1] = pointCloudBuffer[i+1] * 0.001f;
                    data[i+2] = pointCloudBuffer[i+2] * 0.001f;
            }
            pointCloudMat.put(0, 0, data);
            return pointCloudMat;
        }