Home / WPF / Getting started with real-time image processing with C#

Getting started with real-time image processing with C#

One of the interesting fields in industrial automation is real time image processing and computer vision. It’s quite simple to write applications that access to a camera in C#, both with Windows Forms and WPF, and in this article we will see how to connect to a USB camera (or your webcam) or an IP camera using AForge framework, to do a basic image processing. AForge is an open-source framework that is very well documented and easy to get started with. You can find the website at this address: http://www.aforgenet.com/.

Sample application

You can download the sample application on Github at this address: https://github.com/mesta1/AForge-examples

Watch on youtube

Installing AForge

AForge can be downloaded from NuGet, just browse for AForge, and download AForge.Video.DirectShow to connect to camera, and AForge.ImageProcessing to access to image processing libraries.

AForge nuget

Then reference System.Drawing. If you work with Windows Forms, System.Drawing is referenced by default, but to work with WPF, System.Drawing must be referenced manually because AForge works with Bitmaps.

System_Drawing

Connecting to a USB camera (or a webcam)

To connect to the camera you first need to get a the list of devices and connect to the right one.
This can be done with this code:

var videoDevicesList = new FilterInfoCollection(FilterCategory.VideoInputDevice);
foreach (FilterInfo videoDevice in videoDevicesList)
{
    cmbVideoSource.Items.Add(videoDevice.Name);
}

Then when you have a video device, to connect to the camera you have to create a VideoCaptureDevice object by passing a MonikerString, that you can get from the video device:

videoSource = new VideoCaptureDevice(videoDevicesList[cmbVideoSource.SelectedIndex].MonikerString);
videoSource.NewFrame += new NewFrameEventHandler(video_NewFrame);
videoSource.Start();

A video is composed of different images that are displayed at a very fast rate, so to display a video you need an Image control and you have to swap the images. the images correctly is crucial, otherwise the application will consume all the system memory in a few minutes.
To display a video in Windows Forms we will use a PictureBox control; to display it in WPF we will use an Image control. The camera is an asynchronous I/O, so you need to subscribe to the event NewFrame to update the image.
Here is the Windows Form code, assuming that you created a pictureBox1 in the main form:

private void video_NewFrame(object sender,NewFrameEventArgs eventArgs)
{   
    // get new frame
    Bitmap bitmap = (Bitmap)eventArgs.Frame.Clone();    
    // here you can process the frame
    pictureBox1.Image = bitmap;    
}

And here is the WPF code to update the Image control:

private void video_NewFrame(object sender,NewFrameEventArgs eventArgs)
{
    try
    {
        BitmapImage bi;
        using (var bitmap = (Bitmap)eventArgs.Frame.Clone())
        {
            bi = bitmap.ToBitmapImage();
        }
        bi.Freeze(); // avoid cross thread operations and prevents leaks
        Dispatcher.BeginInvoke(new ThreadStart(delegate { videoPlayer.Source = bi; }));
    }
    catch (Exception exc)
    {
        MessageBox.Show("Error on _videoSource_NewFrame:\n" + exc.Message, "Error);
        StopCamera();
    }
}

Connecting to an IP camera

The connection to an IP camera is similar to the connection to a USB camera, but instead of VideoInputDevices you must create a JPEGStream or a MJPEGStream, depending on the specifications of your webcam.

if (UseJpegStream)
{
    _videoSource = new JPEGStream(ConnectionString);
}
else // UseMJpegStream
{
    _videoSource = new MJPEGStream(ConnectionString);
}
_videoSource.NewFrame += new NewFrameEventHandler(video_NewFrame);
_videoSource.Start();

Also you need to know the url of the ip camera to get connected.

Real time image processing

Once you have access to the pictures, Bitmaps in this case, you can apply some filters to start recognizing objects from the camera.
You can find some examples on image processing for recognizing objects, shapes and colors on the AForge image processing sample page.
There are also some examples on surveillance and movement detection on the AForge computer vision samples page.
In the sample application you can find an example of thresholding an image and display it in WPF:

BitmapImage bi;
using (var bitmap = (Bitmap)eventArgs.Frame.Clone())
{
    if (Grayscaled)
    {
        using (var grayscaledBitmap = Grayscale.CommonAlgorithms.BT709.Apply(bitmap))
        {
            bi = grayscaledBitmap.ToBitmapImage();
        }
    }
    else if (Thresholded)
    {
        using (var grayscaledBitmap = Grayscale.CommonAlgorithms.BT709.Apply(bitmap))
        using (var thresholdedBitmap = new Threshold(Threshold).Apply(grayscaledBitmap))
        {
            bi = thresholdedBitmap.ToBitmapImage();
        }
    }
    else // original
    {
        bi = bitmap.ToBitmapImage();
    }
}
bi.Freeze(); // avoid cross thread operations and prevents leaks
Dispatcher.BeginInvoke(new ThreadStart(delegate { videoPlayer.Source = bi; }));

Drawing on an image

find books computer vision
Once you processed an image and extracted the information you need, like position of objects (blobs) in pixels, painting on an image is trivial.
Pick the original Bitmap, create a Pen object and draw geometric figures, like lines, circles and rectangles that connect the pixels.
For example this code draws a red rectangle with a thickness of 1px:

Graphics g = Graphics.FromImage(myBitmap);
var pen = new System.Drawing.Pen(System.Drawing.Color.Red, /*thickness = */ 1);
g.DrawRectangle(pen, x1, y1, x2 - x1, y2 - y1);
pen.Dispose();
g.Dispose();

WPF Post-It detector

post-it detector
This is a complete WPF application that connects to a USB camera to track the Post-it position on a picture.
This makes use of Color Filtering, Thresholding and Blob processing algorithms to find the borders of the post it.
You can select a color from the image by clicking the Color picker button.
Then you can adjust the filtering and thresholding of the processed image to recognize the borders.
The image process is in this method:

private List<IntPoint> FindCorners(Bitmap bitmap)
{
    List<IntPoint> corners = new List<IntPoint>();
    using (var clone = bitmap.Clone() as Bitmap)
    {
        new EuclideanColorFiltering(new AForge.Imaging.RGB((byte)Red, (byte)Green, (byte)Blue), Radius).ApplyInPlace(clone);
        using (var grayscaledBitmap = Grayscale.CommonAlgorithms.BT709.Apply(clone))
        {
            new Threshold(Threshold).ApplyInPlace(grayscaledBitmap);
            if (Inverted)
            {
                new Invert().ApplyInPlace(grayscaledBitmap);
            }
            BlobCounter blobCounter = new BlobCounter();
            blobCounter.FilterBlobs = true;
            blobCounter.MinWidth = 50;
            blobCounter.MinHeight = 50;
            blobCounter.ObjectsOrder = ObjectsOrder.Size;
            blobCounter.ProcessImage(grayscaledBitmap);
            Blob[] blobs = blobCounter.GetObjectsInformation();
            // create convex hull searching algorithm
            GrahamConvexHull hullFinder = new GrahamConvexHull();
            for (int i = 0, n = blobs.Length; i < n; i++)
            {
                List<IntPoint> leftPoints, rightPoints;
                List<IntPoint> edgePoints = new List<IntPoint>();
                // get blob's edge points
                blobCounter.GetBlobsLeftAndRightEdges(blobs[i], out leftPoints, out rightPoints);
                edgePoints.AddRange(leftPoints);
                edgePoints.AddRange(rightPoints);
                // blob's convex hull
                corners = hullFinder.FindHull(edgePoints);
            }
        }
    }
    return corners;
}

More image processing libraries

If AForge is not enough, there are more famous open-source frameworks, like OpenCV in C++, or its C# port Emgu-CV.

Sample application

You can find a sample application on GitHub: https://github.com/mesta1/AForge-examples
There you will find 4 projects:
1) one that connects a USB camera to a Windows Form application,
2) one to connect a USB camera to a WPF application,
3) one to connect an IP camera to a WPF application,
4) and the last one shows how to apply some filter on the displayed image with WPF.

If you are not familiar with launching different projects from Visual Studio, just right-click on a project in the solution explorer and select “Set as Startup project”.

More examples can be found on Codeproject , just by searching for AForge.

Share Button

Leave a Reply