Image Labelling: Where is where, what is what!

In this tutorial, we will try to tell the location (or coordinates) of the green turtle in the image to the computer. We will use this turtle from Sea Turtles Conservation Center Royal Thai Navy (ศูนย์อนุรักษ์พันธุ์เต่าทะเล กองทัพเรือ) as our subject.

The sense of knowing the location

Human and computer perceive locations of things differently. We can describe to a person that the turtle is on the right or at the bottom of the image. (or the bottom right?)

15321424_1378213435524693_1346132572_o
Original turtle image

However, to the computer, it knows that this specific image is x pixels wide and y pixels high. They do not know weather the coordinate (30,30) is on the left, on the right, or at the top left corner of the image. This is where image labelling comes in handy. The image below will describe how coordinates system in the image work. The little ‘green plus sign’ in the middle of the image shows coordinates 100 by 100, and this image dimension was 200 by 200 pixels. For more information on how to create this image, please refer to this code.

coordinates illustrate by MATLAB
coordinates illustrated by MATLAB

The ultimate goal of this tutorial is to labelling the region in an image, if we know the target, we could calculate more useful data from those regions. The region we are trying to label is universally called “region of interest” or ROI. There could be more than one ROI in one image, and there might be ROI inside another ROI as well. It depends on our interests at the moment.

Find the ROI (Segmentation)

Before we could find the ROI of this image, we have to segment the turtle from the background, this process is called segmentation. In this case, we will use simple adaptive thresholding to segment this image. We could do that using the following code.

%% Load the image
I = imread('turtle_bottom_right.jpg');
% convert RGB to gray colormode
G = rgb2gray(I);

%% Adaptive thresholding (segmentation)
threshold = graythresh(G);
BW = im2bw(G,threshold);

You can show the result image here using …

figure, imshow(BW);

But, there is many things wrong about this segmentation process.

  1.  We usually let the interested area be the white (or value = 1), and the background should be dark (or value = 0, not interested).  We can fix that by inverting the color.
  2. Our turtle has some white color, and it could be mistaken to be considered as the background.
  3. Some noises are presented. We need to remove them using morphological operators or region properties.

1. Invert the color

To invert the color, we can do that using simple MATLAB operator ‘~’.

BW2 = ~BW;

The following image shows the image BW on the left and BW2 on the right.

1_invert_color
Comparison of inverted and original black and white image

2. Fill the turtle shell

Filling the ‘holes’ in the white region is a bit tricky. However, MATLAB has simple solution for us as well. The function is called ‘imfill’ and we need to fill ‘holes’ here.

BW_filled = imfill(BW2,'holes');
imfill example
Comparison of filled image and not filled image

If you don’t quite get how ‘imfill’ works yet, you may explore this short MATLAB script that will show you how this function behave.

3. Remove the noises

Note: In step 3, you may need to read this twice, there is a lot of stuff to demonstrate here.

You may spot some noises in the image after the holes are filled. There are many methods to do this. In this tutorial, we will use region properties to determine each region and assess weather it is noises or the ROI. There are 2 methods we are going to demonstrate in this tutorial, the hard way, and the easy way.

Method 1, the hard way

First, you have to imagine that each white spot, including the turtle, is one region. Each regions will have the properties of its own. For example, area or size (in pixels), roundness, or length and angle of major and minor axises.

Example of how region properties works
Example of how region properties works

In this case, it can be clearly seen that area of the turtle is much bigger than the noises. We will use this knowledge as our advantage to separate the ROI from the noises. So first we need to know the size of each region in the image. We could do that using easy function in MATLAB called ‘regionprops’. Please refer to the documentation for further information about this function here.

image_region_props = regionprops(BW_filled,'Area');

Now, the area property of our image is stick into the variable ‘image_region_props’. We want to get those values out so we can sort them and get the area of the biggest region in the image.

image_area = image_region_props.Area;
image_area = sort(image_area,'descend');

You can check ‘image_area’ in the workspace for the value. It should be … , or make the program show it in the command window by entering the following code.

display(image_area(1));

Now, we can only keep the largest area in the image using another MATLAB easy function called ‘bwareafilt’.

BW_roi_method1 = bwareafilt(BW_filled,[image_area(1),Inf]);

Method 2, the easy way

We know the number of region we need, we do not know the size of the region. 

This method will be easier because the MATLAB function can help us determine the biggest region and keep it. This is the same ‘bwareafilt’ function. In the following code, we will tell the function to keep 1 largest region in the image and ignores other regions.

BW_roi_method2 = bwareafilt(BW_filled,1);

Please note that the Method 2 only works in this case because there is only 1 region that we are interested in this image. If we need to preserve many regions with known surface area, we need Method 1 to get that done. In the example image below, we need to use Method 1 to separate small regions from the big regions.

Mixed small and large regions in one image
Mixed small and large regions in one image

So, it is easier to implement Method 2. However, it is important to know how this works like in Method 1 as well. Also, there are other morphological operation (stuff you do with binary image) such as image opening, image closing, and erosion and dilation that could remove noises like these as well. You might want to explore about that on your own.

Segmentation is completed!

Let’s review what we have at this point. We have a clean and clear segmented image. The image we are talking about is binary image (or black-and-white image). We often refer to the part we are interested in as value = 1, and background as value = 0. Now we are going to label them.

It is fair to say that the segmented image we had is good as labelled image. But, we are here to learn on how to label the image, so we will label 2 things here, the background and the image.

How should we label them?

There are two type of data we need to tell here.

  1. The coordinates of the pixel.
  2. The class (or group, or category) of that specific pixel — In this case, there will be 2 class, 0 is background, 1 is the turtle.

You could do something like this table which tells the x and y coordinates along with region number (or the class that region belongs to) but this table will be extensive, consumes a lot of memory, and hard to interpret by either computer or human eyes.

x coordinate

y coordinate

region number

1

1

1

1

2

1

1

3

1

1

4

0

1

5

1

The more established way to do this is to create some matrix that has the exact same dimension with our image. Then we do this

Concept illustration of region labelling
Concept illustration of region labelling

We assign class number to specific pixels which already has its own coordinates. This way we have created  labelling matrix we have been talking about since the beginning of this tutorial.

Now that we get the idea of how to construct a labelling matrix, we will construct one of them. Unfortunately, our binary image with 1 ROI already looks like labelling matrix. In the image below, whites represents the background.

RGB illustration of labelling matrix (one turtle case)

For the sake of demonstration, we will create an image that has two turtle by sticking two image into one image like this.

bind_turtle = [BW_roi_method1;BW_roi_method2];
RGB2 = label2rgb(L2);

Then we will try to make that a label matrix. In this case, we have 2 ROIs, therefore there will be region number 1 and region number 2. Different number of regions will be shown with different colors.  (showing ‘RGB2’)

RGB illustration of labelling matrix (two turtle case)
RGB illustration of labelling matrix (two turtle case)

Please note that the two turtle in the image above could also be the same region by assigning the same class to both of the region. However, the specified case is rare.

Done !

Intermediate result of Labelling 101 Tutorial
Intermediate result of Labelling 101 Tutorial

So, now you can manage to do simple segmentation of the image using thresholding method. You can also label the image. The completed MATLAB script of this tutorial is HERE. Next step is to learn about extracting each region of the image. I will update on how to extract the region later, but if you already figured that out, please share with others in the comment section. Also, please feel free to ask questions as well.

One more thing

We can apply the labelling matrix to the original image in order to segment the image using labelling matrix. In this case there is only one ROI, therefore we will use our only ROI to segment the image. The result segmented image is illustrated below.

Last update: February 11st, 2016 at 2:39PM


Creative Commons License
Labelling 101 – MATLAB Image Labelling Tutorial by Pargorn Puttapirat is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

2 Replies to “Image Labelling: Where is where, what is what!”

  1. อยากได้ตัวหนังสือสีนี้ ม่อนปรับเองหมดเลยอ่อ หรือมีสกินให้โหลด

Leave a Reply