Simple interactive object extraction
Simple Interactive Object Extraction (SIOX) is an algorithm for extracting foreground objects from color photographs with little user interaction. SIOX is a tool in the GIMP (since version 2.3.3) and Inkscape. SIOX originates from a distance learning tool called E-Chalk where an instructor standing in front of an electronic chalkboard is segmentated. The algorithm can also be used for segmentation of videos.
Initially, a free hand selection tool is used to specify the region of interest. It must contain all foreground objects to extract and as few background as possible. The pixels outside the region of interest form the sure background while the inner region define a superset of the foreground, i.e. the unknown region. A so-called foreground brush is then used to mark representative foreground regions. The algorithm outputs a selection mask. The selection can be refined by either adding further foreground markings or by adding background markings using the background brush.
Technically, the algorithm performs the following steps:
- Create a set of representative colors for sure foreground and sure background, the so-called color signatures.
- Assign all image points to foreground or background by a weighted nearest neighbour search in the color signatures.
- Apply some standard image processing operations like erode, dilate, and blur to remove artifacts.
- Find the connected foreground components that are either large enough or marked by the user.
For video segmentation the sure background and sure foreground regions are learned from motion statistics. SIOX also features tools that allow sub-pixel accurate refinement of edges and high texture areas, the so-called "detail refinement brushes".
As with all segmentation (image processing) algorithms, there are always pictures where the algorithm does not yield perfect results. The most critical drawback of SIOX is the color dependence. Although many photos are well-separable by color, the algorithm cannot deal with camouflage. If the foreground and background share many identical shades of similar colors, the algorithm might give a result with parts missing or incorrectly classified foreground.
The algorithm is currently under development at the department of computer science at Freie Universitaet Berlin.