Object co-segmentation

Example video frames and their object co-segmentation annotations (ground truth) in the Noisy-ViDiSeg dataset. Object segments are depicted by the red edge.

In computer vision, object co-segmentation is a special case of image segmentation, which is defined as jointly segmenting semantically similar objects in multiple images or video frames. The problem of separating out a foreground object from the background across all frames of a video is known as video object segmentation. The goal is to label each pixel in all video frames according to whether it belongs to the unknown target object or not. The resulting segmentation is a spatio-temporal object tube delineating the boundaries of the object throughout a video. Such capacity can be useful for a variety of computer vision tasks, such as object centric video summarization, action analysis, video surveillance, and content-based video retrieval.

Challenges

It is often challenging extract segmentation masks of a target/object from a noisy collection of images or video frames, which involves object discovery coupled with segmentation. A noisy collection implies that the object/target is present sporadically in a set of images or the object/target disappears intermittently throughout the video of interest. Most methods either only fulfill video object discovery, or video object segmentation presuming the existence of the object in all video frames. The unrealistically optimistic assumption is often made in these methods, that the target object is present in all (or most) video frames. Therefore, methods robust to a large number of noisy frames (i.e., irrelevant frames devoid of the target object) are urgently needed.

Moreover, most of existing methods emphasized on leveraging the low-level features (i.e., color and motion) or contextual information shared among individual or consecutive frames to find the common regions, and simply employed the short-term motion (e.g., optical flow) between consecutive frames to smooth the spatio-temporal segmentation. Therefore, they often encountered difficulties when the objects exhibit large variations in appearance, motion, size, pose, and viewpoint.

Furthermore, several methods,,,,, employed the mid-level representation of objects (i.e., object proposals) as an additional cue to facilitate the segmentation of the object, with object discovery and object segmentation conveniently isolated as two independent tasks and performed in a two-step manner}. Unfortunately, the disregard of their dependencies often leads to sub-optimal performances, e.g., object segmentation dramatically failing at focusing on the target, object discovery providing wildly inaccurate object proposals.

Dynamic Markov Networks-based Methods

The inference process of the two coupled dynamic Markov networks to obtain the joint video object discovery and segmentation

A joint object discover and co-segmentation framework based on coupled dynamic Markov Networks

A joint object discover and co-segmentation method based on coupled dynamic Markov Networks has been proposed recently, which claims significant improvements in robustness against irrelevant/noisy video frames. A principled probabilistic model is introduced with two coupled dynamic Markov networks, one for discovery and the other for segmentation. When conducting the Bayesian inference on this model using belief propagation, the bi-directional propagation of the beliefs of the object's posteriors on an object proposal graph and a superpixel graph reveals a clear collaboration between these two inference tasks. More specifically, object discovery is conducted through the object proposal graph representing the correlations of object proposals among multiple frames, which is built under the help of the spatio-temporal object segmentation tube obtained by object segmentation on the superpixel graph. Object segmentation is achieved on the superpixel graph representing the connections of superpixels, which is benefited from the spatio-temporal object proposal tube generated by object discovery through the object proposal graph.

Given a video ${\textstyle \mathbf {V} =\{{f}_{t}\}_{t=1}^{T}}$ with a significant number of noisy frames, our goal is to jointly find an object discovery labeling ${\textstyle \mathbf {L} }$ and an object segmentation labeling ${\textstyle \mathbf {B} }$ from ${\textstyle \mathbf {V} }$. ${\textstyle \mathbf {L} =\{\mathbf {L} _{t}\}_{t=1}^{T}}$ is a spatio-temporal region (object) proposal tube of ${\textstyle \mathbf {V} }$. ${\textstyle \mathbf {L} _{t}=\{l_{t,i}\}_{i=1}^{K}}$ is the object discovery label of each frame ${\textstyle f_{t}}$, where ${\textstyle l_{t,i}\in \{0,1\}}$ and ${\textstyle \sum _{i=1}^{K}{l_{t,i}}\leq 1}$, i.e., no more than one region proposal among all the ${\textstyle K}$ proposals in ${\textstyle f_{t}}$ will be identified as the object. ${\textstyle \mathbf {B} =\{\mathbf {B} _{t}\}_{t=1}^{T}}$ is a spatio-temporal object segmentation tube of ${\textstyle \mathbf {V} }$. ${\textstyle \mathbf {B} _{t}=\{b_{t,j}\}_{j=1}^{J}}$ is the object segmentation label of ${\textstyle f_{t}}$, where ${\textstyle b_{t,j}\in \{0,1\}}$ denotes that each of the ${\textstyle J}$ superpixels either belongs to the object (${\textstyle b_{t,j}=1}$) or the background (${\textstyle b_{t,j}=0}$). The image observations associated with ${\textstyle \mathbf {L} }$, ${\textstyle \mathbf {L} _{t}}$, ${\textstyle \mathbf {B} }$, and ${\textstyle \mathbf {B} _{t}}$ are denoted by ${\textstyle \mathbf {O} =\{\mathbf {O} _{t}\}_{t=1}^{T}}$, ${\textstyle \mathbf {O} _{t}=\{o_{t,i}\}_{i=1}^{K}}$, ${\textstyle \mathbf {S} =\{\mathbf {S} _{t}\}_{t=1}^{T}}$ and ${\textstyle \mathbf {S} _{t}=\{s_{t,j}\}_{j=1}^{J}}$, respectively. ${\textstyle o_{t,i}}$ and ${\textstyle s_{t,j}}$ are the representations of a region proposal and a superpixel, respectively.

Specifically, the beneficial information are encouraged to be propagated between the joint inference of ${\textstyle \mathbf {L} }$ and ${\textstyle \mathbf {B} }$, and hence video object discovery and video object segmentation can naturally benefit each other. A Markov network is employed to characterize the joint object discovery and segmentation from ${\textstyle \mathbf {V} }$. The undirected link represents the mutual influence of object discovery and object segmentation, and is associated with a potential compatibility function ${\textstyle \Psi (\mathbf {L,B} )}$. The directed links represent the image observation processes, and are associated with two image likelihood functions ${\textstyle p(\mathbf {O|L} )}$ and ${\textstyle p(\mathbf {S|B} )}$. According to the Bayesian rule, it is easy to obtain $$p(\mathbf {L,B|O,S} )={\frac {1}{Z_{Q}}}\Psi (\mathbf {L,B} )p(\mathbf {O|L} )p(\mathbf {S|B} ),$$ where ${\textstyle {Z_{Q}}}$ is a normalization constant. The above Markov network is a generative model at one time instant.

When putting the above Markov network into temporal context by accommodating dynamic models, we construct two coupled dynamic Markov networks. The subscript ${\textstyle t}$ represents the time index. In addition, we denote the collective image observations associated with the object discovery labels from the beginning to ${\textstyle t}$ by ${\textstyle {\vec {\mathbf {O} }}_{t}=\{\mathbf {O} _{1},\dots ,\mathbf {O} _{t}\}}$, and reversely from the end to ${\textstyle t}$ by ${\textstyle {\overleftarrow {\mathbf {O} }}_{t}=\{\mathbf {O} _{T},\dots ,\mathbf {O} _{t}\}}$. The collective image observations associated with the object segmentation labels are built in the same way, i.e., ${\textstyle {\vec {\mathbf {S} }}_{t}=\{\mathbf {S} _{1},\dots ,\mathbf {S} _{t}\}}$ and ${\textstyle {\overleftarrow {\mathbf {S} }}_{t}=\{\mathbf {S} _{T},\dots ,\mathbf {S} _{t}\}}$. In this formulation, the problem of joint video object discovery and segmentation from a single noisy video is to perform Bayesian inference of the dynamic Markov networks to obtain the marginal posterior probabilities ${\textstyle p(\mathbf {L} _{t}|\mathbf {O} ,\mathbf {S} )}$ and ${\textstyle p(\mathbf {B} _{t}|\mathbf {O} ,\mathbf {S} )}$.

See also

• Image segmentation
• Object detection
• Object-based image analysis
• Video content analysis
• Image analysis
• Activity recognition
• Computer vision