Context-Sensitive Semantic Segmentation

Recognizing and representing objects of certain categories become increasingly important due to the availability of high-resolution imaging technologies and the explosive amount of digital data. In particular, semantic segmentation of given data (i.e.: two dimensional images or three dimensional volumes) labels or extracts objects in the form of contiguous regions with similar semantic interpretation. Hence semantic segmentation offers great rewards from object recognition and image segmentation. However, the combination of difficulties from both fields also yields incredible computational challenges. In practice, the appearance of objects is under the influence of views, poses, colors, shapes, scales, occlusion, illumination conditions and intrinsic imaging limitations. Thus an ideal semantic segmentation should tolerate both the considerable intra-class variance and the noticeable inter-class similarities in terms of appearance. The primary contribution of this thesis is the investigation on context cues that may improve semantic segmentation. I first propose a novel two-stage framework to solve a special problem of semantic segmentation, in which the target objects are much more likely to be observable under the existence of context objects. In the first stage, global salient context objects are segmented using appearance features. The second stage formulates multiple types of context cues, followed by a model that combines both appearance and context cues. I then apply this framework to the problem of spike segmentation and tattoo segmentation, resulting in a cryo-electron tomogram segmentation system and a tattoo classification system. The first system allows biophysicists to significantly accelerate their data processing by replacing manual annotation with semi-automatic segmentation, whereas the second system explores for the first time the possibility of category-level tattoo classification by machine. As shown by these two systems, the proposed models outperform traditional object-centered models that purely focus on appearance features.