Multi-instance Referring Image Segmentation of Scene Sketches based on Global Reference Mechanism

Loading...
Thumbnail Image
Date
2022
Journal Title
Journal ISSN
Volume Title
Publisher
The Eurographics Association
Abstract
Scene sketch segmentation based on referring expression plays an important role in sketch editing of anime industry. While most existing referring image segmentation approaches are designed for the standard task of generating a binary segmentation mask for a single or a group of target(s), we think it necessary to equip these models with the ability of multi-instance segmentation. To this end, we propose GRM-Net, a one-stage framework tailored for multi-instance referring image segmentation of scene sketches. We extract the language features from the expression and fuse it into a conventional instance segmentation pipeline for filtering out the undesired instances in a coarse-to-fine manner and keeping the matched ones. To model the relative arrangement of the objects and the relationship among them from a global view, we propose a global reference mechanism (GRM) to assign references to each detected candidate to identify its position. We compare with existing methods designed for multi-instance referring image segmentation of scene sketches and for the standard task of referring image segmentation, and the results demonstrate the effectiveness and superiority of our approach.
Description

CCS Concepts: Computing methodologies → Scene understanding; Image Segmentation

        
@inproceedings{
10.2312:pg.20221238
, booktitle = {
Pacific Graphics Short Papers, Posters, and Work-in-Progress Papers
}, editor = {
Yang, Yin
and
Parakkat, Amal D.
and
Deng, Bailin
and
Noh, Seung-Tak
}, title = {{
Multi-instance Referring Image Segmentation of Scene Sketches based on Global Reference Mechanism
}}, author = {
Ling, Peng
and
Mo, Haoran
and
Gao, Chengying
}, year = {
2022
}, publisher = {
The Eurographics Association
}, ISBN = {
978-3-03868-190-8
}, DOI = {
10.2312/pg.20221238
} }
Citation