Multi-instance Referring Image Segmentation of Scene Sketches based on Global Reference Mechanism
Loading...
Date
2022
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
The Eurographics Association
Abstract
Scene sketch segmentation based on referring expression plays an important role in sketch editing of anime industry. While most existing referring image segmentation approaches are designed for the standard task of generating a binary segmentation mask for a single or a group of target(s), we think it necessary to equip these models with the ability of multi-instance segmentation. To this end, we propose GRM-Net, a one-stage framework tailored for multi-instance referring image segmentation of scene sketches. We extract the language features from the expression and fuse it into a conventional instance segmentation pipeline for filtering out the undesired instances in a coarse-to-fine manner and keeping the matched ones. To model the relative arrangement of the objects and the relationship among them from a global view, we propose a global reference mechanism (GRM) to assign references to each detected candidate to identify its position. We compare with existing methods designed for multi-instance referring image segmentation of scene sketches and for the standard task of referring image segmentation, and the results demonstrate the effectiveness and superiority of our approach.
Description
CCS Concepts: Computing methodologies → Scene understanding; Image Segmentation
@inproceedings{10.2312:pg.20221238,
booktitle = {Pacific Graphics Short Papers, Posters, and Work-in-Progress Papers},
editor = {Yang, Yin and Parakkat, Amal D. and Deng, Bailin and Noh, Seung-Tak},
title = {{Multi-instance Referring Image Segmentation of Scene Sketches based on Global Reference Mechanism}},
author = {Ling, Peng and Mo, Haoran and Gao, Chengying},
year = {2022},
publisher = {The Eurographics Association},
ISBN = {978-3-03868-190-8},
DOI = {10.2312/pg.20221238}
}