Deep Manifold Alignment for Mid-grain
Sketch based Image Retrieval
published in Asian Conference on Computer Vision (ACCV), December 2018. DOI: 10.1007/978-3-030-20893-6_20
University
of
Surrey, UK.
University of Sao Paulo, Brazil.
Abstract
We present an algorithm for visually searching image collections using free-hand sketched queries. Prior sketch based image retrieval (SBIR) algorithms adopt either a category-level or fine-grain (instance-level) definition of cross-domain similarity --- returning images that match the sketched object class (category-level SBIR), or a specific instance of that object (fine-grain SBIR). In this paper we take the middle-ground; proposing an SBIR algorithm that returns images sharing both the object category and key visual characteristics of the sketched query without assuming photo-approximate sketches from the user. We describe a deeply learned cross-domain embedding in which 'mid-grain' sketch-image similarity may be measured, reporting on the efficacy of unsupervised and semi-supervised manifold alignment techniques to encourage better intra-category (mid-grain) discrimination within that embedding. We propose a new mid-grain sketch-image dataset (MidGrain65c) and demonstrate not only mid-grain discrimination, but also improved category-level discrimination using our approach.
Paper
Deep Manifold Alignment for Mid-grain Sketch based Image Retrieval (pdf).
Supplementary Materials
1. Dataset
MidGrain65c 138 sketches and 1247 relevant images + distracting images from Adobe Stock totalling 100k corpus (800MB) (Download).
2. Pretrained model
Caffe pretrained model with deploy prototxt for image and sketch (80MB)(Download).
3. Code
Python code for feature extraction (Github).
4. BibTeX
@inproceedings{bui2018deep,
title = {Deep manifold alignment for mid-grain sketch based image retrieval},
author = {Bui, Tu and Ribeiro, Leonardo and Ponti, Moacir and Collomosse, John},
booktitle = {Proceedings of the 14th Asian Conference on Computer Vision},
pages={314--329},
year={2018},
publisher={Springer}
}