Séminaire LIGM - Labeling the world

Date : Thursday, May 7 at 11:00

"Labeling the world"

Bryan C. Russell

Adobe Research

Abstract: Much of our 3D visual world is described in auxiliary data, such as online text and maps. In this talk I will describe two works that reason about such auxiliary data, together with reconstructed 3D geometry. In the first part I'll describe an approach for analyzing Wikipedia and other text, together with online photos, to produce annotated 3D models of famous tourist sites. The approach is completely automated, and leverages online text and photo co-occurrences via Google Image Search. It enables a number of interactions, which we demonstrate in a novel 3D visualization tool. Text can be selected to move the camera to the corresponding objects, 3D bounding boxes provide anchors back to the text describing them, and the overall narrative of the text provides a temporal guide for automatically flying through the scene to visualize the world as you read about it. We show compelling results on several major tourist sites. In the second part I'll describe an approach for analyzing annotated maps of a site, together with Internet photos, to reconstruct large indoor spaces of famous tourist sites.

While current 3D reconstruction algorithms often produce a set of disconnected components (3D pieces) for indoor scenes due to scene coverage or matching failures, we make use of a provided map to lay out the 3D pieces in a global coordinate system. Our approach leverages position, orientation, and shape cues extracted from the map and 3D pieces and optimizes a global objective to recover the global layout of the pieces. We introduce a novel crowd flow cue that measures how people move across the site to recover 3D geometry orientation. We show compelling results on major tourist sites.

Bio: Bryan Russell is a Research Scientist in the Creative Technologies Lab at Adobe Research in San Francisco. He received his Ph.D. from MIT in the Computer Science and Artificial Intelligence Laboratory in 2008 under the supervision of Professors William T. Freeman and Antonio Torralba. He was a post-doctoral fellow from 2008-2010 in the INRIA Willow team at the Département d'Informatique of Ecole Normale Supérieure in Paris, France. He was a research scientist with Intel Labs from 2012-2014 as part of the Intel Science and Technology Center for Visual Computing (ISTC-VC) and has been affiliated with the University of Washington since 2011.