masterhead masterhead  masterhead

Portable Lumipen: mobile dynamic projection mapping system


In recent years, Spatial Augmented Reality (SAR) systems for dynamically moving targets have been proposed, and this field is called "dynamic SAR" or "dynamic projection mapping." In these systems, the moving target is recognized by a high-speed camera and the projection onto the target is controlled at high speed, for example, at 1,000fps. By achieving such high responsiveness in sensing and feedback that spatio-temporal consistency between the target and projected image is ensured at the level of human visual perception, these dynamic SAR systems have allowed the development of new application fields in entertainment and user interfaces. However, one problem with most of these conventional SAR systems is that they have restricted spatial movement because they are fixed in a room. Then, we take dynamic SAR a step further. In this research, we use a 3D-stacked vision chip and an optical gaze controller, and realize a portable dynamic SAR system, called "Portable Lumipen." A vision chip is an intelligent imager that has parallel processing elements adjacent to photodetectors, which enables high-speed image capturing and processing on a single chip, and it is a suitable device for a high-speed and small visual feedback system. An optical gaze controller is a unit to change the imaging and projecting direction at high speed by reflecting light at 2-axis motored mirrors, and achieves high-speed alignment of the projection image without the need for a specialized projector or re-rendering of the projection image.

We developed the prototype system of Portable lumipen in Fig.1 and Fig.2, and confirmed that image capturing at 1,000fps, rapid processing and a 3ms response time for projection were achieved on a mobile device. We also propose potential applications using this portable dynamic SAR system. For example, in anywhere surface user interface shown in Fig.3, the system recognizes a gesture and provides a display on an arbitrary object. In Fig.4, a system for taking a selfy with projector-based makeup is shown. The Portable Lumipen can compensate for hand shaking, and realizes animated makeup. Besides, a searchlight that automatically scans the surrounding environment and tracks a detected target while projecting information on it is proposed. Portable dynamic SAR will enhance a variety of applications including entertainment, interface and combination with a drone.

Fig.1 Portable Lumipen
Fig.2 System inside

Fig.3 Anywhere surface UI
Fig.4 Selfy with Makeup by light
Fig.5 Tracking search light


If you want to use the original video, please send an e-mail for copyright permission to .


  1. Leo Miyashita, Tomohiro Yamazaki, Kenji Uehara, Yoshihiro Watanabe and Masatoshi Ishikawa: Portable Lumipen: Dynamic SAR in Your Hand, International Conference on Multimedia and Expo (ICME2018), San Diego, California, USA, 23-27 Jul.
Ishikawa Senoo Laboratory, Department of Information Physics and Computing, Department of Creative Informatics,
Graduate School of Information Science and Technology, University of Tokyo
Ishikawa Senoo Laboratory WWW admin:
Copyright © 2008 Ishikawa Senoo Laboratory. All rights reserved.