"Sensor Fusion" is an inclusive term for technologies that use multiple sensors of the same or different types to extract useful information that cannot be obtained from a single sensor. Strictly speaking, it should be called "Sensor Data Fusion"; however, the shorter "Sensor Fusion" has now become entrenched. The similar terms "Multi-sensors" and "Integration" are also used to describe the concept, and "Binding" in psychology has a similar meaning. Sensor fusion forms the basis for processing structures in "Information Fusion" and "Sensor Networks". Many associated themes have been discussed, such as conjoined sensors, structural theory of sensory information processing using intelligent sensors, architectural theory of network construction and processing hardware, signal and statistical processing related to the computational structure of processing, knowledge processing and artificial intelligence related to logical structure, accommodation theory of learning in the case where the processing structure is unknown, and overall system design.

Conventionally, there was only the serial model, which operates after recognizing the outside world. Recently, however, various types of integrated processing models have been proposed, such as those in which motion is performed in order to realize sensing and those in which processing systems function as upper-level monitors of lower-level sensor feedback systems. These processing architectures are called "Sensory Motor Integration". A massively parallel information processor like the brain is essential for implementing this kind of architecture. In order to construct a sensory motor integration system, we must integrate a sensory system (sensors), a processing system (computers and algorithms), and a kinetic system (actuators) with tight compatibility , and also deal with the entire system comprehensively, involving the outside world and the tasks to be performed. In our laboratory, we develop high-speed robots based on "Sensory Motor Integration" from the standpoint of functional aspects and time characteristics.

Industrial robots are capable of high-speed motion when it comes to "playback motion", i.e., reproducing a prescribed motion , but when sensor feedback, especially visual feedback, is introduced, their motion can be delayed due to the processing required in the visual system. Even in the case of intelligent robots such as humanoids designed to operate like the human body, their whole body movements are also slow due to the slowness of the sensory and recognition systems. Based on the fact that robots, in mechanical terms, are capable of performing motions much faster than the human body, in our laboratory, we are attempting to speed-up robot tasks by making the sensory and recognition systems faster. Our goal in robot research is to build intelligent robots that include sensory and recognition systems and that are able to move so fast that we cannot even see their motion, surpassing the motion speed of the human body.

Beginning with the Turing test, there have been many conventional ways of thinking about how we define "Intelligence". This question arises not only in computers (that is to say, in the information world) but also in the real world. Therefore, here we consider an intelligent system as a system that interacts adaptively with the real world, where a variety of changes in sensory/recognition systems (sensor technology), processing systems (computer technology), and motor/behavior systems (actuator technology) coexist. Since the definition of intelligence mentioned above includes the conventional one and is aimed at the real world, the problem setting is more difficult. There are three key parts to realize an intelligent system: The first is to establish computational theories, especially for building hierarchical parallel distributed architectures. The second is the configuration of the algorithms and information expression rules by which the theories are applied to the real world, particularly including their internal models and information representation, as well as the design of fusion algorithms. The third is to construct hardware that interacts with the real world, such as smart sensors and smart actuators.

In the domain of intelligent systems, Albus suggested a model serving as a structure model of a general intellectual processing system that is inspired by the human brain but surpasses its limitations. The model is based on a parallel distributed architecture in which processing modules for each function are connected with each other in a parallel and hierarchical way and are integrated with sensory, processing, and motor systems. In the model, sensor data input to the sensory and recognition systems is processed in progressively higher hierarchical levels and is passed to the next level as afferent information, gradually increasing the level of abstraction of the information. The processed information, which is regarded as efferent information for the lower-level motor and behavior system, is converted to concrete signals that are passed on to the actuators. In each hierarchical level, information is processed using the corresponding representation and time constant, and feedback loops spanning the multiple levels and having a multiplexed structure are formed. In higher levels, knowledge processing is performed to realize logical structures such as decision and planning. In lower levels, highly parallelized signal processing is performed under constraint conditions that require highly real-time properties. To make effective use of the structure, decomposition of the task in question will be key. (Refer to Task Decomposition below.)

In the construction of intelligent systems having a hierarchical parallel distributed architecture, the whole task needs to be decomposed into modular processes in order to functionally implement these subtasks on the distributed processing modules. This is called task decomposition. Task decomposition is an important subject because the behavior of the intelligent system is affected by the task decomposition method used. However, there is no general solution for task decomposition, and several design concepts have been proposed. As a basic structure, task decomposition is separated into sequential decomposition, which decomposes the task into the sensory system, the processing system, and the motor system, and parallel decomposition, which makes virtual parallel feedback loops. In practice, sequential decomposition and parallel decomposition are often used together. With sequential decomposition, the design of each module is easy, but the whole system becomes slow if there is a heavy processing load. On the other hand, parallel decomposition has the advantage of higher processing speed, but it can be realized only by heuristics. Our laboratory has proposed orthogonal decomposition, which makes the outputs of processing modules independent of time and space by simply summing the outputs of parallel modules and using the sum as the input for a limited number of actuators.

The design concepts for intelligent systems using the high-speed sensor feedback proposed by our laboratory when interacting with real objects (including the physical systems of robots) involve specific dynamics. Therefore, due to the sampling theorem, all of the components of the system need to have sufficiently wide bandwidth relative to the objects' dynamics in order to measure and control the objects perfectly. Dynamics matching means realizing an intelligent system that matches the properties as a whole, by designing the sensory system (sensors), processing system (computers), and motor system (actuators) so that they have sufficiently wide bandwidth to cope with the object's dynamics. If there is a slow module in the system, the whole system is constrained by the dynamics of that module because the system controls the object's dynamics based on imperfect information. Sampling rates of currently available servo controllers are about 1 kHz; therefore, 1 kHz is the rough upper target to be realized in real mechanical intelligent systems.

Real-time processing is to realize processing in a defined time, and it is an essential technology in fast-moving systems in the real world, such as robots. Although parallel processing is effective for high-speed arithmetic processing, some problems such as priority inversion occur due to changes in the execution time of operations and data transfer between processing modules. As a result, the overall processing time is difficult to control, so that it is extremely difficult to make parallel processing compatible with the real-time nature required for a robot. Therefore, at present, in many cases, the process is designed in an ad hoc manner for the target function.

Sensor feedback involves feeding back sensor information, about the environment as well as the robot, to the robot operation. Usually, sensor feedback denotes feedback of information from sensors that capture changes of the outside world and the interaction between the robot and the outside world, or control to reflect such changes of the environment and the object in the robot's action in real time. Conventional industrial robots are designed to repeat the same operation over and over, that is to say, "playback", with certain levels of accuracy and speed, and are evaluated based on their ability to achieve these levels. On the other hand, in sensor feedback mode, robots are not forced to repeat the same movement, so that the accuracy should be evaluated in terms of an absolute accuracy or a relative accuracy, and the robots must be designed by taking into account also the operating speed and the processing time for recognition and understanding in the sensor information processing system. In order to realize a high-speed robot as an intelligent system, a design concept like dynamics matching proposed by this laboratory is necessary, and the introduction of backlash-free mechanisms suitable for the non-repetitive control is essential.

Visual Feedback, which is one type of Sensor Feedback, particularly refers to feedback control for image information. Conventionally, it has been difficult to utilize image information for feedback in real time, because image information, which is two-dimensional, requires a long time for image processing, which affects the feedback rate. However, our High-Speed Image Processing system makes real-time visual feedback possible. Target objects for image information feedback include robots, robot manipulation objects, lights, imaging cameras and so on. Example applications include robot tasks, manipulation control, micro-visual feedback for control of microscope images, active vision, target tracking and so on. A coordinate transform error occurs when using a coordinate transform from an image to the task coordinates via absolute coordinates, but this error can be removed by introducing relative coordinate control between the target and robot in the image.

A Sensor Network is a network in which various kinds of sensors are provided, and their sensor information is utilized. Whereas conventional network nodes are mainly computers, sensor network nodes are sensors that provide sensor information, so in a sensor network it is possible to acquire and utilize real-world information. The idea of a cyber physical system has been proposed. As it stands now, the problem is how to use a conventional network architecture to connect sensor information, and research has involved merely improving protocols. Current network architectures are not, in essence, suitable for realizing basic sensing architectures because of the requirements for real-time performance, space and time density of information, and security. In particular, a sensor network needs to realize task decomposition on a Hierarchical Parallel Distributed Architecture, which is essential for Sensor Fusion and Sensory Motor Integration and is the basis of Real-Time Parallel Processing, and also requires a structure capable of implementing active sensing and intentional sensing based on dynamics matching.

While the term generally has many meanings, in this laboratory, Active Sensing refers to the way that we sense and recognize objects using actuators. When we are confronted with an unknown environment, Active Sensing enables us to sense the environment in advance and provides a lot of benefits. Specifically, the aim is to search for objects (positions, aspects) and avoid locality (configuration) when exploring a comprehensive structure with regional sensors. It is possible to improve the spatial resolution by using high-resolution regional sensors and comprehensive sweeping, optimized sensing of minute structures and surface textures by using actuator-related time-series signals and the responses to them, and recovery of dynamic characteristics, particularly those with differential behavior, by controlling the temporal properties of actuators. Active Sensing is closely related with ideas such as affordance (J.J. Gibson), which proposes that an agent can perform shape recognition and exhibit certain behavior by means of the relation between her/his behavior and the environment that s/he is involved in, by studying the relationship between self-behavior and self-recognition, as well as the ideas of the perceptual cycle (U. Neisser), selective attention, and self-recognition based on proprioception. This concept is called Active Vision and has been gathering a lot of attention in the field of optical research, and is called Haptics in research on tactile perception.

Sensing is a process in which we narrow-down the space in which a solution may exist by using measured values and constraint conditions, or find an optimal solution by statistical processing in cases where a lot of useful information exists, in regard to the solution space, which contains information upon which algorithms may converge. When multidimensional information space is dealt with using a small amount of sensor information, the problem becomes ill-posed, meaning that the amount of useful information is smaller than the amount of information about the target space , and the problem of searching a large information space often occurs. In this case, to constrain the solutions, not only measured values but also past experience or physical constraints are frequently used as constraint conditions. Moreover, as sensing has its own goal, it is also possible to constrain the target information space by using an explicit distribution of the objects as a constraint condition. This method is called Intentional Sensing. This idea was proposed in the Sensor Fusion Project, which ran from 1991 to 1995, and plays an important role in active recognition in Sensory Motor Integration.

A Tactile Sensor is a sensor that is equivalent to a touch receptor under the human skin. It usually means a sensor that measures a pressure distribution on its surface associated with touch, but sensor assemblies consisting of force sensors and temperature sensors, or heat current sensors, also exist. Usually, such sensors measure the distortion of a flexible elastic body. When designing a Tactile Sensor, it is necessary to ensure flexibility while maintaining durability, so that the sensor can conform to many kinds of three-dimensional surface forms, to ensure a large surface area, depending on the circumstances, to design circuit technology for acquiring pressure distribution information, and to decrease the number of cables in order to provide a larger working area. None of these requirements are seen in usual electronic devices. While it is not true to say that fixed sensors are never seen, sensors that are fixed to movable parts exhibit high-activity because motion of the sensors has a large influence on the measurement. The motion that makes a Tactile Sensor work effectively is called a touching motion. The study of perceptual structure, while taking account of haptic sense and motion at the same time, is called Haptics.

This is the general term for high-speed dynamic robot manipulation. The aim is to realize swinging motions that are close to the dynamical limit, which is impossible for conventional slow and quasi-static manipulation systems. Conventionally, the recognition ability and motion capability have not been rapid enough to keep up with high-speed / accelerated movements of the target; thus, even playback or feedforward-driven control systems could realize dynamic manipulation only with limited trajectories. To solve this problem, our laboratory has developed high-speed sensors and actuators that can cover a wide range, and can also perform high-speed and dexterous manipulations with fewer degrees-of-freedom by intentionally utilizing unstable or non-contact states for the target. We aim to create a brand new dynamic manipulation system by getting the maximum performance from sensor-actuator systems.

Sampling means acquiring values of an analog signal (original signal, continuous value) as samples (discrete signals) at certain time intervals. The interval is called the sampling interval, and its reciprocal is called the sampling frequency. The sampling theorem states that, compared with the peak frequency of a certain band-limited signal, it is possible to restore the original signal completely from the sampled values by sampling the original signal at a sampling frequency that is more than twice the peak frequency. Thus, in designing a system, in order to acquire and comprehend an object completely, it is necessary to use a sensing system whose frequency band is more than twice as wide as the frequency band of the target items to be measured , after comprehending or setting the frequency band of these items. In reality, however, it is extremely difficult to set the frequency band or the sampling frequency like the operating frequency of the system. Therefore, by using a control system design that takes account of this problem , it is desirable to set the frequency band not just twice as large but wider. For example, for temporal control management of a robot, some textbooks recommend setting the sampling frequency band to be approximately ten times larger. In the visual feedback in our laboratory, since in many cases the sampling time of the servo controller generally is set to 1 ms, our basic goal is to realize a frame rate with an upper limit of 1,000 fps in visual information processing from the viewpoint of dynamics matching. This means that, theoretically, it is possible to acquire and comprehend the object in a frequency band below 500 Hz, but in terms of control of the object, depending on the dynamical characteristics of the object or the system, we cover a slightly lower frequency band (e.g., a frequency band up to 100 to 500 Hz).

Dynamic Image Control is a technology that presents, in a simple form, some objects and phenomena that cannot normally be seen, by appropriately controlling the optical system, illumination system, and processing systems in response to various phenomena exhibiting dynamical behavior. Since a trade-off exists between the angle of view and resolution in conventional slow fixed imaging systems due to the angle of view being fixed, it is impossible to take images outside the angle of view and to take images at high resolution. In addition, it has not been possible to capture a moving object at high resolution because the images include dynamic phenomena, like the movement of the object. Dynamic Image Control enables us to capture the required images, according to the actual implementation of the system, by compensating for unwanted movements in the images.

It is difficult to manipulate minute objects like microorganisms in microscopy, and operators need to acquire special skills. Micro Visual Feedback enables us to get the required information and images by capturing enlarged images of a minute object at a high frame rate and feeding back information about the object at high speed, with high accuracy, and in a non-contact manner. On the micro scale, the natural frequency of an object and its velocity relative to its size are high; therefore, image processing in Micro Visual Feedback and high-speed actuation performance are especially important elements. It is expected that, with this method, we will be able to manipulate minute objects autonomously without an excessive burden on the operator, which will lead to breakthroughs in microscope observation, inspection, and manipulation.

A microbe can be regarded as a single module that composes a system in combination with an information processing mechanism through high-speed vision. Such modules will enable very large-scale micro-systems that provide flexible and diverse functions as an element in the system. In living organisms, in order to achieve precise detection of environmental changes and allow quick action in response, microbes have developed highly sensitive, highly precise sensors and actuators. In an Organized Bio Module, a microbe is regarded as a bio module in which a highly sensitive sensor and a microminiature actuator are integrated. The development of an interface for associating a processing element with multiple Organized Bio Modules will realize new microsystems in which living organisms and information processing mechanisms are fused.

Active Vision is a technology for obtaining useful image information by adding information or energy to an object. Although there are many definitions, in the broadest definition, it includes image processing using gaze control corresponding to eye movement and image processing in which an object is intentionally irradiated with patterned light using structured illumination. Gaze control includes a method using real-time visual feedback, a method of improving image search efficiency by using stored information, and so on. These methods achieve better performance than those using a fixed camera.

Target tracking is a technique for tracking a moving target and obtaining necessary information about it. In the case where there are multiple targets, it is called multi-target tracking. When the aim is to acquire an image of the moving target, we first capture the target with a high-speed vision system that can track it, then give feedback to the actuator to control the line of sight and fix the target at the center of the field of view. In addition to basic regulation control with two degrees of freedom, i.e., pan and tilt, three-dimensional tracking with stereo vision and monocular three-dimensional tracking with high-speed focusing have also been achieved.

The part in an optical system that limits light entering the image plane, such as the aperture of a camera and the iris of the human eye, is called the "pupil". Connecting multiple optical systems often causes obscuration and image loss because some rays of light that go through the pupil of one system cannot pass through the pupil of another. The Pupil Shift System is placed at the connection between two optical systems so that more rays can pass through both pupils effectively. In the Saccade Mirror, it is located between the small rotating mirrors and the camera.

The Self-Windowing Method is a simple target tracking algorithm designed on the assumption that images are captured at high speed. With high-speed images, by setting a window around the target in a frame, it can be assumed that the target is still in the window at the next frame. This is because the movement of the target in the image plane between frames is small. In the case where the movement of the target on the image plane is larger, by increasing the camera frame rate in order to satisfy the assumption, the search range becomes extremely small, and target extraction can be achieved using a simple matching algorithm. This method could also be applied to three-dimensional tracking in the same manner.

Most cameras and binoculars have a focusing mechanism. Existing optical systems consist of multiple solid lenses, and focus adjustment is achieved by changing the positions of some of the lenses. However, the need to control the positions of the lenses with high accuracy causes some problems; for example, the mechanism becomes complex and is thus difficult to reduce in size, and it is difficult to perform focus adjustment rapidly. A Variable Focus Lens is a new optical device in which a single lens itself has a focusing mechanism. If such a lens could be realized in practice , it could achieve focus adjustment and zoom functions without moving lenses. This would allow the development of extremely small, low-power, high-functionality digital cameras. The Variable Focus Lens is currently an area of active research, including focus adjustment using a change in shape of a plate, film, or interface between two liquids, or by using materials with variable refractive index, such as liquid crystal. In some cases, a whole optical system that achieves focusing by moving lenses mechanically is called a Variable Focus Lens. In this laboratory, however, "Variable Focus Lens" means a device in which the lens itself has a focus adjustment function.

An all-in focus image is an image in which the whole scene is in focus. While it is impossible to capture such an image with a usual camera, it can be computed from a series of differently focused images or from images captured by a special optical system. This area is under active investigation because the following optical constraints demand these approaches. Normally, an image captured by a usual camera has parts that are both in focus and out of focus because the distance between the camera and the object in the scene defines the sharpness of the image. The range of distances at which the object appears acceptably sharp is called the "depth of field" (DOF). It is generally known that a lens with a high magnification, such as a macro lens or a telephoto lens, has a narrow DOF, and it is therefore necessary to adjust the focal length to match the placement of the object. When the object is large, however, the entire object does not appear acceptably sharp in an image because some parts are outside the DOF. In our laboratory, we are conducting research to obtain all-in-focus images using a high-speed liquid variable-focus lens.

Vision Architecture is an academic field established with the aim of examining problems concerning realizing systems from the viewpoint of exploring the applications of high-speed vision technology and practical systems that can recognize the real world and respond in real time. To explore new applications in various fields, it is necessary to create new system technologies that will enable the ideal performance. In order to do that, it is necessary to pursue significant performance and functionality improvements by establishing sophisticated relationships between applications, principles, and devices. Based on this design concept, the Vision Architecture focuses on practical research to explore new applications in various fields by using high-speed image sensing that is superior to the human eye. In concrete terms, we are creating new high-speed recognition and sensing systems and developing new applications in fields such as robotics, inspection, visual media, human interfaces, digital archiving, and so on by utilizing VLSI technology, parallel processing, image recognition, and instrumentation engineering.

A device that realizes general-purpose, fast image processing in a single chip by integrating a photodetector (PD) and a programmable general-purpose digital processing element (PE) at each pixel of an image sensor. It enables high-speed image processing with high real-time capability, in a small, lightweight, and low-power-consumption form factor. This is due to the fully-parallelized structure, which makes raster scanning unnecessary, and the absence of bottlenecks in data transfer between the imaging device and the image processing unit. The processing architecture often adopts the SIMD (Single Instruction Stream, Multiple Data Stream) architecture and has simple A-to-D converters. In addition to devices with general-purpose processing units, devices specially designed for target tracking have been developed.

High-Speed Image Processing is an image processing technology designed to satisfy the demands concerning sampling rates and delays required to visualize dynamical phenomena or to control robots based on visual servoing. In particular, it is capable of processing 100 to 1000 images per second, which is much faster than usual image processing, which processes images at only 30 fps or less. Usual image processing requires prediction or learning because it has lower bandwidth than a fast-moving object. In contrast, High-Speed Image Processing is assumed to have a high frame rate that ensures sufficient bandwidth, so that processing algorithms becomes simpler, achieving a quick response. Generally, high-speed video is a technology that realizes fast imaging and recording. On the other hand, High-Speed Image Processing realizes fast imaging and image processing. For example, real-time visual feedback requires not high-speed video but high-speed image processing with a high frame rate and low latency.

Parallel Image Processing is an image processing technology, which aims at faster processing by parallelization, using dedicated processing architectures to perform the processing, by utilizing the data architecture and the properties of the processing task. Some examples of this technology at the device level include parallelization of processing at the pixel level for calculating feature values, and parallelization of processing at the target level for simultaneous observation of 1000 targets, without scanning, which is serial processing. Some techniques have been introduced to implement Parallel Image Processing, such as SIMD processing utilizing the parallelism at the data level, bit-serial architectures for realizing compact PEs, and reconfigurable architectures for realizing multiple functions. The key factor for Parallel Image Processing is to use these techniques to achieve O(1) processing time for an nxn image.

Scanning involves operations carried out element by element, not in parallel, to obtain, transfer, and display arrayed data. This method has an advantage that a large-scale database can be processed using circuits with low processing and transfer performance. However, this method also suffers from the disadvantage that the processing speed is slow. Generally, an image contains 2D data, and image data is obtained by scanning in rows and columns sequentially. If an image sensor has a global shutter, simultaneity of image acquisition is ensured; however, if an image sensor does not have a global shutter, the timing of the data acquisition depends on the timing of the scanning, and the time lag between data acquisition at the top-left and the bottom-right becomes the total scanning time for the whole image (which is equal to the reciprocal of the frame rate). Furthermore, since the acquisition timing of the top-right data is soon after the acquisition timing of the bottom-left data in the previous frame, we should take care in the image processing to account for this short time. For example, the acquired images will be different when the sensor detects objects that move in the same direction as the scanning direction and objects that move in the opposite direction.

Column Parallel is a parallel processing structure in which a PE is connected and assigned to one column for image data in a 2D array (we call the crosswise direction a row, and the lengthwise direction a column). This structure falls somewhere between a completely parallel processing structure in which a PE is connected to each pixel and a CPU processing structure in which the whole image is scanned and processed by a single computing unit. Although this structure's processing speed is slower than the completely parallel structure because of the column scanning, this structure is capable of processing on the order of milliseconds. Additionally, some types of processor reduce the number of data transmission wires and A/D converters by reducing the number of wires for columns and performing partial scanning. The Intelligent Vision System developed by our laboratory and put into practical use by Hamamatsu Photonics has this type of structure.

A Bit Serial Architecture is one of the processing architectures used in a general-purpose processor. This architecture can perform operations on n bits by using a 1-bit ALU (Arithmetic and Logic Unit) n times sequentially. This kind of architecture was used in the early days of electronic computers, which consisted of a small number of vacuum tubes. These days, the Bit Parallel Architecture, which uses an ALU capable of processing 32 or 64 bits in parallel, is normally used in processors. The Bit Serial Architecture has been introduced into vision chips, which have a limited number of transistors that can be used for each pixel, and also in completely parallel image processing. The processing circuit required for n bits is the same as that required for 1 bit, even though the processing time for n operations is n-times longer than the processing time for 1 bit. High-speed image processing is achieved with a processing time of about 1 ms, which allows some margin for dealing with an increase in processing time.

To speed up processing, the optimal composition of processor circuits varies depending on the algorithm. A Reconfigurable Architecture is an architecture that can reconstruct the optimum circuit configuration in every task for each individual algorithm. This architecture has attracted attention as a technology that achieves flexibility and high-performance implementations by specializing the architecture to match the task. Some of our parallel high-speed image-processing architectures employ a reconfigurable architecture to implement various functions on the same hardware.

An SIMD (Single Instruction Stream and Multiple Data Stream) architecture has parallel data paths connected to parallel processing circuits that are controlled by a single instruction. It contains identical processing elements (PEs) which have their own data paths (pixels in the case of image data) arranged in parallel and are controlled by a single instruction supplied to the whole parallel processing circuit. Images are suitable for this architecture because of their high intrinsic homogeneity. Moreover, a pseudo-MIMD structure, especially with conditional branches for each pixel, can be realized, and global feature values can be extracted in a parallel circuit. From the viewpoint of implementation, the PE design is small because the arrayed PEs have the same design , and the design task is simplified compared with the overall circuit size in an FPGA or single-chip implementation.

CMOS imagers construct images using CMOS switches to sequentially scan the outputs of photodetectors for each pixel. In a CCD (charged coupled device) imager, on the other hand, because charge is transferred to turn on the CCDs in parallel with the outputs of the photodetectors for each pixel, a CCD imager provides simultaneous pixel data, unlike a CMOS imager which does not have a global shutter function. In recent years, CMOS imagers having a global shutter have been developed by improving the circuit design, and they are expected to become more widespread in the future. Moreover, a new pixel architecture called "Active Pixel" has been developed, which is expected to improve not only the switching performance but also the imaging performance. The circuit is constructed of several to about 20 transistors, mainly to achieve better imaging performance, such as improved sensitivity. Also, pixel structures constructed from tens to hundreds of transistors are designed to realize various functionalities, such as image processing, by using such structures, called "Smart Pixels". In our laboratory, we developed a kind of smart pixel device called the "Vision Chip", which has a general-purpose PE (Processing Element) at every pixel.

The frame rate refers to the number of moving images displayed per second. The unit is frames per second (fps). In the old NTSC standard (interlaced), black and white images are 30 fps, and color images are 29.97 fps. In the standard European PAL system, the frame rate is 25 fps, based on the frame rate used for film in theaters, for historical reasons. In the defined depending on the resolution and scan system, including 60 fps, 30 fps, 59.94 fps, 29.97 fps, 50 fps, 25 fps, 24 fps, and 23.98 fps. There are two kinds of scanning: progressive scanning and interlace scanning. The former scans from top to bottom in scanning lines, whereas the latter usually scans the image twice, with the scanning lines shifted by one line each time (written as 2:1).

In general, the spatial resolution of an imager is defined by the number of pixels. Imagers that have over a hundred million pixels are being developed today. However, the higher the resolution becomes, the slower the transfer speed because pixel data is generally read out by scanning. Some measures should be taken to alleviate this problem, though there are some limitations. Namely, there is a trade-off between time resolution and spatial resolution, and there is the question of which resolution should be given priority.

The sensitivity of an imager or digital camera is defined in various way. To begin with, sensitivity in measurement means the ratio of output to input levels, but "good sensitivity" may mean the minimum input value for a meaningful output (sensitivity limit). For that reason, different ways of expressing the sensitivity are used in devices designed for measurement and consumer products, such as digital cameras. In the former, the sensitivity is defined by the output voltage divided by the time-integrated value of the surface illuminance, which depends on the light source. In case of a camera, the output characteristics are considered instead of the illumination intensity at the image plane, but considering its use as a camera, the minimum illumination of the field with a standard candle is often used because the obtained image is more meaningful than the output voltage. In this case, the characteristics of the lens system or the light source are also given because the performance depends on them. Also, in the case of a digital camera, "corresponding ISO speed", which describes the speed corresponding to conventional film speed, or the standard output sensitivity, at which the defined brightness of an object comes up to a defined standard value, are used for the convenience of general users.

A global shutter is an electronic shutter function that allows each pixel data of an imager to be obtained simultaneously. This function is provided in CCD (Charge Coupled Device) imagers, but not in common CMOS imagers. In recent years, however, CMOS imagers having this function have been developed. Image data obtained using a global shutter function ensures simultaneity, so that there is no interference between the direction of the target movement and the scanning direction. Classical mechanical shutters do not ensure simultaneity because of their structure.

Structured Light is an active light pattern used in three-dimensional measurement methods. This method uses a camera and a light projector that projects a known pattern that can be easily identified spatially or temporally, and calculates three-dimensional shape information by extracting the three-dimensional positions of points that correspond to the light pattern reflected on the surface of the target object by using a three-dimensional position extraction method (for example, triangulation). Various types of light pattern have been proposed. For example, a grid of points or lines has been used as an easy way to extract the corresponding points, and a random dot pattern, a two-dimensional M-sequence pattern, or a pattern modulated by color information has been used as a complicated way to extract the corresponding points. In addition, light-section methods that use a fan-shaped pattern in combination with scanning, which are also often used in applications where speed is not important but accuracy is, have also been used. In the case of a mirror, it is necessary to identify the corresponding points while taking account of the specularity of a target surface.

Multi-Target Measurement is a technique for estimating the values of measurement targets (e.g., the 3D positions of a point cloud), by dividing an image into a number of regions and simultaneously analyzing the local variations of each divided region. In particular, when dealing with a large number of small targets, it is possible to realize high-speed multi-point measurement as a whole by executing local processes for each object in parallel over the entire image. Specific examples include inspection of small products or particles, bio-imaging of a large number of cells, blood flow analysis, fluid measurement, observation of microorganisms, manipulation of minute or small objects, detection of dust before cleaning the surface of a substrate, particle observation in the atmosphere, motion measurement using a texture, three-dimensional measurement by structured light, and visible light communication using an image sensor. A dedicated processor operating at a rate of 1,000 frames per second has been developed for measuring more than 1,000 objects at the same time.

Three-dimensional measurement is a technique for obtaining the three-dimensional surface shape of a measurement object. It is also called "shape measurement" or "three-dimensional sensing". In general, there are two types of three-dimensional measurement: the contact type, which uses a mechanical probe that touches the surface of the target object to measure the three-dimensional position by kinematics of the arm, and the non-contact type, which mainly uses optical measurement. Two methods are generally used: In one method, the target is scanned by measuring a single point at a time. In the other method, the multiple points are measured from the image at the same time. With an image- based non-contact method, multi-point three-dimensional measurement technology is required, and we have been working on this in our laboratory with the aim of speeding up the process. In conventional systems, stationary target are usually measured. In contrast, we developed a system that is capable of obtaining the shape of a measurement object in real time at a rate of 1 kHz, even for objects which deform and move. Super-fast real-time 3D sensing is expected to be used in the fields of robotics, industrial product inspection, automobiles, and human interfaces.

Template matching is a kind of image processing that detects an object from an input image. It scans the input image using previously saved patterns of the object, and then performs a matching operation at each scanning point to determine a degree of similarity which enables the system to recognize an object. Several methods have been proposed for determining the degree of similarity, such as SSD (Sum of Squared Difference), SAD (Sum of Absolute Difference), and NCC (Normalized Cross-Correlation). In the case of target tracking, we can make the scan area narrow by utilizing high-speed images and decreasing the computational complexity because the scan area can be assumed to be only an area where an object can move to from its position in the previous frame. Moreover, we can realize high-speed template matching by adopting parallel computing for the matching operation.

Feature extraction is an operation for recognizing/understanding an object in an image. It transforms image patterns to a space with different dimensions, called "feature space", and then performs calculations to extract various features. A value given by this operation is called a feature. Various values have been proposed as features for various purposes. Features are classified roughly into two types: local features and global features. A local feature is calculated and extracted from only neighboring pixel data, and a global feature is computed using all pixel data of an image. A key issue for both is to accelerate the operations. In particular, we need to pay special consideration to global features because they need an integral calculation involving all pixel data (e.g., for moment features).

Moment extraction is a process for extracting a moment of an image, which is one of the features commonly used in image processing. We can utilize this moment feature not only for representing geometric information of an object, such as the size (0th moment), position (1st moment / 0th moment), slope (2nd moment), etc., but also for pattern recognition. We have proposed some schemes where moment features are computed by a massively parallel circuit containing a processing element for every pixel, and these features are used for calculating the center of gravity.

Real-Time Fluid / Particle Measurement is a technology for measuring the motion of a fluid or particles mixed in a fluid in real time. In measurement using image processing, one method is Particle Image Velocimetry (PIV), which is a method for obtaining a time-series flow velocity distribution from location information of particles scattered in a fluid. In cases where particle motion cannot be obtained directly due to a low frame rate, a method for identifying the particle number and sizes statistically by using light scattered by illuminated particles has been employed. In contrast to this method, we developed a method called Real-time Moment-based Analysis of Numerous Objects. This method enables fluid measurement and pattern analysis, which are normally executed offline, to be executed in real time and can measure a higher number of particles.

Book scanning is a technology for digitizing and computerizing the information contained in printed books. With the great advances being made in network search technology and digital books, there is a growing need for computerizing printed books. Generally, current technology is based on copiers or flatbed scanners and is inadequate for computerizing an enormous number of books in terms of speed and convenience. We have proposed a new book computerizing system that scans books continuously without users having to laboriously turn the pages one-by-one. We demonstrated the system experimentally. This technology employs a method in which acquired images are transformed into distortion-free images according to the measured 3D shapes of pages while the user rapidly flicks through the book, and can realize non-destructive book scanning without having to place the book face-down as in conventional flatbed scanning.

Gesture recognition technology recognizes human gesture motions. The objective is to control devices by giving each gesture a meaning. Recognition systems are often targeted at only the limbs or fingers. The former requires high speed, whereas the latter requires high precision. Leading applications of this technology are control of TVs (3 m - 5 m), video games and digital signage (1 m - 3 m), computers (0.3 m - 1 m), and portable devices and car navigation systems (10 cm - 30 cm). In these cases, the usual speed of the extremity of the arm is about 50km/h , and the maximum speeds of the wrist and fingers are about 100 km/h and 150 km/h, respectively. For these recognition applications, not just high-speed image processing but also geometry recognition is often needed. The meanings of gestures are recognized by pattern extraction from time-series location information of representative features.

With advancing technologies such as sensors, displays, and actuators, sensory and motor systems with capabilities far beyond those of human beings will become possible, and the relationship between humans and machines is going to change greatly. In our laboratory we coined the term "Meta Perception" to describe technologies that will enable humans to communicate in new ways with each other by actively using such systems possessing capabilities surpassing those of humans. The conventional approach was to develop systems that are matched with human functions. In contrast, if artificial systems possess capabilities beyond ours, we will need to consider how we process information and what kind of information to provide, with full understanding of the capabilities and structure of the human sensory and motor systems. By realizing such systems, humans can start to engage with information that we could not previously perceive or recognize.

The Smart Laser Scanner aims a laser beam scanned to form a particular spatial pattern toward the target object to be measured, and captures the shape and motion of the target by capturing intensity variations of the reflected light with a single photosensor element. The light beam can be made to move in a way that corresponds to the target by applying adaptive motion to the light beam. This system is capable of measuring three degrees of freedom from the direction of the light beam and the intensity of the reflected light and is capable of 3D position measurement, 2D feature tracking of targets, and so on. To correspond to the design of adaptive motion, this system is capable of creating meaningful motions. In particular, with the high-speed performance that the system achieves, the laser beam can be moved in response to the motion of the target object, which can also be applied to interactive interfaces.

When performing optical measurements, Sensing Display is a method of realizing sensing and display functions simultaneously, in order to use illumination and a laser beam for measurement and as a light source through the same optical system as the display system. For instance, this allows images of measured blood vessels to be displayed on the skin surface, to implement measurement of blood vessel images above the skin with an infrared laser beam. This method has the advantage of not causing any position misalignment between sensing and displaying.

Up to now, displays just show information in some form or another to present it to users, and users input their intentions via keyboards and mice, not the display. An interactive display enables them to input their intentions directly, allowing the user and the system to communicate with each other directly. Inputting to the display directly means that the coordinate system of the display corresponds to that of the input device, so that a coordinate transformation carried out in the user's head, which is needed when using keyboards or mice, becomes unnecessary, allowing an intuitive and comfortable interface to be implemented. The latency time between the user's operation and the information display strongly affects the operating sensation. Because of this, in this laboratory we use a high-speed vision system to keep the input latency below a few milliseconds, allowing the implementation of a high-trackability display system that can track high speed motion.

Proprioception is the ability of humans to determine the progress of their own motor commands through their own sensory tract when commands are given by their brain. Proprioception plays an imporant role in recognizing the state of one's own body, as well as recognizing something unusual, the presence of another person, and one's surroundings. Inside our brain, a motor command, as efferent information, is transmitted to the motor system, and at the same time, is copied to the recognition system. This process, called efferent copy, is used for identifying the outside world and the body, as compared with the sensory tract which involves afferent information. In designing human interfaces, how proprioception is implemented is one of the indexes used to evaluate the system, and proprioception plays an important role in self-recognition.

In human interfaces, when we reflect ourselves in a virtual world in some form or another, it is a measure of the realism of the virtual world whether or not we can understand the representation as our own. In reference to texture and dynamics of representation, as well as latency and synchronization of display, ensuring proprioception is critical. It is also related to discrimination between meum et tuum, paradoxically, and a clear definition is too difficult a problem. In general, our selves in the virtual world have some information gaps with those in the real world, and discrimination between meum et tuum is determined by the relation between the degree of those gaps and the processing in the human brain. So, regarding implementation of human interfaces, implementation of an easy self-recognition system is one of the indexes for evaluating the system.

Haptics is an academic field involving processing structures of perception and recognition related to the tactile sense. The tactile receptors on the skin can be regarded as sensors whose measured variables are mechanical contact pressure and a surface map of heat flow and temperature. However, one cannot obtain information about an object only by holding one's hand over it; it is not until one moves one's hand and touches the object that we can get information about the object. The movement of the hands like this is called a touching motion, and it is considered as one typical example of active sensing, as a manifestation of motion for recognition. That is, we need to consider that a receptor on the skin and the touching motion are united in tactile perception. A manifestation of the touching motion is required for several functions, such as a preparation act for recognition, compensation of the dynamic behavior of sensors, evasion of locality, improvement of spatial resolution, and recognition of surface texture.

Popular image displays are presented on a flat plane, and interactions are bounded by the two dimensions in this plane. On the other hand, by introducing a freely deformable structure to display an environment on which images are presented, it becomes possible to offer a totally new interface, that is, a three-dimensional user interface. Concretely, besides conventional ordinary multi-touch control, new controls such as push and deformation become possible, and also it becomes possible to present an image on a curved surface, which is generated depending on the user's interaction with the display. This will lead to various next-generation interactive digital media environments.