Optoelectronic graded neurons for perceiving dynamic motion

May 2023

The tiny visual systems of flying insects have inspired PolyU researchers to develop optoelectronic graded neurons for perceiving dynamic motion, enriching the functions of vision sensors for agile response.

Biological visual systems can effectively perceive motion in a complicated environment with high energy efficiency. Particularly, flying insects have high flicker function frequency (FFF) and could perceive objects with high motion speeds. This nature inspiration leads to advancing machine vision systems with very economical hardware resource. Conventional machine vision system for action recognition typically involves complex artificial neural networks such as “spatial” and “temporal” stream computation architectures.

PolyU research team showed that optoelectronic graded neurons can perform high information transmission rate (>1000 bit/s) and fuse spatial and temporal information at sensory terminals. Significantly, the research finding empowers the functionalities that are unavailable in conventional image sensors.

Bioinspired in-sensor computing

Machine vision systems usually consist of hardware with physically separated image sensors and processing units. However, most sensors can only output “spatial” frames without fusing “temporal” information. Acute motion recognition requires “spatial” and “temporal” stream information to be transferred to and fused in the processing units. Therefore, this bioinspired in-sensor motion perception brings the progress in motion processing, which have been a computational challenge with considerable demands on computational resources.

The PolyU research “Optoelectronic graded neurons for bioinspired in-sensor motion perception” is published on Nature Nanotechnology. The research team has focused on studies on in-sensor computing to process visual information at sensory terminals. In other previous studies, the team demonstrated the contrast enhancement of static images and visual adaptation to different light intensities.

Flying insects such as Drosophila with a tiny vision system can agilely recognize a moving object much faster than human. Specifically, its visual system consists of non-spiking graded neurons (retina-lamina) that have a much higher information transmission rate (R) than the spiking neurons in the human visual system. The tiny vision system of insect greatly decreases the signal transmission distance between the retina (sensor) and brain (computation unit).

Essentially, the graded neurons enable efficient encoding of temporal information at sensory terminals, which reduces the transfer of abundant vision data of fusing spatiotemporal (spatial and temporal) information in a computation unit. This bioinspired agile motion perception leads to the research team to develop artificially optoelectronic graded neurons for in-sensor motion perception.

Highly accurate motion recognition

High accurate motion recognition is essential for machine applications such as for automated vehicles and surveillance systems. The research found that the charge dynamics of shallow trapping centres in MoS₂ phototransistors emulate the characteristics of graded neurons, showing an information transmission rate of 1200 bit s⁻¹ and effectively encoding temporal light information.

By encoding the spatiotemporal information and feeding the compressive images into an artificial neural network, the accuracy of action recognition reaches 99.2%, much higher than the recognition achieved with conventional image sensors (~50%).

The research unleashes challenge in motion processing which demands considerable computational resources. Now, the artificially graded neurons enable direct sensing and encoding of the temporal information. The bioinspired vision sensor array can encode spatiotemporal visual information and display the contour of the trajectory, enable the perception of motion with limited hardware resources.

Getting inspiration from agile motion perception of the insect visual systems, the research brings significant progress in the transmission speed and processing of integrated static and dynamic motion for machine vision systems in an intelligent way.

References:
Nature Nanotechnology 18 (2023). Optoelectronic graded neurons for bioinspired in-sensor motion perception.

Nature Nanotechnology 14, 776-782 (2019). Optoelectronic resistive random access memory for neuromorphic vision sensors.

Nature Electronics 5, 84-91 (2022). Bioinspired in-sensor visual adaptation for accurate perception.