Retinomorphic sensor explained

Retinomorphic sensors are a type of event-driven optical sensor which produce a signal in response to changes in light intensity, rather than to light intensity itself.^[1] This is in contrast to conventional optical sensors such as charge coupled device (CCD) or complementary metal oxide semiconductor (CMOS) based sensors, which output a signal that increases with increasing light intensity. Because they respond to movement only, retinomorphic sensors are hoped to enable faster tracking of moving objects than conventional image sensors, and have potential applications in autonomous vehicles, robotics, and neuromorphic engineering.^{[2] [3] [4] [5]}

Naming and history

The first so-called artificial retina were reported in the late 1980's by Carver Mead and his doctoral students Misha Mahowald, and Tobias Delbrück.^[6] These silicon-based sensors were based on small circuits involving differential amplifiers, capacitors, and resistors. The sensors produced a spike and subsequent decay in output voltage in response to a step-change in illumination intensity. This response is analogous to that of animal retinal cells, which in the 1920's were observed to fire more frequently when the intensity of light was changed than when it was constant.^[7] The name silicon retina has hence been used to describe these sensors.^[8]

The term retinomorphic was first used in a conference paper by Lex Akers in 1990.^[9] The term received wider use by Stanford Professor of Engineering Kwabena Boahen, and has since been applied to a wide range of event-driven sensing strategies.^[10] The word is analogous to neuromorphic, which is applied to hardware elements (such as processors) designed to replicate the way the brain processes information.

Operating principles

There are several retinomorphic sensor designs which yield a similar response. The first designs employed a differential amplifier which compared the input signal from of a conventional sensor (e.g. a phototransistor) to a filtered version of the output, resulting in a gradual decay if the input was constant. Since the 1980's these sensors have evolved into much more complex and robust circuits.^[11]

A more compact design of retinomorphic sensor consists of just a photosensitive capacitor and a resistor in series.^[12] The output voltage of these retinomorphic sensors,

V _{out} , is defined as voltage dropped across the resistor. The photosensitive capacitor is designed to have a capacitance which is a function of incident light intensity. If a constant voltage V _{in} , is applied across this RC circuit it will act as a passive high-pass filter and all voltage will be dropped across the capacitor (i.e. V _{out} =0 ). After a sufficient amount of time, the plates of the capacitor will be fully charged with a charge Q = \pm C _{dark} (V _{in} - V _{out} ) on each plate, where C _{dark} is the capacitance in the dark. Since V _{out} =0 under constant illumination, this can be simplified to Q = \pm C _{dark} V _{in} .

If light is then applied to the capacitor it will change capacitance to a new value:

C _{light} . The charge that the plates can accommodate will therefore change to Q = \pm C _{light} (V _{in} - V _{out} ) , leaving a surplus / deficit of charge on each plate. The excess charge will be forced to leave the plates, flowing either to ground or the input voltage terminal. The rate of charge flow is determined by the resistance of the resistor R , and the capacitance of the capacitor. This charge flow will lead to a non-zero voltage being dropped across the resistor and hence a non-zero V _{out} . After the charge stops flowing the system returns to steady-state, all the voltage is once again dropped across the capacitor, and V _{out} =0 again.For a capacitor to change its capacitance under illumination, the dielectric constant of the insulator between the plates,^[13] or the effective dimensions of the capacitor, must be illumination-dependent. The effective dimensions can be changed by using a bilayer material between the plates, consisting of an insulator and a semiconductor. Under appropriate illumination conditions the semiconductor will increase its conductivity when exposed to light, emulating the process of moving the plates of the capacitor closer together, and therefore increasing capacitance. For this to be possible, the semiconductor must have a low electrical conductivity in the dark, and have an appropriate band gap to enable charge generation under illumination. The device must also allow optical access to the semiconductor, through a transparent plate (e.g. using a transparent conducting oxide).

Applications

See main article: Event camera.

Conventional cameras capture every part of an image, regardless of whether it is relevant to the task. Because every pixel is measured, conventional image sensors are only able to sample the visual field at relatively low frame rates, typically 30 - 240 frames per second. Even in professional high speed cameras used for motion picture, the frame rate is limited to a few 10's of thousands of frames per second for a full resolution image. This limitation could represent a performance bottleneck in the identification of high speed moving objects. This is particularly critical in applications where rapid identification of movement is critical, such as in autonomous vehicles.

By contrast, retinomorphic sensors identify movement by design. This means that they do not have a frame rate and instead are event-driven, responding only when needed. For this reason, retinomorphic sensors are hoped to enable identification of moving objects much more quickly than conventional real-time image analysis strategies. Retinomorphic sensors are therefore hoped to have applications in autonomous vehicles,^{[14] [15]} robotics,^[16] and neuromorphic engineering.^[17]

Theory

Retinomorphic sensor operation can be quantified using similar techniques to simple RC circuits, the only difference being that capacitance is not constant as a function of time in a retinomorphic sensor.^[18] If the input voltage is defined as

V _{in} , the voltage dropped across the resistor as V _{out} , and the voltage dropped across the capacitor as V _{C} , we can use Kirchhoff's Voltage Law to state:V _{in} = V _{C} + V _{out}

Defining the current flowing through the resistor as

I , we can use Ohm's Law to write:V _{in} = V _{C} + IR

From the definition of current, we can then write this in terms of charge,

Q , flowing off the bottom plate:V _{in} = V _{C} + R (dQ)/(dt)

where

t is time. Charge on the capacitor plates is defined by the product of capacitance, C , and the voltage across the capacitor, V _{C} , we can hence say: V _{in} = V _{C} + R (d)/(dt) (CV _{C} )

Because capacitance in retinomorphic sensors is a function of time,

C cannot be taken out of the derivative as a constant. Using the product rule, we get the following general equation of retinomorphic sensor response: V _{in} = V _{C} + R [ C (dV _{C})/(dt) + V _{C} (dC)/(dt) ]

or, in terms of the output voltage:

V _{out} = R [ C (dV _{C})/(dt) + V _{C} (dC)/(dt) ]

Response to a step-change in intensity

While the equation above is valid for any form of

C(t) , it cannot be solved analytically unless the input form of the optical stimulus is known. The simplest form of optical stimulus would be a step function going from zero to some finite optical power density P at a time t = t _{0} . While real-world applications of retinomorphic sensors are unlikely to be accurately described by such events, it is a useful way to understand and benchmark the performance of retinomorphic sensors. In particular, we are primarily concerned with the maximum height of the V _{out} immediately after the light has been turned on.

In this case the capacitance could be described by:

C(t) = \begin{cases} C _{dark} , & for t<t _{0} \ C _{light} , & for t\geq t _{0} \end{cases}

The capacitance under illumination will depend on

P . Semiconductors are known^[19] to have a conductance, G , which increases with a power-law dependence on incident optical power density: G \propto P ^{\gamma} , where \gamma is a dimensionless exponent. Since G is linearly proportional to charge density, and capacitance is linearly proportional to charges on the plates for a given voltage, the capacitance of a retinomorphic sensor also has a power-law dependence on P . The capacitance as a function of time in response to a step function, can therefore be written as: C(t) = \begin{cases} C _{dark} , & for t<t _{0} \ C _{dark} + \alpha _{0} P ^{\gamma} , & for t\geq t _{0} \end{cases}

where

\alpha _{0} is the capacitance prefactor. For a step function we can re-write our differential equation for V _{in} as a difference equation: V _{in} = V _{C} + R [ C (\Delta V _{C})/(\Delta t) + V _{C} (\Delta C)/(\Delta t) ]

where

\Delta V _{C} is the change in voltage dropped across the capacitor as a result of turning on the light, \Delta C is the change in capacitance as a result of turning on the light, and \Delta t is the time taken for the light to turn on. The variables V _{C} and C are defined as the voltage dropped across the capacitor and the capacitance, respectively, immediately after the light has been turned on. I.e. V _{C} is henceforth shorthand for V _{C} (t _{0} ) , and C is henceforth shorthand for C(t _{0} ) . Assuming the sensor has been held in the dark for sufficiently long before the light is turned on, the change in V _{C} can hence be written as: \Delta V _{C} = V _{C} - V _{in}

Similarly, the change in

C can be written as \Delta C = \alpha _{0} P ^{\gamma}

Putting these into the difference equation for

V _{in} :V _{in} = V _{C} + R [ C (V _{C} - V _{in})/(\Delta t) + V _{C} (\alpha _{0} P ^{\gamma})/(\Delta t) ] Multiplying this out:V _{in} \Delta t = V _{C} \Delta t + R [ C(V _{C} - V _{in} ) + V _{C} \alpha _{0} P ^{\gamma} ]

Since we are assuming the light turns on very quickly we can approximate

\Delta t ≈ 0 . This leads to the following:V _{C} = V _{in} (C _{dark} + \alpha _{0} P ^{\gamma})/(C _{dark} +2 \alpha _{0} P ^{\gamma})

Using the relationship

V _{in} = V _{C} + V _{out} , this can then be written in terms of the output voltage:V _{max} = V _{in} (\alpha _{0} P ^{\gamma})/(C _{dark} +2 \alpha _{0} P ^{\gamma})

Where we have defined the peak height as

V _{max} = V _{out} (t _{0} ) , since he peak occurs immediately after the light has been turned on.

The retinomorphic figure of merit,

Λ , is defined as the ratio of the capacitance prefactor and the capacitance of the retinomorphic sensor in the dark:Λ= (\alpha _{0})/(C _{dark})

With this parameter, the inverse ratio of peak height to input voltage can be written as follows:

(V _{in})/(=2+ V _{max}) (1)/(Λ P ^{\gamma})

The value of

\gamma will depend on the nature of recombination in the semiconductor,^[20] but if band-to-band recombination dominates and the charge density of electrons and holes are equal, \gamma =0 . 5 . For systems where this is approximately true^[21] the following simplification to the above equation an be made:(V _{in})/(=2+ V _{max}) (1)/(Λ P ^{1/2})

This equation provides a simple method for evaluating the retinomorphic figure of merit from experimental data. This can be carried out by measuring the peak height,

V _{max} , of a retinomorphic sensor in response to a step change in light intensity from 0 to P , for a range of values P . Plotting V _{in} /V _{max} as a function of P ^{-1/2} should yield a straight line with a gradient of 1 / Λ . This approach assumes that V _{max} is linearly proportional to V _{in} .

See also

• Active-pixel sensor
• Charge coupled device
• Event camera
• Neuromorphic engineering
• Optical sensor
• Photodiode

References

1. Posch. Christoph. Serrano-Gotarredona. Teresa. Linares-Barranco. Bernabe. Delbruck. Tobi. 2014. Retinomorphic Event-Based Vision Sensors: Bioinspired Cameras With Spiking Output. Proceedings of the IEEE. 102. 10. 1470–1484. 10.1109/JPROC.2014.2346153. 11441/102353. 11513955. 1558-2256. free.
2. The Silicon Retina. 2021-12-28. Scientific American. 1991. 10.1038/scientificamerican0591-76. en. Mahowald. Misha A.. Mead. Carver. 264. 5. 76–82. 2052936. 1991SciAm.264e..76M .
3. Book: Gilder, George F.. The Silicon Eye. 2005. W. W. Norton & Company. 978-0-393-05763-8. en.
4. Web site: Hambling. David. AI vision could be improved with sensors that mimic human eyes. 2021-10-28. New Scientist. en-US.
5. Web site: An eye for an AI: Optic device mimics human retina. 2021-10-28. BBC Science Focus Magazine. en.
6. Mead. Carver A.. Mahowald. M. A.. 1988-01-01. A silicon model of early visual processing. Neural Networks. en. 1. 1. 91–97. 10.1016/0893-6080(88)90024-X. 0893-6080.
7. Adrian. E. D.. Matthews. Rachel. 1927. The action of light on the eye. The Journal of Physiology. en. 63. 4. 378–414. 10.1113/jphysiol.1927.sp002410. 1469-7793. 1514941. 16993896.
8. Delbruck. T.. 1993. Silicon retina with correlation-based, velocity-tuned pixels. IEEE Transactions on Neural Networks. 4. 3. 529–541. 10.1109/72.217194. 18267755. 1941-0093.
9. Book: Rao. A.. Akers. L.A.. 1990 IJCNN International Joint Conference on Neural Networks . A retinomorphic VLSI smart sensor for invariant geometric object recognition . 1990. https://ieeexplore.ieee.org/document/5726918. 949–954 vol.2. 10.1109/IJCNN.1990.137961. 35554142 .
10. Book: Boahen, K.. Proceedings of Fifth International Conference on Microelectronics for Neural Networks . Retinomorphic vision systems . 1996. https://ieeexplore.ieee.org/document/493766. 2–14. 10.1109/MNNFS.1996.493766. 0-8186-7373-7. 62609792.
11. Posch. Christoph. Serrano-Gotarredona. Teresa. Linares-Barranco. Bernabe. Delbruck. Tobi. 2014. Retinomorphic Event-Based Vision Sensors: Bioinspired Cameras With Spiking Output. Proceedings of the IEEE. 102. 10. 1470–1484. 10.1109/JPROC.2014.2346153. 11441/102353. 11513955. 1558-2256. free.
12. Trujillo Herrera. Cinthya. Labram. John G.. 2020-12-07. A perovskite retinomorphic sensor. Applied Physics Letters. 117. 23. 233501. 10.1063/5.0030097. 2020ApPhL.117w3501T . 230546095. 0003-6951. free.
13. Dillman. Norman. 1965. Photodielectric effect in semiconductors. PhD Thesis.
14. Web site: IOM3. Optical sensor mimics human eye for self-driving cars. 2021-12-29. www.iom3.org.
15. Web site: Simple Eyelike Sensors Could Make AI Systems More Efficient. 2021-12-29. Inside Science. 8 December 2020 . en.
16. Web site: 2021-01-05. Bigger boost in robot's field of view. 2021-12-29. ZME Science. en-US.
17. Web site: 2021-01-18. Perovskite sensor sees more like the human eye. 2021-12-29. Physics World. en-GB.
18. Trujillo Herrera. Cinthya. Labram. John G. 2021-09-16. Quantifying the performance of perovskite retinomorphic sensors. Journal of Physics D: Applied Physics. 54. 47. 475110. 10.1088/1361-6463/ac1d10. 237541793. 0022-3727.
19. Web site: Photoelectronic Properties of Semiconductors Materials science. 2021-12-28. Cambridge University Press. en.
20. Balberg. I.. 1994-01-15. The two carriers' mobility‐lifetime products and their light intensity dependencies in hydrogenated amorphous silicon. Journal of Applied Physics. 75. 2. 914–923. 10.1063/1.356447. 1994JAP....75..914B . 0021-8979.
21. Levine. Igal. Gupta. Satyajit. Brenner. Thomas M.. Azulay. Doron. Millo. Oded. Hodes. Gary. Cahen. David. Balberg. Isaac. 2016-12-15. Mobility–Lifetime Products in MAPbI3 Films. The Journal of Physical Chemistry Letters. 7. 24. 5219–5226. 10.1021/acs.jpclett.6b02287. 27973905.