Sensors are used for estimating the state of the autonomous system as well as its operating environment. As shown in the chapter on intelligent control depending on particular architecture sensor data processing might vary from architecture to architecture, but all of them implement the simple Sense-Decide-Act cycle. In general, sensors provide information measuring the same phenomena as sensors of biological systems – light, sound, physical orientation, muscle stretch, etc… In terms of information acquired by the sensor, they might be grouped into internal and external sensors, where internal provide data on the state of the autonomous system itself, while external sensors provide data on the operating environment of the autonomous system. The most commonly used sensor types are discussed below ^[1].

This group of sensors uses ultrasound waves and their feature of reflection from objects/obstacles. Knowing the sound wave propagation time, it is possible to calculate the distance to the first obstacle on the wave’s route. To do it, it is necessary to measure the time between emission and receiving time moments of the sound impulse. Therefore, the senor in its essence is a do-called time of flight sensors or simply ToF.

Sonar sensors differ mainly by the wavelength of the impulse. Depending on particular configurations changes to distance and wave propagation speed of the impulse. It must be emphasized, that speed of sound is different in different environments (in terms of density), at different altitudes, and at different temperatures. Usually, the time difference is measured by the onboard processing unit, which is more complex environments that enable temperature and motion compensation of the sensor itself. These sensors are used as simple contactless bumping sensors or in more complex scenarios as “sound radars” enabling reveal high dualization of robot environment especially in high-density environments like water in underwater applications.

Lidars (Light detection and ranging) sensors are very widely used in autonomous systems. In the same way, as sonars, Lidars exploit time differences. However, they might use other measuring techniques as well. Therefore, several types of Lidars sensors might be used in autonomous systems:

Pulse Lidars use the time of flight principle in the same way as sonars do. Knowing the speed of light gives enough information to calculate the distance from the object hit by the laser ray. Another mechanism used in scanning lasers is a rotating prism, which enables to control of the angle of the emitted laser pulse. Thereby both angle and distance might be estimated, which provides data to calculate the relative position of the object hit by the laser ray.

Continuous-wave amplitude Modulated (CWAM) Lidars exploit phase shift of continuous intensity-modulated laser signal. In this case, the phase shift provides in its essence the same information difference of time when the actual phase has been emitted and observed.

Continuous-wave frequency modulated (CWFM) Lidars mixes emitted and reflected signals using the principle of heterodyne via heterodyning (a method of mixing two frequencies). Using frequency shifts it is possible to estimate object motion speed and direction.

Other types of Lidars are derivates of the mentioned ones

Since the laser ray is very compact the sensing resolution is much higher than sonar sensors could provide. Another advantage is a relative energy efficiency enabling the use of Lidars even to scan the object at significant distances. Currently, the market provides single beam Lidars, 2D/3D scanning Lidars. Currently, even 4D Lidars are in development to provide object motion data along with simple distance. This feature would allow capturing a very important piece of missing information, especially in the autonomous car domain.

Radars use radio signals and their features to estimate the distance to the object, its speed, and direction of motion. Mainly two types of radars are used in autonomous systems – pulses radars and frequency modulation radars.

Pulsed radars in the same way as sonars or pulse Lidars, pulse radars use the time difference of emitted and received signal pulses enabling them to estimate the distance to the object detected.

Frequency modulated Continuous wave (FMCW) radars use frequency modulated signal, which might vary from 30 GHz to 300 GHz. The emitted signal is mixed with the received signal to produce a so-called intermediate frequency signal of IF. IF signal is used to estimate object range, speed, and direction. Dedicated high-resolution FMCW radars are used to receive radar images enabling not only to detect but also to recognize the objects detected. Sometimes these radars are called broad-band radars or imaging radars. Currently mainly broad-band radars are used in combination with multiple receiving antennas enabling effective operation with different frequencies.

Digital cameras, like web cameras, are used to visual information about the surrounding environment. It might be a simple solution as using a cell phone and as complex as using stereo vision systems of time-synchronized cameras. Frame-based digital cameras are composed of lens and sensor matrix, where each element is called a pixel – a photo-sensitive semiconductor element.

Single-camera solution uses a single digital camera to obtain a series of frames, which enable to recognize an object in each frame, compare their position relative to the autonomous system, and thus enables to estimate object relative speeds and displacements throughout the series of the frames. This is the most simples and the most imprecise solution due to the imperfection of cameras, limited frames per second, sensitivity of the given sensor, and other parameters.

Stereo vision systems are using two horizontally aligned cameras, which are time-synchronized (frames are taken simultaneously). Time synchronization minimizes the difference between frames. Horizontal alignment allows observing a distant object from a slightly different angle, which creates a slightly different frame. These differences – binocular disparity - allow us to calculate point location in a 3D environment like the human brain does working with natural vision sensors – eyes. Acquisition of data of the third dimension requires additional calculations and inevitably additional computing power on-board.

Unfortunately, the mentioned systems suffer from several significant disadvantages:

• Motion blur – caused by motion and sensor sensitivity. The less sensitivity the higher blur effects might be observed. Blurred images decrease object detection and distance estimation precision;
• Lens distortion – distorts images in an unpredictable way as a result of imperfection of manufacturing;
• Frames per second – fewer frames per second, less accurate the derived estimates will be;
• Changes of light condition from one frame to another, which complicates the overall processing. One of the obvious results is changes in colors, which reduces the usability of the frames detected.

Event-based cameras allow avoiding all of the mentioned disadvantages at a cost of more complicated data processing. The essence of the working principle is similar to the natural light-sensing retina in eyes of biological systems, where only differences of light intensity are submitted instead of the whole frame. Thus, motion blur as a phenomenon, and the related unwanted phenomena are lost. Therefore, the cameras might be an excellent option for visual autonomous system pose-estimation applications. Unfortunately, there is a price–algorithmic complexity of data interpretation.

Image is taken from an experimental agriculture robot that uses multiple time-synchronized sensors – Single RGB camera (upper-left), Event-based camera (upper-right – reds pixels with increasing intensity, blue ones with decreasing), Lidar (lower-left), and stereo vision camera (lower-left). Image and video produced in ERDF 1.1.1.2 “Post-doctoral Research Aid”, project num. FLPP Lzp-2018/1-0482.

IMUs are the core of the modern autonomous system's internal sensors, which use different electronic devices to produce data of robot accelerating forces towards 3 axes as well as angular accelerations and angular positions. To do so, IMUs use 3D accelerometers, 3D gyroscopes, and sometimes magnetometers. Today IMUs are exploiting different technical solutions, where the most affordable are MEMS (Micro-Electro-Mechanical Mechanical System) systems. MEMS gyroscopes use lithographically constructed versions of one or more of the vibrating mechanisms i.e. tuning forks, vibrating wheels, or resonant solids of various designs ^[2]. This design uses the Coriolis effect – a vibrating body tends to maintain its vibration plane even if its supporting body plane changes (the autonomous system has moved). As a result, some forces are created for sensor bases, which are measured to determine the rotation rate. Currently, the most precise gyroscopic sensor available is fiber-optic gyroscope (FOG), which exploits the Sagnac effect ^[3], thus performing as a mechanical gyroscope. The basic principle is the use of two laser beams injected into a fiber optical channel with a significant length (5km). Due to the Sagnac effect if the sensor is rotating one of the beans experiences a slightly shorter path, which results in a phase shift. The phase shift is measured using the interferometry method, which results in an angular velocity estimate. Despite various measuring methods, IMUs suffer from inherent problem–error accumulation, which provides a systematic error to pose estimation of the autonomous system. Therefore, usually in the outdoor autonomous system, additional sensors like GNSS (Global Navigation Satellite System) are used as an additional source of position information to mitigate the accumulation and keep within reasonable limits.

Rotary encoders are widely used in ground systems, providing an additional relatively precise and reliable source of displacement estimate. The main purpose of the sensor is to provide output data on wheel or shaft angular displacement. There are two main types of rotary encoders – absolute and incremental ^[4]. As all sensors, rotary sensors have several main technologies:

• Mechanical – in its essence they are potentiometers, enabling to encode the full or several full rotations as a continuous output signal. Due to the building principle used the sensor’s main disadvantage is wearing out due to internal friction;
• Optical – it uses opto-pair to detect a reflected signal from the rotating disk (mounted on the shaft) or a light going through the disk through dedicated gaps, thus providing a series of impulses while the shaft is rotating;
• On- and Off-axis magnetic – these types of senor use stationary or rotary magnets and exploit the hall effect to sense changes in a magnetic field.

Using one of the designs absolute sensors allow to encode every single rotation angle with a unique code, while incremental produce a series of impulses. In the case of incremental encoding usually a quadrature A_B phase shifts are used to determine both direction and displacement.

In autonomous systems, as discussed previously, sensors are used to acquire important data about the system itself and its surrounding environment, which is needed to make proper decisions. One of the fundamental needs in autonomous systems is keeping track of the system in a given environment. If the environment’s map is known then the task is to find distinctive features of the environment (corners, monuments, crossings, etc…), and knowing their relative position to the autonomous vehicle it is possible to locate the vehicle on the map using a technique – triangulation. However, in reality, a predefined reliable map is very rarely available. Therefore, the map has to be constructed during the exploration of the environment, i.e. the vehicle simultaneously constructs the map finds its position on the map. This process is known as Simultaneous Localization And Mapping (SLAM). Depending on sensors used to acquired data about the environment as well as depending on computational resources, there is rather a rich palette of the algorithms available. Most of them try to employ a kind of data approximation to tackle the problem, which is clearly a – chicken and egg problem (what is the first map or location on the map?). The overall process might be split into several steps:

• Unordered List ItemSensing the environment before any action is executed. This helps to acquire the first data and probably find there some distinctive features like corners in office or crossings in an open traffic application;
• Execution of motion which provides motion data from IMU, rotary encoders, or other internal sensors, that provide data on the actual motion of the system. One might imagine this step as moving for short enough time with closed eyes;
• Location calculation and map update this step is the most complicated since it combines the map data acquired before, with sensed motion data and uses this data to update the map.

Implementation of the last step is the essence of all SLAM algorithms. Since the sensors provide data with some error due to imperfection of the sensor, the lasts step is done based on some kind of posterior probability estimation of the vehicle pose and for the parameters of the map. Algorithms differ in their way to use some statistical estimates or non-statistical estimates. Other pose approximation methods achieve improved computational efficiency by using simple bounded-region representations of uncertainty. Using even a single 2D scanning Lidar system autonomous vehicle could produce a map like this one:

Of course, not always the selected algorithm and its implementation provide enough precision, which might end up with maps far from perfect, like this one:

Sometimes significant development time is spent on tuning parameters of the algorithms to get the best possible result for a given application scenario.