The list of book contributors is presented below.

• Rahul Razdan, Ph.D
• Mohsen Malayjerdi, Ph.D

• Roman Czyba, Ph. D., DSc., Eng.
• Piotr Czekalski, Ph. D., Eng.
• Tomasz Grzejszczak, Ph. D., Eng.

• Agris Nikitenko, Ph. D., Eng.
• Karlis Berkolds, M. sc., Eng.
• Larisa Survilo, M. sc., Eng.

• Raivo Sell, Ph. D., ING-PAED IGIP

• Tomasz Siwy, CEO

• Ing. Libor Přeučil, CSc. (Ing. = Master of Engineering, CSc. = Ph. D.)
• Ing. Karel Košnar, Ph.D. (Ing. = Master of Engineering)

• Raivo Sell, Ph. D., ING-PAED IGIP

This content was implemented under the project: SafeAV - Harmonizations of Autonomous Vehicle Safety Validation and Verification for Higher Education.

Project number: 2024-1-EE01-KA220-HED-000245441.

Consortium

• ITT Group, Tallinn, Estonia (Coordinator).
• Silesian University of Technology, Gliwice, Poland,
• Riga Technical University, Riga, Latvia,
• Czech Technical University in Prague, Prague, Czech Republic
• Tallinn University of Technology, Tallinn, Estonia
• Prodron, Gliwice, Poland

Erasmus+ Disclaimer
This project has been funded with support from the European Commission.
This publication reflects the views only of the author, and the Commission cannot be held responsible for any use which may be made of the information contained therein.

Copyright Notice
This content was created by the SafeAV Consortium 2024–2027.
The content is copyrighted and distributed under CC BY-NC Creative Commons Licence and is free for non-commercial use.

[raivo]Please fill in some introduction

The book comprises a comprehensive guide for a variety of education levels. A brief classification of the contents regarding target groups may help in a selective reading of the book and ease finding the correct chapters for the desired education level. To inform a reader about the proposed target group, icons are assigned to the top headers of the chapters. The list of icons and their reflection on the target groups is presented in the table 1.

Table 1: List of icons presenting content classification and corresponding target groups

Icon	Target group
	Bachelor and Engineering level students
	Masters students

[rczyba][✓ rczyba, 2025-09-18]

Autonomous systems are systems capable of operating independently, making decisions and adapting to a changing environment without direct human intervention. This means that they can collect information about the environment, process it and, based on it, perform tasks in accordance with programmed goals, for a longer period of time and without the need for constant supervision. A key feature of autonomous systems is their ability to self-regulate and operate independently of external control. Autonomous systems use sensors (e.g. cameras, radars, ultrasonic sensors) to collect information about the environment. The collected data are processed, and decisions regarding further action are made on their basis. What exactly is autonomy? The autonomy of a system can be defined as its ability to act according to its own goals, norms, internal states, and knowledge, without external human intervention. This means that autonomous systems are not limited to robots or unmanned vehicles, well known as drones. This definition actually includes any automatic functions that can reduce the level of workload or support the person driving the vehicle.

Autonomous systems are those that make decisions and operate without human intervention based on data and predetermined rules. Such solutions use advanced technologies such as artificial intelligence, machine learning, neural networks, Internet of Things, and others to perform tasks independently. Autonomous systems are today's Industry 4.0 and are used in various areas, from robotics, through transport and logistics, to medicine and education. An example would be an autonomous car that makes decisions on its own based on data from sensors, or an autonomous transport vehicle (AGV, or Automated Guided Vehicles) designed to safely and efficiently transport loads in a warehouse, without the need for operator supervision. Another application of autonomous systems are production systems that, based on data from industrial sensors, automatically control production processes, control machines and optimize production. This allows for shortening production times, reducing production costs and increasing product quality. Autonomous systems are also used in transport and logistics, where they enable faster and more efficient delivery of goods. Thanks to the Internet of Things and monitoring systems, every stage of transport can be tracked, from loading to delivery, which allows for better control of the process. Autonomous systems are also used in medicine, where they act as a doctor's advisor and enable faster and more precise diagnosis of diseases, as well as more effective treatment of patients. These systems are also used in education, where they allow the adaptation of the teaching process to the individual needs of students and improvement of the teaching process.

Autonomous systems are becoming an increasingly important part of our lives, and their development and application will have an increasing impact on the future. This study focuses on autonomous control of unmanned ground, aerial, and marine vehicles.

[rczyba][✓ rczyba, 2025-10-22]

Autonomous vehicles, also known as self-driving vehicles, are platforms that can move and make decisions without direct human intervention. They use advanced technologies such as artificial intelligence, sensors, cameras, and navigation systems to analyze their surroundings and make decisions in real time. Autonomy of mobile vehicles means the ability to independently determine their own control signals. In the case of mobile platforms, this primarily involves the ability to plan a route to a destination. In order to make these decisions as correct and optimal as possible, it is crucial to have a good understanding of the environment surrounding the vehicle, which in practice is often unknown and unpredictable. Autonomous vehicles have the potential to increase road safety, reduce traffic jams, and improve the availability of transport for people who cannot drive. In recent years, many companies, including Tesla, Waymo, and Uber, have been investing in the development of these technologies, leading to intensive research and testing on public roads. As technology advances, autonomous vehicles may become more and more common in our daily lives.

Unmanned vehicles, both ground, aerial, water and underwater, are versatile structures used in various fields, from monitoring areas to search and rescue operations (SAR). Thanks to the development of a wide range of high-precision sensors and algorithms for estimating movement parameters, unmanned vehicles have the ability to localize, detect objects and avoid obstacles. New technologies are gaining importance, especially in the context of the development of vehicle autonomy. Autonomous navigation of unmanned vehicles is proving to be a significant challenge. This is a complex and dynamically developing area of technology that has the potential to revolutionize many industries and applications.

The growing importance of autonomy in various sectors is a clear and visible phenomenon. This is not surprising considering the advantages of these systems. Platforms with a high level of autonomy can operate in environments with poor or limited communication capabilities, significantly reducing their dependence on remote control stations. This capability, along with a greater operational range, is one of the key attributes for mission execution. As autonomy levels increase, there will be a shift from the classic individual remote control operational model to the swarm concept, in which a single pilot/operator controls more than one vehicle simultaneously.

Global transport has been undergoing dynamic transformation in recent years. Key trends shaping it include mobility, multimodality, sharing, ecology, and autonomy. The aging population and the need for multitasking will also have a significant impact on transport in the coming years, providing additional impetus for the development of autonomous means of transport. Autonomous transport is no longer a futuristic vision, but a reality that surrounds us. Automated metro systems, shuttle buses, and drones are all examples of the use of autonomous devices to transport people and goods. For now, these are niche solutions, but with further technological development, we can expect the autonomy of other modes of transport, including the most popular road passenger transport. The acceleration in the development of autonomous transport systems is linked to the rapid development of machine learning techniques in recent years. Increased computing power, combined with increasingly affordable cameras and sensors, has enabled the emergence of self-learning artificial intelligence systems capable of analyzing road conditions and making decisions, including those related to vehicle control. As technology advances, investments in autonomous vehicles are increasing. Although the largest companies have yet to create a finished product or generate a profit, investors are hopeful of an autonomous road revolution in the near future. Level 2 vehicles (advanced driver assistance systems) are already available for sale, while work on fully autonomous vehicles is underway. In November 2019, the first driverless taxi service was launched in Phoenix, USA (in a limited area and for a limited number of customers). In Poland, the most intensive progress in autonomous transport involves drones. In 2018, the Central European Drone Demonstrator was established in the Upper Silesian-Zagłębie Metropolis, an initiative for the development of urban drone transport, including autonomous transport.

Autonomous transport relies on sensors that analyze the surroundings. Sensors in autonomous vehicles constitute a large percentage of the device's production cost, making these vehicles expensive. Vehicle sensors also generate a huge data stream, the analysis of which requires significant computing power, thus increasing energy consumption. Autonomous vehicle traffic relies on continuous vehicle-to-vehicle and vehicle-to-infrastructure communication. Recent years have been characterized by intense competition between two vehicle-to-vehicle communication systems: ITS-G5 (based on the Wi-Fi standard) and C-V2X (based on 4G or 5G mobile networks). The development of autonomous road transport will necessitate equipping road infrastructure with appropriate sensors to coordinate vehicle traffic. The development of 5G technology offers a significant opportunity to improve V2I (Vehicle to Infrastructure) technology. In the field of drones, miniaturized ADS-B aviation technology, successfully utilizing LTE, has proven highly effective. A major challenge related to the development of autonomous transport will be the emphasis on defending against cyber threats. The possibility of remotely taking control of a vehicle, data theft, cyberespionage, and viruses that confuse sensors – these are just some of the threats modern society must face. Another challenge will be creating appropriate regulations that will regulate liability for potential damages, as well as the rules for sharing transport infrastructure between different types of vehicles. Such regulations must respond to public expectations and concerns. Adapting the infrastructure to the needs of autonomous vehicles will be a challenge and a cost, including equipping it with appropriate sensors and perhaps even redesigning it more broadly. While it is not yet clear what scope of transport will be automated, the gradual introduction of autonomous solutions to road, rail, maritime, and air transport is certain.

Due to the specific requirements of the operational environment, autonomous land, air and sea vehicles will be discussed separately in the following subsections.

Over the past two decades, the rapid evolution of digital technologies has transformed the design, deployment, and operation of autonomous systems. The advancements in artificial intelligence (AI), robotics, and advanced sensors have driven the emergence of intelligent platforms, which, depending on their application domain and specifics, are capable of operating with limited or no human intervention. This transformation spans across ground, aerial, and marine environments — each presenting distinct challenges yet sharing a common architectural foundation centred on perception, decision-making, and control ^[1,2]. Currently, systems are no longer perceived and designed as isolated machines but integral parts of a broader digital ecosystem involving cloud computing, edge processing, and distributed intelligence ^[3]. The ongoing Industry 4.0 and emerging Industry 5.0 initiatives emphasise the fusion of human–machine collaboration, sustainability, and adaptability, all of which depend heavily on robust and modular system architectures. A key enabler of this transformation is the system’s architecture — the structured framework that defines how system components interact, communicate, and evolve. In autonomous systems, architecture governs how sensor data is interpreted, how decisions are made in uncertain environments, and how control actions are executed safely and reliably. For instance, in self-driving cars, architectural layers coordinate LiDAR, camera, and radar inputs to produce real-time navigation decisions; in drones, they manage flight stability and mission autonomy; and in underwater robots, they handle communication delays and localisation challenges ^[4,5]. Further, the architectures of the autonomous systems and the related topics are discussed in the following order:

Software architecture represents the high-level structure of a system, outlining the organisation of its components, their relationships, and the guiding principles governing their design and evolution. In autonomous systems, architecture defines how perception, planning, and control modules interact to achieve autonomy while maintaining safety, reliability, and performance ^[6]. The primary purpose of an autonomous system’s software architecture is to ensure:

• Scalability: The ability to integrate new sensors, algorithms, or mission modules.
• Interoperability: Compatibility with other systems and communication protocols.
• Maintainability: Ease of updating or modifying individual modules.
• Safety and fault tolerance: Robustness against sensor failure, communication loss, or software bugs.
• Real-time responsiveness: Capability to process environmental data and respond within strict temporal limits.

Architectural design in autonomous systems typically follows several universal principles:

1. Modularity: Systems are divided into well-defined modules (e.g., perception, localisation, path planning, control), allowing independent development and testing. Enables functional isolation, expandability and higher maintainability of the given system.
2. Abstraction: Functional details are hidden behind interfaces, promoting flexibility and reuse. In its essence, the higher the abstraction, the easier system development, testing and applications. It also reduces design complexity at every abstraction layer.
3. Layering: Tasks are grouped by level of abstraction — for instance, hardware interfaces at the lowest level and mission planning at the highest. Besides the functional abstraction, layering enables different technical implementations at different levels, which is needed to address different reaction speeds, reduce decision-making delays and more effective internal communications and data processing at different levels.
4. Standardisation: Adoption of middleware standards (e.g., ROS, DDS, MOOS) facilitates interoperability across platforms. Besides, the mentioned standardisation enables avoiding vendor locks and reduces overall costs due to increased competition.
5. Data-centric communication: Modern architectures rely on publish/subscribe paradigms to manage distributed communication efficiently.

Middleware serves as the backbone that connects diverse modules, ensuring efficient data exchange and synchronisation. Prominent middleware systems in autonomous vehicles include:

• ROS (Robot Operating System): An open-source framework providing a modular structure for robotic applications, including perception, planning, and control ^[7].
• DDS (Data Distribution Service): A real-time communication standard widely used in aerospace and defence systems, supporting deterministic data exchange ^[8].
• MOOS-IvP: A marine-oriented autonomy framework designed for mission planning and vehicle coordination in autonomous underwater and surface vehicles ^[9].
• AUTOSAR Adaptive Platform: A standard architecture for automotive systems emphasising safety, reliability, and scalability ^[10].

These middleware platforms not only promote interoperability but also enforce architectural patterns that ensure predictable performance across heterogeneous domains.

Most autonomous systems follow a hierarchical layered architecture:

Depending on functional tasks system’s architecture is split into multiple layers to abstract functionality and technical implementation as discussed above. Below is a schema of a generic architecture to get a better understanding of typical tasks at different layers.

Modern autonomous systems increasingly integrate machine learning (ML) techniques for perception and decision-making. Deep neural networks enable real-time object detection, semantic segmentation, and trajectory prediction ^[11]. However, these data-driven methods also introduce architectural challenges:

• Increased computational load requiring edge GPUs or dedicated AI accelerators.
• The need for robust validation and explainability to ensure safety.
• Integration with deterministic control modules in hybrid architectures.

Thus, many systems adopt hybrid designs, combining traditional rule-based or dynamics-based control with data-driven inference modules, balancing interpretability and adaptability

Autonomous systems, regardless of their physical domain — ground, aerial, or marine — share a common architectural logic that structures how data flows from sensors to decision-making and control units. This section explores the typical functional architecture and the operational pipeline that enable autonomous behaviour.

The foundation of most autonomous systems lies in the Sense–Plan–Act (SPA) paradigm, introduced in early robotics and still relevant today. It slightly extends a general controllable system’s architecture by introducing explicit steps of sensing, decision making and acting. This model organises operations into three distinct but interdependent stages:

1. Sense: Gather and interpret data from sensors to build an internal representation of the environment.
2. Plan: Use that representation to decide on goals, trajectories, or actions.
3. Act: Execute those actions through control and actuation mechanisms.

This conceptual model remains universal across domains and can be represented as follows.

The SPA cycle forms the basis for layered architectures in autonomous vehicles, where each layer refines or extends one part of this process. In practice, modern architectures also include learning and communication layers that enhance adaptability and collaboration between agents ^[13].

A simplified data-flow architecture illustrates how perception data transitions through layers to produce control actions via extending the SPA architecture to a more applicable one :

This closed-loop interaction ensures that systems continuously update their understanding of the world, adjusting behaviour based on environmental changes or feedback from control execution ^[14].

Architectural organisation also depends on whether processing is centralised or distributed:

• Centralised Architectures:
  • All decision-making occurs on a single computing unit.
  • Easier synchronisation but can become a bottleneck.
  • Common in small-scale robots and drones.
• Distributed Architectures:
  • Tasks are distributed among multiple computing nodes or agents.
  • Enhanced scalability and fault tolerance.
  • Essential for multi-vehicle coordination (swarm robotics, fleet operations).

Middleware frameworks like ROS 2 and DDS are inherently designed to support distributed computation, enabling decentralised data exchange in real time ^[15]).

Safety is a critical design consideration. Redundant architectures replicate essential components (e.g., dual sensors, parallel computing paths) to ensure operation even during failures. For example, aircraft autopilot systems employ triple-redundant processors and cross-monitoring logic ^[16]. Similarly, marine vehicles use redundant navigation sensors to counter GPS outages caused by water interference. Architectural safety mechanisms include:

• Failover controllers
• Health monitoring nodes
• Watchdog timers
• Self-diagnostics and logging subsystems

These ensure resilience, especially in mission-critical or human-in-the-loop systems.

Reference architectures serve as standardised templates guiding the design of specific systems. They establish a common vocabulary, promote interoperability, and enable systematic validation.

Robot Operating System (ROS) provides a modular, publish–subscribe communication infrastructure widely adopted in research and industry. Its successor, ROS 2, adds real-time capabilities, security features, and DDS-based communication, making it suitable for production-grade autonomous systems ^[17]).

The ROS 2 architecture provides several advantages, including component-level independence from the provider, modularity enabling easier development as well as large community and libraries of packages for deployment. ROS 2 is now the backbone for major open-source projects such as Autoware.AI (autonomous driving) and PX4-Autopilot (UAV control).

In the automotive sector, the AUTOSAR (AUTomotive Open System ARchitecture) standard defines a scalable, service-oriented architecture supporting high-performance applications such as automated driving ^[18].

The AUTOSAR Adaptive Platform is conceptually a middleware. AUTOSAR Adaptive Platform provides services to Adaptive Applications beyond those available from the underlying operating system, drivers, and extensions. One of the distinctive features of the AUTOSTAR is its Real-time and safety-critical compliance (ISO 26262) as well as use of SOME/IP and DDS for service-based communication purposes. AUTOSAR is widely implemented in production autonomous vehicles from manufacturers like BMW, Volkswagen, and Toyota ^[19].

The JAUS (Joint Architecture for Unmanned Systems) is a U.S. Department of Defense standard (SAE AS5669A) defining a message-based, modular architecture for interoperability among unmanned systems ^[20]. It is domain-agnostic, supporting aerial, ground, and marine vehicles. JAUS defines:

• A component-based hierarchy (Subsystem → Node → Component → Service)
• Standardised message sets for communication
• Cross-domain interoperability

JAUS remains influential in defence and research projects where multiple unmanned vehicles must coordinate under a unified framework. Due to its straightforward and easy-to-implement architecture, it has been adopted by different systems and domains

MOOS (Mission Oriented Operating Suite) combined with IvP (Interval Programming) forms a robust architecture for marine autonomy, developed at MIT and used in NATO and U.S. Navy programs ^[21].

IvP (Interval Programming) Helm provides decision-making capabilities based on models provided by the developers, while MOOS DB provides access to data and decisions collected by Applications. Client applications (or just Applications are the main functionality, and modularity drives the ability to integrate different functions, logically isolating them and providing access through a common communications space. The communications are enabled using pMOOSBridge middleware. All of the mentioned allow an asynchronous behaviour-based response to a changing environment and higher flexibility, easier maintainability and to some extent future-proof solutions.

Recent trends combine multiple reference architectures to exploit their strengths. For example:

• ROS–AUTOSAR bridges enable integration between experimental and production-grade systems.
• DDS–MQTT hybrids connect real-time robotics with cloud-based IoT analytics.
• ROS–MOOS integrations allow cross-domain cooperation between underwater and surface robots ^[22].

Such hybridization reflects the growing need for flexibility and cross-domain interoperability in modern autonomous systems.

Autonomous systems operate across diverse environments that impose unique constraints on perception, communication, control, and safety. While all share a foundation in modular, layered architectures, the operational domain strongly influences how these layers are implemented ^[23,24]. Some of the most important challenges and differences are listed in the following table:

Aerial autonomous systems include Unmanned Aerial Vehicles (UAVs), drones, and autonomous aircraft. Their software architectures must ensure flight stability, real-time control, and safety compliance while supporting mission-level autonomy ^[25]. UAV architectures are often tightly coupled with flight control hardware, leading to a split architecture:

1. Onboard system (real-time control and perception)
2. Ground control system (mission management, supervision)

Some of the most popular architectures:

PX4 Autopilot An open-source flight control stack supporting multirotors, fixed-wing, and VTOL aircraft. The PX4 architecture is divided into Flight Stack (estimation, control) and Middleware Layer (uORB) for data communication ^[26]). The technical implementation of the architecture ensures compatibility with MAVLink communication and ROS 2 integration, making it a very popular and widely used solution.

ArduPilot In comparison, the ArduPilot is a Modular architecture with layers for HAL (Hardware Abstraction Layer), Vehicle-Specific Code, and Mission Control. The technical implementation are widely used by the community and used in research and commercial UAVs for mapping, surveillance, and logistics ^[27].

Still, some challenges remain:

• Safety and Redundancy: Flight-critical functions must survive component failures.
• Communication Constraints: Limited bandwidth and intermittent connectivity.
• Energy Efficiency: Trade-offs between payload weight and computational power.
• Airspace Regulation: Compliance with UAV Traffic Management (UTM) systems ^[28].

Ground autonomous systems encompass self-driving cars, unmanned ground vehicles (UGVs), and delivery robots. Their architectures must manage complex interactions with dynamic environments, multi-sensor fusion, and strict safety requirements ^[29]. A ground vehicle’s software stack integrates high-level decision-making with low-level vehicle dynamics, ensuring compliance with ISO 26262 functional safety standards ^[30]. One of the reference architectures used is Autoware.AI (and its successor Autoware.Auto), which is an open-source reference architecture for autonomous driving built on ROS/ROS 2. It implements all functional modules required for L4 autonomy, including:

• Perception (object recognition, segmentation)
• Planning (route, behaviour, trajectory)
• Control (PID, MPC)
• Simulation and visualisation tools

Autoware emphasises modularity, allowing integration with hardware-in-the-loop (HIL) simulators and real vehicle platforms ^[31]). Currently, the automotive industry is using several standards to foster the development and practical implementations of future autonomous ground transport systems:

• AUTOSAR Adaptive Platform: Provides safety-certified, service-oriented design.
• ISO 26262: Functional safety standard ensuring risk assessment and hazard analysis.
• SAE J3016: Defines levels of driving automation (0–5).
• OpenDrive / OpenScenario: Data models for simulation and testing.

Due to the environmental complexity, in the autonomous ground vehicles domain, the following main challenges still remain:

• Sensor Fusion Complexity: Handling heterogeneous sensor data in urban environments.
• Uncertainty and Prediction: Managing unpredictable behaviours of pedestrians and other vehicles.
• Computation Load: Real-time inference on limited onboard computing resources.
• V2X Communication: Integration with smart infrastructure and other vehicles.

Marine autonomous vehicles operate in harsh, unpredictable environments characterised by communication latency, limited GPS access, and energy constraints. They include AUVs (Autonomous Underwater Vehicles), ASVs (Autonomous Surface Vehicles) and ROVs (Remotely Operated Vehicles). These vehicles rely heavily on acoustic communication and inertial navigation, requiring architectures that can operate autonomously for long durations without human intervention (^[32].

The reference architecture is based on the MOOS (Mission-Oriented Operating Suite) IvP architecture discussed previously. It provides interprocess communication and logging, while IvP Helm enables a decision-making engine using behaviour-based optimisation via IvP functions. The architecture supports distributed coordination (multi-vehicle missions) and robust low-bandwidth communication (^[32]. The architecture is extensively used in NATO CMRE and MIT Marine Robotics research ^[33].

While the overall trend is to take advantage of modularity, abstraction and reuse, the are significant differences among the application domains.

An important trend in recent years is the convergence of architectures across domains. Unified software platforms (e.g., ROS 2, DDS) now allow interoperability between aerial, ground, and marine systems, enabling multi-domain missions such as coordinated search-and-rescue (SAR) operations. The integration of AI, edge computing, and cloud-based digital twins has blurred domain boundaries, giving rise to heterogeneous fleets of autonomous agents working collaboratively. Aerial systems look after stability, lightweight real-time control, and airspace compliance; open stacks like PX4/ArduPilot show how flight-critical loops coexist with higher-level autonomy. Ground systems exploit dense, dynamic scenes, heavy sensor fusion, and functional safety; stacks like Autoware illustrate a full L4 pipeline from localisation to MPC-based control. Marine systems suffer from low-bandwidth communications, GPS-denied navigation, and long-endurance missions; MOOS-IvP’s shared-database and behaviour-arbitration approach fits these realities. Summarising, a successful autonomy is based on sound software architecture instead of any particular single algorithm. The developed frameworks provide practical blueprints that can be adapted, mixed, and extended to meet mission demands across air, land, and sea.

[rczyba][✓ rczyba, 2025-10-20]

Autonomous technologies and robotics are redefining possibilities, improving efficiency and safety across sectors. Advanced applications such as self-driving vehicles, crop and harvesting robots rely on precise GNSS/GPS positioning and require centimeter-level accuracy to function properly. As the domain of autonomous applications expands, the ability to use reliable, real-time GNSS/GPS correction services becomes not only useful, but essential. Mapping and localization algorithms, as well as sensor models, enable vehicle orientation even in unfamiliar environments. Route planning and optimization algorithms, as well as obstacle avoidance algorithms, enable vehicles to reach their destinations independently. The development of autonomous driving technology also involves the introduction of systems that enable communication between self-driving cars, as well as between the AVs and their surroundings. Autonomous driving technology is constantly evolving, and among the greatest challenges associated with developing fully functional self-driving cars is the dependence of individual sensors' performance on weather conditions.

A significant challenge is how to navigate autonomously by unmanned vehicles in environments with limited or no access to localization data. Autonomous navigation without GNSS is a complex and rapidly evolving technology area that has the potential to revolutionize many industries and applications. Key technologies and methods for navigation that lack GNSS information include inertial navigation systems (INS), vision-based localization, Lidar, and indoor localization systems. Promising results are also provided by SLAM technology, which is used to simultaneously determine the vehicle's position (location) and build a map of the environment in which it moves. Each of the technologies mentioned above has its advantages and disadvantages, but none of them provides a complete overview of the current state of the surrounding world. Although autonomous navigation technology without GNSS has many advantages, it also encounters challenges. These include, among others, difficulties in accurate measurement in an unknown or changing environment and problems with sensor calibration, which can lead to navigation errors.

In recent decades, much research and technology has been developed for various autonomous systems, including airborne, ground-based, and naval systems (see Figure 11). Much of this technology has already reached maturity and can be implemented on unmanned platforms, while others are still in the research and development phase.

Domain-specific challenges in the autonomy of vehicles include several technical, safety, regulatory, and ethical issues unique to different operational environments. Key challenges are:

• Sensor Limitations and Perception
  • Accurate object detection in diverse weather conditions (rain, fog, snow).
  • Distinguishing between road users like pedestrians, cyclists, and animals.
  • Handling sensor occlusions and blind spots.
• Complex and Dynamic Environments
  • Navigating complex urban settings with unpredictable human behavior.
  • Managing construction zones, roadworks, and unexpected obstacles.
  • Adapting to high traffic density and erratic driver behaviors.
• Localization and Mapping
  • Achieving precise real-time localization in GPS-denied environments (tunnels, urban canyons).
  • Maintaining up-to-date high-definition maps amid road changes.
• Decision-Making and Planning
  • Ensuring safe, compliant, and contextually appropriate decisions.
  • Handling edge cases like jaywalking pedestrians or unusual vehicle maneuvers.
• Communication and Cybersecurity
  • Secure vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communications.
  • Protecting against hacking or malicious cyber-attacks.
• Regulatory and Legal Challenges
  • Navigating diverse legal frameworks across regions.
  • Defining liability in accidents involving autonomous vehicles.
  • Achieving standardization and interoperability.
• Ethical Considerations
  • Decision-making in unavoidable accident scenarios.
  • Privacy concerns related to data collection and sharing.
• Infrastructure and Standardization
  • Lack of uniform infrastructure to support vehicle-to-infrastructure communication.
  • Variability in road signage, markings, and infrastructure standards.
• Testing, Validation, and Certification
  • Developing comprehensive testing protocols for safety assurance.
  • Validating autonomous systems in all possible scenarios.
• Human Factors and User Acceptance
  • Ensuring passenger trust and understanding of autonomous system limits.
  • Managing transition of control between human and automation.

Solving these domain-specific challenges is crucial for the safe and reliable deployment of autonomous vehicles in the various operational scenarios of our lives. In the following subchapters, specific challenges are determined taking into account the vehicle type and its operating environment.

Building public trust in autonomous vehicles will be a huge challenge. In 2025, an online preference survey collected 235 responses from across Europe ^[35]. Respondents were asked whether they would feel comfortable traveling in an autonomous car. According to over half of the respondents, driving a driverless autonomous car would make them feel uncomfortable. Distrust of machines is primarily fueled by the prospect of losing control over one's fate, which is one of the fundamental philosophical issues in the context of artificial intelligence development. Therefore, it seems crucial for sociologists and psychologists to thoroughly examine this fear and develop a plan to educate the public. Furthermore, placing excessive reliance on automated systems delays drivers' reaction times in crisis situations and compromises their willingness to take manual control. Increased trust also leads to longer driver reaction times to roadside warnings.

Another considered advantage of introducing autonomous vehicles in public transport would be the reduction of fatal accidents. According to statistics, driver error contributes to 75–90% of all road accidents. Eliminating driver error could significantly reduce fatalities among drivers and passengers. However, critics of this approach point out that automation can only correct some human errors, not eliminate them entirely. However, forecasts regarding increasing access to this technology are optimistic. According to the EU Transport Commissioner, by 2030, roads in member states will be shared by cars with conditional automation and standard vehicles. The prospect of full automation is another 10–15 years away. The research performed using the autonomous Blees bus ^[36] constructed in Gliwice, Poland, confirms the results of the survey and allows us to look optimistically into the future.

Software-based vehicle (SV) is a term used to describe, among other things, cars whose parameters and functions are controlled by software. This is the result of the car's ongoing evolution from a purely mechanical product to a software-based electronic device (see Figure 13).

Today's premium vehicles can require up to 150 million lines of code distributed across over a hundred electronic control units, and utilize an ever-increasing number of sensors, cameras, radars, and lidar sensors. Lower-end cars have only slightly fewer. Three powerful trends—electrification, automation, and digitalization—are fundamentally changing customer expectations and forcing manufacturers to invest in software to meet them.

As driver assistance systems become automated and vehicles acquire autonomous driving capabilities, the demand for software also grows. The more sophisticated content consumers expect from their infotainment systems, the more digital content the vehicle must manage. And because such a vehicle, as part of the Internet of Things (IoT), transmits increasing amounts of data to and from the cloud, software is required to process and manage it.

There are two basic software development methods: the traditional waterfall model and a newer approach called agile, which is key to the automotive industry's transformation toward “software-defined vehicles.”

In the waterfall approach, software development proceeds through distinct, sequential phases. These phases include requirements definition, implementation, integration, and testing. The waterfall method has numerous drawbacks: it is not flexible enough to keep up with the pace of change in today's automotive industry; it does not emphasize close collaboration and rapid feedback from internal business teams and external partners; and it does not ensure testing occurs early enough in the project. The ability to test complex software systems is particularly important in the automotive industry, where engineers conduct advanced simulations of real-world driving conditions—software-in-the-loop, hardware-in-the-loop, and vehicle-in-the-loop—before a vehicle hits the road.

The agile approach represents a cultural and procedural shift from the linear and sequential waterfall approach. Agile is iterative, collaborative, and incorporates frequent feedback loops. Organizations using agile form small teams to address specific, high-priority business requirements. Teams often work in a fixed, relatively short iterative cycle, called a Sprint. Within this single iteration, they produce a complete and tested increment of a software product based on specific business needs, ready for stakeholder review.

Both waterfall and agile approaches can utilize the V-model for software development, sequentially progressing through the design, implementation, and testing phases. The difference is that the waterfall approach uses the V-model as a single-pass process throughout the project, while the agile approach may implement the V-model within each Sprint.

Software-defined vehicles (SDVs) open up enormous opportunities, allowing end customers to enjoy a wide range of cutting-edge safety, comfort, and convenience features. Well-organized cybersecurity management must go hand in hand with the development of SDVs. Software developers must ensure security in every area, regardless of the security of a specific application.

Cybersecurity is a relatively new concept in the automotive industry [1]. While automakers began introducing electronically controlled steering and braking systems into their vehicles, the likelihood of threats increased, connectivity opened the door to significantly greater risks. Direct vehicle connections to the internet are cited as a source of cyber threats, but indirect connections, such as those to a mobile phone connected via USB or Bluetooth, are often overlooked. Even a vehicle that appears to have no connectivity at all can be equipped with a wireless tire pressure monitoring system or on-board diagnostic module that allows access to vehicle information.

Here are some key areas related to this topic:

• Secure updates: To ensure consumers have access to the most efficient features, today's vehicles download software updates wirelessly from the cloud, so these updates must be secure. Public key infrastructure (PKI) is a mechanism that allows manufacturers to digitally sign software so that the receiving system can verify its authenticity. Using a secret digital key, the manufacturer encrypts the software before releasing it. When the vehicle downloads the software, it uses another, publicly available key to verify the content. A complex algorithm ensures that only content signed with the secret key can be verified with the public key.
• Safe system boot: Manufacturers must also ensure secure system boot. Once the vehicle is started, the system must verify the authenticity and integrity of the software. This means the system must ensure that the code was created by the manufacturer and not by an attacker.
• Secure vehicle network: As vehicles become increasingly complex and software-defined, many applications running on them will use the same processors and networks to transmit data between different processing nodes. For example, some infotainment applications might require vehicle speed and navigation data, while other applications might need battery management information. Having an in-vehicle network and connecting to cloud networks via mobile network and Wi-Fi requires vehicles to secure these connections on multiple layers. The lowest layer is Media Access Control Security (MACsec), which establishes a bidirectional encrypted connection between two directly communicating devices. MACsec can operate extremely quickly, encrypting and decrypting information at wire speeds using specialized hardware. The next higher layer is Internet Protocol Security (IPsec), which operates at the network layer to authenticate and encrypt data packets between network nodes with IP addresses. Using IPsec can help protect data flowing across the network—through a router, to the cloud, and beyond—not just on the physical link between two points. Moving up the stack, manufacturers can use Transport Layer Security (TLS). This protocol operates at the network layer, where processes communicate without being bound to IP addresses, making the security mechanism more flexible. TLS is now widely used in internet communications, and vehicles should use it when connecting to the cloud.

As autonomous driving becomes more common, implementing even higher levels of message encryption may become increasingly important. For example, a user could send a message to a vehicle requesting it be picked up from a specific address. This message would be cryptographically signed and delivered securely. New protocols can help with this, even involving multiple clouds if necessary. Furthermore, as automotive companies begin to hire more developers to build various software functions, ensuring no interference between applications becomes crucial. This is achieved by using hypervisors, containers, and other technologies to decouple software, even on shared hardware.

Recent advances in vehicle radar technology will soon lead to a fundamental increase in radar capabilities, significantly enhancing the radar-centric approach to advanced driver assistance systems (ADAS).

There are two primary ways to improve ADAS system perception. We can either improve how the system interprets sensor data using machine learning, or improve the quality and accuracy of real-world sensor data. Combining these approaches creates a synergistic effect, creating a reliable model of the environment that vehicles can use to make intelligent driving decisions. While machine learning is constantly evolving, so is radar sensor technology. One specific technology is the use of 3D air-waveguide technology in radar antennas to capture more precise signals and increase range. In automotive radar systems, 3D air-waveguide antennas help efficiently scan the surrounding area with radar signals and receive weak echoes from the surrounding environment with low loss. By reducing losses in the transmitted and received signal, air-waveguide antennas enable the use of a more sensitive sensor while maintaining the same small physical radar footprint.

Radars with advanced 3D air-waveguide antenna technology could support a high-resolution perception mode, enabling self-parking. Early implementations of self-parking systems rely on ultrasonic sensors to measure the width of parking spaces. This often means the vehicle must drive past the space to determine if it is the right size before backing into it. With ambient perception software that utilizes improved radar, the vehicle would be able to determine the space's dimensions before driving past it, allowing it to park directly in it.

By utilizing advanced antenna technology, high-speed data transmission support, and enhanced software, Aptiv’s seventh-generation radar family provides an excellent foundation for building the next generation of automated driving vehicles.

At the same time, in the case of high-speed NOA radars, the detection range of the traditional radar is only 210 meters when traveling at high speed, while the detection range of the precise 4D radar reaches 280 meters (the parameters of some manufacturers even exceed 300 meters), which allows for earlier identification of targets, thereby solving the problem of the rigid requirement for forward perception.

Currently, brands such as Ideal, Changan, and Weilai are leading the widespread adoption of 4D radars. Among them, all Weilai NT3 platform models will be equipped with 4D radars as standard, with a maximum detection range of up to 370 meters. Furthermore, the three millimeter-wave distributed radar arrays in the Huawei Zunjie S800 can also utilize 4D technology.

Moreover, according to Huawei, the distributed architecture further improves performance. Its detection capabilities surpass those of existing 4D millimeter-wave radars, its high-confidence detection range in rainy and foggy conditions has been increased by 60%, and the detection latency of frontal and side targets has been reduced by 40%.

[rczyba][✓ rczyba, 2025-10-12]

Autonomy of unmanned systems refers to their ability to self-manage, make decisions, and complete tasks with minimal or no human intervention. The scope of autonomy ranges from zero to full capability, often defined through models, and encompasses four fundamental functions: perception, orientation, problem-solving (planning), and action. Advances in autonomy enable unmanned systems to learn, adapt to changing environmental conditions, and perform complex tasks, driving innovation in various fields.

There are several ways to classify autonomy levels based on various criteria. In 2014, the American organization Society of Automotive Engineers (SAE) International adopted a classification of six levels of autonomous driving, which was subsequently modified in 2016. Based on a decision by the National Highway Traffic Safety Administration (NHTSA), this is the officially applicable standardization in the United States, which is also the most popular in studies on autonomous driving technologies in Europe.

To clarify the situation, SAE International has defined 5 levels of automation for autonomous vehicles, which have been adopted as an industry standard (see Figure 17).

• Level 0: The driver has full control of the vehicle and there are no automated systems.

• Level 1: Also known as “hands-on,” the driver controls all standard driving functions, such as steering, acceleration, braking, and parking. Some automated systems, such as cruise control, parking assist, and lane-keeping assist, will be built into the car. The driver must monitor their surroundings and be able to take full control at any time.

• Level 2: “Hands-free” automation means that the automated system can take full control of the vehicle, steering, accelerating, and braking. However, the driver must be ready to take control of the vehicle if necessary. The “hands-free” principle shouldn't be taken literally, and the SAE recommends that the hands remain in contact with the steering wheel to confirm that the driver is ready to take control.

• Level 3: Level 3 is referred to as “without looking into the eyes” automation. The driver can focus on activities other than driving, such as using a phone or watching a movie. The automated system will be able to respond to situations requiring immediate action, such as emergency braking, but the driver will still need to intervene if notified by the technology.

• Level 4: The next level is “mind-off” automation. It's essentially similar to Level 3 in that the driver doesn't need to monitor their surroundings. In fact, they can fall asleep, as driver intervention isn't required, even in emergency situations. However, this level of autonomy is only supported in limited areas or under specific circumstances, such as traffic jams.

• Level 5: Level 5 means “steering wheel optional.” The car is fully autonomous and requires no human intervention.

In general, autonomy or autonomous capability is defined in the context of decision-making or self-governance within a system. According to the Aerospace Technology Institute (ATI), autonomous systems can essentially decide independently how to achieve mission objectives, without human intervention ^[39]. These systems are also capable of learning and adapting to changing operating environment conditions. However, autonomy may depend on the design, functions, and specifics of the mission or system ^[40]. Autonomy can be broadly viewed as a spectrum of capabilities, from zero autonomy to full autonomy. The Pilot Authorization and Task Control (PACT) model assigns authorization levels, from level 0 (full pilot authority) to level 5 (full system autonomy), also used in the automotive industry for autonomous vehicles (see Figure 18).

Levels of autonomy in drone technology are typically divided into five distinct levels, each representing a gradual increase in the drone's ability to operate independently.

• Level 0: The pilot has complete control over every movement. The platforms are always 100% manually controlled.

• Level 1 – Remote Control: The pilot retains control of overall operations and vehicle safety. However, the drone can take over one or more essential functions for a specified period. While the pilot does not have continuous control of the vehicle and never simultaneously controls speed and direction, it can assist with navigation and/or maintain altitude and position. The drone is supported by GNSS for flight stabilization, and all inputs regarding direction, altitude, and speed are entered manually. Obstacle detection functions are available at this level, but avoidance is performed manually by the pilot.

• Level 2 - Automated Flight Control (Assisted Autonomy): A pre-programmed flight path is transmitted to the autopilot, and the drone begins its mission, flying along waypoints. The pilot is still responsible for the safe control of the vehicle and must be ready to take control of the drone if an unexpected event occurs. However, under certain conditions, the drone itself can take control of the drone’s heading, altitude, and speed. The pilot still has full control, including monitoring airspace, flight conditions, and responding to emergencies. Most manufacturers currently build drones at this level, where the platform can assist with navigation functions and allow the pilot to disengage from certain tasks.

• Level 3 - Partial Autonomy (Semi-Autonomous): Similar to Level 2, the drone can fly autonomously, but the pilot must be alert and ready to take control at any time. The drone notifies the pilot of the need for intervention, acting as an emergency system. This level means that the drone can perform all functions “under certain conditions.”

• Level 4 - Cognitive Autonomy (Advanced Semi-Autonomous): A drone can be controlled by a human, but it doesn't always have to be. Under the right conditions, it can fly autonomously at all times. It's expected that the drone will have redundant systems so that if one system fails, it will continue to operate. At this level of autonomy, the “sense and avoid” function becomes “sense and navigate.” This means that the drone detects obstacles along its flight path and actively avoids contact by changing its flight trajectory.

• Level 5 – Full Autonomy: The drone controls itself independently in any situation, without the need for human intervention. This includes full automation of all flight tasks in all conditions. Currently, such drones do not yet exist. However, it is expected that in the near future, they will be able to utilize artificial intelligence tools for flight planning—in other words, autonomous learning systems with the ability to modify routine behavior.

Another general but useful model describing autonomy levels in unmanned systems is the Autonomy Levels for Unmanned Systems (ALFUS) model ^[42]. European Union Aviation Safety Agency (EASA), in one of its technical reports, provided some information on autonomy levels and guidelines for human-autonomy interactions. According to EASA, the concept of autonomy, its levels, and human-autonomous system interactions are not established and remain actively discussed in various areas (including aviation), as there is currently no common understanding of these terms ^[43]. Since these concepts are still somewhat developmental, this becomes a huge challenge for the unmanned aircraft regulatory environment as they remain largely unestablished.

The classification of autonomy levels in multi-drone systems is somewhat different. In multi-drone systems, several drones cooperate to perform a specific task. Designing multi-drone systems requires that individual drones have an increased level of autonomy. The classification of autonomy levels is directly related to the division into flights performed within the pilot's or observer's line of sight (VLOS) and flights performed beyond the pilot's line of sight (BVLOS), where particular attention is paid to flight safety. One way to address the autonomy issue is to classify the autonomy of drones and multi-drone systems into levels related to the hierarchy of tasks performed ^[44]. These levels will have standard definitions and protocols that will guide technology development and regulatory oversight. For single-drone autonomy models, two distinct levels are proposed: the vehicle control layer (Level 1) and the mission control layer (Level 2), see Figure 20. Multi-drone systems, on the other hand, have three levels: single-vehicle control (Level 1), multi-vehicle control (Level 2), and mission control (Level 3). In this hierarchical structure, Level 3 has the lowest priority and can be overridden by Levels 2 or 1.

[rahulrazdan][✓ rczyba, 2025-10-12]

Figure 1

In society, products operate within the confines of a legal governance structure. The legal governance structure is one of the great inventions of civilization and its primary role is to funnel disputes from unstructured expression and perhaps even violence to the domain of courts (figure 1). To be effective, legal governance structures must be perceived as fair and predictable. The objective of fairness is obtained by a number of methods such as due process procedures, transparency and public proceedings, and Neutral decision-makers (judges, juries, arbitrators). The objective of predictability is achieved by the use of the concept of precedence. Precedence is the idea that past rulings are given heavier weight relative to decision making, and it is an extraordinary event to diverge from precedence. Precedence gives the legal system stability. The combination of fairness and predictability shifts the dispensation of disputes to a more orderly process which promotes societal stability.

How does this mechanically work and how does this connect to product development ?

Figure 2

As shown in figure 2, there are three major stages. First, legal frameworks are established by law-making bodies (legislators). However, in practice, legislators cannot specify all aspects and empower administrative entities (regulators) to codify the details of law. Finally, regulators often do not have the technical knowledge to codify all aspects of the law and rely on independent industry groups such as Society for Automotive Engineering (SAE) or Institute of Electrical and Electronics Engineers (IEEE) for technical knowledge. Second, in the field, disputes arise and must be adjudicated by the legal system. The typical process is a trial, under the strict processes established for fairness. The result of the trial is to apply the facts to the legal frameworks and apply a judgement. The facts of the case can result in three potential outcomes. In the first situation, the facts are covered by the legal framework, so there is no further action relative to the governance structure. In the second case, the facts expose an “edge” condition in the governance structure. In this situation, the court looks for previous cases which might fit (the concept of precedence) and uses that to make its judgement. If such a case does not exist, the court can establish precedence with its judgement in this case. This has the effect of weighing the future decisions as well. Finally, in rare situations, the facts of the case are in a field which is so new that there is not much in the way of body of law. In these situation, the courts may make a judgement, but often there is a call for law-making bodies to establish deeper legal frameworks.

In fact, autonomous vehicles (AVs) are considered to be one of these situations. Why ? In traditional automobiles, the body of law connected to product liability is connected to the car, and the liability of actions using the car is connected to the driver. Further, Product liability is often managed at the federal level and driver licensing more locally. However, surprisingly, as the figure below shows, there is a body of law dealing with autonomous vehicles from the distant past. In the days of horses, there were accidents, and a sophisticated liability structure emerged. In this structure, there was a concept that if a person directed his horse into an accident, then the driver was at fault. However, if a bystander did something to “spook” the horse, it was the bystander's fault. Finally, there was also the concept of “no-fault” when a horse unexpectedly went rogue. A discerning reader may well understand that this body of law emerges from a deep understanding of the characteristics of a horse. In legal terms, it creates an “expectation.' What are the “expectations” for a modern autonomous vehicle ? This is currently a highly debated point in the industry.

Overall, whatever value products provide to their consumers is weighed against the potential harm caused by the product, and leads to the concept of legal product liability. While laws diverge across various geographies, the fundamental tenets have key elements of expectation and harm. Expectation as judged by “reasonable behavior given a totality of the facts” attaches liability. As an example, the clear expectation is that if you stand in front of a train, it cannot stop instantly while this is not the expectation for most autonomous driving situations. Harm is another key concept where AI recommendation systems for movies are not held to the same standards as autonomous vehicles. The governance framework for liability is mechanically developed through legislative actions and associated regulations. The framework is tested in the court system under the particular circumstances or facts of the case. To provide stability to the system, the database of cases and decisions are viewed as a whole under the concept of precedence. Clarification on legal points is set by the appellate legal system where arguments on the application of the law are decided what sets precedence.

What is an example of this whole situation ? Consider the airborne space with the figure above where the governance framework consists of enacted law (in this case US) with associated cases providing legal precedence, regulations, and industry standards. Any product in the airborne sector, must be compliant to release their solution to the marketplace.

Ref:

1. Razdan, R., (2019) “Unsettled Technology Areas in Autonomous Vehicle Test and Validation,” Jun. 12, 2019, EPR2019001.
2. Razdan, R., (2019) “Unsettled Topics Concerning Automated Driving Systems and the Transportation Ecosystem,” Nov 5, 2019, EPR2019005.
3. Ross, K. Product Liability Law and its effect on product safety. In Compliance Magazine 2023, [Online]. Available: https://incompliancemag.com/product-liability-law-and-its-effect-on-product-safety/

[rahulrazdan][✓ rahulrazdan, 2025-06-16]

As discussed in the governance module, whatever value products provide to their consumers is weighed against the potential harm caused by the product, and leads to the concept of legal product liability. From a product development perspective, the combination of laws, regulations, legal precedence form the overriding governance framework around which the system specification must be constructed [3]. The process of validation ensures that a product design meets the user's needs and requirements, and verification ensures that the product is built correctly according to design specifications.

Fig. 1. V&V and Governance Framework. The Master V&V(MaVV) process needs to demonstrate that the product has been reasonably tested given the reasonable expectation of causing harm. It does so using three important concepts [4]:

1. Operational Design Domain (ODD): This defines the environmental conditions and operational model under which the product is designed to work.
2. Coverage: This defines the completeness over the ODD to which the product has been validated.
3. Field Response: When failures do occur, the procedures used to correct product design shortcomings to prevent future harm.

As figure 1 shows, the Verification & Validation (V&V) process is the key input into the governance structure which attaches liability, and per the governance structure, each of the elements must show “reasonable due diligence.” An example of unreasonable ODD would be for an autonomous vehicle to give up control a millisecond before an accident.

Fig. 2. Execution is space.

Mechanically, MaVV is implemented with a Minor V&V (MiVV) process consisting of:

1. Test Generation: From the allowed ODD, test scenarios are generated.
2. Execution: This test is “executed” on the product under development. Mathematically, a functional transformation which produces results.
3. Criteria for Correctness: The results of the execution are evaluated for success or failure with a crisp criteria-for-correctness.

In practice, each of these steps can have quite a bit of complexity and associated cost. Since the ODD can be a very wide state space, intelligently and efficiently generating the stimulus is critical. Typically, in the beginning, stimulus generation is done manually, but this quickly fails the efficiency test in terms of scaling. In virtual execution environments, pseudo-random directed methods are used to accelerate this process. In limited situations, symbolic or formal methods can be used to mathematically carry large state spaces through the whole design execution phase. Symbolic methods have the advantage of completeness but face algorithmic computational explosion issues as many of the operations are NP-Complete algorithms.

The execution stage can be done physically (such as test track above), but this process is expensive, slow, has limited controllability and observability, and in safety critical situations, potentially dangerous. In contrast, virtual methods have the advantage of cost, speed, ultimate controllability and observability, and no safety issues. The virtual methods also have the great advantage of performing the V&V task well before the physical product is constructed. This leads to the classic V chart shown in figure 1. However, since virtual methods are a model of reality, they introduce inaccuracy into the testing domain while physical methods are accurate by definition. Finally, one can intermix virtual and physical methods with concepts such as Software-in-loop or Hardware-in-loop. The observable results of the stimulus generation are captured to determine correctness. Correctness is typically defined by either a golden model or an anti-model. The golden model, typically virtual, offers an independently verified model whose results can be compared to the product under test. Even in this situation, there is typically a divergence between the abstraction level of the golden model and the product which must be managed. Golden model methods are often used in computer architectures (ex ARM, RISCV). The anti-model situation consists of error states which the product cannot enter, and thus the correct behavior is the state space outside of the error states. An example might be in the autonomous vehicle space where an error state might be an accident or violation of any number of other constraints. The MaVV consists of building a database of the various explorations of the ODD state space, and from that building an argument for completeness. The argument typically takes the nature of a probabilistic analysis. After the product is in the field, field returns are diagnosed, and one must always ask the question: Why did not my original process catch this issue? Once found, the test methodology is updated to prevent issues with fixes going forward. The V&V process is critical in building a product which meets customer expectations and documents the need for “reasonable” due diligence for the purposes of product liability in the governance framework.

Finally, the product development process is typically focused on defining an ODD and validating against that situation. However, in modern times, an additional concern is that of adversarial attacks (cybersecurity). In this situation, an adversary wants to high jack the system for nefarious intent. In this situation, the product owner must not only validate against the ODD, but also detect when the system is operating outside the ODD. After detection, the best case scenario is to safely redirect the system to the ODD space. The risk associated with cybersecurity issues typically split at three levels for cyber-physical systems:

1. OTA Security: If an adversary can take manipulate the Over the Air (OTA) software updates, they can take over mass number of devices quickly. An example worst case situation would be a Tesla OTA which turns Tesla's into collision engines.
2. Remote Control Security: If the adversary can take over a car remotely, they can cause harm to the occupants as well as third-parties.
3. Sensor Spoofing: In this situation, the adversary uses local physical assets to fool the sensors of the target. GPS jamming or spoofing are active examples.

In terms of governance, some reasonable due-diligence is expected to be provided by the product developer in order to minimize these issues. The level of validation required is dynamic in nature and connected to the norm in the industry.

[rahulrazdan][✓ raivo.sell, 2025-09-18]

In terms of domains, the Operational Design Domain (ODD) is the driving factor, and typically have two dimensions. The first is the operational model and the second is the physical domain (ground, airborne, marine, space). In terms of ground, Passenger AVs are perhaps the most well-known face of autonomy, with robo-taxi services and self-driving consumer vehicles gradually entering urban environments. Companies like Waymo, Cruise, and Tesla have taken different approaches to ODDs. Waymo’s fully driverless cars operate in sunny, geo-fenced suburbs of Phoenix with detailed mapping and remote supervision. Cruise began service in San Francisco, originally operating only at night to reduce complexity. Tesla’s Full Self Driving (FSD) Beta aims for broader generalization, but it still relies heavily on driver supervision and is limited by weather and visibility challenges.

Transit shuttles, though less publicized, have quietly become a practical application of AVs in controlled environments. These low-speed vehicles typically operate in geo-fenced areas such as university campuses, airports, or business parks. Companies like Navya, Beep, and EasyMile deploy shuttles that follow fixed routes and schedules, interacting minimally with complex traffic scenarios. Their ODDs are tightly defined: they may not operate in rain or snow, often run only during daylight, and avoid high-speed or mixed-traffic conditions. In many cases, a remote operator monitors operations or is available to intervene if needed. Delivery robots represent a third class of autonomous mobility—compact, lightweight vehicles designed for last-mile delivery. Their ODDs are perhaps the narrowest, but that’s by design. These robots, from companies like Starship, Kiwibot, and Nuro, navigate sidewalks, crosswalks, and short street segments in suburban or campus environments. They operate at pedestrian speeds (typically under 10 mph), carry small payloads, and avoid extreme weather, high traffic, or unstructured terrain. Because they don’t carry passengers, safety thresholds and regulatory oversight can differ significantly.

Weather is a particularly limiting factor across all autonomous systems. Rain, snow, fog, and glare interfere with LIDAR, radar, and camera performance—especially for smaller robots that operate close to the ground. Most AV deployments today restrict operations to fair-weather conditions. This is especially true for delivery robots and transit shuttles, which often halt operations during storms. While advanced sensor fusion and predictive modeling promise improvements, true all-weather autonomy remains a significant technical challenge. The intersection of weather and autonomy is an active research area [1]

Another ODD dimension is time of day. Nighttime operation brings unique difficulties for AVs: reduced visibility, increased pedestrian unpredictability, and in urban areas, more erratic driver behavior. Some systems (like Waymo in Chandler, AZ) now operate 24/7, but most deployments—particularly delivery robots and shuttles—remain restricted to daylight hours. Tesla's FSD does operate at night, but it still requires human oversight. Infrastructure also shapes ODDs in crucial ways. Many AV systems depend on high-definition maps, lane-level GPS, and even smart traffic signals to guide their decisions. In geo-fenced environments—where the route and surroundings are highly predictable—this infrastructure dependency is manageable. But for broader ODDs, where environments may change frequently or lack digital maps, achieving safe autonomy becomes much harder. That’s why passenger AVs today generally avoid rural areas, unpaved roads, or newly constructed zones.

Regulatory environments further shape ODDs. In the U.S., states like California, Arizona, and Florida have developed AV testing frameworks, but each differs in what it permits. For instance, California limits fully driverless vehicles to certain urban zones with strict reporting requirements. Delivery robots are often regulated at the city level—some cities allow sidewalk bots, others ban them outright. Transit shuttles often receive special permits for low-speed operation on limited routes. These regulatory boundaries translate directly into ODD constraints.

In terms of physical domains, Ground-based autonomous systems, especially in automotive contexts, are the most commercially visible. Self-driving vehicles operate in human-dense environments, requiring perception systems to identify pedestrians, cyclists, vehicles, and traffic infrastructure. Validation here relies heavily on scenario-based testing, simulation, and controlled pilot deployments. Standards like ISO 26262 (functional safety), ISO/PAS 21448 (SOTIF), and UL 4600 (autonomy system safety) guide safety assurance. Regulatory frameworks are evolving state-by-state or country-by-country, with Operational Design Domain (ODD) restrictions acting as practical constraints on deployment.

Autonomous aircraft (e.g., drones, urban air mobility platforms, and optionally piloted systems) must operate in highly structured, safety-critical environments. Validation involves rigorous formal methods, fault tolerance analysis, and conformance with aviation safety standards such as DO-178C (software), DO-254 (hardware), and emerging guidance like ASTM F38 and EASA's SC-VTOL. Airspace governance is centralized and mature, often requiring type certification and airworthiness approvals. Unlike automotive systems, airborne autonomy must prove reliability in loss-of-link scenarios and demonstrate fail-operational capabilities across flight phases. Autonomous surface and underwater marine systems face unstructured and communication-constrained environments. They must operate reliably in GPS-denied or RF-blocked conditions while detecting obstacles like buoys, vessels, or underwater terrain. Validation is more empirical, often involving extended sea trials, redundancy in navigation systems, and adaptive mission planning. IMO (International Maritime Organization) and classification societies like DNV are working on Maritime Autonomous Surface Ship (MASS) regulatory frameworks, though global standards are still nascent. The dual-use nature of marine autonomy (civil and defense) adds governance complexity. Space-based autonomous systems (e.g., planetary rovers, autonomous docking spacecraft, and space tugs) operate under extreme constraints: communication delays, radiation exposure, and no real-time human oversight. Validation occurs through rigorous testing on Earth-based analog environments, formal verification of critical software, and fail-safe design principles. Governance falls under national space agencies (e.g., NASA, ESA) and international frameworks like the Outer Space Treaty. Assurance relies on mission-specific autonomy envelopes and pre-defined decision trees rather than reactive autonomy.

Governance also differs. Aviation and space operate within centralized, internationally coordinated regulatory systems (ICAO, FAA, EASA, NASA), while ground autonomy remains highly fragmented across jurisdictions. Maritime governance is progressing but lacks harmonization. Space governance, although anchored in treaties, increasingly contends with commercial activity and national interests, demanding updated risk management protocols.

Emerging efforts like the SAE G-34/SC-21 standard for AI in aviation, NASA's exploration of adaptive autonomy, and ISO’s work on AI functional safety indicate a trend toward domain-agnostic principles for validating intelligent behavior. There is growing recognition that autonomous systems, regardless of environment, need rigorous testing of edge cases, clarity of system intent, and real-time assurance mechanisms.

Ref:

[1] Vargas, J.; Alsweiss, S.; Toker, O.; Razdan, R.; Santos, J. An Overview of Autonomous Vehicles Sensors and Their Vulnerability to Weather Conditions. Sensors 2021, 21, 5397. https://doi.org/10.3390/s21165397

[rahulrazdan][✓ rahulrazdan, 2025-06-16]

In most cases, the generic V&V process must grapple with massive ODD spaces, limited execution capacity, and high cost of evaluation. Further, all of this must be done in a timely manner to make the product available to the marketplace. Traditionally, the V&V regimes have been bifurcated into two broad categories: Physics-Based and Decision-Based. We will discuss the key characteristics of each now.

A. TRADITIONAL PHYSICS-BASED EXECUTION

For MaVV, the critical factors are the efficiency of the MiVV “engine” and the argument for the completeness of the validation. Historically, mechanical/non-digital products (such as cars or airplanes) required sophisticated V&V. These systems were examples of a broader class of products which had a Physics-Based Execution (PBE) paradigm. In this paradigm, the underlying model execution (including real life) has the characteristics of continuity and monotonicity because the model operates in the world of physics. This key insight has enormous implications for V&V because it greatly constrains the potential state-space to be explored. Examples of this reduction of state-space include: 1) Scenario Generation: One need only worry about the state space constrained by the laws of physics. Thus, objects which defy gravity cannot exist. Every actor is explicitly constrained by the laws of physics. 2) Monotonicity: In many interesting dimensions, there are strong properties of monotonicity. As an example, if one is considering stopping distance for braking, there is a critical speed above which there will be an accident. Critically, all the speed bins below this critical speed are safe and do not have to be explored. Mechanically, in traditional PBE fields, the philosophy of safety regulation (ISO 26262 [5], AS9100 [6], etc.) builds the safety framework as a process, where 1) failure mechanisms are identified; 2) a test and safety argument is built to address the failure mechanism; 3) there is an active process by a regulator (or documentation for self-regulation) which evaluates these two, and acts as a judge to approve/decline. Traditionally, faults considered were primarily mechanical failure. As an example, the flow for validating the braking system in an automobile through ISO 26262 would have the following steps: 1) Define Safety Goals and Requirements (Concept Phase): Hazard Analysis and Risk Assessment (HARA): Identify potential hazards related to the braking system (e.g., failure to stop the vehicle, uncommanded braking). Assess risk levels using parameters like severity, exposure, and controllability. Define Automotive Safety Integrity Levels (ASIL) for each hazard (ranging from ASIL A to ASIL D, where D is the most stringent). Define safety goals to mitigate hazards (e.g., ensure sufficient braking under all conditions). 2) Develop Functional Safety Concept: Translate safety goals into high-level safety requirements for the braking system. Ensure redundancy, diagnostics, and fail-safe mechanisms are incorporated (e.g., dual-circuit braking or electronic monitoring). 3) System Design and Technical Safety Concept: Break down functional safety requirements into technical requirements, design the braking system with safety mechanisms like (Hardware {e.g., sensors, actuators. Software (e.g., anti-lock braking algorithms). Implement failure detection and mitigation strategies (e.g., failover to mechanical braking if electronic control fails). 4) Hardware and Software Development: Hardware Safety Analysis (HSA): Validate that components meet safety standards (e.g., reliable braking sensors). Software Development and Validation: Use ISO 26262-compliant processes for coding, verification, and validation. Test braking algorithms under various conditions. 5) Integration and Testing: Perform verification of individual components and subsystems to ensure they meet technical safety requirements. Conduct integration testing of the complete braking system, focusing on: Functional tests (e.g., stopping distance), Safety tests (e.g., behavior under fault conditions), and Stress and environmental tests (e.g., heat, vibration). 6) Validation (Vehicle Level): Validate the braking system against safety goals defined in the concept phase. Perform real-world driving scenarios, edge cases, and fault injection tests to confirm safe operation. Verify compliance with ASIL-specific requirements. 7) Production, Operation, and Maintenance: Ensure production aligns with validated designs, implement operational safety measures (e.g., periodic diagnostics, maintenance), monitor and address safety issues during the product's lifecycle (e.g., software updates). 8) Confirmation and Audit: Use independent confirmation measures (e.g., safety audits, assessment reviews) to ensure the braking system complies with ISO 26262.

Finally, the regulations have a strong idea of safety levels with Automotive Safety Integrity Levels (ASIL). Airborne systems follow a similar trajectory (pun intended) with the concept of Design Assurance Levels (DALs). A key part of the V&V task is to meet the standards required at each ASIL level. Historically, a sophisticated set of V&V techniques has been developed to verify traditional automotive systems. These techniques included well-structured physical tests, often validated by regulators, or sanctioned independent companies (ex TUV-Sud [7]). Over the years, the use of virtual physics-based models has increased to model design tasks such as body design [8] or tire performance [9]. The general structure of these models is to build a simulation which is predictive of the underlying physics to enable broader ODD exploration. This creates a very important characterization, model generation, predictive execution, and correction flow. Finally, because the execution is highly constrained by physics, virtual simulators can have limited performance and often require extensive hardware support for simulation acceleration. In summary, the key underpinnings of the PBE paradigm from a V&V point of view are: 1) Constrained and well-behaved space for scenario test generation 2) Expensive physics based simulations 3) Regulations focused on mechanical failure 4) In safety situations, regulations focused on a process to demonstrate safety with a key idea of design assurance levels

[pczekalski][✓ pczekalski, 2025-10-29]

Drones' cybersecurity covers all aspects of IT security systems, but due to their autonomous operations and the physical presence of potentially dangerous devices, they could have a far greater impact on outcomes, including life-threatening incidents. This is related to their physical presence, including commonly relatively high weight (compared to the human body), high operational speeds and thus large impact energy.

Below, we briefly describe the most important areas and list domain-specific challenges. UAV applications grow in both well-established and new environments, presenting unforeseen vulnerabilities. A compromise of a single device (e.g., a smart-enabled car on a highway) or multiple devices (e.g., a swarm of drones during a show) may have serious, even fatal consequences, not only for their owners but also for others. This potential is being used massively during the war in Ukraine, which has been going on since 2022, where drones (UAV, UGV, USV) are one of the primary methods of attacks and defence.
Autonomous systems vary in size and complexity, and thus differ in vulnerability to hacking and potential environmental harm in the event of compromise. Unauthorised access may have a dual nature and related consequences:

• hacking of an unmanaged system and its intentional use with a different target than formerly planned (e.g. flight plan change) - done usually via professional hackers that study the system and its vulnerabilities,
• hacking of an unmanaged system with unintentional results of disturbance of processed tasks (algorithms) that may result in disruption of operation and even involuntary destruction, usually done by accident during hacking.

Both cases are raising serious dangers to life and property.

In the table 2, we present a list of selected, recent incidents involving autonomous or semi-autonomous systems, with a short description.

Cybersecurity for drones includes all their components (hardware and software), procedures, and operations. Below is in a table 3, there is a short list of those components with characteristics:

Technically, drones are a blend of robotics and ICT and thus pose domain-specific cybersecurity challenges and threats, which we juxtapose in the table 4 along with estimates of potential impact and mitigation strategies. Many of them are identical or similar to the embedded systems, AI and IoT domains.

put your contents here

[karlisberkolds]

Autonomous operations require instant delivery of the surrounding environment. Either it is just the location when executing a predefined path, e.g., a flight plan or challenging ad-hoc decisions such as collision avoidance, sensors need to deliver accurate and correct information to the motion controller.

Sensors play a dual role in autonomous vehicles: some monitor the device's internal state (e.g., power source condition), while others are essential for autonomous operation, delivering environmental information (e.g. current altitude). Depending on the application domain, those sensors are either popular, industrial-grade, or military-grade. Selecting the proper class when building an autonomous system is driven by its application, risk mitigation and cost: different classes and qualities of sensors are used in camera drones and different ones in autonomous flight control systems for aircrafts, while in both cases they may perform exactly the same function and even provide the same accuracy, virtually delivering the same data in optimal work conditions.
Industrial and military-grade applications usually require multiplexed sensors that are resistant to interference and highly reliable (with high MTBF); however, this comes at the cost of certification, higher build quality, and more sophisticated protection against interference and cybersecurity threats.

Most sensors used in autonomous vehicles are cross-domain, with different capabilities, ranges, and accuracies. Those sensors are commonly shared with IoT and embedded systems.
Below is a list (table 5) of the most common sensors and their applications. We discuss application-specific differences in the following chapter. For more information, examples, technical details, and their applications, please refer to https://iot-open.eu/introduction-to-the-iot-coursebook-2nd-edition/.

Some sensors are domain-specific; for example, UUVs typically do not use GPS/GNSS-based navigation while underwater. A pressure sensor used for altitude and immersion sensing has different structural requirements and ranges for aerial and underwater vehicles. To perform an operation, an aerial drone requires at least a 6DOF (degrees of freedom) IMU; commonly, it is equipped with a 9DOF or 10DOF IMU. It is not the case when considering UGV. Refer to table 6 for an explanation on IMU construction:

The IMU itself does not ensure positioning; external impacts on the vehicle may cause its drift, particularly in low-friction environments such as air and water. Thus, an external reference system is required to ensure the drone's positioning, either planar, linear or 3-dimensional. Besides ensuring proper drone operation, positioning (and thus navigation) is an essential component of autonomous operations. It is common to use hybrid and combined positioning systems, as one of the positioning systems may fail or become inaccurate (e.g. GNSS signal jamming or wheel slippage).

Below, we discuss common positioning techniques that are domain-specific (table 7:

Many sensors are application-specific; for example, a drone used to monitor air quality will have onboard sensors for gases and air pollution, such as PM2.5 and PM10. Virtually any manufactured sensors may appear here but they provide application logic, separate from the drone's logic and its construction.

Communication between sensors, actuators and components of the autonomous systems is performed using a variety of protocols. Sensors and actuators are usually connected to the MCU (control unit) with one of the embedded protocols:

• low-speed and local procolos, presented in tables 8,
• industrial grade and field protocols, presented in 9.

Detailed description of those protocols can be found, e.g. in ^[45] and it is shared with IoT and embedded systems.

Computing units in autonomous vehicles present a hierarchical structure.

At the top of the control algorithms, there are the biggest and most powerful devices, such as:

• NVIDIA Jetson series (Nano, Xavier, Orin) – edge AI + GPU acceleration
• NVIDIA DRIVE (AGX / Pegasus) – automotive-grade ADAS/autonomy platform
• Qualcomm Snapdragon Ride – automotive SoC for perception & planning
• Intel Mobileye EyeQ – vision and ADAS processor
• AMD/Xilinx Zynq Ultrascale+ – FPGA + ARM SoC for high-throughput tasks
• Google/Waymo custom compute platform – cloud-grade edge processors
• Huawei MDC – automotive compute platform
• Tesla Full Self-Driving Computer – custom AI chips (FSD/autopilot)

Their role is to provide high computing capabilities that enable unmanned vehicles to analyse their surrounding environment and adapt control algorithms to the current situation. It is common nowadays to implement AI-based algorithms that utilise vision, LIDAR, radar, and synthetic data provided by other devices located nearby or far, to inform decisions on how to behave in current conditions. Modern autonomous vehicles challenge this decision locally. Some of them utilise remote computing and remote links, typically when they lack rich local computing resources, and thus, part of their autonomy is located remotely.
High-level, main computing units provide means for abstract logic of the autonomous device, such as object recognition, obstacle detection, target identification, collision avoidance and many other. Common architectures are ARM, x86, x64, GPU acceleration (NVIDIA CUDA) and FPGA.

Those devices are of the IoT fog class, more powerful than end-node and edge devices, and still able to implement multiple algorithms in parallel. Used to implement indirect control logic, telemetry, communication, etc. They are commonly integrated with low-level, real-time modules. Examples of these devices include autopilots (such as Pixhawk, PX4, Ardupilot, and ROD-based embedded computers), as well as DJI flight controllers. Common architecture is ARM, sometimes low-powered x86 and x64. GPU acceleration is rare and usually not present.

Their nature is to ensure the vehicle's elementary operations and states, such as stabilisation, vehicle dynamics, parameter monitoring, actuator control and low-level communication. Those are usually MCUs, such as STM32 (common in UAVs), TI, ARM Cortex-M series, PIC, AVR/ATMEL, and ESP - virtually any MCU family known from the embedded systems and IoT domains are present here.

Besides the aforementioned ones, there is a variety of specialised computing hardware, usually application and domain-specific, e.g., radar and LIDAR processing units, software-defined radio components, sonar processors, and a number of communication-related controllers, such as CAN bus controllers, GSM communication and acoustic modems.

Over the last two decades, autonomous systems, from self-driving vehicles to unmanned aerial drones and underwater robots, have been transforming industries such as logistics, defence, agriculture, and transportation. This transformation is driven by the convergence of hardware miniaturisation, AI-enabled software, and interconnected supply ecosystems that allow global collaboration ^[46,47]. As systems evolve, the line between mechanical, electrical, and computational components has blurred, making hardware integration and supply chain orchestration central to the success of any autonomous platform. Whether in aerial, ground, or marine domains, achieving reliable autonomy depends as much on hardware compatibility and component sourcing as on algorithms and control logic. Autonomous platforms typically rely on tightly coupled subsystems, including sensors, embedded processors, actuators, power systems, and communication modules, each sourced from specialised manufacturers across the globe. Ensuring these heterogeneous components work seamlessly requires an integration process supported by robust supply chain management. As such, understanding hardware integration and supply chain considerations is vital not only for engineers but also for system architects, project managers, and policymakers involved in developing safe, scalable autonomous technologies.

Hardware integration refers to the systematic process of combining multiple physical components and subsystems — such as sensors, controllers, communication modules, and actuators — into a unified operational platform. The goal is to ensure that the integrated system performs its intended functions reliably and efficiently ^[48] In autonomous systems, this process goes beyond mechanical assembly. It includes:

• Electrical integration: ensuring signal and power compatibility among components.
• Communication integration: establishing data exchange between sensors, processors, and actuators via standardised protocols (e.g., CAN, Ethernet, DDS).
• Functional integration: aligning subsystems to fulfil overall mission requirements.
• Environmental integration: designing for ruggedness, temperature tolerance, and vibration resistance.

The performance of an autonomous system is fundamentally determined by how well its hardware is integrated: Any mismatch in interface, timing, or reliability can cascade into systemic failure. Hence, hardware integration establishes the bridge between the physical and computational worlds — transforming a collection of devices into a functioning autonomous agent ^[49]. Hardware integration in autonomous systems is complex due to heterogeneity, tight coupling, and high reliability demands. Major challenges include:

The hardware integration process typically follows iterative V-model or spiral development cycles:

• Requirements definition (functional, electrical, environmental).
• Subsystem design (sensor, computation, actuation).
• Interface design and standardisation (protocols, pinouts, connectors).
• System assembly and calibration.
• Hardware-in-the-loop (HIL) simulation for verification.
• Integration testing and validation under real-world conditions.

Each iteration refines both design and performance, ensuring compliance with standards such as ISO 26262 (automotive), DO-254 (aerospace), or MIL-STD-810G (defence environmental testing).

Due to the rapid technological development, hardware integration requires components or processes that were not needed previously. For instance:

• System-on-Chip (SoC) integration: Combining CPUs, GPUs, and AI accelerators in single chips (e.g., NVIDIA Xavier, Tesla FSD).
• Sensor fusion hardware accelerators: Dedicated ASICs and FPGAs for real-time data aggregation.
• Modular hardware platforms: Plug-and-play architectures like NVIDIA Jetson and PX4 stack.
• Edge AI co-design: Optimising computation between embedded hardware and cloud services.
• Standardised communication backbones: Time-sensitive networking (TSN) for deterministic data exchange.

These trends support scalable, reusable architectures that can adapt to rapidly evolving AI and robotics ecosystems. The chapter is structured as follows:

In the context of modern engineering and technology industries, a supply chain refers to the network of organisations, resources, and processes involved in the design, procurement, manufacturing, and delivery of a product or system ^[50]. For autonomous systems, the supply chain includes everything from electronic components (sensors, chips, batteries) and mechanical parts (motors, frames) to software dependencies and data provisioning. Since the autonomous systems are multi-disciplinary, a typical supply chain is global. The components and services are sourced globally, making the supply chain geographically distributed and highly interdependent. Effective management ensures that every subsystem arrives on time, meets specifications, and can be integrated seamlessly ^[51]. The main processes within an autonomous systems supply chain include:

• Procurement: Identifying and acquiring necessary parts and raw materials.
• Production Planning: Scheduling and coordinating the assembly of subsystems.
• Quality Assurance (QA): Verifying compliance with technical and safety standards.
• Logistics: Transportation, warehousing, and inventory management.
• Aftermarket Services: Maintenance, software updates, and replacement part logistics.

In highly complex domains such as aerospace and automotive, these processes must align with standards such as AS9100, ISO 9001, and IATF 16949 for quality assurance and traceability.

Autonomous systems depend heavily on specialised components such as LiDARs, high-density batteries, and embedded processors. Many of these have limited global suppliers, creating vulnerability to shortages or geopolitical disruptions ^[52].

The Semiconductor Bottleneck The semiconductor supply chain crisis (2020–2023) revealed how fragile technology manufacturing can be. Autonomous vehicles and drones rely on advanced microprocessors and GPUs fabricated using sub-10nm processes available only in a few facilities globally (TSMC, Samsung, Intel). Disruptions in this sector ripple across the entire autonomy industry ^[53].

Environmental and Ethical Constraints Supply chains for autonomy-related technologies often rely on materials such as lithium, cobalt, and rare earth metals used in sensors and batteries. Ethical sourcing, sustainability, and carbon accountability are now critical supply chain dimensions ^[54].

The Rise of Supply Chain Cybersecurity As hardware and software become interconnected, supply chain cybersecurity has emerged as a critical risk domain. Compromised firmware or cloned microcontrollers can introduce vulnerabilities deep within a system’s hardware root of trust ^[55]. Security frameworks such as NIST SP 800-161, ISO/IEC 27036, and Cybersecurity Maturity Model Certification (CMMC) are being applied to mitigate these threats.

Autonomous systems add several unique layers of complexity to both hardware integration and supply chain management:

Multi-Vendor Dependency A single autonomous platform may use components from dozens of vendors — from AI accelerators to GNSS modules. Managing version control, firmware updates, and hardware compatibility across this ecosystem requires multi-tier coordination and continuous configuration tracking ^[56].

Safety-Critical Certification Hardware must meet safety and regulatory certifications, such as:

• ISO 26262 (automotive functional safety)
• DO-254 (aerospace hardware design assurance)
• IEC 61508 (industrial functional safety)

Each certification adds cost, time, and documentation requirements.

Real-Time and Deterministic Performance Integration must guarantee low-latency, deterministic behaviour — meaning that sensors, processors, and actuators must communicate within microsecond precision. This influences hardware selection and network design ^[57].

Rapid Technology Obsolescence AI and embedded computing evolve faster than mechanical systems. Components become obsolete before the platform’s lifecycle ends, forcing supply chains to manage technology refresh cycles and long-term component availability planning ^[58].

The most important challenges and possible solutions are summarised in the following table:

Hardware integration is a structured, iterative process designed to ensure that all components — sensors, processors, actuators, and communication modules — work together seamlessly and safely. In autonomous systems, integration must account for functional, electrical, mechanical, and software–hardware interfaces simultaneously ^[59,60]. The following sections describe standardised integration procedures used across automotive, robotics, and aerospace industries.

Integration begins with defining functional requirements, interface specifications, and testing criteria. Key steps:

• Define subsystem roles: sensors, computation, actuation, communication, and power.
• Specify interfaces: voltage levels, connectors, data rates, protocols (e.g., CAN, UART, Ethernet, SPI).
• Set performance constraints: latency budgets, power envelopes, fault tolerance, safety margins.
• Establish test metrics: accuracy, reliability, thermal tolerance, and MTBF (mean time between failures).

This stage aligns stakeholders (hardware engineers, software developers, and supply managers) under a shared system model — often implemented using Model-Based Systems Engineering (MBSE) tools such as SysML, MATLAB Simulink, or Enterprise Architect.

Each subsystem is designed and tested individually, using simulated or mock environments:

• Sensor modules undergo calibration and noise testing.
• Processing boards are benchmarked for computational throughput and thermal dissipation.
• Actuation subsystems (motors, servos, hydraulics) are tested for response time and precision.

Prototype interfaces are validated through Hardware-in-the-Loop (HIL) or Software-in-the-Loop (SIL) setups to ensure cross-compatibility before full integration.

At this stage, physical and electrical integration occur:

• Assembly of sensor suites onto mechanical structures (vehicle body, UAV frame, or marine hull).
• Routing of wiring harnesses, data buses, and power lines to minimize interference.
• Installation of embedded computers, gateways, and controllers.

Integration testing ensures that the full system operates as expected under diverse conditions. Testing methods include:

• Unit and subsystem tests: Validate functionality of individual modules.
• Hardware-in-the-loop (HIL) testing: Real-time simulation of sensors and actuators.
• Environmental tests: Shock, vibration, and thermal chamber evaluations per MIL-STD-810G or ISO 16750.
• Electromagnetic compatibility (EMC) testing: Assess cross-interference between radios, processors, and power systems.
• Calibration: Align sensor measurements (LiDAR–camera extrinsics, IMU alignment).

Testing outcomes feed back into design revisions, forming a closed integration loop that improves reliability iteratively.

Upon successful validation, systems undergo formal verification and certification processes. Common frameworks include:

• Automotive: ISO 26262 (safety), ISO/PAS 21448 (SOTIF – Safety of Intended Functionality).
• Aerospace: DO-254 and DO-178C (hardware and software assurance).
• Marine: DNV-ST-0358 (Autonomous and Remotely Operated Vessels Certification).

Compliance ensures that systems meet functional safety, traceability, and documentation requirements for commercial or defence deployment ^[61]

While continuous integration (CI) originated in software, it is now applied to hardware development. Through hardware CI pipelines, designs are automatically:

• Simulated and synthesised (FPGA/ASIC flows).
• Deployed to prototype boards for regression testing.
• Linked with software CI systems (e.g., Jenkins, GitLab CI).

This convergence of software and hardware pipelines accelerates innovation cycles in autonomous platforms.

Supply Chain Management (SCM) refers to the strategic coordination of procurement, production, logistics, and distribution processes to ensure timely and cost-effective delivery of materials and systems ^[62]. In the context of autonomous systems, SCM strategies must handle highly specialised components, short technology cycles, and stringent quality standards. The SCOR model, developed by the Supply Chain Council (SCC), is a widely used framework for designing and evaluating supply chains ^[63].

Each phase integrates digital tools and real-time analytics to ensure supply resilience and performance traceability.

Lean SCM focuses on minimising waste (time, material, cost) across the chain while maximising value for the customer ^[64]. In autonomous system production, Lean methods include:

• Kanban scheduling for just-in-time component delivery.
• Standardised work procedures for repetitive integration steps.
• Continuous improvement (Kaizen) loops based on test feedback.

Lean thinking improves agility in responding to rapid technological changes and component obsolescence.

Recent developments have introduced Agile Supply Chain concepts, emphasising adaptability, visibility, and rapid reconfiguration ^[65]. Digital Supply Chain (DSC) technologies such as:

• IoT-based asset tracking
• Blockchain-enabled traceability
• AI-driven demand forecasting
• Digital twins of supply networks

enable dynamic optimisation and early detection of disruptions — crucial for fast-moving fields like autonomous robotics.

Supply chain risk management (SCRM) in autonomous systems involves proactive identification and mitigation of disruptions:

• Supplier diversification: multiple qualified vendors for critical components.
• Regionalisation: developing local manufacturing hubs.
• Inventory buffers: maintaining safety stock for high-risk parts.
• Scenario simulation: modelling responses to geopolitical or pandemic-related events.

AI-based SCRM tools (e.g., Resilinc, Everstream) now monitor supplier health and logistics delays in real time.

Many companies are moving toward vertical integration, controlling multiple stages of the supply chain. For instance:

• Tesla manufactures its own battery packs and AI chips.
• DJI designs in-house flight controllers and optical sensors.

This approach increases supply security and reduces dependency on third parties, though it requires substantial capital investment.

Sustainability in supply chains focuses on reducing carbon footprint, ensuring ethical sourcing, and promoting recyclability ^[66]. Key practices:

• Lifecycle assessments (LCA) for environmental impact.
• Closed-loop recycling for electronic waste.
• Supplier sustainability audits.
• ISO 14001 certification for environmental management systems.

Effective hardware integration and supply chain management are tightly interwoven. Integration depends on having high-quality, compatible components, while supply chains rely on robust feedback from integration and testing to forecast needs, reduce waste, and maintain reliability. Modern SCM frameworks, particularly Lean, Agile, and Digital models, offer strategies to make the autonomy industry more resilient, sustainable, and responsive.

[rahulrazdan]

Autonomous vehicles place extraordinary demands on their sensing stack. Cameras, LiDARs, radars, and inertial/GNSS units do more than capture the environment—they define the limits of what the vehicle can possibly know. A planner cannot avoid a hazard it never perceived, and a controller cannot compensate for latency or drift it is never told about. Sensor validation therefore plays a foundational role in safety assurance: it characterizes what the sensors can and cannot see, how those signals are transformed into machine-interpretable entities, and how residual imperfections propagate into system-level risk within the intended operational design domain (ODD).

In practice, validation bridges three layers that must remain connected in the evidence trail. The first is the hardware layer, which concerns intrinsic performance such as resolution, range, sensitivity, and dynamic range; extrinsic geometry that pins each sensor into the vehicle frame; and temporal behavior including latency, jitter, timestamp accuracy, and clock drift. The second is the signal-to-perception layer, where raw measurements are filtered, synchronized, fused, and converted into maps, detections, tracks, and semantic labels. The third is the operational layer, which tests whether the sensing system—used by the autonomy stack as deployed—behaves acceptably across the ODD, including rare lighting, weather, and traffic geometries. A credible program links evidence across these layers to a structured safety case aligned with functional safety (ISO 26262), SOTIF (ISO 21448), and system-level assurance frameworks, making explicit claims about adequacy and known limitations.

The overarching aim is not merely to pass tests but to bound uncertainty and preserve traceability. For each modality, the team seeks a quantified understanding of performance envelopes: how detection probability and error distributions shift with distance, angle, reflectivity, ego speed, occlusion, precipitation, sun angle, and electromagnetic or thermal stress. These envelopes are only useful when translated into perception key performance indicators and, ultimately, into safety metrics such as minimum distance to collision, time-to-collision thresholds, mission success rates, and comfort indices. Equally important is traceability from a system-level outcome back to sensing conditions and processing choices—so a late failure can be diagnosed as calibration drift, timestamp skew, brittle ground filtering, overconfident tracking, or a planner assumption about obstacle contours. Validation artifacts—calibration reports, timing analyses, parameter-sweep results, and dataset manifests—must therefore be organized so that claims in the safety case are backed by reproducible evidence.

The bench begins with geometry and time. Intrinsic calibration (for cameras: focal length, principal point, distortion; for LiDAR: channel angles and firing timing) ensures raw measurements are geometrically meaningful, while extrinsic calibration fixes rigid-body transforms among sensors and relative to the vehicle frame. Temporal validation establishes timestamp accuracy, cross-sensor alignment, and end-to-end latency budgets. Small timing mismatches that seem benign in isolation can yield multi-meter spatial discrepancies during fusion, particularly when tracking fast-moving actors or when the ego vehicle is turning. Modern stacks depend on this foundation: a LiDAR–camera fusion pipeline that projects point clouds into image coordinates requires both precise extrinsics and sub-frame-level temporal alignment to avoid ghosted edges and misaligned semantic labels. Calibration is not a one-off event; temperature cycles, vibration, and maintenance can shift extrinsics, and firmware updates can alter timing. Treat calibration and timing as monitorable health signals with periodic self-checks—board patterns for cameras, loop-closure or NDT metrics for LiDAR localization, and GNSS/IMU consistency tests—to catch drift before it erodes safety margins.

Validation must extend beyond the sensor to the pre-processing and fusion pipeline. Choices about ground removal, motion compensation, glare handling, region-of-interest cropping, or track-confirmation logic can change effective perception range and false-negative rates more than a nominal hardware swap. Controlled parameter sensitivity studies are therefore essential. Vary a single pre-processing parameter over a realistic range and measure how first-detection distance, false-alarm rate, and track stability evolve. These studies are inexpensive in simulation and surgical on a test track, and they surface brittleness early, before it appears as uncomfortable braking or missed obstacles in traffic. Notably, changes to LiDAR ground-filter thresholds can shorten the maximum distance at which a stopped vehicle is detected by tens of meters, shaving seconds off reaction time and elevating risk—an effect that should be measured and tied explicitly to safety margins.

Perception KPIs must be defined with downstream decisions in mind. Aggregate AUCs are less informative than scoped statements such as “stopped-vehicle detection range at ninety-percent recall under dry daylight urban conditions.” Localization health is better expressed as a time-series metric correlated with map density and scene content than as a single RMS figure. The aim is to generate metrics a planner designer can reason about when setting buffers and behaviors. These perception-level KPIs should be linked to system-level safety measures—minimum distance to collision, collision occurrence, braking aggressiveness, steering smoothness—so that changes in sensing or pre-processing can be convincingly shown to increase or decrease risk.

Miles driven is a weak proxy for sensing assurance. What matters is which situations were exercised and how well they cover the risk landscape. Scenario-based validation replaces ad-hoc mileage with structured, parameterized scenes that target sensing stressors: low-contrast pedestrians, vehicles partially occluded at offset angles, near-horizon sun glare, complex specular backgrounds, or rain-induced attenuation. Scenario description languages allow these scenes to be specified as distributions over positions, velocities, behaviors, and environmental conditions, yielding reproducible and tunable tests rather than anecdotal encounters. Formal methods augment this process through falsification—automated searches that home in on configurations most likely to violate monitorable safety properties, such as maintaining a minimum separation or confirming lane clearance for a fixed dwell time. This formalism pays two dividends: it turns vague requirements into properties that can be checked in simulation and on track, and it exposes precise boundary conditions where sensing becomes fragile, which are exactly the limitations a safety case must cite and operations must mitigate with ODD constraints.

High-fidelity software-in-the-loop closes the gap between abstract scenarios and the deployed stack. Virtual cameras, LiDARs, and radars can drive the real perception software through middleware bridges, enabling controlled reproduction of rare cases, precise occlusions, and safe evaluation of updates. But virtual sensors are models, not mirrors; rendering pipelines may fail to capture radar multipath, rolling-shutter distortions, wet-road reflectance, or the exact beam divergence of a specific LiDAR. The simulator should therefore be treated as an instrument that requires its own validation. A practical approach is to maintain paired scenarios: for a subset of tests, collect real-world runs with raw logs and environmental measurements, then reconstruct them in simulation as faithfully as possible. Compare detection timelines, track stability, and minimum-distance outcomes, and quantify the divergence with time-series metrics such as dynamic time warping on distance profiles, discrepancies in first-detection timestamps, and divergence in track IDs. The goal is not to erase the sim-to-real gap—an unrealistic aim—but to bound it and understand where simulation is conservative versus optimistic.

Because budgets are finite, an efficient program adopts a two-layer workflow. The first layer uses faster-than-real-time, lower-fidelity components to explore large scenario spaces, prune uninformative regions, and rank conditions by estimated safety impact. The second layer replays the most informative cases in a photorealistic environment that streams virtual sensor data into the actual autonomy stack and closes the control loop back to the simulator. Both layers log identical KPIs and time-aligned traces so results are comparable and transferable to track trials. This combination of breadth and fidelity uncovers corner cases quickly, quantifies their safety implications, and yields ready-to-execute test-track procedures for final confirmation.

Modern validation must encompass accidental faults and malicious interference. Sensors can be disrupted by spoofing, saturation, or crafted patterns; radars can suffer interference; GPS can be jammed or spoofed; IMUs drift. Treat these as structured negative test suites, not afterthoughts. Vary spoofing density, duration, and geometry; inject glare or saturation within safe experimental protocols; simulate or hardware-in-the-loop radar interference; and record how perception KPIs and system-level safety metrics respond. The objective is twofold: quantify degradation—how much earlier does detection fail, how often do tracks drop—and evaluate defenses such as cross-modality consistency checks, health-monitor voting, and fallbacks that reduce speed and increase headway when sensing confidence falls below thresholds. This work connects directly to SOTIF by exposing performance-limited hazards amplified by adversarial conditions, and to functional safety by demonstrating safe states under faults.

Validation produces data, but assurance requires an argument. Findings should be organized so that each top-level claim—such as adequacy of the sensing stack for the defined ODD—is supported by clearly scoped subclaims and evidence: calibrated geometry and timing within monitored bounds; modality-specific detection and tracking KPIs across representative environmental strata; quantified sim-to-real differences for critical scenes; scenario-coverage metrics that show where confidence is high and where operational mitigations apply; and results from robustness and security tests. Where limitations remain—as they always do—they should be stated plainly and tied to mitigations, whether that means reduced operational speed in heavy rain beyond a specified attenuation level, restricted ODD where snow eliminates lane semantics, or explicit maintenance intervals for recalibration.

A final pragmatic recommendation is to treat validation data as a first-class product. Raw logs, configuration snapshots, and processing parameters should be versioned, queryable, and replayable. Reproducibility transforms validation from a hurdle into an engineering asset: when a perception regression appears after a minor software update, the same scenarios can be replayed to pinpoint the change; when a new sensor model is proposed, detection envelopes and safety margins can be compared quickly and credibly. In this way, the validation of perception sensors becomes a disciplined, scenario-driven program that ties physical sensing performance to perception behavior and ultimately to system-level safety outcomes, while continuously informing design choices that make the next round of validation faster and more effective.

[rahulrazdan][✓ rahulrazdan, 2025-06-16]

Figure 1

As discussed in previously, there is an intimate interaction between laws, regulations, and technology to build a governance framework which assigns liability and manages shared resources. In the case of the mechanical world, laws and regulations in transportation connect to traffic laws and the traffic infrastructure. With the development of wireless communications, sensing, and now wireless power, there is a need to build a governance structure for this space. In the US, the primary legal basis was the communication act passed in 1934 which created the regulator (Federal Communications Commission [FCC]). The FCC manages the radio spectrum (figure 1) through a range of regulatory and technical actions to ensure its efficient and interference-free use. It allocates specific frequency bands for various services—such as broadcasting, cellular, satellite, public safety, and amateur radio—based on national needs and international agreements. The FCC issues licenses to commercial and non-commercial users, setting terms for power limits, coverage areas, and operating conditions. It also conducts spectrum auctions to assign frequencies for commercial use, such as 5G, while reserving portions for public services and unlicensed uses like Wi-Fi. In addition, the FCC enforces rules to prevent harmful interference, coordinates spectrum sharing and repurposing efforts, and leads initiatives like dynamic spectrum access and band reallocation to adapt to evolving technological demands. To enforce these standards, the FCC requires many devices to undergo testing and certification before they can be marketed or sold in the United States. This process is carried out by FCC-recognized testing laboratories, known as accredited Conformity Assessment Bodies (CABs), which evaluate products against applicable Part 15 or Part 18 regulations, among others. Certified devices must meet limits on emissions, immunity, and specific absorption rate (SAR) when applicable. Once a product passes testing, the lab submits a report to a Telecommunications Certification Body (TCB), which issues the FCC ID and authorizes the product for sale. These labs play a critical role in ensuring compliance, supporting innovation while maintaining spectrum integrity and public safety.

FCC Part 15 and Part 18 differ primarily in the type and purpose of radio frequency (RF) emissions they regulate. Part 15 governs devices that intentionally or unintentionally emit RF energy for communication purposes, such as Wi-Fi routers, Bluetooth devices, and computers. These devices must not cause harmful interference and must accept interference from licensed users. In contrast, Part 18 regulates Industrial, Scientific, and Medical (ISM) equipment that emits RF energy not for communication, but for performing physical functions like heating, welding, or medical treatments—examples include microwave ovens and RF diathermy machines. While both parts limit electromagnetic interference, Part 15 devices operate under stricter emissions limits due to their proximity to communication bands, whereas Part 18 devices are allowed higher emissions in designated ISM frequency bands. Additionally, health and safety regulations for Part 18 equipment are typically overseen by other agencies such as the FDA or OSHA, while the FCC focuses on interference mitigation.

Figure 2

A key instrument for electromagnetic testing is an anechoic chamber (figure 2). An anechoic chamber is a specialized, sound- and radio wave-absorbing enclosure designed to create an environment free from reflections and external interference. Its walls, ceiling, and floor are typically lined with wedge-shaped foam or ferrite tiles that absorb electromagnetic or acoustic waves, depending on the application. For radio frequency (RF) testing, the chamber is constructed with conductive materials (like steel or copper) to form a Faraday cage, isolating it from external RF signals. In acoustic chambers, sound-absorbing foam eliminates echoes and simulates free-field conditions. Anechoic chambers are critical in industries such as telecommunications, defense, aerospace, and consumer electronics, where they are used to test antenna performance, electromagnetic compatibility (EMC), emissions compliance, radar systems, or audio equipment in highly controlled, repeatable conditions. The chamber ensures that test measurements reflect only the characteristics of the device under test (DUT), without environmental interference

What are the implications for automakers ?

In modern vehicles, electronics are no longer confined to infotainment or engine control—sensors, communication modules, and controllers are now central to vehicle safety and performance. These systems emit and receive electromagnetic energy, which can result in electromagnetic interference (EMI) if not properly managed. EMI can compromise safety-critical applications like radar-based adaptive cruise control or camera-based lane keeping. Sensor technologies introduce unique EMI challenges. Radar and lidar sensors, which are critical for driver assistance and autonomous systems, must not only avoid interference with each other but must also operate within spectrum allocations defined by the FCC and global bodies like the ITU. Similarly, cameras and ultrasonic sensors are susceptible to noise from nearby power electronics, especially in electric vehicles. EMI from poorly shielded cables or high-frequency switching components can cause data corruption, missed detections, or degraded signal integrity—raising both functional safety and regulatory concerns.

From a communications standpoint, FCC-compliant system design must also consider interoperability and coexistence. In a vehicle packed with Bluetooth, Wi-Fi, GPS, DSRC or C-V2X, and cellular modules, maintaining RF harmony requires careful frequency planning, shielding, and filtering. The FCC’s evolving rules for the 5.9 GHz band—reallocating portions from DSRC to C-V2X—illustrate how regulatory frameworks directly impact product architecture. OEMs must track these developments and validate that their communication modules not only operate within approved frequency bands but also do not emit spurious signals that could violate FCC emission ceilings.

To meet FCC standards while ensuring high system reliability, automotive developers must embed EMI considerations early in the design cycle. Pre-compliance testing, EMI-aware PCB layout, and component-level certification all contribute to a smoother path to regulatory approval. Moreover, aligning FCC requirements with international automotive EMC standards—like CISPR 25 and UNECE R10—helps ensure global market readiness. As vehicles grow increasingly software-defined, connected, and autonomous, managing EMI through smart engineering and regulatory foresight will be a critical enabler of innovation, safety, and compliance.

As discussed, FCC regulations are primarily focused on electromagnetic interference. However, if RF energy has the potential to cause health issues, other regulators are involved. Health and safety regulation for FCC Part 18 devices—such as microwave ovens and medical RF equipment—is primarily handled by agencies. The Food and Drug Administration (FDA) oversees radiation-emitting electronic products to ensure they meet safety standards for human exposure, particularly for consumer appliances and medical devices. The Occupational Safety and Health Administration (OSHA) establishes workplace safety limits for RF exposure to protect employees who operate or work near such equipment. Meanwhile, the National Institute for Occupational Safety and Health (NIOSH) conducts research and provides guidance on safe RF exposure levels in occupational settings. While the FCC regulates RF emissions from Part 18 devices to prevent interference with licensed communication systems, it relies on these other agencies to ensure that the devices do not pose health risks to users or workers.

In the case of vehicle makers, part 18 health issues manifest themselves in use-models such as wireless power delivery where SAR levels may impact safety directly.

Figure 3

Finally, while the examples used above are from a US context, similar structures exist in all other geographies.

[rahulrazdan][✓ rahulrazdan, 2025-06-16]

In product development, the initial focus is on functionality and differentiated value. As discussed in the governance sections, the next stage is to make sure the product conforms within the appropriate regulatory frameworks connected to safety and shared usage. The final stage and perhaps the most important stage is that of consistently delivering and supporting the product in the marketplace. To consistently deliver the product, one must manage the supply chain which drives the forward delivery of the product. In addition, as customers interact with the product, there is a reverse flow which involves reparability, diagnostics, and in most situations safe disposal. Figure 1 below shows these flows. Finally, the connection of the product to the environment must be adjusted periodically in a process known as calibration.

Figure 1

For most products, the mechanical component supply chain, maintenance, and calibration have a well-formed rich history. Today, the big megatrend which is impacting all of these aspects of product designs is the introduction of electronics (hardware, software, and now AI). We will now consider the impact of electronics on the product delivery function. The underlying components for electronics are semiconductors. The semiconductor industry is driven by several important underlying factors: Lithography dominates the cost of advanced process nodes in semiconductors, which has a massive impact on the cost of designing and building a modern manufacturing facility. The cost of lithography also has an impact on design where the size of design to be built, the associated electronic design automation tools, and mask sets also scale up with advanced nodes. Additionally, given the costs, the economics of semiconductor design is that custom advanced digital semiconductors only make economic sense for markets with high volume. This hard economic fact has optimized the semiconductor industry towards consumer-driven short lifecycle products. Thus, in these very high-volume consumer markets, large numbers of staff from semiconductor companies work with their system counterparts to effectively co-design the final product.

As the process reaches advanced nodes, the opportunity cost of maintaining older designs and fabrications becomes very high. Thus, most modern semiconductor vendors start end-of-life processes for parts on older process nodes. Various governments which want to start semiconductor operations build capability on lagging nodes. Today, the massive investment by China on semiconductor fabrication has created a near monopoly for lagging nodes. This process has the potential to provide an immense supply of parts but creates an external, perhaps unreliable, dependency. Even worse, the lack of supply creates a dark market which enables counterfeiting and sometimes even security issues.

Lastly, in general, chips are built for the reliability of the consumer marketplace and specialty process changes are expensive and rare. The combination of these factors results in a situation where the consumer marketplace dominates the definition, investment, and specifications for the semiconductor industry. As a simple illustration of scale, Apple itself spends more on research and development than the top defense prime contractors combined. We broadly refer to the non-consumer marketplace as long lifecycle (LLC) markets, and the divergences between short lifecycle and LLC are listed in Table 1.

Table 1. Short lifecycle versus LLC products.

Meanwhile, software (and now AI) are key drivers of value for products with two critical features: OTA updates and open-source development models. As discussed earlier, OTA processes provides a very powerful capability to dynamically update the product in the field but create an issue for V&V as well as cybersecurity. In terms of the supply chain, software components become part of the supply chain and must be managed carefully. Open-source communities are the second powerful force because they allow for crowdsourcing of innovation. Open-source systems (e.g., Linux) have become dominant despite the efforts of major industrial players or even countries. Open-source environments provide enormous value and often cannot be ignored, yet the challenge for safety-focused, cyber-physical systems is the ability to manage the issues of V&V, cybersecurity, and the software supply chain. As these megatrends from the IT world crash like big waves into the traditional LLC world, the consequences are profound in many dimensions (i.e., industry structure, economic incentives, geopolitical factors). To illustrate this, Table 2 outlines the well-known supplier structure for the automotive industry with OEMs and Tier 1, 2, and 3 suppliers.

Table 2. Traditional automotive supply chain.

Today, OEMs consume whole subsystems from Tier 1 vendors who are providing an increasing amount of electronics content. Since the systems are built separately, the integrated product is an accumulation of the various systems provided by the vendors. The result is an explosion in semiconductor product skews and large complexity in the software stack. The consequences include a massive exposure to obsolescence for semiconductors and software, higher cost due to a lack of integration, and a dearth of future flexibility. Meanwhile, the automotive industry is moving rapidly towards the concept of a Software Defined Vehicles (SDV) with a release structure very similar to IT server products.

Traditionally, for new products, the automotive industry has focused on concepts of lean manufacturing and “just-in-time” inventory management which prioritizes minimizing inventory levels at all stages of production. In a world dominated by OEM-driven demand, this paradigm worked reasonably well. However, with the accelerated usage of electronics, automotive OEMs increasingly find themselves managing a supply chain where they are the minority drivers for demand, or worse—the primary drivers for legacy semiconductors. This effect was most pronounced following the COVID-19 pandemic when the supply for semiconductors could not keep up with the demand.

Further, much like the US Department of Defense, automotive companies traditionally require chips with automotive grade certification. Automotive-grade components require stringent compliances. (Passive components need AEC Q200, ASILI/ISO 26262 Class B, IATF 16949 qualification while active components, including automotive chips, should be compliant with AEC Q100, ASILI/ISO 26262 Class B, IATF 16949 standards.) However, these requirements are not embraced by the much bigger consumer marketplace, and the divergence imposes a large constraint on the automotive supply chain. For similar reasons, the Department of Defense responded to this reality with an aggressive commercial off-the-shelf (COTS) approach [43]. From a supply chain management perspective, LLC designers must manage the issues of supply chain by employing techniques such as the following:

1. Parts obsolescence: A smaller number of programmable parts minimize inventory skews and the aggregation of function around a small number of programmable parts raises the volume of these parts and thus minimizes the chances for parts obsolescence.
2. Redundancy for reliability: Reliability can be greatly enhanced using redundancy within and of multiple programmable devices. Similar to storage via a redundant array of independent disks, one can leave large parts of a programmable fabric unprogrammed, and dynamically move functionality based on detected failures.
3. Future function: Programmability enables the use of OTA updates to adjust functionality dynamically. This is critical for building strong aftersales business models and remote maintenance.

In terms of supply chain and software, the traditional aerospace sector has commonly lagged behind the automotive sector in terms of absorption and impact of electronics. Much like automotive, airborne systems had been absorbing semiconductors from legacy nodes and largely working with proprietary software. However, the introduction of AI for aircraft safety systems—and certainly the newer form factors of drones and delivery vehicles—has accelerated this shift. From a semiconductor perspective, markets such as airborne, marine, and space systems must largely use standard parts, typically at a very low level of integration, to build final system function. The result is higher cost and lower performance. Finally, the increased use of electronics for sensors has introduced an additional requirement for in-field calibration.

put your contents here

[karlisberkolds]

Modern autonomous systems — from self-driving cars and unmanned aerial vehicles (UAVs) to marine robots and industrial co-bots — depend fundamentally on software architectures capable of real-time sensing, decision-making, and control. While mechanical and electronic components define what a system can do, the software stack defines how it does it — how it perceives the world, interprets data, plans actions, and interacts safely with its environment ^[67,68]. Autonomy software differs from conventional embedded or enterprise software in several critical ways:

• It operates under strict real-time constraints.
• It must integrate data from heterogeneous sensors.
• It needs robust fault tolerance and safety compliance.
• It supports continuous learning and adaptation through AI.
• It often spans distributed systems, connecting vehicles, edge servers, and cloud services.

This combination of safety-critical engineering and AI-driven decision-making makes autonomy software one of the most challenging areas in modern computing.

Autonomy software must achieve four key functional objectives ^[69,70]:

• Perception – sensing and interpreting the environment (via LiDAR, radar, cameras, sonar, etc.).
• Localisation – estimating the system’s precise position and orientation in the world.
• Planning – generating paths or behaviours that fulfil mission goals while avoiding obstacles.
• Control – executing actions safely and stably, compensating for environmental changes.

Each of these objectives corresponds to distinct software layers and modules in the autonomy stack.

These characteristics drive architectural decisions and the choice of frameworks (e.g., ROS, AUTOSAR Adaptive, DDS).

Autonomy software is layered, combining multiple software technologies:

• Low-level embedded firmware (real-time control and drivers).
• Middleware and communication frameworks (data exchange and scheduling).
• High-level AI algorithms (perception, decision-making).
• Supervisory or cloud-level systems (fleet management, updates, data analytics).

The combination of these layers forms the autonomy software stack, which enables complex behaviour while maintaining reliability. A defining aspect of autonomy software is its reliance on middleware — frameworks that manage interprocess communication (IPC), data distribution, and time synchronisation across distributed computing nodes. Some of the widely used standards:

• ROS / ROS 2 (Robot Operating System): Modular publish-subscribe communication.
• DDS (Data Distribution Service): Real-time, QoS-based messaging standard.
• MQTT / ZeroMQ: Lightweight protocols for cloud and IoT integration.

A complete software stack is a layered collection of software components, frameworks, and libraries that work together to deliver a complete set of system functionalities. Each layer provides services to the layer above it and depends on the layer below it. Middleware, which is an essential part of the multi-layered architectures, ensures that all layers of the software stack can exchange information deterministically and safely ^[71]. In autonomous systems, the software stack enables integration between:

• Hardware abstraction (sensors, actuators, compute units),
• Middleware (communication and data exchange),
• Application-level modules (AI perception, planning, control), and
• System management (diagnostics, simulation, fleet updates).

It’s the backbone that allows autonomy to function as a cohesive system rather than a set of disconnected modules (Quigley et al., 2009; Maruyama et al., 2016). From a technical perspective, the software stack defines how functionality, data flow, and control are structured within the system.

Each layer isolates complexity by providing a clean interface to the one above.

• Developers can modify one layer (e.g., perception algorithms) without altering lower layers (e.g., drivers).
• Enables easier testing, debugging, and reuse across multiple vehicles or applications.

Autonomous systems rely on real-time responses. The stack architecture ensures:

• Timely communication between perception, planning, and control.
• Deterministic scheduling using RTOS kernels or real-time Linux patches ^[72].
• Synchronisation of data streams from multiple sensors using timestamping and time-sensitive networking (TSN).

Middleware such as ROS 2 or DDS standardises interprocess communication. This allows different vendors’ software modules (e.g., LiDAR driver from Company A and planner from Company B) to work together.

Stack layering supports redundant paths for safety-critical functions. If a perception node fails, a backup process may take over seamlessly — ensuring resilience, especially in aerospace and automotive systems ^[73].

A layered design allows developers to:

• Integrate software components progressively.
• Test them using Hardware-in-the-Loop (HIL) and Software-in-the-Loop (SIL) methods.
• Use simulators (e.g., CARLA, Gazebo, AirSim) to validate upper-layer modules without risking hardware damage.

From a software engineering management perspective, a defined software stack provides structure and governance for the development process, which provides the following main advantages: Division of Labour. Teams can specialise by layer — e.g., one group handles perception, another control, another middleware. This parallelises development and allows use of domain expertise without interference.

Reusability and Version Control Reusable modules and APIs speed up development. Tools like Git, Docker, and CI/CD pipelines ensure traceability, maintainability, and fast updates across distributed teams.

Scalability and Lifecycle Management A well-structured stack can be extended with new sensors or algorithms without re-architecting the entire system. Lifecycle management tools (e.g., ROS 2 launch systems, AUTOSAR Adaptive manifests) maintain version consistency and dependency control.

Quality Assurance (QA) and Certification Layered software stacks make it easier to apply quality control and compliance frameworks, such as: ISO 26262 (Automotive safety software), DO-178C (Aerospace software) or IEC 61508 (Functional safety in automation). Each layer can be validated separately, simplifying documentation and certification workflows.

Cost and Risk Reduction When multiple projects share a unified software stack, the cost of testing, validation, and maintenance drops significantly. This approach underpins industry-wide initiatives like AUTOSAR, which standardises vehicle software to lower integration costs.

In large autonomy projects (e.g., Waymo, Tesla), the software stack also serves as an organisational structure. Teams are aligned with layers:

• Perception team handles sensor fusion and computer vision.
• Planning & Control team develops behavioural logic and trajectory generation.
• Systems team manages middleware and data distribution.
• Infrastructure team maintains OS builds, simulation, and DevOps pipelines.

Thus, the software stack doubles as both a technical architecture and an organisational map for coordination and accountability ^[74].

The Robot Operating System 2 (ROS 2) exemplifies how modular software stacks are implemented:

• Application Layer: Navigation, mapping, perception nodes.
• Middleware Layer: DDS for data exchange, QoS configuration.
• OS Layer: Linux or RTOS with POSIX APIs.
• Hardware Abstraction: Drivers for cameras, IMUs, LiDARs.
• Build & Deployment Layer: CMake, Colcon, Docker for reproducibility.

This layered model has become the foundation for numerous autonomous systems in academia and industry — from mobile robots to autonomous vehicles ^[75]).

In essence, a software stack is not merely a technical artefact — it’s a strategic enabler that aligns engineering processes, organisational structure, and long-term sustainability of autonomous platforms. Autonomy software stack and Development and Maintenance challenges are discussed in the following chapters:

A typical autonomy software stack is organised into hierarchical layers, each responsible for a specific subset of functions — from low-level sensor control to high-level decision-making and fleet coordination. Although implementations differ across domains (ground, aerial, marine), the core architectural logic remains similar:

• Perception Layer – sensing and understanding the environment.
• Localisation Layer – determining position and orientation.
• Planning Layer – deciding what actions to take.
• Control Layer – executing those actions through actuators.
• System Layer – managing communication, hardware, and runtime.
• Infrastructure Layer – providing simulation, cloud services, and DevOps.

This layered design aligns closely with both robotics frameworks (ROS 2) and automotive architectures (AUTOSAR Adaptive).

In Figure 1, the main software layers and their functions are depicted.

Hardware Abstraction Layer (HAL) The HAL provides standardised access to hardware resources. It translates hardware-specific details (e.g., sensor communication protocols, voltage levels) into software-accessible APIs. This functionality typically includes:

• Managing device drivers for LiDARs, radars, cameras, IMUs, and other devices needed for the autonomous system.
• Monitoring power systems and diagnostics, which includes behaviour change triggers if needed.
• Providing time-stamped sensor data streams. The real-time or time-stamped data is the essential part of any data-driven system to ensure proper work of time-series analysis algorithms including deep learning methods.
• Handling real-time control of actuators (motors, servos, brakes).

HAL ensures portability — software modules remain agnostic to specific hardware vendors or configurations ^[78].

Operating System (OS) and Virtualisation Layer The OS layer manages hardware resources, process scheduling, and interprocess communication (IPC) as well as real-time operation, alert and trigger raising using watchdog processes. Here, data processing parallelisation is one of the keys to ensuring resources for time-critical applications. Autonomous systems often use:

• Linux (Ubuntu or Yocto-based) for flexibility.
• Real-Time Operating Systems (RTOS) like QNX or VxWorks for safety-critical timing.
• Containerization (Docker, Podman) for software isolation and modular deployment.

Time-Sensitive Networking (TSN) extensions and PREEMPT-RT patches ensure deterministic scheduling for mission-critical tasks ^[79].

Middleware / Communication Layer The middleware layer serves as the data backbone of the autonomy stack. It manages communication between distributed software modules, ensuring real-time, reliable, and scalable data flow. IN some of the mentioned architectures middleware is the central distinctive feature of the architecture. Popular middleware technologies:

• ROS 2 (Robot Operating System 2): Uses DDS (Data Distribution Service) for modular publish–subscribe communication.
• DDS (Data Distribution Service): Industry-standard middleware for real-time, QoS-based data exchange.
• CAN, Ethernet, MQTT: Used for in-vehicle and external communication.
• Logging and telemetry systems (ROS bags, DDS recorders).
• Fault detection and recovery mechanisms.
• QoS (Quality of Service) configuration for bandwidth and latency management.

Control & Execution Layer The control layer translates planned trajectories into actuator commands while maintaining vehicle stability. It closes the feedback loop between command and sensor response. Key modules:

• Low-level control: PID, LQR, or MPC controllers for steering, throttle, braking, or propulsion.
• High-level control: Converts trajectory plans into real-time setpoints.
• State estimation: Uses Kalman filters or observers to correct control deviations.

Safety-critical systems often employ redundant controllers and monitor nodes to prevent hazardous conditions ^[80].

Autonomy Intelligence Layer This is the core of decision-making in the stack. It consists of several interrelated subsystems:

These components interact through middleware and run either on edge computers (onboard) or cloud-assisted systems for extended processing ^[81].

Application & Cloud Layer At the top of the stack lies the application layer, which extends autonomy beyond individual vehicles:

• Fleet management (monitoring, task assignment).
• Over-the-air (OTA) updates for software and firmware.
• Cloud-based simulation and analytics.
• Data collection for machine learning retraining.

Frameworks like AWS RoboMaker, NVIDIA DRIVE Sim, and Microsoft AirSim bridge onboard autonomy with cloud computation.

Autonomy systems rely on data pipelines that move information between layers in real time.

Each stage includes feedback loops to ensure error correction and safety monitoring ^{[82, 83]}.

ROS 2-Based Stack (Research and Prototyping)

• Used in academic and industrial R&D.
• Flexible and modular, ideal for simulation and experimental platforms.
• Integration with Gazebo, RViz, and DDS middleware.

AUTOSAR Adaptive Platform (Automotive)

• Industry-grade framework for production vehicles.
• Service-oriented architecture with real-time OS and safety mechanisms.
• Supports ISO 26262 compliance and multi-core systems.

MOOS-IvP (Marine Autonomy)

• Middleware focused on marine robotics.
• Behaviour-based architecture with mission planning (IvP Helm).
• Optimised for low-bandwidth communication and robustness ^[84].

Hybrid Cloud-Edge Architectures

• Combine onboard autonomy with cloud processing (for model training or high-level optimisation).
• Used in large-scale fleet operations (e.g., logistics robots, aerial mapping).
• Requires secure communication channels and data orchestration ^[85].

This closed-loop data exchange ensures real-time responsiveness, robust error recovery, and cross-module coherence.

Developing and maintaining an autonomous software stack is a long-term, multidisciplinary endeavour. Unlike conventional software, autonomy stacks must handle:

• Continuous real-time operations,
• Massive sensory data streams,
• Hardware dependencies, and
• Strict safety, security, and regulatory constraints.

These constraints make the software lifecycle for autonomy uniquely complex — spanning from initial research prototypes to industrial-grade, certified systems.

Even with knowledge of autonomous software stacks, their development is still associated with significant and challenging problems. Through their mitigation and applications of different solutions, the autonomous systems become both expensive to design and develop as well as hard to maintain. The following are the most significant challenges.

Real-Time Performance and Determinism Autonomous systems require deterministic behaviour: decisions must be made within fixed, guaranteed time frames. However, high computational demands from AI algorithms often conflict with real-time guarantees ^[86]. Key Issues:

• Latency from AI inference (e.g., deep neural networks).
• Non-deterministic middleware scheduling.

Timing mismatches across sensor and control loops. Mitigation:

• Use of real-time operating systems (RTOS), priority-based scheduling, and hardware acceleration (FPGAs, TPUs).
• Middleware with Quality of Service (QoS) guarantees, like DDS.

Scalability and Software Complexity As systems evolve, the number of nodes, processes, and data streams grows exponentially. For instance, a modern L4 autonomous vehicle may contain >200 software nodes exchanging gigabytes of data per second. Problems:

• Dependency conflicts between packages.
• Increasing memory and bandwidth requirements.
• Complexity of debugging distributed systems.

Solutions:

• Modular, microservice-based architectures.
• Container orchestration (Docker, Kubernetes).
• Digital twin platforms for system-level simulation and validation ^[87].

Integration of AI and Classical Control AI-based perception and classical control must coexist smoothly. While AI modules (e.g., neural networks) handle high-dimensional perception, classical modules (e.g., PID, MPC) ensure predictable control. Challenge:

• Integrating data-driven and rule-based components introduces uncertainty, interpretability issues, and difficulties in certification ^[88].

Best Practices:

• Use hybrid architectures combining interpretable models with learned features.
• Introduce runtime monitors for anomaly detection and fallback behaviours.
• Implement explainable AI (XAI) for safety audits.

Safety, Verification, and Certification Autonomous systems must conform to standards like the mentioned ISO 26262 (automotive functional safety), DO-178C (aerospace software certification) and IEC 61508 (industrial safety). Challenges:

• AI systems lack deterministic traceability.
• Validation of all operational scenarios is practically impossible.
• Continuous software updates complicate certification cycles.

Emerging Solutions:

• Simulation-based verification using virtual environments.
• Formal verification of decision modules (model checking, theorem proving).
• Modular certification frameworks (AUTOSAR Adaptive, Safety Element out of Context – SEooC).

Cybersecurity and Software Integrity Autonomous platforms are connected via V2X, cloud APIs, and OTA updates — creating multiple attack surfaces ^[89]. Risks:

• Compromised firmware or middleware components.
• Spoofed sensor inputs (GPS, LiDAR).
• Supply chain vulnerabilities (counterfeit software libraries).

Countermeasures:

• Secure boot and hardware root-of-trust mechanisms.
• Encrypted communication (TLS, DDS Secure).
• Software Bill of Materials (SBOMs) for dependency tracking.
• Compliance with NIST SP 800-161 and ISO/IEC 27036 standards.

Continuous Maintenance and Updates Unlike static embedded systems, autonomy software evolves continuously. Developers must maintain compatibility across versions, hardware platforms, and fleets already deployed in the field. Maintenance Practices:

• Continuous Integration/Continuous Deployment (CI/CD) pipelines for testing and rolling updates.
• Over-the-Air (OTA) update frameworks for vehicles and drones.
• Configuration Management Databases (CMDBs) to track software–hardware combinations.
• Digital twins to test updates before live deployment.

Data Management and Scalability AI-driven autonomy relies on vast datasets for training, simulation, and validation. Managing, labelling, and securing this data is an ongoing challenge ^[92]. Issues:

• Storage and transfer of multi-terabyte sensor data.
• Bias and imbalance in datasets.
• Traceability of model versions and training data.

Approaches:

• Cloud data lakes with edge pre-processing.
• MLOps workflows for dataset versioning and reproducibility.
• Federated learning for privacy-preserving model updates.

Human–Machine Collaboration and Ethical Oversight Autonomy software doesn’t exist in isolation — it interacts with human operators, passengers, and society. Thus, software design must incorporate transparency, accountability, and explainability. Key Considerations:

• Human–machine interface (HMI) design.
• Ethical AI decision frameworks.
• Liability and failover protocols during edge cases.

The software lifecycle typically follows a continuous evolution model:

The goal is continuous evolution with stability, where systems can adapt without losing certification or reliability.

The software lifecycle defines the complete process by which software is conceived, developed, deployed, maintained, and eventually retired. In the context of modern engineering — particularly for complex systems such as autonomous platforms, embedded systems, or enterprise solutions — understanding the lifecycle is essential to ensure quality, reliability, and maintainability. The lifecycle acts as a roadmap that guides project teams through stages of development and management. Each stage defines specific deliverables, milestones, and feedback loops, ensuring that the software evolves in a controlled, traceable, and predictable way ^[93].

“The software lifecycle refers to a structured sequence of processes and activities required to develop, maintain, and retire a software system.” — ^[94] In other words, the lifecycle describes how a software product transitions from idea to obsolescence — incorporating all the engineering, management, and maintenance steps along the way. The lifecycle ensures:

• Consistency: A common framework for teams and stakeholders.
• Quality assurance: Enables verification and validation at each stage.
• Traceability: Establishes clear relationships between requirements, design, code, and tests.
• Risk management: Provides checkpoints to detect and correct issues early.
• Scalability: Supports integration of multiple teams, technologies, and versions.

In regulated domains like aerospace, automotive, and medical devices, adherence to a defined lifecycle is also a legal requirement for certification and compliance (e.g., ISO/IEC 12207, DO-178C, ISO 26262).

The main lifecycle models, Configuration management and configuration management tools are discussed in the following chapters:

Different industries and projects adopt specific lifecycle models based on their goals, risk tolerance, and team structure. The most widely used models are explained in this chapter.

The Waterfall Model is one of the earliest and most widely recognised software lifecycle models. It follows a linear sequence of stages where each phase must be completed before the next begins ^[95].

Advantages:

• Clear structure and documentation.
• Easy to manage for small, stable projects.
• Suitable for regulated environments (e.g., aerospace, defence).

Limitations:

• Inflexible to changes once development begins.
• Late discovery of integration or requirement issues.

An evolution of the waterfall approach, the V-Model emphasises testing and validation at each development stage. Each “downward” step (development) has a corresponding “upward” step (testing/validation).

Advantages:

• Strong focus on verification and validation (V&V).
• Ideal for safety-critical systems (e.g., ISO 26262, DO-178C).
• Provides traceability between design and testing phases.

Limitations:

• Requires well-defined requirements upfront.
• Difficult to adapt to rapid changes.

Instead of completing the whole system in one sequence, the iterative model develops the product through multiple cycles or increments. Each iteration delivers a working version that can be reviewed and refined. Advantages:

• Early delivery of functional prototypes.
• Easier adaptation to requirement changes.
• Continuous stakeholder feedback.

Limitations:

• Higher integration overhead.
• May require complex configuration management (each iteration produces new versions).

Agile development (e.g., Scrum, Kanban, Extreme Programming) emphasises collaboration, adaptability, and customer feedback. It replaces rigid processes with iterative cycles known as sprints.

Core Principles ^[96]:

• Individuals and interactions over processes and tools.
• Working software over comprehensive documentation.
• Customer collaboration over contract negotiation.
• Responding to change by following a plan.

Advantages:

• High flexibility and customer involvement.
• Continuous delivery of value.
• Improved responsiveness to market and technology changes.

Challenges:

• Requires disciplined teams and strong communication.
• Less suitable for safety-critical certification unless paired with hybrid models (e.g., Agile + V-Model).

Introduced by Boehm ^[97], the Spiral Model combines iterative development with risk analysis. Each loop of the spiral represents one phase of the process, with risk evaluation at its core.

Advantages:

• Focused on risk reduction.
• Suitable for large, complex systems.
• Allows progressive refinement and flexibility.

Limitations:

• Complex management and documentation.
• Requires expertise in risk assessment.

Modern systems increasingly adopt DevOps — integrating development, testing, deployment, and operations into a continuous cycle. This model leverages automation, CI/CD pipelines, and cloud-native

Advantages:

• Rapid and reliable delivery.
• Real-time monitoring and feedback integration.
• Continuous improvement of deployed systems.

Challenges:

• Requires cultural and organisational transformation.
• Demands sophisticated toolchains and automation infrastructure.

= Comparative Overview =

In software engineering, Configuration Management (CM) refers to the systematic process of identifying, organising, controlling, and tracking all changes made to a software system throughout its lifecycle. It ensures that:

• The correct versions of software components are used.
• Changes are controlled, reviewed, and documented.
• The entire system remains consistent and reproducible across teams and environments.

According to ISO/IEC/IEEE 828:2012, CM is defined as: “A discipline applying technical and administrative direction and surveillance to identify and document the functional and physical characteristics of a configuration item, control changes to those characteristics, and record and report change processing and implementation status.”

In other words, Configuration Management keeps the software stable while it evolves. Configuration management exists to:

• Ensure traceability – every change can be traced to its origin (e.g., requirement, issue report, or design change).
• Prevent chaos – without CM, multiple developers could overwrite each other’s work or deploy incompatible versions.
• Enable collaboration – teams distributed globally can work on the same product using consistent artefacts.
• Maintain compliance – in safety-critical domains (automotive, aerospace), regulatory standards require version control and change tracking.
• Support automation – CI/CD pipelines rely on version-controlled repositories and configuration metadata.

To understand CM, several foundational terms must be defined.

Configuration Item (CI) A Configuration Item is any component of the system that is subject to configuration control. Examples include:

• Source code files
• Build scripts and binaries
• Databases and configuration files
• Test scripts and reports
• Documentation and requirement specifications
• Firmware or deployed container images

Each CI is uniquely identified, versioned, and tracked over time ^[100].

Baseline A baseline is a formally approved version of one or more configuration items that serves as a reference point. Once established, any changes to the baseline must follow a defined change control process. Types of baselines:

• Functional baseline – system requirements approved for development.
• Allocated baseline – subsystem design completed and approved.
• Product baseline – tested and released version ready for delivery.

Baselines create stability checkpoints in the lifecycle ^[101].

Version Control Version control systems (VCS), such as Git, Mercurial, or Subversion, track and manage modifications to source code and other files. They enable:

• Team collaboration.
• Historical traceability.
• Branching and merging for feature development.

Version control forms the technical backbone of configuration management.

Change Management Change management defines how modifications are proposed, evaluated, approved, and implemented. Typical steps:

• Request – a developer or stakeholder submits a change request (CR).
• Impact analysis – assess implications on cost, schedule, and performance.
• Approval – change control board (CCB) reviews and authorises.
• Implementation and verification – perform and test the change.
• Documentation and closure – record and archive results.

This structured approach ensures accountability and quality control ^[102].

Configuration Audit A configuration audit verifies that the configuration items and documentation:

• Match the approved baseline.
• Have passed required verification steps.
• Are properly labelled and traceable.

Two common types:

• Functional Configuration Audit (FCA): Confirms functional requirements are met.
• Physical Configuration Audit (PCA): Confirms physical configuration matches documentation.

Audits maintain integrity and compliance, especially in defence and aerospace projects ^[103].

Even though CM brings structure and order, it faces numerous practical challenges, particularly in distributed and complex systems.

Complexity and Scale Modern systems can contain millions of lines of code, hundreds of dependencies, and multiple configurations for different platforms. Managing all these variations manually is infeasible. Example: An autonomous vehicle might include distinct configurations for:

• Development simulations
• Real-time embedded control
• Cloud analytics backend

Solution: Automated configuration management with metadata-driven tools (e.g., Ansible, Puppet, Kubernetes Helm).

Multiple Development Streams In large projects, teams work on multiple branches or versions simultaneously (e.g., development, testing, release). This increases the risk of:

• Code divergence and merge conflicts.
• Dependency mismatches.
• Integration bottlenecks.

Solution:

• Enforce branching strategies (GitFlow, trunk-based development).
• Integrate CI/CD pipelines for automated testing and builds.

Hardware–Software Interdependencies In embedded or cyber-physical systems, configurations depend on hardware variants (processors, sensors, memory). Maintaining alignment between software builds and hardware specifications is difficult. Mitigation:

• Maintain configuration matrices mapping software versions to hardware variants.
• Employ digital twins for verification ^[104].

Frequent Updates and Continuous Delivery In the DevOps era, software may be updated multiple times per day across thousands of devices. Each update must maintain consistency and rollback capability. Challenge:

• Balancing speed (Agile/DevOps) with control (safety and certification).

Solution:

• Versioned deployment pipelines.
• Canary releases and A/B testing.
• Immutable infrastructure (containers and images stored as CIs).

Data and Configuration Drift Configuration drift occurs when the system’s actual state deviates from its documented configuration — common in dynamic, cloud-based systems. Causes:

• Manual changes bypassing automation.
• Untracked dependencies.
• Environment-specific overrides.

Prevention:

• Use Infrastructure as Code (IaC) for full traceability.
• Periodic configuration audits and compliance scanning (e.g., Chef InSpec, AWS Config).

Regulatory and Compliance Demands In domains like aerospace, medical, and automotive, configuration management is a compliance requirement under standards such as ISO/IEC/IEEE 12207, ISO 26262 or IEC 61508 Challenge:

• Maintaining traceability between requirements, code, and tests through continuous change cycles.

Solution:

• Use integrated Application Lifecycle Management (ALM) platforms (e.g., IBM DOORS, Siemens Polarion) that link requirements, commits, and test cases.

Human and Organisational Factors The most difficult aspect of CM is often cultural, not technical. Teams may resist documentation or formal change control due to perceived bureaucracy. As a result:

• Changes occur without review.
• Records become incomplete.
• Knowledge is lost during turnover.

Solution:

• Establish clear CM policies and training.
• Promote configuration discipline as a core part of engineering culture.
• Integrate CM practices seamlessly into daily workflows (e.g., automated pull requests and code reviews).

Configuration management (CM) is not a single activity but a cyclic process integrated into the entire software lifecycle. The ISO/IEC/IEEE 828:2012 standard identifies four principal activities:

• Configuration Identification
• Configuration Control
• Configuration Status Accounting
• Configuration Audit

In modern practice, a fifth step — Configuration Verification and Review — is also added for continuous improvement and compliance.

Configuration Identification The first step in CM defines what needs to be managed. It involves:

• Listing all Configuration Items (CIs) (e.g., code, documents, libraries, binaries).
• Assigning each CI a unique identifier (e.g., version tag, build ID).
• Structuring these items into a configuration hierarchy.

Example hierarchy:

Tools & Techniques:

• Version Control Systems: Git, SVN, Mercurial.
• Artefact Repositories: JFrog Artifactory, Nexus.
• Configuration Databases (CMDBs): ServiceNow CMDB, BMC Helix.

Goal: Create a clear inventory of every managed artefact and its dependencies.

Tools and Techniques:

• Issue & Change Tracking: Jira, Redmine, Azure DevOps, Bugzilla.
• Code Review Systems: GitHub Pull Requests, Gerrit, GitLab Merge Requests.
• Workflow Automation: Jenkins, GitHub Actions, Bamboo.

Goal: Ensure that every change is reviewed, justified, and properly recorded before being implemented.

Configuration Status Accounting (CSA) CSA provides visibility into the current state of configurations across the project. It records which versions of CIs exist, where they are stored, and what changes have occurred. Typical outputs include:

• Version histories and change logs.
• Baseline status reports.
• Release documentation and distribution tracking.

Tools & Techniques:

• Version Reporting Tools: Git log, Git tag, or custom CI/CD reports.
• Automated Dashboards: Grafana, Kibana, Jenkins build monitor.
• ALM Suites: IBM Rational Team Concert, Siemens Polarion.

Goal: Provide transparency and traceability, so project managers and auditors can reconstruct the exact configuration of any product version at any point in time.

Configuration Audit A Configuration Audit ensures the product conforms to its baseline and that all changes were properly implemented and documented. It verifies:

• Each configuration item matches its specification.
• All documentation is updated.
• No unauthorised modifications exist.

There are two types:

1. Functional Configuration Audit (FCA): Confirms the system performs as intended.
2. Physical Configuration Audit (PCA): Confirms that the physical implementation matches the design documentation.

Tools & Techniques:

• Automated Compliance Tools: Chef InSpec, OpenSCAP, AWS Config.
• Manual Audits: Checklists and review boards.
• Digital Twin Validation: Comparing digital models with deployed assets ^[107].

Goal: Ensure integrity, consistency, and compliance across the entire configuration baseline.

Configuration Review and Verification This optional step closes the CM loop. It assesses whether CM processes are effective and aligned with project objectives. Activities include:

• Reviewing CM documentation and records.
• Evaluating tool performance and automation coverage.
• Identifying gaps or inefficiencies for improvement.

Tools:

• CM maturity models (CMMI, ISO/IEC 15504).
• Quality management platforms (e.g., Atlassian Confluence for audit documentation).

Goal: Support continuous improvement and process optimisation.

Modern CM relies heavily on automation and integration tools to manage complexity and enforce discipline across teams. These tools can be categorized by function.

Version Control Systems (VCS)

Build and Continuous Integration Tools

Infrastructure and Environment Management

Artifact and Release Management

Configuration Tracking and Documentation

Example – An integrated CM Workflow:

Toolchain Integration for Autonomous Systems In autonomous platforms (e.g., UAVs, vehicles), CM tools are often integrated with:

• Simulation tools (Gazebo, CARLA) for versioned testing.
• Digital twins for validation of real-world behaviour.
• Edge deployment systems for over-the-air (OTA) updates.

This hybrid approach ensures consistent software across all nodes — from cloud services to embedded controllers ^[108].

Even mature organisations often encounter challenges in lifecycle and configuration management:

Organisations that succeed in embedding CM practices view them not as bureaucracy, but as enablers of reliability and trust.

[raivo.sell][✓ raivo.sell, 2025-09-18]

B. TRADITIONAL DECISION-BASED EXECUTION As cyber-physical systems evolved, information technology (IT) rapidly transformed the world. Electronics design trends revolutionized industries, starting with centralized computing led by firms like IBM and DEC. These technologies enhanced productivity for global business operations, significantly impacting finance, HR, and administrative functions, eliminating the need for extensive paperwork.

Fig. 3. Electronics Megatrends.

The next wave in economy shaping technologies consisted of edge computing devices (red in Figure 3) such as personal computers, cell phones, and tablets. With this capability, companies such as Apple, Amazon, Facebook, Google, and others could add enormous productivity to the advertising and distribution functions for global business. Suddenly, one could directly reach any customer anywhere in the world. This mega-trend has fundamentally disrupted markets such as education (online), retail (ecommerce), entertainment (streaming), commercial real estate (virtualization), health (telemedicine), and more. The next wave of electronics is the dynamic integration with physical assets, and thus even enabling autonomy.

Fig. 4. Progression of System Specification (HW, SW, AI).

As shown in Figure 4, within electronics, there has been a progression of system function construction where the first stage was hardware or pseudo-hardware (FPGA, microcode). The next stage involved the invention of a processor architecture upon which software could imprint system function. Software was a design artifact written by humans in standard languages (C, Python, etc.). The revolutionary aspect of the processor abstraction allowed a shift in function without the need to shift physical assets. However, one needed legions of programmers to build the software. Today, the big breakthrough with Artificial Intelligence (AI) is the ability to build software with the combination of underlying models, data, and metrics. In their basic form, IT systems were not safety critical, and the similar levels of legal liability have not attached to IT products. However, the size and growth of IT is such that problems in large volume consumer products can have catastrophic economic consequences [10]. Thus, the V&V function was very important. IT systems follow the same generic processes for V&V as outlined above, but with two significant differences around the execution paradigm and source of errors. First, unlike the PBE paradigm, the execution paradigm of IT follows a Decision Based Execution mode (DBE). That is, there are no natural constraints on the functional behavior of the underlying model, and no inherent properties of monotonicity. Thus, the whole massive ODD space must be explored which makes the job of generating tests and demonstrating coverage extremely difficult. To counter this difficulty, a series of processes have been developed to build a more robust V&V structure. These include: 1) Code Coverage: Here, the structural specification of the virtual model is used as a constraint to help drive the test generation process. This is done with software or hardware (RTL code). 2) Structured Testing: A process of component, subsection, and integration testing has been developed to minimize propagation of errors. 3) Design Reviews: Structured design reviews with specs and core are considered best practice.

A good example of this process flow is the CMU Capability Maturity Model Integration (CMMI) [11] which defines a set of processes to deliver quality software. Large parts of the CMMI architecture can be used for AI when AI is replacing existing SW components. Finally, testing in the DBE domain decomposes into the following philosophical categories: “Known knowns:” Bugs or issues that are identified and understood, “Known unknowns” Potential risks or issues that are anticipated but whose exact nature or cause is unclear, and “Unknown unknowns” Completely unanticipated issues that emerge without warning, often highlighting gaps in design, understanding, or testing. The last category being the most problematic and most significant for DBE V&V. Pseudo-random test generation has been a key technique used as a method to expose this category [12]. In summary, the key underpinnings of the DBE paradigm from a V&V point of view are: 1) Unconstrained and not well-behaved execution space for scenario test generation, 2) Generally, less expensive simulation execution (no physical laws to simulate), 3) V&V focused on logical errors not mechanical failure 4) Generally, no defined regulatory process for safety critical applications. Most software is “best efforts,” 5) “Unknown-unknowns” a key focus of validation.

A key implication of the DBE space is that the idea from the PBE world of building a list of faults and building a safety argument for them is antithetical to the focus of DBE validation.

[raivo.sell][✓ raivo.sell, 2025-09-18]

A. TRADITIONAL PHYSICS-BASED EXECUTION For MaVV, the critical factors are the efficiency of the MiVV “engine” and the argument for the completeness of the validation. Historically, mechanical/non-digital products (such as cars or airplanes) required sophisticated V&V. These systems were examples of a broader class of products which had a Physics-Based Execution (PBE) paradigm. In this paradigm, the underlying model execution (including real life) has the characteristics of continuity and monotonicity because the model operates in the world of physics. This key insight has enormous implications for V&V because it greatly constrains the potential state-space to be explored. Examples of this reduction of state-space include: 1) Scenario Generation: One need only worry about the state space constrained by the laws of physics. Thus, objects which defy gravity cannot exist. Every actor is explicitly constrained by the laws of physics. 2) Monotonicity: In many interesting dimensions, there are strong properties of monotonicity. As an example, if one is considering stopping distance for braking, there is a critical speed above which there will be an accident. Critically, all the speed bins below this critical speed are safe and do not have to be explored. Mechanically, in traditional PBE fields, the philosophy of safety regulation (ISO 26262 [5], AS9100 [6], etc.) builds the safety framework as a process, where 1) failure mechanisms are identified; 2) a test and safety argument is built to address the failure mechanism; 3) there is an active process by a regulator (or documentation for self-regulation) which evaluates these two, and acts as a judge to approve/decline. Traditionally, faults considered were primarily mechanical failure. As an example, the flow for validating the braking system in an automobile through ISO 26262 would have the following steps: 1) Define Safety Goals and Requirements (Concept Phase): Hazard Analysis and Risk Assessment (HARA): Identify potential hazards related to the braking system (e.g., failure to stop the vehicle, uncommanded braking). Assess risk levels using parameters like severity, exposure, and controllability. Define Automotive Safety Integrity Levels (ASIL) for each hazard (ranging from ASIL A to ASIL D, where D is the most stringent). Define safety goals to mitigate hazards (e.g., ensure sufficient braking under all conditions). 2) Develop Functional Safety Concept: Translate safety goals into high-level safety requirements for the braking system. Ensure redundancy, diagnostics, and fail-safe mechanisms are incorporated (e.g., dual-circuit braking or electronic monitoring). 3) System Design and Technical Safety Concept: Break down functional safety requirements into technical requirements, design the braking system with safety mechanisms like (Hardware {e.g., sensors, actuators. Software (e.g., anti-lock braking algorithms). Implement failure detection and mitigation strategies (e.g., failover to mechanical braking if electronic control fails). 4) Hardware and Software Development: Hardware Safety Analysis (HSA): Validate that components meet safety standards (e.g., reliable braking sensors). Software Development and Validation: Use ISO 26262-compliant processes for coding, verification, and validation. Test braking algorithms under various conditions. 5) Integration and Testing: Perform verification of individual components and subsystems to ensure they meet technical safety requirements. Conduct integration testing of the complete braking system, focusing on: Functional tests (e.g., stopping distance), Safety tests (e.g., behavior under fault conditions), and Stress and environmental tests (e.g., heat, vibration). 6) Validation (Vehicle Level): Validate the braking system against safety goals defined in the concept phase. Perform real-world driving scenarios, edge cases, and fault injection tests to confirm safe operation. Verify compliance with ASIL-specific requirements. 7) Production, Operation, and Maintenance: Ensure production aligns with validated designs, implement operational safety measures (e.g., periodic diagnostics, maintenance), monitor and address safety issues during the product's lifecycle (e.g., software updates). 8) Confirmation and Audit: Use independent confirmation measures (e.g., safety audits, assessment reviews) to ensure the braking system complies with ISO 26262.

Finally, the regulations have a strong idea of safety levels with Automotive Safety Integrity Levels (ASIL). Airborne systems follow a similar trajectory (pun intended) with the concept of Design Assurance Levels (DALs). A key part of the V&V task is to meet the standards required at each ASIL level. Historically, a sophisticated set of V&V techniques has been developed to verify traditional automotive systems. These techniques included well-structured physical tests, often validated by regulators, or sanctioned independent companies (ex TUV-Sud [7]). Over the years, the use of virtual physics-based models has increased to model design tasks such as body design [8] or tire performance [9]. The general structure of these models is to build a simulation which is predictive of the underlying physics to enable broader ODD exploration. This creates a very important characterization, model generation, predictive execution, and correction flow. Finally, because the execution is highly constrained by physics, virtual simulators can have limited performance and often require extensive hardware support for simulation acceleration. In summary, the key underpinnings of the PBE paradigm from a V&V point of view are: 1) Constrained and well-behaved space for scenario test generation 2) Expensive physics based simulations 3) Regulations focused on mechanical failure 4) In safety situations, regulations focused on a process to demonstrate safety with a key idea of design assurance levels.

[raivo.sell][✓ raivo.sell, 2025-09-18]

A. AI COMPONENT VALIDATION Both the automotive and airborne spaces have reacted to AI by viewing it as “specialized Software” in standards such as ISO 8800 [14] and [13]. This approach has the great utility of leveraging all the past work in generic mechanically safety and past work in software validation. However, now, one must manage the issue of how to handle the fact that we have a data generated “code” vs conventional programming code. In the world of V&V, this difference is manifested in three significant aspects: coverage analysis, code reviews, and version control. TABLE III V&V Technique Software AI/ML Coverage Analysis: Code Structure provides basis of coverage No structure Code Reviews: Crowd source expert knowledge No Code to Review Version Control Careful construction/release Very Difficult with data

These differences generate an enormous issue for intelligent test generation and any argument for completeness. This is an area of active research, and two threads have emerged: 1) Training Set Validation: Since the final referenced component is very hard to analyze, one approach is to examine the training set and the ODD to find interesting tests which may expose the cracks between them [16]. 2) Robustness to Noise: Either through simulation or using formal methods [17], the approach is to assert various higher-level properties and use these to test the component. An example in object recognition might be to assert the property that an object should be recognized independent of orientation. Overall, developing robust methods for AI component validation is quite an active and unsolved research topic for “fixed” function AI components. That is, AI components where the function is changing with active version control. Of course, many AI applications prefer a model where the AI component is constantly morphing. Validating the morphing situation is a topic of future research.

B. AI SPECIFICATION

For well-defined systems with an availability of system level abstractions, AI/ML components significantly increase the difficulty of intelligent test generation. With a golden spec, one can follow a structured process to make significant progress in validation and even gate the AI results with conventional safeguards. Unfortunately, one of the most compelling uses of AI is to employ it in situations where the specification of the system is not well defined or not viable using conventional programming. In these Specification Less /ML (SLML) situations, not only is building interesting tests difficult, but evaluating the correctness of the results creates further difficulty. Further, most of the major systems (perception, location services, path planning, etc.) in autonomous vehicles fall into this category of system function and AI usage. To date, there have been two approaches to attack the lack of specification problem: Anti-Spec and AI-Driver. 1) Anti-Spec In these situations, the only approach left is to specify correctness through an anti-spec. The simplest anti-spec is to avoid accidents. Based on some initial work by Intel, there is a standard, IEEE 2846, “Assumptions for Models in Safety-Related Automated Vehicle Behavior” [18] which establishes a framework for defining a minimum set of assumptions regarding the reasonably foreseeable behaviors of other road users. For each scenario, it specifies assumptions about the kinematic properties of other road users, including their speed, acceleration, and possible maneuvers. Challenges include an argument for completeness, a specification for the machinery for checking against the standard, and the connection to a liability governance framework. 2) AI-Driver While IEEE 2846 comes from a bottom-up technology perspective, Koopman/Widen [19] have proposed the concept of defining an AI driver which must replicate all the competencies of a human driver in a complex, real-world environment. Key points of Koopman’s AI driver concept include:

a) Full Driving Capability: The AI driver must handle the entire driving task, including perception (sensing the environment), decision-making (planning and responding to scenarios), and control (executing physical movements like steering and braking). It must also account for nuances like social driving norms and unexpected events. b) Safety Assurance: Koopman stresses that AVs need rigorous safety standards, similar to those in industries like aviation. This includes identifying potential failures, managing risks, and ensuring safe operation even in the face of unforeseen events. c) Human Equivalence: The AI driver must meet or exceed the performance of a competent, human driver. This involves adhering to traffic laws, responding to edge cases (rare or unusual driving scenarios), and maintaining situational awareness at all times. d) Ethical and Legal Responsibility: An AI driver must operate within ethical and legal frameworks, including handling situations that involve moral decisions or liability concerns. e) Testing and Validation: Koopman emphasizes the importance of robust testing, simulation, and on-road trials to validate AI driver systems. This includes covering edge cases, long-tail risks, and ensuring that systems generalize across diverse driving conditions. Overall, it is a very ambitious endeavor and there are significant challenges to building this specification of a reasonable driver. First, the idea of a “reasonable” driver is not even well encoded on the human side. Rather, this definition of “reasonableness” is built over a long history of legal distillation, and of course, the human standard is built on the understanding of humans by other humans. Second, the complexity of such a standard would be very high and it is not clear if it is doable. Finally, it may take quite a while of legal distillation to reach some level of closure on a human like an “AI-Driver.” Currently, the state-of-art for specification is relatively poor for both ADAS and AV. ADAS systems, which are widely proliferated, have massive divergences in behavior and completeness. When a customer buys ADAS, it is not entirely clear what they are getting. Tests by industry groups such as AAA, consumer reports, and IIHS have shown the significant shortcomings of existing solutions [20]. In 2024, IIHS introduced a ratings program to evaluate the safeguards of partial driving automation systems. Out of 14 systems tested, only one received an acceptable rating, highlighting the need for improved measures to prevent misuse and ensure driver engagement [21]. Today, there is only one non process oriented regulation in the marketplace, and this is the NHTSA regulations around AEB [22].

[bertlluk][✓ bertlluk, 2025-06-25]

This chapter introduces the perception, mapping, and localization in the context of autonomous vehicles and usage of different sensor modalities. It examines the determination of vehicle position, position and activities of other participants in the traffic, understanding of the surrounding scenes, scene mapping and map-keeping for navigation, (referred in detail in 4.1) applications of AI, and possible sources of uncertainty and instability (mainly referred in 4.1 and 4.3).

Object detection is the fundamental perception function that allows an autonomous vehicle to identify and localize relevant entities in its surroundings. It converts raw sensor inputs into structured semantic and geometric information, forming the basis for higher-level tasks such as tracking, prediction, and planning. By maintaining awareness of all objects within its operational environment, the vehicle can make safe and contextually appropriate decisions.

Detected objects may include:

• Dynamic agents such as other vehicles, cyclists, and pedestrians.
• Static or quasi-static structures such as curbs, barriers, and traffic signs.
• Unexpected obstacles such as debris, animals, or construction equipment.

Each detection typically includes a semantic label, a spatial bounding box (2D or 3D), a confidence score, and sometimes velocity or orientation information. Accurate detection underpins all subsequent stages of autonomous behavior; any missed or false detection may lead to unsafe or inefficient decisions downstream.

Object detection relies on a combination of complementary sensors, each contributing distinct types of information and requiring specialized algorithms.

Cameras provide dense visual data with rich color and texture, essential for semantic understanding. Typical camera-based detection methods include:

• Feature-based algorithms such as Histogram of Oriented Gradients (HOG), Scale-Invariant Feature Transform (SIFT), and Speeded-Up Robust Features (SURF), used in early lane and pedestrian detection systems.
• Stereo vision for depth estimation and obstacle localization by triangulating disparity between image pairs.
• Optical flow analysis for detecting motion and estimating object trajectories in video sequences.
• Classical machine learning approaches like Support Vector Machines (SVM) or AdaBoost combined with handcrafted features for real-time pedestrian detection.
• Modern convolutional and transformer-based networks (covered in Section 5.2), which directly infer 2D and 3D bounding boxes from images.

Cameras are indispensable for interpreting traffic lights, signs, lane markings, and human gestures, but their performance can degrade under low illumination, glare, or adverse weather conditions.

LiDAR (Light Detection and Ranging) measures distances by timing laser pulse returns, producing dense 3D point clouds. LiDAR-based object detection methods focus on geometric reasoning:

• Point clustering approaches such as Euclidean Cluster Extraction and Region Growing group nearby points into potential objects.
• Model-based fitting, including RANSAC for detecting planes, poles, or cylindrical objects.
• Voxelization methods, which discretize 3D space into volumetric grids for feature extraction.
• Projection methods, where point clouds are projected into 2D range or intensity images for segmentation.
• Shape fitting and contour tracking, which match clusters to known templates such as vehicle rectangles or pedestrian ellipsoids.

LiDAR’s precise geometry enables accurate distance and shape estimation, but sparse returns or partial occlusions can challenge classification performance.

Radar (Radio Detection and Ranging) provides long-range distance and velocity information using radio waves. Its unique Doppler measurements are invaluable for tracking motion, even in fog, dust, or darkness. Typical radar-based detection techniques include:

• Constant False Alarm Rate (CFAR) processing, which statistically distinguishes true reflections from noise in range–Doppler maps.
• Range–Doppler clustering to group radar detections corresponding to individual objects.
• Kalman and particle filtering for multi-target tracking using velocity estimates.
• Micro-Doppler analysis, which identifies signatures of specific motion types such as walking pedestrians or spinning wheels.
• Radar–vision association, where radar detections are correlated with camera-based bounding boxes for class refinement.

Radar systems are especially important for early hazard detection and collision avoidance, as they function effectively through adverse weather and poor visibility.

Ultrasonic and sonar sensors detect objects through acoustic wave reflections and are particularly useful in environments where optical or electromagnetic sensing is limited. They are integral not only to ground vehicles for close-range detection but also to surface and underwater autonomous vehicles for navigation, obstacle avoidance, and terrain mapping.

For ground vehicles, ultrasonic sensors operate at short ranges (typically below 5 meters) and are used for parking assistance, blind-spot detection, and proximity monitoring. Common methods include:

• Time-of-Flight (ToF) ranging, where the distance to an obstacle is computed from the delay between transmitted and received sound waves.
• Amplitude and phase analysis, used to infer surface hardness or material properties.
• Triangulation and beamforming, which estimate object direction and shape using multiple transducers.
• Echo profiling, classifying obstacles based on their acoustic return signatures.

For surface and underwater autonomous vehicles, sonar systems extend these principles over much longer ranges and through acoustically dense media. Typical sonar-based detection methods include:

• Multibeam echo sounding, producing 3D point clouds of the seafloor or underwater structures.
• Side-scan sonar, generating high-resolution acoustic images for obstacle and object identification along a vehicle’s trajectory.
• Forward-looking sonar (FLS), providing real-time obstacle avoidance for surface or submersible vehicles.
• Synthetic Aperture Sonar (SAS), achieving enhanced spatial resolution by coherently combining multiple sonar pings as the vehicle moves.
• Acoustic Doppler Velocity Log (ADVL) integration, allowing both obstacle detection and precise motion estimation.

These acoustic systems are essential in domains where electromagnetic sensing (e.g., camera, LiDAR, radar) is unreliable — such as murky water, turbid environments, or beneath the ocean surface. Although sonar has lower spatial resolution than optical systems and is affected by multipath and scattering effects, it offers unmatched robustness in low-visibility conditions. As with other sensors, regular calibration, signal filtering, and environmental adaptation are necessary to maintain detection accuracy across varying salinity, temperature, and depth profiles.

Object detection outputs can be represented in different coordinate systems and abstraction levels:

• 2D detection localizes objects in image coordinates, suitable for semantic reasoning.
• 3D detection estimates object dimensions, position, and orientation in the world frame, enabling motion planning and distance-based safety checks.
• Bird’s-Eye-View (BEV) detection projects all sensor data onto a unified ground-plane map, simplifying spatial reasoning across multiple sensors. Multi-modal sensor fusion in Bird’s Eye View (BEV) representation, which projects multi-sensor data into a unified ground-plane coordinate system, has become the leading approach for 3D object detection.

Hybrid systems combine these paradigms—for example, camera-based semantic labeling enhanced with LiDAR-derived 3D geometry—to achieve both contextual awareness and metric accuracy.

A standard object detection pipeline in an autonomous vehicle proceeds through the following stages:

1. Data acquisition and preprocessing — raw sensor data are collected, filtered, timestamped, and synchronized.
2. Feature extraction and representation — relevant geometric or visual cues are computed from each modality.
3. Object hypothesis generation — candidate detections are proposed based on motion, clustering, or shape priors.
4. Classification and refinement — hypotheses are validated, labeled, and refined based on fused sensory evidence.
5. Post-processing and temporal association — duplicate detections are merged, and tracking ensures temporal consistency.

The pipeline operates continuously in real time (typically 10–30 Hz) with deterministic latency to meet safety and control requirements.

No single sensor technology can capture all aspects of a complex driving scene under all circumstances, diverse weather, lighting, and traffic conditions. Therefore, data from multiple sensors is fused (combined) to obtain a more complete, accurate, and reliable understanding of the environment than any single sensor could provide alone.

Each sensor modality has distinct advantages and weaknesses:

• Cameras provide high-resolution color and texture information, essential for recognizing traffic lights, signs, and object appearance, but are sensitive to lighting and weather.
• LiDAR delivers precise 3D geometry and range data, allowing accurate distance estimation and shape reconstruction, yet is affected by rain, fog, and reflective surfaces.
• Radar measures object velocity and distance robustly, even in poor visibility, but has coarse angular resolution and may struggle with small or static objects.
• GNSS provides global position but suffers from signal blockage and reflection in e.g., urban canyons, tunnels, and under tree canopies.
• IMU provides motion estimation but is prone to drift and accumulated error.

By fusing these complementary data sources, the perception system can achieve redundancy, increased accuracy, and fault tolerance — key factors for functional safety (ISO 26262).

Sensor fusion can be focused on complementarity – different sensors contribute unique, non-overlapping information and redundancy – overlapping sensors confirm each other’s measurements, improving reliability. As multiple sensor modalities are used, both goals can be achieved.

Accurate fusion depends critically on spatial and temporal alignment among sensors.

• Extrinsic calibration determines the rigid-body transformations between sensors (translation and rotation). It is typically estimated through target-based calibration (e.g., checkerboard or reflective spheres) or self-calibration using environmental features.
• Intrinsic calibration corrects sensor-specific distortions, such as lens aberration or LiDAR beam misalignment.
• Temporal synchronization ensures that all sensor measurements correspond to the same physical moment, using hardware triggers, shared clocks, or interpolation.

Calibration errors lead to spatial inconsistencies that can degrade detection accuracy or cause false positives. Therefore, calibration is treated as part of the functional safety chain and is regularly verified in maintenance and validation routines.

Fusion can occur at different stages in the perception pipeline, commonly divided into three levels:

• Data-level fusion combines raw signals from sensors before any interpretation, providing the richest input but also the heaviest computational load.
• Feature-level fusion merges processed outputs such as detected edges, motion vectors, or depth maps, balancing detail with efficiency.
• Decision-level fusion integrates conclusions drawn independently by different sensors, producing a final decision that benefits from multiple perspectives.

The mathematical basis of sensor fusion lies in probabilistic state estimation and Bayesian inference. Typical formulations represent the system state as a probability distribution updated by sensor measurements. Common techniques include:

• Kalman Filter (KF) and its nonlinear extensions, the Extended Kalman Filter (EKF) and Unscented Kalman Filter (UKF), which maintain a Gaussian estimate of state uncertainty and iteratively update it as new sensor data arrive.
• Particle Filter (PF), which uses a set of weighted samples to approximate arbitrary non-Gaussian distributions.
• Bayesian Networks and Factor Graphs, which represent dependencies between sensors and system variables as nodes and edges, enabling large-scale optimization.
• Deep Learning–based Fusion, where neural networks implicitly learn statistical relationships between sensor modalities through backpropagation rather than explicit probabilistic modeling.

Deep learning has significantly advanced sensor fusion. Neural architectures learn optimal fusion weights and correlations automatically, often outperforming hand-designed algorithms. For example:

• BEVFusion fuses LiDAR and camera features into a top-down BEV representation for 3D detection.
• TransFusion uses transformer-based attention to align modalities dynamically.
• DeepFusion and PointPainting project LiDAR points into the image plane, enriching them with semantic color features.

End-to-end fusion networks can jointly optimize detection, segmentation, and motion estimation tasks, enhancing both accuracy and robustness. However, deep fusion models require large multimodal datasets for training and careful validation to ensure generalization and interpretability.

The existence of a certain form of world model is a fundamental prerequisite for autonomous navigation during the execution of various tasks, since their definitions are often directly or indirectly related to the spatial configuration of the environment and the specification of partial goal positions. The world model thus represents a reference frame relative to which goals are defined.

When discussing an environmental model, it is also necessary to consider its suitable representation in relation to the specific problem being solved. The problem of representing and constructing an environmental model from imprecise sensory data must be viewed from several perspectives:

- Representation. The perceived environment must be stored in a suitable data structure corresponding to the complexity of the environment (e.g., empty halls, furnished rooms). When working with raw, unprocessed sensory data, a compact data representation is required.

- Uncertainty in data. For reliable use of the constructed model, it is desirable to maintain a description of uncertainty in the data, which arises from the processing of noisy sensory information. Methods for data fusion can be advantageously used here to reduce overall uncertainty. Data can be merged from different sources or from different time intervals during task execution.

- Computational and data efficiency. The data structures used must be efficiently updatable, typically in real time, even in relatively large environments and at various levels of resolution. It is almost always necessary to find an appropriate compromise between method efficiency (model quality) and computational and data demands.

- Adaptability. The representation of the environmental model should be as application-independent as possible, which is, however, a very demanding task. In real situations, the environmental representation is often tailored directly to the task for which the model is used, so that the existence of the map enables or significantly increases the efficiency of solving the selected problem. The ways of representing environment maps can be divided according to their level of abstraction into the following types:

- Sensor-based maps work directly with raw sensory data or only with their relatively simple processing. Raw sensor data are used at the level of reactive feedback in subsystems for collision avoidance and resolution, where fast responses are required (David, 1996); thus, they are mainly applied at the motion control level.

A typical example of a sensor-based map is the occupancy grid – a two-dimensional array of cells, where each cell represents a square area of the real world and determines the probability that an obstacle exists in that region.

- Geometric maps describe objects in the environment using geometric primitives in a Cartesian coordinate system. Geometric primitives include segments (lines), polygons, arc sections, splines, or other curves. Additional structures such as visibility graphs or rectangular decompositions are often built on top of this description to support robot path planning. Geometric maps can also be used for robot localization or exported into CAD tools, where further corrections create a complete CAD model of the environment for non-robotic applications.

- Topological maps achieve a high level of abstraction. They do not store information about the absolute coordinates of objects but rather record the topological relationships between objects in the environment. This representation focuses on connectivity and adjacency between places rather than precise geometric positions.

- Symbolic. maps contain information that the robot typically cannot directly obtain through its sensors — usually abstract data that allow a certain degree of natural-language communication with the robot. These maps are commonly implemented in a declarative language such as *Prolog* or *LISP*.

The environmental model can be created in many ways, from manual data collection to fully automatic mapping procedures. However, the difference in time efficiency is significant.

During the construction of an environmental model — mapping — several basic tasks must be solved. One of them is the choice of a suitable movement trajectory that ensures the vehicle's sensor system acquires a sufficient amount of data from the entire mapped space. This issue is addressed by planning algorithms for exploration and space coverage. The choice of a suitable trajectory can also be replaced by remote teleoperation, where the operator decides which parts of the mapped area the robot should visit. Another necessary condition for consistent mapping is the ability to localize. By its nature, the mapping and global localization problem can be described as a “chicken and egg” problem.

The intertwined nature of localization and mapping arises from the fact that inaccuracies in the vehicle's motion affect errors and the accuracy of sensory measurements used as input to mapping algorithms. As soon as the robot moves, the estimate of its current position becomes affected by noise caused by motion inaccuracy. The perceived absolute position of objects inserted into the map is then influenced both by measurement noise and by the error in the estimated current position of the robot. The mapping error caused by inaccurate localization can further reduce the accuracy of localization methods that depend on the map. In this way, both tasks influence each other.

In general, the error of the estimated robot trajectory is strongly correlated with errors in the constructed map — that is, an accurate map cannot be built without sufficiently precise, reliable, and robust localization, and vice versa.

An essential component of spatial orientation and environmental understanding for an autonomous vehicle is localization within that environment. Localization is defined as the process of estimating the vehicle’s current coordinates (position) based on information from available sensors. Suitable sensors for localization are those that allow obtaining either information about the vehicle's relative position with respect to the environment or about its own motion within the environment.

To obtain information about ego motion, inertial and odometric sensors are used. These provide an approximate initial estimate of the position. Inertial sensors infer position from acceleration and rotation, while odometric sensors measure directly the length of the trajectory traversed by the vehicle. However, both measurement principles suffer from significant measurement errors.

As the primary sensors for accurate position determination, laser range finders (scanners) are commonly used. These measure the direct distance to obstacles relative to which the robot orients itself. In recent years, camera-based systems have also begun to emerge as viable alternatives or complements to laser range sensors.

The vehicle's coordinates in the working environment can be referenced either to an existing environment model (map) or to a chosen reference frame. The reference is often, for example, the vehicle's initial position.

If the vehicle's initial position is known, or if only relative coordinates with respect to the starting point are important, we speak of the position tracking problem (also called continuous localization). These localization tasks correct odometric errors accumulated during movement, thus refining the vehicle's current position estimate.

In the opposite case — when the initial position is unknown and we need to determine the vehicle's absolute coordinates within the world model — we speak of the global localization problem.

One of the fundamental differences between global localization and position tracking lies in the requirement for a predefined world model. While global localization requires such a model, position tracking does not necessarily depend on it — it can, for example, utilize a model being gradually constructed during movement.

The most general form of the localization task is the so-called “kidnapped robot problem”, which generalizes both position tracking and global localization. In this task, the vehicle must not only continuously track its position but also detect situations in which it has been suddenly displaced to an unknown location — without the event being directly observable by its sensors.

The method must therefore be capable of detecting, during continuous localization, that the vehicle is located in a completely different position than expected, and at that moment, switch to a different localization strategy. This new strategy typically employs global localization algorithms, which, after some time, re-establish the vehicle's true position in the environment. Once the correct position has been determined, localization can continue using a less computationally demanding position-tracking strategy.

[bertlluk][✓ bertlluk, 2025-06-25]

Advances in AI, especially the convolutional neural network, allow us to process raw sensory information and recognize objects and categorize them into classes with higher levels of abstraction (pedestrians, cars, trees, etc.). Taking these categories into account allows autonomous vehicles to understand the scene and reason about future actions of the vehicle as well as about the other participants in road traffic and make assumptions on/predictions of their possible interactions. This section elaborates on the comparison of commonly used methods, their advantages, and weaknesses.

Traditional perception pipelines used hand-crafted algorithms for feature extraction and rule-based classification (e.g., edge detection, optical flow, color segmentation). While effective for controlled conditions, these systems failed to generalize to the vast variability of real-world driving — lighting changes, weather conditions, sensor noise, and unexpected objects.

The advent of deep learning revolutionized perception by enabling systems to learn features automatically from large datasets rather than relying on manually designed rules. Deep neural networks, trained on millions of labeled examples, can capture complex, nonlinear relationships between raw sensor inputs and semantic concepts such as vehicles, pedestrians, and traffic lights.

In an autonomous vehicle, AI-based perception performs several core tasks:

• Object detection – recognizing and localizing entities of interest.
• Semantic and instance segmentation – classifying every pixel or point in the scene.
• Tracking – following detected entities over time.
• Scene understanding – integrating spatial, semantic, and temporal information into a coherent representation.
• Behavioral prediction – anticipating the likely trajectories and intentions of dynamic agents.

Deep learning architectures form the computational backbone of AI-based perception systems in autonomous vehicles. They enable the extraction of complex spatial and temporal patterns directly from raw sensory data such as images, point clouds, and radar returns. Different neural network paradigms specialize in different types of data and tasks, yet modern perception stacks often combine several architectures into hybrid frameworks.

Convolutional Neural Networks are the most established class of models in computer vision. They process visual information through layers of convolutional filters that learn spatial hierarchies of features — from edges and corners to textures and object parts. CNNs are particularly effective for object detection, semantic segmentation, and image classification tasks. Prominent CNN-based architectures used in autonomous driving include:

• ResNet and EfficientNet for general feature extraction,
• Faster R-CNN and YOLO families for real-time object detection,
• U-Net and DeepLab for dense semantic segmentation.

While cameras capture two-dimensional projections, LiDAR and radar sensors produce three-dimensional point clouds that require specialized processing. 3D convolutional networks, such as VoxelNet and SECOND, discretize space into voxels and apply convolutional filters to learn geometric features. Alternatively, point-based networks like PointNet and PointNet++ operate directly on raw point sets without voxelization, preserving fine geometric detail. These models are critical for estimating the shape and distance of objects in 3D space, especially under challenging lighting or weather conditions.

Transformer networks, initially developed for natural language processing, have been adapted for vision and multimodal perception. They rely on self-attention mechanisms, which allow the model to capture long-range dependencies and contextual relationships between different parts of an image or between multiple sensors. In autonomous driving, transformers are used for feature fusion, bird’s-eye-view (BEV) mapping, and trajectory prediction. Notable examples include DETR (Detection Transformer), BEVFormer, and TransFusion, which unify information from cameras and LiDARs into a consistent spatial representation.

Driving is inherently a dynamic process, requiring understanding of motion and temporal evolution. Recurrent Neural Networks (RNNs), particularly Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) models, are used to process sequences of observations and capture temporal dependencies. They are common in object tracking and motion prediction, where maintaining consistent identities and velocities of moving objects over time is essential. More recent architectures use temporal convolutional networks or transformers to achieve similar results with greater parallelism and stability.

Graph Neural Networks extend deep learning to relational data, representing scenes as graphs where nodes correspond to agents or landmarks and edges encode spatial or behavioral relationships. This structure makes GNNs well suited for modeling interactions among vehicles, pedestrians, and infrastructure elements. Models such as VectorNet, Trajectron++, and Scene Transformer use GNNs to learn dependencies between agents, supporting both scene understanding and trajectory forecasting.

Modern perception systems often combine multiple architectural families into unified frameworks. For instance, a CNN may extract image features, a point-based network may process LiDAR geometry, and a transformer may fuse both into a joint representation. These hierarchical and multimodal architectures enable robust perception across varied environments and sensor conditions, providing the high-level scene understanding required for safe autonomous behavior.

The effectiveness of 'AI-based perception' systems depends fundamentally on the quality, diversity, and management of data used throughout their development lifecycle. As deep neural networks do not rely on explicit programming, but they learn to interpret the environment from large, annotated datasets, data becomes the foundation of reliable perception for autonomous vehicles.

Robust perception requires exposure to the full range of operating conditions that a vehicle may encounter. Datasets must include variations in:

• Sensor modalities – data from cameras, LiDAR, radar, GNSS, and IMU, reflecting the multimodal nature of perception.
• Environmental conditions – daytime and nighttime scenes, different seasons, weather effects such as rain, fog, or snow.
• Geographical and cultural contexts – urban, suburban, and rural areas; diverse traffic rules and road signage conventions.
• Behavioral diversity – normal driving, aggressive maneuvers, and rare events such as jaywalking or emergency stops.
• Edge cases – rare but safety-critical situations, including near-collisions or sensor occlusions.

A balanced dataset should capture both common and unusual situations to ensure that perception models generalize safely beyond the training distribution. Because collecting real-world data for every possible scenario is impractical and almost impossible, simulated or synthetic data are often used to supplement real-world datasets. Photorealistic simulators such as CARLA, LGSVL, or AirSim allow the generation of labeled sensor data under controlled conditions, including rare or hazardous events. Synthetic data helps to fill gaps in real-world coverage and supports transfer learning, though domain adaptation is often required to mitigate the so-called sim-to-real gap — differences between simulated and actual sensor distributions.

Supervised learning models rely on accurately annotated datasets, where each image, frame, or point cloud is labeled with semantic information such as object classes, bounding boxes, or segmentation masks. Annotation quality is critical: inconsistent or noisy labels can propagate systematic errors through the learning process. Modern annotation pipelines combine human labeling with automation — using pre-trained models, interactive tools, and active learning to accelerate the process. High-precision labeling is particularly demanding for LiDAR point clouds and multi-sensor fusion datasets, where 3D geometric consistency must be maintained across frames.

Data used in autonomous driving frequently includes imagery of people, vehicles, and property. To comply with privacy regulations and ethical standards, datasets must be anonymized by blurring faces and license plates, encrypting location data, and maintaining secure data storage. Fairness and inclusivity in dataset design are equally important to prevent bias across geographic regions or demographic contexts.

Scene understanding is a process by which an autonomous agent interprets its environment as a coherent model — integrating environment map, objects, semantics, and dynamics into a structured representation that supports decision-making. It is the bridge between raw perception and higher-level autonomy functions such as planning, prediction, and control.

The goal of scene understanding is to transform fragmented sensor detections into a meaningful, temporally consistent model of the surrounding scene.

Scene understanding often relies on multi-layered representations:

• Geometric Layer – 3D occupancy grids or bird’s-eye-view (BEV) maps of the environment.
• Semantic Layer – class labels for objects and surfaces.
• Relational Layer – links, relations and dependencies among entities.
• Temporal Layer – short-term evolution and motion prediction.
• Behavioral Layer – inferred intentions and possible maneuvers.

The relational layer captures how entities within a traffic scene interact with one another and with the static environment. While lower layers (geometric and semantic) describe what exists and where it is, the relational layer describes how elements relate — spatially, functionally, and behaviorally.

Spatial relation describes e.g. mutual distance, relative velocity, and possible collision of trajectories. Functional relations describe when one entity modifies, limits, or restricts functions of another, e.g., traffic lanes modify the movement of vehicles, railing restricts the movement of pedestrians, etc.

These relations can be explicitly represented by scene graphs, where nodes represent entities and edges represent relationships, or encoded in different types of neural networks, e.g., visual-language models.

Scene understanding must maintain temporal stability across frames. Flickering detections or inconsistent semantic labels can lead to unstable planning. Techniques include temporal smoothing, cross-frame data association to maintain consistent object identities, or memory networks that preserve contextual information across time.

The temporal part of the scene understanding is tightly coupled with motion prediction and forecasting future trajectories of all dynamic agents. Two primary approaches are Physics-based models (e.g., constant-velocity, bicycle models), which are simple and interpretable, but limited in complex interactions, and learning-based models, where data-driven networks capture contextual dependencies and multiple possible futures (e.g., MultiPath, Trajectron++, VectorNet).

Instability of perception, mapping, and localization is caused mainly by the uncertainty of the sensor readings. The object recognition could provide different results over time, even though the scene is static, due to sensor noise. The map could be distorted by erroneous readings of GNSS caused by the reflection of signals. Localization could be degraded by occlusions or unexpected objects.

There are several sources of uncertainty: sensor noise, error readings, model uncertainty, environment randomness, occlusions, adversarial attacks, and intention estimation errors for other traffic participants. This section categorizes and describes them and provides an introduction to how these phenomena can be handled.

There are two main types of uncertainty: aleatoric and epistemic.

Aleatoric uncertainty represents the stochastic nature or randomness of the world. Sensor readings are always affected by different types of noise, and many processes in the environment are of a stochastic nature.

Epistemic uncertainty, or systematic uncertainty, arises from imprecise or incomplete models. The model cannot explain the observed data completely or precisely.

Sensor noise is a significant source of instability in the AV systems. All the sensors providing digital signals add quantization noise inevitably to the measurement, as the nature of the real environment is continuous. To minimize the effects of quantization, a higher resolution is used, causing, on the other hand, a significant increase in computational complexity.

But quantization is not only a source of noise in the sensors. Different physical processes caused noise, e.g., interference, statistical quantum fluctuations, etc. These types of noise are random and usually have a normal distribution. Therefore, the noise can be reduced by averaging over multiple measurements and other similar methods, but it could introduce a time delay and distortion of data.

Cheaper sensors are usually more prone to noise. But even the best sensors are not noiseless. According to ISO 26262, HARA should be performed, safety goals should be set, and finally, hardware safety requirements should be specified. The hardware specification may include the noise ratio, etc., of the sensor.

Sensor fusion is often used to cope with sensor noise. Different sensors use different physical processes for measurement, and therefore, the noise is different as well. A combination of different sensor modalities can improve the measurement.

Another source of uncertainty is the limitation of sensing, such as occlusion or limited visibility. The limitation of the perception subsystem limits the performance of the systems relying on the perception, as the information is only partial or not provided at all. There are various approaches to deal with partial occlusion, like object tracking and prediction, the Kalman filter, or sensor fusion. Especially when we take the weather into account, different sensor modalities could work better for different weather conditions. A combination of radar, visible light camera, and infrared cameras of different wavelengths could effectively diminish the effects of the harsh weather conditions ( e.g., fog).

A similar type of uncertainty is when we have only partial information about the intentions of other agents in the traffic. The prediction of the future vehicle’s trajectory is always a combination of the physical limits of the vehicle and the intention of its driver. The physical limits could be known to a high degree of certainty; the intention of the driver is always only estimated from the observations of previous actions.

Another sources of uncertainty are traffic regulations themselves. Because the real-world road environment is too intricate to quantify all the regulations one by one, there are ambiguities in traffic regulations that do not provide quantitative criteria.

A specific source of instability, especially in the context of neural networks, is an adversarial attack. An adversarial attack is an intentional modification of the environment, causing networks to misrecognize an object as a different class or not detect it at all. Even a little pattern added to the environment (e.g. traffic sign) could cause misrecognition.

Adversarial attacks are especially threatening for autonomous driving systems, which may harm human life. The robustness of autonomous driving systems against adversarial attacks is called SOTIF (Safety Of The Intended Functionality) and is covered by international standards such as ISO 21448.

[bertlluk]CTU: Help needed. We don't know what the intent was behind this chapter.

This section presents a practical, simulation-driven approach to validating the perception, mapping (HD maps/digital twins), and localization layers of an autonomous driving stack. The core idea is to anchor tests in the operational design domain (ODD), express them as reproducible scenarios, and report metrics that connect module-level behavior to system-level safety.

We decompose the stack into Perception (object detection/tracking), Mapping (HD map/digital twin creation and consistency), and Localization (GNSS/IMU and vision/LiDAR aiding) and validate each with targeted KPIs and fault injections. The evidence is organized into a safety case that explains how module results compose at system level. Tests are derived from the ODD and instantiated as logical/concrete scenarios (e.g., with a scenario language like Scenic) over the target environment. This gives you systematic coverage and reproducible edge-case generation while keeping hooks for standards-aligned arguments (e.g., ISO 26262/SOTIF) and formal analyses where appropriate.

The objective is to quantify detection performance—and its safety impact—across the ODD. In end-to-end, high-fidelity (HF) simulation, we log both simulator ground truth and the stack’s detections, then compute per-class statistics as a function of distance and occlusion. Near-field errors are emphasized because they dominate braking and collision risk. Scenario sets should include partial occlusions, sudden obstacle appearances, vulnerable road users, and adverse weather/illumination, all realized over the site map so that failures can be replayed and compared.

• KPIs: precision/recall per class and distance bin; time-to-detect and time-to-react deltas; TTC availability and whether perceived obstacles trigger sufficient braking distance.
• Search strategy: use low-fidelity (LF) sweeps for breadth (planner-in-the-loop, simplified sensors) and confirm top-risk cases in HF with full sensor simulation before any track trials.

Figure 1 explains object comparison. Green boxes are shown for objects captured by ground truth, while Red boxes are shown for objects detected by the AV stack. Threshold-based rules are designed to compare the objects. It is expected to provide specific indicators of detectable vehicles in different ranges for safety and danger areas.

Validation begins with how the map and digital twin are produced. Aerial imagery or LiDAR is collected with RTK geo-tagging and surveyed control points, then processed into dense point clouds and classified to separate roads, buildings, and vegetation. From there, you export OpenDRIVE (for lanes, traffic rules, and topology) and a 3D environment for HF simulation. The twin should be accurate enough that perception models do not overfit artifacts and localization algorithms can achieve lane-level continuity.

Key checks include lane topology fidelity versus survey, geo-consistency in centimeters, and semantic consistency (e.g., correct placement of occluders, signs, crosswalks). The scenarios used for perception and localization are bound to this twin so that results can be reproduced and shared across teams or vehicles. Over time, you add change-management: detect and quantify drifts when the real world changes (construction, foliage, signage) and re-validate affected scenarios.

Here, the focus is on the robustness of ego-pose to sensor noise, outages, and map inconsistencies. In simulation, you inject GNSS multipath, IMU bias, packet dropouts, or short GNSS blackouts and watch how quickly the estimator diverges and re-converges. Similar tests perturb the map (e.g., small lane-mark misalignments) to examine estimator sensitivity to mapping error.

The following is a short KPI list:

• Pose error & drift: per-frame position/orientation error, drift rate during GNSS loss.

• Continuity: lane-level continuity at junctions and during sharp maneuvers.

• Recovery: re-convergence time and heading stability after outages.

• Safety propagation: impact on distance-to-collision (DTC), braking sufficiency, and rule-checking (e.g., lane keeping within margins).

The current validation methods perform a one-to-one mapping between the expected and actual locations. As shown in Fig. 2, for each frame, the vehicle position deviation is computed and reported in the validation report. Later parameters, like min/max/mean deviations, are calculated from the same report. In the validation procedure, it is also possible to modify the simulator to embed a mechanism to add noise in the localization process to check the robustness and validate its performance.

A two-stage workflow balances coverage and realism. First, use LF tools (e.g., planner-in-the-loop with simplified sensors and traffic) to sweep large grids of logical scenarios and identify risky regions in parameter space (relative speed, initial gap, occlusion level). Then, promote the most informative concrete scenarios to HF simulation with photorealistic sensors for end-to-end validation of perception and localization interactions. Where appropriate, a small, curated set of scenarios is carried to closed-track trials. Success criteria are consistent across all stages, and post-run analyses attribute failures to perception, localization, prediction, or planning so fixes are targeted rather than generic.

put your contents here

[momala]

[tgrzejszczak][✓ raivo.sell, 2025-09-18]

The control system of an autonomous vehicle is the final arbiter of safety, translating high-level plans and decisions into precise, real-time actions that govern the vehicle's movement. It is responsible for managing the vehicle's speed, steering, acceleration, and braking, ensuring that the vehicle follows the planned trajectory accurately and safely, even in the face of disturbances, sensor noise, and dynamic environmental changes. The effectiveness and robustness of the control strategy are paramount to overall vehicle safety. This sub-chapter explores the two primary paradigms shaping modern vehicle control: classical control strategies and AI-based control strategies, examining their principles, applications, safety implications, and the ongoing convergence between them.

Classical control strategies form the bedrock of modern vehicle control systems. These methods rely on mathematical models of the vehicle dynamics and well-established principles from control theory, primarily developed in the 20th century. Their strength lies in their mathematical rigor, transparency, and well-understood stability properties.

• Model-Based Approach: Classical control typically requires a mathematical model describing the vehicle's behavior (e.g., how steering angle affects lateral position, how throttle input affects speed). These models are often linearized around operating points or use simplified representations (like the bicycle model for lateral dynamics).
• Feedback Control: The core idea is feedback: measure the current state (e.g., actual speed, actual steering angle, lateral position error), compare it to the desired state (setpoint or reference trajectory), compute an error, and generate a control action (e.g., throttle command, steering angle command) to minimize this error.
• Key Techniques:
  • PID (Proportional-Integral-Derivative) Control: Perhaps the most ubiquitous control algorithm. It calculates the control action based on the proportional error (current difference), integral of past errors (eliminates steady-state error), and derivative of the error (anticipates future error, dampens oscillations). Widely used for speed control, yaw rate control, and simple steering tasks.
  • LQR (Linear Quadratic Regulator): An optimal control technique that finds the control inputs that minimize a cost function, typically balancing tracking error and control effort (or control energy). Requires a linear model of the system and definitions for the cost function weights. Often used for trajectory tracking and stabilization.
  • State Estimation: Techniques like Kalman Filters are essential companions to classical control. They fuse data from multiple sensors (IMU, GPS, wheel speed sensors) to estimate the vehicle's state (position, velocity, orientation, etc.) accurately, which is then fed into the controller. This is crucial for handling sensor noise and providing a reliable state estimate for the controller.
  • Sliding Mode Control (SMC): A robust control technique designed to handle uncertainties and disturbances by forcing the system state to “slide” along a predefined surface, making the system insensitive to certain variations.

• Predictability and Stability: Classical control methods offer strong theoretical guarantees regarding stability and performance, provided the system model is accurate and operating conditions remain within the model's assumptions. This predictability is a significant safety advantage.
• Transparency: The logic of classical controllers (e.g., PID gains, LQR cost function) is often interpretable by engineers. This makes it easier to understand *why* the controller is behaving in a certain way, facilitating verification, validation, and debugging.
• Maturity and Proven Track Record: These techniques have been extensively used and refined in safety-critical systems (like automotive engine control units, ABS, ESC) for decades, demonstrating their reliability under well-defined conditions.

• Model Dependency: Performance heavily relies on the accuracy of the vehicle model. Complex, highly nonlinear dynamics (tire slip, aerodynamic forces, suspension effects) are difficult to model precisely across all operating conditions.
• Handling Uncertainty: Classical methods can struggle with significant model uncertainties, unmodeled dynamics, and large external disturbances (e.g., sudden wind gusts, icy patches) unless specifically designed for robustness (like SMC).
• Complexity in High Dimensions: Designing and tuning classical controllers for complex, high-dimensional systems (like full vehicle dynamics with multiple degrees of freedom) can become computationally intensive and require significant expertise.
• Limited Adaptability: Standard classical controllers are typically designed for specific operating regimes and may not adapt well to drastically changing conditions without re-tuning or redesign.

AI-based control strategies leverage machine learning and artificial intelligence techniques to learn control policies directly from data or simulations, often bypassing the need for explicit, hand-crafted mathematical models. This data-driven approach offers potential advantages in handling complexity and adaptability.

• Data-Driven Approach: AI controllers learn the mapping from sensor inputs (or estimated states) to control outputs by analyzing large datasets or through simulation-based training. They discover complex, non-linear relationships that might be difficult or impossible to capture with traditional modeling.
• Learning from Experience: Techniques like Reinforcement Learning (RL) allow agents (the AI controller) to learn optimal policies by interacting with an environment (simulator or real vehicle) and receiving rewards or penalties for their actions. The goal is to maximize cumulative reward, which can be defined to align with safety and performance objectives.
• Function Approximation: Neural networks are a common tool in AI control, acting as flexible function approximators. They can learn the complex mapping from state to control action without requiring a predefined model structure.
• Key Techniques:
  • Reinforcement Learning (RL): Learns a policy (control strategy) through trial and error. The agent explores actions, observes the resulting state and reward, and updates its policy to favor actions that lead to higher cumulative rewards. Deep RL combines RL with deep neural networks to handle high-dimensional state spaces (like raw sensor data).
  • Supervised Learning for Control: Can be used if expert demonstrations of desired control behavior are available. The AI learns to mimic these demonstrations.
  • Model Predictive Control (MPC) with Learned Models: While MPC itself is a classical optimization-based control technique, AI can be used to learn the prediction model of the vehicle dynamics, potentially capturing complex non-linearities better than hand-crafted models.
  • Neural Network Controllers: Directly using neural networks to output control commands based on the current state estimate.

• Potential for Handling Complexity: AI can learn to control highly complex, non-linear systems where deriving accurate classical models is intractable.
• Adaptability and Generalization: AI controllers, especially those trained on diverse data or simulations, may generalize better to unseen situations or adapt to gradual changes in the vehicle or environment.
• Learning Optimal Behaviors: RL, in particular, can potentially learn control policies that are optimal with respect to a carefully designed reward function, potentially outperforming hand-tuned classical controllers.

• Black-Box Nature: AI controllers, particularly deep neural networks, can act as “black boxes.” It can be difficult to understand *why* they make specific control decisions, which complicates verification, validation, and debugging – critical steps for safety certification.
• Verification and Validation (V&V): Ensuring the safety and robustness of AI controllers is a major challenge. Standard V&V techniques for classical systems are often not directly applicable. Guaranteeing stability, performance bounds, and safety across all possible operating conditions is difficult.
• Data Dependency and Bias: Performance heavily depends on the quality and diversity of the training data. Biases in the data can lead to unsafe behaviors in real-world scenarios not represented in the training set.
• Robustness to Adversarial Attacks and Novel Situations: AI models can be vulnerable to adversarial inputs designed to fool them. Their performance in truly novel situations (out-of-distribution data) is often unpredictable.
• Safety Guarantees: Providing formal, mathematical proofs of safety (e.g., stability, collision avoidance) for complex AI controllers is an active area of research and remains largely unsolved for production systems.

In practice, a purely classical or purely AI-based control system is rare. Instead, a hybrid approach is often employed, leveraging the strengths of both paradigms:

• AI for High-Level Strategy, Classical for Low-Level Execution: AI might be used in the planning or decision-making layers to determine the desired trajectory or maneuvers, while classical controllers (like LQR or PID) handle the precise low-level actuation (steering, throttle, braking) based on the AI's output. This keeps the safety-critical low-level control transparent and predictable.
• AI for Model Estimation, Classical for Control: AI can be used to learn a more accurate or adaptive model of the vehicle dynamics, which is then fed into a classical controller like MPC.
• AI for Exception Handling: Classical controllers handle normal driving conditions, while an AI component (potentially an RL agent) is trained to handle rare or complex edge cases that the classical controller might struggle with.
• Hybrid Controllers: Combining elements of both, for example, using a neural network to tune the gains of a PID controller in real-time based on the driving conditions.

The choice between classical and AI-based control strategies, or a hybrid approach, has profound implications for the safety of autonomous vehicles.

• Transparency vs. Performance Trade-off: There is often a trade-off between the interpretability and verifiability of classical methods and the potential performance and adaptability of AI methods. Safety requires careful consideration of this trade-off.
• Robustness and Reliability: Both approaches must demonstrate robustness to sensor failures, actuator limitations, environmental disturbances, and unexpected interactions. Classical methods offer more established theoretical frameworks for robustness analysis, while AI methods require ongoing research into robust learning and control.
• Verification and Validation (V&V): Developing rigorous V&V processes for AI-based components is critical. This includes simulation testing across diverse scenarios, hardware-in-the-loop (HIL) testing, and potentially new techniques like formal verification or safety certificates based on training procedures.
• Certification: Regulatory bodies require evidence of safety. The black-box nature of AI makes traditional certification pathways challenging, necessitating new standards and methodologies.
• Future Trends: Research is actively focused on making AI controllers more transparent (e.g., via explainable AI), more robust (e.g., via adversarial training, safe exploration in RL), and better integrated with classical methods. There is also growing interest in “learning-to-control” approaches that combine model learning with control policy learning.

Classical control strategies provide a foundation of predictability, stability, and transparency, making them essential for safety-critical low-level vehicle control. AI-based control strategies offer the potential to handle unprecedented complexity and adaptability, learning optimal behaviors from data. Neither approach is a silver bullet; each has distinct strengths and weaknesses regarding safety. The future of safe autonomous vehicle control likely lies in sophisticated hybrid systems that intelligently combine the rigor of classical control with the power of AI, all underpinned by rigorous verification, validation, and a relentless focus on ensuring robust and predictable behavior in the real world. The ongoing development and integration of these strategies are key to achieving the high levels of safety required for widespread deployment of autonomous vehicles.

[tgrzejszczak][✓ raivo.sell, 2025-09-18]

While decision-making algorithms determine *what* high-level goal the autonomous vehicle should pursue (e.g., reach destination, avoid obstacle, follow lane), motion planning and behavioral algorithms translate these goals into specific, executable paths and maneuvers within the dynamic and complex environment. This sub-chapter delves into these critical components, exploring how they generate safe, efficient, and predictable trajectories and behaviors for the vehicle. The interplay between planning the path and deciding the behavior is fundamental to the safe operation of autonomous vehicles, requiring algorithms that can handle uncertainty, react to other road users, and comply with traffic rules.

Behavioral algorithms form the higher-level decision-making layer that interprets the vehicle's goals and the perceived environment to choose appropriate driving behaviors. They determine *what* the vehicle should do next and *when* to do it, such as deciding to change lanes, yield, accelerate, or stop.

• Finite State Machines (FSMs): A classic approach where the vehicle's behavior is modeled as a set of discrete states (e.g., `FollowLane`, `PrepareLaneChangeLeft`, `ExecuteLaneChange`, `Yield`, `Stop`) and transitions between them based on predefined conditions (e.g., “If safe gap detected AND driver intent is left lane, transition from `FollowLane` to `PrepareLaneChangeLeft`”). FSMs offer simplicity and clarity but can struggle with complex, overlapping, or gradual behaviors.
• Hierarchical State Machines: Extend FSMs by organizing states into layers, allowing for more complex and modular behavior representation. Higher layers might handle overall mission goals, while lower layers manage specific maneuvers.
• Behavior Trees (BTs): A more modern and flexible alternative to FSMs. BTs use a tree structure with nodes representing conditions, actions, or control flow (sequences, selectors). They are better suited for handling parallel behaviors and complex decision logic common in driving scenarios.
• Rule-Based Systems: Utilize a set of “if-conditions-then-actions” rules derived from traffic laws, heuristics, or expert knowledge. For example, “If red light detected AND vehicle is within stopping distance, then apply emergency braking.” These can be combined with other methods.
• Goal-Based and Utility-Based Approaches: These methods evaluate different possible behaviors based on their desirability (utility) in achieving the overall goal while considering constraints like safety, comfort, and efficiency. They can select the behavior that maximizes a defined objective function.
• Reinforcement Learning (for Behavior): Similar to its use in control, RL can be applied to learn behavioral policies. An agent learns to choose actions (behaviors) that maximize a reward signal based on interactions with a simulated or real environment. This can potentially discover complex, human-like behaviors but faces similar challenges regarding safety guarantees and interpretability.

• Rule Compliance: Algorithms must ensure compliance with traffic laws and regulations (e.g., stopping at red lights, yielding right-of-way, speed limits).
• Predictability: Behaviors should be predictable to other road users, enhancing cooperative driving and reducing confusion.
• Consistency: The vehicle should react consistently to similar situations, building trust and predictability.
• Robustness to Uncertainty: Algorithms must handle uncertainty in perception (e.g., occluded objects, sensor noise) and predict the behavior of other agents (e.g., predicting if a pedestrian will cross).
• Ethical Considerations: In unavoidable conflict scenarios, behavioral algorithms may implicitly or explicitly need to consider ethical priorities, although formalizing these is a significant challenge.

• Complexity of Driving Scenarios: Real-world driving involves intricate social interactions, ambiguous situations, and unexpected events that are hard to capture with simple rules or states.
• Handling Uncertainty and Prediction: Accurately predicting the intentions and future paths of other dynamic agents (pedestrians, cyclists, other vehicles) is notoriously difficult and crucial for safe interaction.
• Scalability: As the number of possible behaviors and environmental factors increases, the complexity of the behavioral logic grows significantly.
• Human-Like Behavior: Capturing the nuanced, sometimes imperfect but generally safe and cooperative behaviors of human drivers remains a challenge.

Once a behavioral decision is made (e.g., “change lane left”), the motion planner is responsible for generating a specific, feasible, and safe trajectory that executes this behavior. It answers the question of *how* to move from the current state to the desired state within the constraints of the environment and the vehicle itself.

• Grid-Based Methods (e.g., A*, D*, D* Lite): Discretize the environment into a grid. Algorithms like A* search for the shortest path from start to goal while avoiding obstacles represented as occupied grid cells. Variants like D* can replan efficiently when the map changes. These are computationally efficient but can be inaccurate if the grid resolution is too coarse or if the vehicle's footprint is large relative to the grid.
• Sampling-Based Methods (e.g., RRT, RRT*, PRM): These are particularly popular for high-dimensional vehicle state spaces (position, orientation, velocity). They randomly sample configurations in the state space and connect them if they are collision-free, gradually building a roadmap (PRM) or a rapidly exploring random tree (RRT). RRT* aims to find asymptotically optimal paths. These methods are good at handling complex geometries and high dimensions but can be sensitive to sampling density and may require replanning.
• Optimization-Based Methods (e.g., Model Predictive Control - MPC, Trajectory Optimization): Formulate the pathfinding problem as an optimization task. Define an objective function (e.g., minimize time, distance, jerk, control effort) subject to constraints (collision avoidance, kinematic/dynamic limits, smoothness). MPC solves this optimization problem over a finite prediction horizon at each time step, making it suitable for real-time control and handling moving obstacles. These methods can generate smooth, high-quality trajectories but can be computationally intensive.
• Potential Field Methods: Treat the goal as an attractive force and obstacles as repulsive forces. The vehicle navigates by following the resultant force field. Simple and intuitive, but can suffer from local minima (getting stuck) and may produce jerky trajectories.
• Lattice-Based Planning: Pre-compute a graph (lattice) of feasible paths or “primitives” the vehicle can execute based on its kinodynamic constraints. Planning then involves searching this pre-computed lattice for a sequence of primitives connecting the start to the goal. This can be efficient but might limit the planner's flexibility.

• Collision Avoidance: The primary safety goal is to ensure the generated trajectory does not result in collisions with static or dynamic obstacles, including a safe buffer distance.
• Feasibility: The trajectory must be physically executable by the vehicle, respecting its kinematic (steering, turning radius) and dynamic (acceleration, deceleration, speed) constraints.
• Smoothness and Comfort: Trajectories should be smooth to ensure passenger comfort and reduce wear on the vehicle. This often involves minimizing jerk (rate of change of acceleration).
• Predictability: The planned path should be predictable, both for the autonomous vehicle itself (maintaining consistency) and for other road users observing it.
• Reactivity and Replanning: The planner must be able to react quickly to changes in the environment (e.g., a new obstacle appearing) and replan a safe trajectory in real-time.

• Computational Complexity: Finding optimal or even feasible paths in high-dimensional state spaces with complex constraints in real-time is computationally demanding.
• Uncertainty in Perception: Motion planning relies heavily on accurate and up-to-date perception. Errors or delays in perception can lead to unsafe plans.
• Dynamic Environments: Planning must account for the movement of other agents, requiring prediction and often necessitating frequent replanning.
• Balancing Goals: Planners must balance potentially conflicting objectives like safety, efficiency (time/distance), comfort, and adherence to traffic rules.
• Generalization: The planner should perform well across diverse environments and traffic scenarios, not just those it was explicitly designed for.

Behavioral algorithms and motion planners are deeply intertwined and operate in a continuous loop:

1. Perception: The vehicle senses its environment.
2. Decision-Making/Behavioral Layer: Analyzes the environment and current goals to select a high-level behavior (e.g., “prepare for left lane change”).
3. Motion Planning Layer: Takes the current state, the target behavior's goal state (e.g., position in the left lane), and the perceived environment to generate a feasible, safe, and smooth trajectory.
4. Control Layer: Takes the generated trajectory (or reference points on it) and commands the vehicle's actuators (steering, throttle, brake) to follow it.
5. Monitoring & Replanning: The system continuously monitors the execution, perception updates, and any deviations, potentially triggering replanning at either the behavioral or motion planning level.

This tight coupling is essential. The behavioral layer provides the “intent,” while the motion planner provides the “execution plan.” A failure or limitation in one layer can compromise the safety and effectiveness of the other. For example, an overly aggressive behavioral decision might lead the motion planner to generate an unsafe trajectory, while a motion planner that is too conservative might prevent the behavioral layer from making progress.

Ensuring the safety of the planning and behavioral components is paramount and presents unique challenges:

• Verification and Validation (V&V): Rigorously testing planners and behavioral algorithms across a vast range of scenarios (including edge cases and rare events) is critical but extremely difficult. Simulation is key, but ensuring it covers all relevant real-world possibilities is an ongoing challenge.
• Handling Uncertainty: Both perception uncertainty and the inherent unpredictability of other road users must be explicitly handled. This involves robust planning techniques, prediction models with confidence bounds, and potentially conservative fallback behaviors.
• Ethical and Social Considerations: The choices made by behavioral algorithms (e.g., who to yield to, how assertive to be) have social and ethical dimensions that need careful consideration and potentially stakeholder input.
• Explainability: Understanding *why* an autonomous vehicle chose a specific behavior or planned a particular path is important for debugging, trust, and potentially for interaction with humans.
• Future Trends: Research is moving towards more integrated, learning-based approaches where AI models might learn both behavioral policies and motion planning strategies simultaneously from data. There is also a focus on multi-agent planning, where the vehicle explicitly models and coordinates with other agents in the environment. Ensuring safety within these more complex and less transparent systems remains a core focus.

Motion planning and behavioral algorithms are the intelligent core that guides autonomous vehicles through the complexities of the real world. Behavioral algorithms decide the appropriate high-level actions based on goals and the environment, while motion planners generate the precise, safe, and feasible paths to execute those actions. Both face significant challenges related to complexity, uncertainty, computational demands, and safety assurance. The successful integration and continuous refinement of these algorithms, underpinned by rigorous testing and validation, are essential steps towards achieving the high levels of safety required for autonomous vehicles to operate reliably and deploy widely. Their evolution will continue to be a critical driver in the development of safe autonomous mobility.

—

[momala]

Planning and control are where intent becomes motion. A planning stack selects a feasible, safety-aware trajectory under evolving constraints; the control stack turns that trajectory into actuation while respecting vehicle dynamics and delays. Validating these layers is therefore about much more than unit tests: it is about demonstrating, with evidence, that the combined decision–execution loop behaves safely and predictably across the intended operational design domain (ODD). In practice, this requires two complementary ideas. First, a digital twin of the vehicle and environment that is accurate enough to make simulation a meaningful predictor of real behavior. Second, a design-of-experiments (DOE)–driven scenario program that stresses the decision and control logic where it matters most, and converts outcomes into monitorable, quantitative metrics. Your V&V suite frames both: scenario descriptions feed a co-running simulator with the under-test algorithms, the digital twin (vehicle and environment) is loaded as an external asset, and the outcome is a structured validation report rather than anecdotal test logs.

Planning/control V&V must also navigate the mix of deterministic dynamics and stochastic perception/prediction. At the component level, your framework treats detection, control, localization, mission planning, and low-level control as distinct abstractions, yet evaluates them in the context of Newtonian physics—explicitly trading fidelity for performance depending on the test intent. This modularity enables validating local properties (e.g., trajectory tracking) while still measuring system-level safety effects (e.g., minimum distance to collision).

A final principle is lifecycle realism. A digital twin is not just a CAD model; it is a live feedback loop receiving data from the physical system and its environment, so the simulator remains predictive as the product evolves. The same infrastructure that generates scenarios can replay field logs, inject updated vehicle parameters, and reflect map changes, enabling continuous V&V of planning and control post-deployment.

The V&V workflow begins with a formal scenario description: functional narratives are encoded in a human-readable DSL (e.g., M-SDL/Scenic), then reduced to logical parameter ranges and finally to concrete instantiations selected by DOE. This ensures tests are reproducible, shareable, and traceable from high-level goals down to the numeric seeds that define a specific run. The simulator co-executes these scenarios with under the test algorithms inside the digital twin, and the V&V interface collects vehicle control signals, virtual sensor streams, and per-run metrics to generate the verdicts required by the safety case.

To maintain broad coverage without sacrificing realism, validations can be done using a two-layer approach shown in Figure 1. A low-fidelity (LF) layer (e.g., SUMO) sweeps wide parameter grids quickly to reveal where planning/control begins to stress safety constraints; a high-fidelity (HF) layer (e.g., a game engine simulator like CARLA with the control software in the loop) then replays the most informative cases with photorealistic sensors and closed-loop actuation. Both layers log the same KPIs, so results are comparable and can be promoted to track tests when warranted. This division of labor is central to scaling scenario space while maintaining end-to-end realism for planning and control behaviors like cut-in/out, overtaking, and lane changes.

Formal methods strengthen this flow. In the simulation-to-track pipeline, scenarios and safety properties are specified formally (e.g., via Scenic and Metric Temporal Logic), falsification synthesizes challenging test cases, and a mapping executes those cases on a closed track^[111]. In published evidence, a majority of unsafe simulated cases reproduced as unsafe on track, and safe cases mostly remained safe—while time-series comparisons (e.g., DTW, Skorokhod metrics) quantified the sim-to-real differences relevant to planning and control. This is exactly the kind of transferability and measurement discipline a planning/control safety argument needs.

Finally, environment twins are built from aerial photogrammetry and point-cloud processing (with RTK-supported georeferencing), yielding maps and 3D assets that match the real campus, so trajectory-level decisions (overtake, yield, return-to-lane) are evaluated against faithful road geometries and occlusion patterns^[112].

Mission-level planning validation starts from a start–goal pair and asks whether the vehicle reaches the destination via a safe, policy-compliant trajectory. Your platform publishes three families of evidence: (i) trajectory-following error relative to the global path; (ii) safety outcomes such as collisions or violations of separation; and (iii) mission success (goal reached without violations). This couples path selection quality to execution fidelity.

At the local planning level, your case study focuses on the planner inside the autonomous software. The planner synthesizes a global and a local path, then evaluates them based on predictions from surrounding actors to select a safe local trajectory for maneuvers such as passing and lane changes. By parameterizing scenarios with variables such as the initial separation to the lead vehicle and the lead vehicle’s speed, you create a grid of concrete cases that stress the evaluator’s thresholds. The outcomes are categorized by meaningful labels—Success, Collision, Distance-to-Collision (DTC) violation, excessive deceleration, long pass without return, and timeout—so that planner tuning correlates directly with safety and comfort.

Control validation links perception-induced delays to braking and steering outcomes. Your framework computes Time-to-Collision (Formula) along with the simulator and AV-stack response times to detected obstacles. Sufficient response time allows a safe return to nominal headway; excessive delay predicts collision, sharp braking, or planner oscillations. By logging ground truth, perception outputs, CAN bus commands, and the resulting dynamics, the analysis separates sensing delays from controller latency, revealing where mitigation belongs (planner margins vs. control gains).

A necessary dependency is localization health. Your tests inject controlled GPS/IMU degradations and dropouts through simulator APIs, then compare expected vs. actual pose per frame to quantify drift. Because planning and control are sensitive to absolute and relative pose, this produces actionable thresholds for safe operation (e.g., maximum tolerated RMS deviation before reducing speed or restricting maneuvers).

Finally, your program extends to low-level control via HIL-style twins. A Simulink-based network of virtual ECUs and data buses sits between Autoware’s navigation outputs and simulator actuation. This lets you simulate bus traffic, counters, and checksums; disable subsystems (e.g., steering module) to provoke graceful degradation; and compare physical ECUs against their twin under identical inputs to detect divergence. It is an efficient route to validating actuator-path integrity without building a full physical rig.

On the TalTech iseAuto shuttle, the digital twin (vehicle model, sensor suite, and campus environment) is integrated with LGSVL/Autoware through a ROS bridge so that “photons-to-torque” loops are exercised under realistic scenes before any track test. Scenarios are distributed over the campus xodr network using Scenic/M-SDL; multiple events can be chained within a scenario to probe planner behaviors around parked vehicles, slow movers, or oncoming traffic. Logging is aligned to the KPIs above so outcomes are comparable across LF/HF layers and re-runnable when planner or control parameters change.

In practice, this has yielded a concise, defensible narrative for planning & control safety: (1) what was tested (formalized scenarios across a structured parameter space); (2) how it was tested (two-layer simulation with a calibrated digital twin and, when necessary, track execution); (3) what happened (mission success, DTC minima, TTC profiles, braking/steering transients, localization drift); and (4) why it matters (evidence that tuning or algorithmic changes move the decision–execution loop toward or away from safety). The same framework has been used to analyze adversarial stresses on rule-based local planners, reinforcing that planning validation must include robustness to distribution shifts and targeted perturbations.

As a closing reflection, the approach acknowledges that simulation is not the world—so it measures the gap. By transporting formally generated cases to the track and comparing time-series behaviors, the program both validates planning/control logic and calibrates the digital twin itself, using discrepancies to guide model updates and ODD limits. That is the hallmark of modern control & planning V&V: scenario-driven, digitally twinned, formally grounded, and relentlessly comparative to reality.

[momala]

Simulation is indispensable in autonomous-vehicle validation because it lets us probe safety-critical behavior without exposing the public to risk, but simulation alone is only as persuasive as its predictive value. A simulator that cannot anticipate how the real system behaves—because of poor modeling, missing variability, or unmeasured assumptions—does not provide credible evidence for a safety case. This is why we pair simulation with formal methods: a discipline for specifying scenarios and safety properties with mathematical precision, generating test cases systematically, and measuring how closely simulated outcomes match track or road trials. In our program, the digital twin of the vehicle and its operating environment acts as the concrete “world model,” while formal specifications direct the exploration of that world to the places where safety margins are most likely to fail.

Treating the digital twin as a live feedback loop is central to maintaining predictive value over time. The twin ingests logs and environmental data from the physical shuttle, updates maps and vehicle parameters, and feeds those data back into the simulator so that new tests reflect actual wear, calibration drift, and environmental change. This continuous synchronization turns simulation into an ongoing assurance activity rather than a one-off milestone.

Building such twins is non-trivial. Our workflow constructs environment twins from aerial photogrammetry with RTK-supported georeferencing, then processes point clouds into assets capable of driving a modern simulator. The resulting model can be used across many AVs and studies, amortizing the cost of data collection and asset creation while preserving the fidelity needed for planning, perception, and control validation.

Formal methods begin by making requirements executable. We express test intent as a distribution over concrete scenes using the SCENIC language, which provides geometric and probabilistic constructs to describe traffic, occlusions, placements, and behaviors. A SCENIC program defines a scenario whose parameters are sampled to generate test cases; each case yields a simulation trace against which temporal properties—our safety requirements—are monitored. This tight loop, implemented with the VERIFAI toolkit, supports falsification (actively searching for violations), guided sampling, and clustering of outcomes for test selection.

In practice, the pipeline unfolds as follows. We first assemble the photorealistic simulated world and dynamics models from HD maps and 3D meshes. We then formalize scenarios in SCENIC and define safety properties as monitorable metrics—often using robust semantics of Metric Temporal Logic (MTL), which provide not just a pass/fail verdict but a quantitative margin to violation. VERIFAI searches the parameter space, records safe and error tables, and quantifies “how strongly” a property held or failed; these scores guide which cases deserve promotion to track tests. This process transforms vague test ideas (“test passing pedestrians”) into a concrete population of parameterized scenes with measurable, comparable outcomes.

Our project also leverages scenario distribution over maps: using OpenDRIVE networks of the TalTech campus, SCENIC instantiates the same behavioral narrative—say, overtaking a slow or stopped vehicle—at diverse locations, ensuring that lane geometry, curbside clutter, and occlusions vary meaningfully while the safety property remains constant. The result is a family of tests that stress the same planning and perception obligations under different geometric and environmental embeddings.

A formal pipeline is only convincing if simulated insights transfer to the track. After falsification, we select representative safe/unsafe cases through visualization or clustering of the safe/error tables and implement them on a closed course with controllable agents. Notably, the same SCENIC parameters (starting pose, start time, velocities) drive hardware actors on the track as drove agents in simulation, subject to physical limitations of the test equipment. This parity enables apples-to-apples comparisons between simulated and real traces.

We then quantify the sim-to-real gap using time-series metrics such as dynamic time warping and the Skorokhod distance to compare trajectories, first-detection times, and minimum-distance profiles. In published results, trajectories for the same test were qualitatively similar but showed measurable differences in separation minima and timing; moreover, even identical simulations can diverge when the autonomy stack is non-deterministic, a reality that the methodology surfaces rather than hides. Understanding this variance is a virtue: tests with lower variance are more reproducible on track, while highly variable tests reveal sensitivity in planning, perception, or prediction that merits redesign or tighter ODD limits.

This formal sim-to-track pipeline does more than label outcomes; it helps diagnose causes. By replaying logged runs through the autonomy stack’s visualization tools, we can attribute unsafe behavior to perception misses, unstable planning decisions, or mispredictions, and then target those subsystems in subsequent formal campaigns. In one case set, the dominant failure mode was oscillatory planning around a pedestrian, discovered and characterized through this exact loop of scenario specification, falsification, track execution, and trace analysis.

Exhaustive testing is infeasible, so we combine multiple fidelity levels to balance breadth with realism. Low-fidelity (LF) platforms sweep large scenario grids quickly to map where safety margins begin to tighten; high-fidelity (HF) platforms (e.g., LGSVL/Unity integrated with Autoware) replay the most informative LF cases with photorealistic sensors and closed-loop control. Logging is harmonized so that KPIs and traces are comparable across levels, and optimization or tuning derived from LF sweeps is verified under HF realism before any track time is spent. In extensive experiments, thousands of LF runs revealed broad patterns, but only HF replays uncovered subtle interactions that flipped outcomes—evidence that fidelity matters exactly where the safety case will later be challenged.

This workflow sits within a DOE-driven V&V suite that treats the digital twin and scenario engine as programmable assets. Scenario definitions, vehicle models, and evaluation logic are versioned; control-loop delays, TTC profiles, and collision metrics are computed consistently per run; and the same infrastructure can be extended downward into hardware-in-the-loop experiments of low-level control paths to test actuator-path integrity under identical scene conditions. In our project platform, the simulator co-runs with Autoware, accepts parameterized scenarios through a public interface, and emits validation reports that roll up from frame-level signal checks to mission-level success, closing the traceability chain from formal property to system outcome.

Just as important as capability is honesty about limits. Our own survey and case study argue for explicit attention to abstraction choices, modeling assumptions, and convergence questions for AI-based components. The literature and our results stress that simulation’s value depends on calibrated models, careful measurement of non-determinism, and disciplined mapping to the real world; formal methods help precisely because they make these assumptions visible, testable, and comparable over time. The digital-twin perspective then turns those measurements into an engine for continuous improvement, updating the twin as the physical system and environment evolve.

put your contents here

[raivo.sell][✓ raivo.sell, 2025-10-20]

This chapter focuses on how autonomous vehicles interact and communicate with people and their surrounding environment. As automation removes the human driver from the control loop, new forms of Human–Machine Communication (HMC) are required to ensure transparency, trust, and safety. The chapter examines how information is exchanged between vehicles, passengers, pedestrians, operators, and fleet managers through a variety of interfaces and communication modes. It introduces conceptual and practical frameworks such as Human–Machine Interfaces (HMI), the Language of Driving (LoD), and public acceptance mechanisms that together define how autonomy becomes understandable and socially integrated in everyday mobility.

This chapter explores the specificities of Human–Machine Interaction (HMI) in the context of autonomous vehicles (AVs). It examines how HMI in autonomous vehicles differs fundamentally from traditional car dashboards. With the human driver no longer actively involved in operating the vehicle, the challenge arises: *how should AI-driven systems communicate effectively with passengers, pedestrians, and other road users?*

HMI in AVs extends far beyond the driver’s dashboard. It defines the communication bridge between machines, people, and infrastructure — shaping how autonomy is perceived and trusted. Effective HMI determines whether automation is experienced as *intelligent and reliable* or *opaque and alien.*

Traditional driver interfaces were designed to support manual control. In contrast, autonomous vehicles must communicate *intent*, *status*, and *safety* both inside and outside the vehicle. The absence of human drivers requires new communication models to ensure safe interaction among all participants.

This section addresses the available communication channels and discusses how these channels must be redefined to accommodate the new paradigm. Additionally, it considers how various environmental factors—including cultural, geographical, seasonal, and spatial elements—impact communication strategies.

A key concept in this transformation is the Language of Driving (LoD) — a framework for structuring and standardizing how autonomous vehicles express awareness and intent toward humans (Kalda et al., 2022).

Understanding how humans perceive the world is crucial for autonomous vehicles to communicate effectively. Human perception is multimodal — combining sight, sound, motion cues, and social awareness. By studying these perceptual mechanisms, AV designers can emulate intuitive human signals such as:

• Body orientation or focus of attention.
• Gesture and trajectory anticipation.
• Subtle speed or direction changes as non-verbal cues.

Such behaviorally inspired signaling helps AVs become socially legible, supporting *shared understanding* on the road.

Driving is a social act. Culture, norms, and environment shape how humans interpret signals and movements. Autonomous vehicles may need to adapt their communication style — from light colors and icons to audio tones and message phrasing — depending on cultural and regional expectations.

Research explores whether AVs could adopt human-like communication methods, such as digital facial expressions or humanoid gestures, to support more natural interactions in complex social driving contexts.

Modern HMI systems increasingly rely on artificial intelligence, including large language models (LLMs), to process complex situational data and adapt communication in real time. AI enables:

• Context-aware dialogue systems for passengers and operators.
• Adaptive message prioritization based on urgency and environment.
• Natural language explanations of AV behavior (e.g., *“Slowing down for crossing pedestrian”*).

The evolution toward AI-mediated interfaces marks a shift from fixed UI design toward *conversational and contextual* vehicle communication.

[raivo.sell]

While the previous section described the foundations and goals of HMI, this section focuses on how autonomous vehicles communicate with various stakeholders and through which modes. These interactions can be categorized by user type, purpose, and proximity.

The vehicle–passenger interface supports comfort, awareness, and accessibility. It replaces the human driver’s social role by providing:

• Visual or auditory cues explaining system decisions (e.g., “Yielding to pedestrian”).
• Clear indications of route, stops, and operational mode.
• Options for emergency stop, help request, or trip feedback.

Passenger communication must balance automation with reassurance. In an Estonian field study (Kalda, Sell & Soe, 2021), over 90% of first-time AV users reported feeling safe and willing to ride again when the interface clearly explained the vehicle’s actions.

The vehicle–pedestrian interface (V2P) substitutes human cues such as eye contact or gestures. The *Language of Driving* (Kalda et al., 2022) proposes using standardized visual symbols, light bars, or projections to express intent:

• Green arrows — invitation to cross.
• White pulses — awareness of pedestrian presence.
• Red cross — do not cross / vehicle in motion.

Pedestrian communication must remain universal and intuitive, avoiding dependence on text or language comprehension.

At current autonomy levels (L3–L4), a safety operator interface remains essential. Two variants exist:

• Onboard HMI: allows manual control, displays alerts, and ensures quick handover.
• Teleoperation station: enables remote monitoring and control via secure networks.

Teleoperation acts as a *bridge* between human oversight and full autonomy — essential for handling ambiguous traffic or emergency scenarios.

A dedicated maintenance interface enables technicians to safely inspect and update the vehicle:

• Sensor and actuator diagnostics.
• Log analysis and system replay.
• Secure firmware updates and access control.

Such interfaces ensure traceability, reliability, and compliance with safety regulations.

Fleet-level interfaces provide centralized control and analytics for multiple vehicles. They support:

• Mission planning and route monitoring.
• Predictive maintenance using vehicle telemetry.
• Integration with smart city and MaaS platforms.

These tools operate mainly over remote communication channels, relying on secure data infrastructure.

Autonomous vehicle interaction can be divided into direct (local) and remote (supervisory) communication:

To ensure that human–machine communication is intuitive and safe, several universal design principles apply:

• Transparency: reveal intent and system state clearly.
• Consistency: uniform behavior across environments.
• Accessibility: accommodate diverse users and abilities.
• Multimodality: combine light, sound, and motion cues.
• Security and privacy: protect both human and machine data.

When applied systematically, these principles make autonomous systems understandable, predictable, and trustworthy.

References

Kalda, K.; Pizzagalli, S.-L.; Soe, R.-M.; Sell, R.; Bellone, M. (2022). *Language of Driving for Autonomous Vehicles.* Applied Sciences, 12(11), 5406. [https://doi.org/10.3390/app12115406](https://doi.org/10.3390/app12115406)

Kalda, K.; Sell, R.; Soe, R.-M. (2021). *Use Case of Autonomous Vehicle Shuttle and Passenger Acceptance.* Proceedings of the Estonian Academy of Sciences, 70(4), 429–435. [https://doi.org/10.3176/proc.2021.4.09](https://doi.org/10.3176/proc.2021.4.09)

The Language of Driving (LoD) describes the implicit and explicit signals that allow autonomous vehicles and humans to understand each other in mixed traffic [1–3].

Driving behavior can be analyzed as a layered communication system:

• Phonetics: visible cues such as lights or motion rhythm.
• Semantics: the meaning of those cues (e.g., yield, proceed).
• Pragmatics: how meaning changes with context and environment.

An autonomous vehicle must infer human intent and simultaneously display legible intent of its own [2].

Driving “languages” vary globally; hence interfaces must maintain universal meaning while allowing local adaptation [1]. Behavior should be recognizable but not anthropomorphic, preserving clarity across cultures [3].

Field experiments using light-based cues have shown that simple color and motion patterns effectively communicate awareness and yielding. Participants reported improved understanding when signals were consistent and redundant across modalities [2].

Formalizing LoD as a measurable framework is essential for verification, standardization, and interoperability of automated behavior [3].

References: [1] Razdan, R. et al. (2020). *Unsettled Topics Concerning Human and Autonomous Vehicle Interaction.* SAE EDGE Research Report EPR2020025. [2] Kalda, K., Sell, R., Soe, R.-M. (2021). *Use Case of Autonomous Vehicle Shuttle and Passenger Acceptance.* Proc. Estonian Academy of Sciences, 70 (4). [3] Kalda, K., Pizzagalli, S.-L., Soe, R.-M., Sell, R., Bellone, M. (2022). *Language of Driving for Autonomous Vehicles.* *Applied Sciences*, 12 (11).

[raivo.sell][✓ raivo.sell, 2025-10-20]

The integration of autonomous vehicles (AVs) into everyday traffic introduces both technological and societal challenges. While automated driving systems aim to eliminate human error and improve efficiency, the perceived safety and acceptance of these systems remain crucial for their widespread adoption. Ensuring that people *trust* the technology is equally important as ensuring that the technology *functions safely*.

Safety in autonomous mobility can be divided into two interdependent aspects:

• Technical Safety – the reliability of sensors, algorithms, and mechanical systems that guarantee correct perception, planning, and control under varying traffic conditions.
• Perceived Safety – the degree to which passengers and other road users feel safe when interacting with or observing an autonomous vehicle.

Even if an AV operates flawlessly according to standards and regulations, users may still hesitate to use it unless the system communicates its actions clearly and behaves predictably. Thus, *trust* emerges as a measurable component of safety.

Public acceptance is closely linked to how transparently the system communicates its intentions and limitations. People expect autonomous vehicles to behave in a consistent and understandable manner — signalling when yielding, stopping, or resuming motion. Clear visual or auditory cues from the vehicle’s human–machine interface (HMI) can substantially increase user confidence.

Equally important is transparent communication from operators and authorities regarding how safety is managed, what happens in case of system failures, and how data is used. Misinformation or uncertainty during incidents may quickly erode public trust even if no technical fault has occurred.

Empirical research has shown that direct experience with AVs strongly increases trust. In one Estonian field study (Kalda, Sell & Soe, 2021), the majority of first-time users reported a high sense of safety and comfort, with over 90% indicating willingness to use autonomous shuttles again after their initial ride.

Such results confirm that personal experience and well-managed demonstrations are key factors in shaping public perception. People who interact directly with autonomous vehicles tend to transition from curiosity to trust, whereas those without exposure often remain cautious or skeptical. This highlights the importance of continuous testing, education, and public engagement.

Public acceptance extends beyond safety alone. It also encompasses questions of responsibility, fairness, accessibility, and societal impact. Autonomous transport must be inclusive and understandable to all citizens — regardless of age, digital literacy, or physical ability.

Ethical transparency, clear rules of accountability, and human-centered interface design all contribute to societal readiness for automation. Collaboration between engineers, psychologists, communication experts, and policy-makers is therefore essential to define a holistic framework of *social safety*.

Ensuring public confidence in autonomous mobility requires a balanced approach:

• Continuous safety validation through simulation, testing, and monitoring.
• Transparent governance and open communication with the public.
• Education and awareness activities that explain system limits and benefits.
• Ethical and inclusive design principles ensuring accessibility and fairness.

When these dimensions align, public acceptance evolves naturally, transforming initial curiosity and caution into trust and habitual use. The success of future autonomous mobility therefore depends not only on technological excellence but also on how well society understands and embraces it.

Reference: Kalda, K.; Sell, R.; Soe, R.-M. (2021). *Use Case of Autonomous Vehicle Shuttle and Passenger Acceptance.* Proceedings of the Estonian Academy of Sciences, 70(4), 429–435. [https://doi.org/10.3176/proc.2021.4.09](https://doi.org/10.3176/proc.2021.4.09)

[raivo.sell]

Verification and Validation (V&V) of Human–Machine Interfaces (HMI) in autonomous vehicles ensure that communication between humans and intelligent systems is safe, intuitive, and consistent. While functional safety standards focus on the correct operation of sensors and control logic, HMI validation extends this to human comprehension, usability, and behavioral response [1–3].

The goal of HMI V&V is to confirm that:

• Users correctly interpret the information and cues provided by the vehicle.
• System feedback supports timely and safe human reactions.
• Communication remains effective under diverse environments and user conditions.

The validation process therefore combines *technical testing* with *human-centered evaluation*.

Verification addresses whether the interface behaves as intended. Typical methods include:

• Simulation-based testing – verification of visual, audio, and tactile signals within virtual driving scenarios.
• Scenario-based validation – predefined interaction cases between AVs and pedestrians or passengers tested systematically.
• Software-in-the-loop (SIL) / Hardware-in-the-loop (HIL) – to evaluate timing and synchronization of multimodal feedback.
• Failure mode testing – analysis of degraded communication (e.g., light or network failure) and fallback behavior [2].

Verification ensures consistency, latency limits, and redundancy across modalities before any user testing is performed.

Validation focuses on how people actually experience and understand the interface. This involves iterative testing with human participants in controlled and real-world environments [1–3]. Approaches include:

• Usability studies – measurement of comprehension time, task completion, and error rate.
• Eye-tracking and physiological monitoring – assessing attention and cognitive workload.
• Questionnaires and interviews – evaluating perceived safety, clarity, and trust.

Results are analyzed to refine signal patterns, color codes, and message phrasing to improve intuitiveness and reduce confusion.

High-fidelity simulation environments enable early-stage evaluation of HMI without physical prototypes. Tools integrate virtual pedestrians, lighting, and weather to test how design choices influence visibility and legibility [3]. Virtual validation supports:

• Rapid comparison of alternative communication concepts.
• Testing rare or hazardous scenarios ethically.
• Correlating behavioral metrics with simulated responses.

These techniques shorten development cycles and allow data-driven interface improvement.

To make validation reproducible, quantitative metrics are defined, such as:

• Comprehension rate (% of participants interpreting cues correctly).
• Reaction latency (time to respond to a signal).
• Confidence index (subjective trust level).
• Error frequency (number of misinterpretations per test run).

Standardized metrics enable benchmarking across projects and support regulatory assessment of AV communication readiness.

HMI validation does not end with prototype testing. Field data from pilot deployments provide valuable feedback loops for ongoing improvement [2]. By combining simulation, real-world performance, and user analytics, HMI systems evolve continuously as technology and user expectations mature.

Effective verification and validation bridge the gap between technical functionality and human understanding. By ensuring that communication is accurate, interpretable, and trusted, these processes contribute directly to the safe and responsible deployment of autonomous mobility [1–3].

References: [1] Razdan, R. et al. (2020). *Unsettled Topics Concerning Human and Autonomous Vehicle Interaction.* SAE EDGE Research Report EPR2020025.

[2] Kalda, K., Sell, R., Soe, R.-M. (2021). *Use Case of Autonomous Vehicle Shuttle and Passenger Acceptance.* Proc. Estonian Academy of Sciences, 70 (4).

[3] Kalda, K., Pizzagalli, S.-L., Soe, R.-M., Sell, R., Bellone, M. (2022). *Language of Driving for Autonomous Vehicles.* Applied Sciences, 12 (11).

put your contents here

Validation and verification (V&V) are critical processes in systems engineering and software development that ensure a system meets its intended purpose and functions reliably. Verification is the process of evaluating whether a product, service, or system complies with its specified requirements—essentially asking, “Did we build the system right?” It involves activities such as inspections, simulations, tests, and reviews throughout the development lifecycle. Validation, on the other hand, ensures that the final system fulfills its intended use in the real-world environment—answering the question, “Did we build the right system?” This typically includes user acceptance testing, field trials, and performance assessments under operational conditions. Together, V&V help reduce risks, improve safety and quality, and increase confidence that a system will operate effectively and as expected. In the context of autonomous systems, V&V combines two historical trends. The first from the mechanical systems and the second more recent one from classical digital decision systems. Finally, AI adds further complexity to the testing of the digital decision systems.

For traditional safety-critical systems in automotive, the evolution of V&V has been closely linked to regulatory standards frameworks such as ISO 26262. Key elements of this framework include:

1. System Design Process: A structured development assurance approach for complex systems, incorporating safety certification within the integrated development process.
2. Formalization: The formal definition of system operating conditions, functionalities, expected behaviors, risks, and hazards that must be mitigated.
3. Lifecycle Management: The management of components, systems, and development processes throughout their lifecycle.

The primary objective was to meticulously and formally define the system design, anticipate expected behaviors and potential issues, and comprehend the impact over the product's lifespan.

With the advent of conventional software paradigms, safety-critical V&V adapted by preserving the original system design approach while integrating software as system components. These software components maintained the same overall structure of fault analysis, lifecycle management, and hazard analysis within system design. However, certain aspects required extension. For instance, in the airborne domain, standard DO-178C, which addresses “Software Considerations in Airborne Systems and Equipment Certification,” updated the concept of hazard from physical failure mechanisms to functional defects, acknowledging that software does not degrade due to physical processes. Also revised were lifecycle management concepts, reflecting traditional software development practices. Design Assurance Levels (DALs) were incorporated, allowing the integration of software components into system design, functional allocation, performance specification, and the V&V process, akin to SOTIF in the automotive industry.

Table one above shows the difference between ISO 26262 and SOTIF. In general, the fundamental characteristics of digital software systems are problematic in safety critical systems. However, the IT sector has been a key megatrend which has transformed the world over the last 50 years. In the process, it has developed large ecosystems around semiconductors, operating systems, communications, and application software. At this point, using these ecosystems is critical to nearly every product’s success, so mixed-domain safety critical products are now a reality. Mixed Domain structures can be classified in three broad paradigms each of which have very different V&V requirements: Mechanical Replacement (Big Physical, small Digital), Electronic Adjacent (separate Physical and Digital), autonomy (Big Digital, small Physical). Drive-by-Wire functionality is an example of the mechanical replacement paradigm where the implementation of the original mechanical functionality is done by electronic components (HW/SW). In their initial configurations, these mixed electronic/mechanical systems were physically separated as independent subsystems. In this configuration, the V&V process looked very similar to the traditional mechanical verification process.

The paradigm of separate physical subsystems has the advantage of V&V simplification and safety, but the large disadvantage of component skew and material cost. Thus, an emerging trend has been to build underlying computational fabrics with networking and virtually (through software) separate functionality. From a V&V perspective, this means that the virtual backbone which maintains this separation (ex: RTOS) must be verified to a very high standard. Infotainment systems are an example of Electronics Adjacent integration. Generally, there is an independent IT infrastructure working with the safety critical infrastructure, and from a V&V perspective, they can be validated separately. However, the presence of infotainment systems enables very powerful communication technologies (5G, Bluetooth, etc.) where the cyber-physical system can be impacted by external third parties. From a safety perspective, the simplest method for maintaining safety would be to physically separate these systems. However, this is not typically done because a connection is required to provide “over-the-air” updates to the device. Thus, the V&V capability must again verify the virtual safeguards against malicious intent are robust. Finally, the last level of integration is in the context of autonomy. In autonomy, the processes of sensing, perception, location services, path planning envelope the traditional mechanical functionality.

Moving beyond software, AI has built a “learning” paradigm. In this paradigm, there is a period of training where the AI machine “learns” from data to build its own rules, and in this case, learning is defined on top of traditional optimization algorithms which try to minimize some notion of error. This effectively is data driven software development as shown in figure below. However,there are profound differences between AI software and conventional software. The introduction of AI generated software introduces significant issues to the V&V task as shown in table 2 below.

[raivo.sell][✓ raivo.sell, 2025-09-18]

As discussed earlier, generic V&V process consists of testing the product under test within the ODD. This is generally done with a number of techniques. The central paradigm is to generate a test, execute the test, and have a clear criteria for correctness. Three major styles of intelligent test generation are currently active: physical testing, real-world seeding, and virtual testing.

1. Physical Testing :Typically, physical scaling is the most expensive method to verify functionality. However, Tesla has built a flow where their existing fleet is a large distributed testbed. Using this fleet, Tesla's approach to autonomous driving uses a sophisticated data pipeline and deep learning system designed to process vast amounts of sensor data efficiently. In this flow, the scenario under construction is the one driven by the driver, and the criterion for correctness is the driver's corrective action. Behind the scenes, the global verification flow can be managed by large databases and supercomputers (DoJo) . By employing this methodology, Tesla knows that its scenarios are always valid. However, there are challenges with this approach. First, the real world moves very slowly in terms of new unique situations. Second, by definition the scenarios seen are very much tied to the market presence of Tesla, so not predictive of new situations. Finally, the process of capturing data, discerning an error, and building corrective action is non-trivial. At the extreme, this process is akin to taking crash logs from broken computers, diagnosing them, and building the fixes.
2. Real-World Seeding: Another line of test generation is to use physical situations as a seed for further virtual testing. Pegasus, the seminal project initiated in Germany, took such an approach. The project emphasized a scenario-based testing methodology which used observed data from real-world conditions as a base. Another similar effort comes from Warwick University with a focus on test environments, safety analysis, scenario-based testing, and safe AI. One of the contributions from Warwick is Safety Pool Scenario Database. Databases and seeding methods, especially of interesting situations, offer some value, but of course, their completeness is not clear. Further, databases of tests are very susceptible to be over optimized by AI algorithms.
3. Virtual Testing: Another important contribution was ASAM OpenSCENARIO 2.0 which is a domain-specific language designed to enhance the development, testing, and validation of Advanced Driver-Assistance Systems (ADAS) and Automated Driving Systems (ADS). A high-level language allows for a symbolic higher level description of the scenario with an ability to grow in complexity by rules of composition. Underneath the symbolic apparatus are pseudo-random test generation which can scale the scenario generation process. The randomness also offers a chance to expose “unknown-unknown” errors.

Beyond component validation, there have been proposed solutions specifically for autonomous systems such as UL 4600, “Standard for Safety for the Evaluation of Autonomous Products.” Similar to ISO 26262/SOTIF, UL 4600 has a focus on safety risks across the full lifecycle of the product and introduces a structured “safety case” approach. The crux of this methodology is to document and justify how autonomous systems meet safety goals. It also emphasizes the importance of identifying and validating against a wide range of real-world scenarios, including edge cases and rare events. There is also a focus on including human-machine interactions.

What kind of testing infrastructure is required to execute on these various methodologies ?

The baseline for automotive physical testing are facilities for crash testing, road variations, and weather effects. These are generally in private and shared test tracks around the world. For autonomy, several levels of test infrastructure have emerged around the topics of sensors, test tracks, and virtual simulation.

Figure: Anechoic Chamber

For sensors, important equipment includes: - Anechoic Chambers: These chambers are characterized by their anechoic (echo-free) interior, meaning they are designed to completely absorb sound or electromagnetic waves to eliminate reflections from the walls, ceiling, and sometimes the floor. - Fully Anechoic Chambers (FAC): These chambers have all interior surfaces (walls, ceiling, and floor) covered with RF absorbing materials, creating an environment free from reflections. They are ideal for high-precision measurements like antenna testing or situations where a free-space environment is needed. - Semi-Anechoic Chambers (SAC): In this type, the walls and ceiling are covered with absorbing materials, while the floor remains reflective (often a metal ground plane). This reflective floor helps simulate real-world environments, such as devices operating on the ground. Semi-anechoic chambers are commonly used for general EMC (Electromagnetic Compatibility) testing. - RF Shielded Rooms (Faraday Cages): These are enclosed rooms designed to block the entry or exit of electromagnetic radiation. They are constructed with a conductive shield (typically copper or other metals) around the walls, ceiling, and floor, minimizing the entry or exit of electromagnetic interference (EMI). They are a fundamental component of many EMI testing facilities. - Reverberation Chambers: These chambers intentionally use resonances and reflections within the chamber to create a statistically uniform electromagnetic field. They can accommodate larger and more complex test setups and are particularly useful for immunity testing where the device is exposed to interference from all directions. However, their performance can be limited at lower frequencies.

Figure: Zalazone Autonomous Test Track

In terms of test tracks, traditional test tracks which were used for purposes for mechanical testing have been extended for testing autonomy functions. A recent example shown in the figure above is ZalaZONE, a large test track located in Hungary. ZalaZONE integrates both conventional vehicle testing infrastructure and next-generation smart mobility features. One of its standout components is the Smart City Zone, which simulates real-world urban environments with intersections, roundabouts, pedestrian crossings, and public transport scenarios. This allows for comprehensive testing of urban-level autonomy, V2X communication, and AI-driven mobility solutions in a controlled yet realistic environment. The facility includes a dedicated highway and rural road section to support the evaluation of higher-speed autonomous functions such as adaptive cruise control, lane-keeping, and safe overtaking. A high-speed oval enables long-duration endurance testing and consistent-speed trials for autonomous or connected vehicles. The dynamic platform provides a flat, open space for vehicle dynamics testing, such as automated emergency braking, evasive maneuvers, and trajectory planning, while both wet and dry handling courses allow for testing on varied friction surfaces under critical scenarios. ZalaZONE is also equipped with advanced V2X and 5G infrastructure, including roadside units (RSUs) and edge computing systems, to enable real-time communication and data exchange between vehicles and infrastructure—critical for cooperative driving and sensor validation. Additionally, an off-road section supports testing for SUVs, military vehicles, and trucks in rough terrain conditions. The facility is complemented by EMC testing capabilities and plans for climate-controlled testing chambers, enhancing its support for environmental and regulatory testing. ZalaZONE also provides integration with simulation and digital twin environments. Through platforms such as IPG CarMaker and AVL tools, developers can carry out software-in-the-loop (SIL) and hardware-in-the-loop (HIL) testing in parallel with on-track validation.

Figure: Carla Simulator

Finally, a great deal of simulation is done virtually. Simulation plays a critical role in the development and validation of autonomous vehicles (AVs), allowing developers to test perception, planning, and control systems in a wide range of scenarios without physical risk. Among the most prominent tools is CARLA, an open-source simulator built for academic and research use, known for its realistic urban environments, support for various sensors (LiDAR, radar, cameras), and integration with ROS. It’s widely adopted for prototyping and reinforcement learning in AVs. In the commercial space, “rFpro” is a leading choice for OEMs and Tier-1 suppliers, offering photorealistic environments and precise sensor modeling with sub-millimeter accuracy—essential for validating sensor fusion algorithms. Similarly, “IPG CarMaker” and “dSPACE ASM” provide powerful closed-loop environments ideal for testing vehicle dynamics and ADAS features, especially in hardware-in-the-loop (HIL) and software-in-the-loop (SIL) setups. These tools are tightly integrated with MATLAB/Simulink and real-time hardware for embedded control testing. For large-scale and safety-critical simulations, platforms like “VIRES VTD” and “Applied Intuition” are favored due to their compliance with industry standards like ASAM OpenX and ISO 26262, and their ability to model thousands of edge-case scenarios. “NVIDIA DRIVE Sim”, built on the Omniverse platform, is used to generate synthetic data for training and validating neural networks and digital twins, offering GPU-accelerated realism that aids perception system testing. Finally, simulators like “Cognata” and “MathWorks' Automated Driving Toolbox” serve niche but critical roles—Cognata provides city-scale environments for scenario testing and safety validation, while MathWorks' tools are widely used for algorithm development and control prototyping, especially in academia and early-stage design. Each simulator has a specific focus—some prioritize sensor realism, others full-system integration or large-scale scenario generation—so selection depends on whether the goal is research, real-time control testing, or safety validation for deployment.

[raivo.sell][✓ raivo.sell, 2025-09-18]

In terms of challenges, autonomy is very much in the early innings. Broadly speaking. the challenges can be split into three broad categories. First, the core technology elements within the autonomy pipeline (sensors, location services, perception, and path planning, the algorithms and methodology for demonstrating safety, and finally business economics.

Autonomous vehicles rely on a suite of sensors—such as LiDAR, radar, cameras, GPS, and ultrasonic devices—to perceive and interpret their surroundings. However, each of these sensor types faces inherent limitations, particularly in challenging environmental conditions. Cameras struggle with low light, glare, and weather interference like rain or fog, while LiDAR can suffer from backscatter in fog or snow. Radar, though more resilient in poor weather, provides lower spatial resolution, making it less effective for detailed object classification. These environmental vulnerabilities reduce the reliability of perception systems, especially in safety-critical scenarios. Another major challenge lies in the integration of multiple sensor types through sensor fusion. Achieving accurate, real-time fusion demands precise temporal synchronization and spatial calibration, which can drift over time due to mechanical or thermal stresses. Furthermore, sensors are increasingly exposed to cybersecurity threats. GPS and LiDAR spoofing, or adversarial attacks on camera-based recognition systems, can introduce false data or mislead decision-making algorithms, necessitating robust countermeasures at both the hardware and software levels. Sensor systems also face difficulties with occlusion and semantic interpretation. Many sensors require line-of-sight to function properly, so their performance degrades in urban settings with visual obstructions like parked vehicles or construction. Even when objects are detected, understanding their intent—such as whether a pedestrian is about to cross the street—remains a challenge for machine learning models. Meanwhile, high-resolution sensors generate vast data streams, straining onboard processing and communication bandwidth, and creating trade-offs between resolution, latency, and energy efficiency. Lastly, practical concerns such as cost, size, and durability hinder mass adoption. LiDAR units, while highly effective, are often expensive and mechanically complex. Cameras and radar must also be ruggedized to withstand weather and vibration without degrading in performance. Compounding these issues is the lack of standardized validation methods to assess sensor reliability under varied real-world conditions, making it difficult for developers and regulators to establish trust and ensure safety across diverse operational domains.

The “perception system” is at the core of autonomous vehicle functionality, enabling the car to understand and interpret its surroundings in real time. It processes data from multiple sensors—cameras, LiDAR, radar, and ultrasonic devices—to detect, classify, and track objects. The perception system struggles with “semantic understanding and edge cases.” While object detection and classification have improved with deep learning, these models often fail in rare or unusual scenarios—like an overturned vehicle, a pedestrian in costume, or construction detours. Understanding the context and intent behind actions (e.g., whether a pedestrian is about to cross) is even harder. This lack of true situational awareness can lead to poor decision-making and is a key challenge for Level 4 and 5 autonomy. Also, the “computational burden” of real-time perception—especially with high-resolution inputs—creates constraints in terms of processing power, thermal management, and latency. Balancing model accuracy with speed, and ensuring system performance across embedded platforms, is a persistent engineering challenge.

Location services—often referred to as localization—are essential to autonomous vehicles (AVs), enabling them to determine their precise position within a map or real-world environment. While traditional GPS offers basic positioning, autonomous vehicles require “centimeter-level accuracy,” robustness, and real-time responsiveness, all of which present significant challenges. One major challenge is the “limited accuracy and reliability of GNSS (Global Navigation Satellite Systems)” such as GPS, especially in urban canyons, tunnels, or areas with dense foliage. Buildings can block or reflect satellite signals, leading to multi-path errors or complete signal loss. While techniques like Real-Time Kinematic (RTK) correction and augmentation via ground stations improve accuracy, these solutions can be expensive, infrastructure-dependent, and still prone to failure in GNSS-denied environments. To compensate, AVs often combine GPS with “sensor-based localization,” including LiDAR, cameras, and IMUs (inertial measurement units), which enable map-based and dead-reckoning approaches. Sensor-based dead reckoning using IMUs and odometry can help bridge short GNSS outages, but “drift accumulates over time,” and errors can compound, especially during sharp turns, vibrations, or tire slippage. Finally, “map-based localization” depends on the availability of high-definition (HD) maps that include detailed features like lane markings, curbs, and traffic signs. These maps are costly to build and maintain, and they can become outdated quickly due to road changes, construction, or temporary obstructions—leading to mislocalization.

Path planning in autonomous vehicles is a complex and safety-critical task that involves determining the vehicle's trajectory from its current position to a desired destination while avoiding obstacles, complying with traffic rules, and ensuring passenger comfort. One of the most significant challenges in this area is dealing with dynamic and unpredictable environments. The behavior of other road users—such as pedestrians, cyclists, and human drivers—can be erratic, requiring the planner to continuously adapt in real time. Predicting these agents' intentions is inherently uncertain and often leads to either overly cautious or unsafe behavior if misjudged. Real-time responsiveness is another major constraint. Path planning must be executed with low latency while factoring in a wide range of considerations including traffic laws, road geometry, sensor data, and vehicle dynamics. This requires balancing optimality, safety, and computational efficiency within strict time limits. Additionally, the planner must account for the vehicle’s physical constraints such as turning radius, acceleration, and braking limits, especially in complex maneuvers like unprotected turns or obstacle avoidance. Another persistent challenge is operating with incomplete or noisy information. Sensor occlusion, poor weather, or localization drift can obscure critical details such as road markings, traffic signs, or nearby objects. Planners must therefore make decisions under uncertainty, which adds complexity and risk. Moreover, the vehicle must navigate complex and often-changing road topologies—like roundabouts, construction zones, or temporary detours—where map data may be outdated or ambiguous. Finally, the need for continuous replanning introduces issues of robustness and comfort. The path planning system must frequently adjust trajectories to respond to new inputs, but abrupt changes can degrade ride quality or destabilize the vehicle. All of this must be done while maintaining rigorous safety guarantees, ensuring that every planned path can be verified as collision-free and legally compliant. Developing a system that meets these demands across diverse environments and edge cases remains one of the toughest challenges in achieving fully autonomous driving.

Algorithms and Methodology for Safety:

A major bottleneck remains the inability to fully validate AI behavior, with a need for more rigorous methods to assess completeness, generate targeted test cases, and bound system behavior. Advancements in explainable AI, digital twins, and formal methods are seen as promising paths forward. Additionally, current systems lack scalable abstraction hierarchies—hindering the ability to generalize component-level validation to system-level assurance. To build trust with users and regulators, the industry must also adopt a “progressive safety framework,” clearly showing continuous improvement, regression checks during over-the-air (OTA) updates, and lessons learned from real-world failures.

In terms of “V&V test apparatuses,” both virtual and physical tools are emphasized. Virtual environments will play a key role in supporting evolving V&V methodologies, necessitating ongoing work from standards bodies like ASAM. Physical test tracks must evolve to not only replicate real-world scenarios efficiently but also validate the accuracy of their virtual counterparts—envisioned through a “movie set” model that can quickly stage complex scenarios. Another emerging concern is “electromagnetic interference (EMI),” especially due to the widespread use of active sensors. Traditional static EMI testing methods are insufficient, and there is a need for dynamic, programmable EMI testing environments tailored to cyber-physical systems.

Finally, a rising concern is around cybersecurity in autonomous systems. These systems introduce systemic vulnerabilities that span from hardware to software, necessitating government-level oversight. Key sensor modalities like LiDAR, GPS, and radar are susceptible to spoofing, and detecting such threats is an urgent research priority. The V&V process itself must evolve to minimize exposure to adversarial attacks, effectively treating security as an intrinsic constraint within system validation, not an afterthought.

Business Models and Supply Chain:

Robo-taxis, or autonomous ride-hailing vehicles, represent a promising use case for autonomous vehicle (AV) technology, with the potential to transform urban mobility by offering on-demand, driverless transportation. Key use models include urban ride-hailing in city centers, first- and last-mile transit to connect riders with public transportation, airport and hotel shuttle services in geofenced areas, and mobility on closed campuses like universities or corporate parks. These models aim to increase vehicle utilization, reduce transportation costs, and offer greater convenience, particularly in environments where human-driver costs are a major factor. However, the business challenges are substantial. The development and deployment of robo-taxi fleets require enormous capital investment in hardware, software, testing, and infrastructure. Operational costs remain high, particularly in the early stages when human safety drivers, detailed maps, and limited deployment zones are still necessary. Regulatory uncertainty also hampers scalability, with different jurisdictions applying inconsistent safety, insurance, and operational standards. This makes expansion slow and costly.

In addition, consumer trust in autonomous systems remains fragile. High-profile incidents have raised safety concerns, and many riders may be hesitant to use driverless vehicles, especially in unfamiliar or emergency situations. Infrastructure constraints—such as poor road markings or limited connectivity—further limit the environments in which robo-taxis can operate reliably. Meanwhile, the path to profitability is challenged by competitive fare pricing, fleet maintenance logistics, and integration with broader transportation networks. Overall, while robo-taxis offer significant long-term promise, their success hinges on overcoming a complex mix of technological, regulatory, and business barriers.

The evolving economics of the semiconductor industry pose a significant challenge for low-volume markets, where custom chip development is often not cost-effective. As a result, autonomous and safety-critical systems must increasingly rely on Commercial Off-The-Shelf (COTS) components, making it essential to develop methodologies that can ensure security, reliability, and performance using these standardized parts. This shift places greater emphasis on designing systems that are resilient and adaptable, even without custom silicon. Additionally, traditional concerns like field maintainability, lifetime cost, and design-for-supply-chain practices—common in mechanical and industrial engineering—must now be applied to electronics and embedded systems. As electronic components dominate modern products, a more holistic design approach is needed to manage downstream supply chain implications. The trend toward software-defined vehicles reflects this need, promoting deeper integration between hardware and software suppliers. To further enhance supply chain resilience, there's a push to standardize around a smaller set of high-volume chips and embrace flexible, programmable hardware fabrics that integrate digital, analog, and software elements. This architecture shift is key to mitigating supply disruptions and maintaining long-term system viability. Finally, “maintainability” also implies the availability of in-field repair facilities which must be upgraded to handle autonomy.

[raivo.sell][✓ raivo.sell, 2025-09-18]

Autonomy is part of the next big megatrend in electronics which is likely to change society. As a new technology, there are a large number of open research problems. These problems can be classified in four broad categories: Autonomy hardware, Autonomy Software, Autonomy Ecosystem, and Autonomy Business models. In terms of hardware, autonomy consists of a mobility component (increasingly becoming electric), sensors, and computation.

Research in sensors for autonomy is rapidly evolving, with a strong focus on “sensor fusion, robustness, and intelligent perception.” One exciting area is “multi-modal sensor fusion,” where data from LiDAR, radar, cameras, and inertial sensors are combined using AI to improve perception in complex or degraded environments. Researchers are developing uncertainty-aware fusion models that not only integrate data but also quantify confidence levels, essential for safety-critical systems. There's also growing interest in “event-based cameras” and “adaptive LiDAR,” which offer low-latency or selective scanning capabilities for dynamic scenes, while self-supervised learning enables autonomous systems to extract semantic understanding from raw, unlabeled sensor data. Another critical thrust is the development of resilient and context-aware sensors. This includes sensors that function in all-weather conditions, such as “FMCW radar” and “polarization-based vision,” and systems that can detect and correct for sensor faults or spoofing in real-time. Researchers are also exploring “terrain-aware sensing,” “semantic mapping,” and “infrastructure-to-vehicle (I2V)” sensor networks to extend situational awareness beyond line-of-sight. Finally, sensor co-design—where hardware, placement, and algorithms are optimized together—is gaining traction, especially in “edge computing architectures” where real-time processing and low power are crucial. These advances support autonomy not just in cars, but also in drones, underwater vehicles, and robotic systems operating in unstructured or GPS-denied environments.

In terms of computation, exciting research focuses on enabling real-time decision-making in environments where cloud connectivity is limited, latency is critical, and power is constrained. One prominent area is the “co-design of perception and control algorithms with edge hardware,” such as integrating neural network compression, quantization, and pruning techniques to run advanced AI models on embedded systems (e.g., NVIDIA Jetson, Qualcomm RB5, or custom ASICs). Research also targets “dynamic workload scheduling,” where sensor processing, localization, and planning are intelligently distributed across CPUs, GPUs, and dedicated accelerators based on latency and energy constraints. Another major focus is on “adaptive, context-aware computing,” where the system dynamically changes its computational load or sensing fidelity based on situational awareness—for instance, increasing compute resources during complex maneuvers or reducing them during idle cruising. Related to this is “event-driven computing” and “neuromorphic architectures” that mimic biological efficiency to reduce energy use in perception tasks. Researchers are also exploring “secure edge execution,” such as trusted computing environments and runtime monitoring to ensure deterministic behavior under adversarial conditions. Finally, “collaborative edge networks,” where multiple autonomous agents (vehicles, drones, or infrastructure nodes) share compute and data at the edge in real time, open new frontiers in swarm autonomy and decentralized intelligence.

Finally, as there is a shift towards “software defined vehicles,” there is an increasing need to develop computing hardware architectures bottom-up with critical properties of software reuse and underlying hardware innovation. This process mimics computer architectures in information technology, but does not exist in the world of autonomy today.

In terms of software, important system functions such as perception, path planning, and location services sit in software/AI layer. While somewhat effective, AV stacks are quite a bit less effective then a human who can navigate the world spending only about a 100 watts of power. There are a number of places where humans/machine autonomy differ. These include:

1. Focus: Humans have the idea of focus and peripheral vision…whereas AVs monitor all directions all the time. This has implications on power, data, and computation
2. Movement based Perception: Humans use movement as a key signature for identification. In contrast, current perception engines effectively try to work on static photos.
3. Perception based recognition: Humans use an expectation of the future movement of objects to limit computation. This technique has advantages in computation, but is not currently used in AVs.

Thus, in addition to traditional machine learning techniques, newer AI architectures with properties of robustness, power/compute efficiency, and effectiveness are open research problems.

In terms of Ecosystem, key open research problems exist in areas such as safety validation, V2X communication, and ecosystem partners.

Verification and validation (V\&V) for autonomous systems is evolving rapidly, with key research focused on making AI-driven behavior both “provably safe and explainable.” One major direction involves “bounding AI behavior” using formal methods and developing “explainable AI” (XAI) that supports safety arguments regulators and engineers can trust. Researchers is also focused on “rare and edge-case scenario generation” through adversarial learning, simulation, and digital twins, aiming to create test cases that challenge the limits of perception and planning systems. Defining new “coverage metrics”—such as semantic or risk-based coverage—has become crucial, as traditional code coverage doesn’t capture the complexity of non-deterministic AI components. Another active area is “scalable system-level V&V,” where component-level validation must support higher-level safety guarantees. This includes “compositional reasoning,” contracts-based design, and model-based safety case automation. The integration of digital twins for closed-loop simulation and real-time monitoring is enabling continuous validation even post-deployment. In parallel, “cybersecurity-aware V&V” is emerging, focusing on spoofing resilience and securing the validation pipeline itself. Finally, standardization of simulation formats (e.g., OpenSCENARIO, ASAM) and the rise of “test infrastructure-as-code” are laying the groundwork for scalable, certifiable autonomy, especially under evolving regulatory frameworks like UL 4600 and ISO 21448.

One of the ecosystem aids to autonomy maybe connection to the infrastructure and of course, in mixed human/machine environments there is the natural Human Machine Interface (HMI). Key research in V2X (Vehicle-to-Everything) for autonomy centers on enabling cooperative behavior and enhanced situational awareness through low-latency, secure communication. A major area of focus is on “reliable, high-speed communication” via technologies like “C-V2X and 5G/6G,” which are critical for supporting time-sensitive autonomous functions such as coordinated lane changes, intersection management, and emergency response. Closely linked is the development of “edge computing architectures,” where V2X messages are processed locally to reduce latency and support real-time decision-making. Research is active in “cooperative perception,” where vehicles and infrastructure share sensor data to extend the field of view beyond occlusions, enabling safer navigation in complex urban environments. Another core research direction is the integration of “smart infrastructure and digital twins,” where roadside sensors provide real-time updates to HD maps and augment vehicle perception. This is essential for detecting dynamic road conditions, construction zones, and temporary signage. In parallel, ensuring “security and privacy in V2X communication” is a growing concern. Work is underway on encrypted, authenticated protocols and on methods to detect and respond to malicious actors or faulty data. Finally, standardization and interoperability are vital for large-scale deployment; efforts are focused on harmonizing communication protocols across vendors and regions and on developing robust, scenario-based testing frameworks that incorporate both simulation and physical validation. Finally, an open research issue is the tradeoff between individual autonomy and dependence on an infrastructure. Associated with infrastructure dependence are open issues of legal liability, business model, or cost.

Human-Machine Interface (HMI) for autonomy remains an area with several open research and design challenges, particularly around trust, control, and situational awareness. One major issue is how to build “appropriate trust and transparency” between users and autonomous systems. Current interfaces often fail to clearly convey the vehicle’s capabilities, limitations, or decision-making rationale, which can lead to overreliance or confusion. There's a delicate balance between providing sufficient information to promote understanding and avoiding cognitive overload. Additionally, ensuring “safe and intuitive transitions of control,” especially in Level 3 and Level 4 autonomy, remains a critical concern. Drivers may take several seconds to re-engage during a takeover request, and the timing, modality, and clarity of such prompts are not yet standardized or optimized across systems. Another set of challenges lies in maintaining “situational awareness” and designing “adaptive, accessible interfaces.” Passive users in autonomous systems tend to disengage, losing track of the environment, which can be dangerous during unexpected events. Effective HMI must offer context-sensitive feedback using visual, auditory, or haptic cues while adapting to the user’s state, experience level, and accessibility needs. Moreover, autonomous vehicles currently lack effective ways to interact with external actors—such as pedestrians or other drivers—replacing human cues like eye contact or gestures. Developing standardized, interpretable external HMIs, a language of driving, remains an active area of research. Finally, a lack of unified metrics and regulatory standards for evaluating HMI effectiveness further complicates design validation, making it difficult to compare systems or ensure safety across manufacturers.

Finally, autonomy will have implications on topics such as civil infrastructure guidance, field maintenance, interaction with emergency services, interaction with disabled and young riders, insurance markets, and most importantly the legal profession. There are many research issues underlying all of these topics.

In terms of business models, use models and their implications for supply chain are open research problems. For example, for the supply chain, the critical technology is semiconductors which is highly sensitive to very high volume. For example, the largest market in mobility, the auto industry, is approx. 10% of semiconductor volume, and the other forms (airborne, marine, space) are orders-of-magnitude lower. From a supply chain point perspective, a small number of skews which service a large market are ideal. The research problem is: What should be the nature of these very scalable components. In terms of end-markets, autonomy in traditional transportation is likely to lead to a reduction in unit volume. Why? With autonomy, one can get much higher utilization (vs the < 5% in today's automobiles). However, it is also likely that autonomy unleashes a broad class of solutions in markets such as agriculture, warehouses, distribution, delivery, and more. Micromobility applications in particular offer some interesting options for very high volumes. The exact nature of the applications is an open research problem.