| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| en:safeav:avt:challenges [2025/08/02 02:13] – rahulrazdan | en:safeav:avt:challenges [2025/10/20 18:01] (current) – raivo.sell |
|---|
| {{:en:iot-open:czapka_m.png?50| Masters (2nd level) classification icon }} | {{:en:iot-open:czapka_m.png?50| Masters (2nd level) classification icon }} |
| |
| <todo @raivo.sell></todo> | <todo @raivo.sell #raivo.sell:2025-09-18></todo> |
| |
| {{:en:safeav:avt:av_stack.jpg?600|}} | {{:en:safeav:avt:av_stack.jpg?600|}} |
| The "perception system" is at the core of autonomous vehicle functionality, enabling the car to understand and interpret its surroundings in real time. It processes data from multiple sensors—cameras, LiDAR, radar, and ultrasonic devices—to detect, classify, and track objects. The perception system struggles with "semantic understanding and edge cases." While object detection and classification have improved with deep learning, these models often fail in rare or unusual scenarios—like an overturned vehicle, a pedestrian in costume, or construction detours. Understanding the context and intent behind actions (e.g., whether a pedestrian is about to cross) is even harder. This lack of true situational awareness can lead to poor decision-making and is a key challenge for Level 4 and 5 autonomy. Also, the "computational burden" of real-time perception—especially with high-resolution inputs—creates constraints in terms of processing power, thermal management, and latency. Balancing model accuracy with speed, and ensuring system performance across embedded platforms, is a persistent engineering challenge. | The "perception system" is at the core of autonomous vehicle functionality, enabling the car to understand and interpret its surroundings in real time. It processes data from multiple sensors—cameras, LiDAR, radar, and ultrasonic devices—to detect, classify, and track objects. The perception system struggles with "semantic understanding and edge cases." While object detection and classification have improved with deep learning, these models often fail in rare or unusual scenarios—like an overturned vehicle, a pedestrian in costume, or construction detours. Understanding the context and intent behind actions (e.g., whether a pedestrian is about to cross) is even harder. This lack of true situational awareness can lead to poor decision-making and is a key challenge for Level 4 and 5 autonomy. Also, the "computational burden" of real-time perception—especially with high-resolution inputs—creates constraints in terms of processing power, thermal management, and latency. Balancing model accuracy with speed, and ensuring system performance across embedded platforms, is a persistent engineering challenge. |
| |
| Location services—often referred to as localization—are essential to autonomous vehicles (AVs), enabling them to determine their precise position within a map or real-world environment. While traditional GPS offers basic positioning, autonomous vehicles require "centimeter-level accuracy," robustness, and real-time responsiveness, all of which present significant challenges. T | Location services—often referred to as localization—are essential to autonomous vehicles (AVs), enabling them to determine their precise position within a map or real-world environment. While traditional GPS offers basic positioning, autonomous vehicles require "centimeter-level accuracy," robustness, and real-time responsiveness, all of which present significant challenges. |
| One major challenge is the "limited accuracy and reliability of GNSS (Global Navigation Satellite Systems)" such as GPS, especially in urban canyons, tunnels, or areas with dense foliage. Buildings can block or reflect satellite signals, leading to multi-path errors or complete signal loss. While techniques like Real-Time Kinematic (RTK) correction and augmentation via ground stations improve accuracy, these solutions can be expensive, infrastructure-dependent, and still prone to failure in GNSS-denied environments. To compensate, AVs often combine GPS with "sensor-based localization," including LiDAR, cameras, and IMUs (inertial measurement units), which enable map-based and dead-reckoning approaches. Finally, "map-based localization" depends on the availability of high-definition (HD) maps that include detailed features like lane markings, curbs, and traffic signs. These maps are costly to build and maintain, and they can become outdated quickly due to road changes, construction, or temporary obstructions—leading to mislocalization. Sensor-based dead reckoning using IMUs and odometry can help bridge short GNSS outages, but "drift accumulates over time," and errors can compound, especially during sharp turns, vibrations, or tire slippage. Additionally, environmental factors like heavy rain, snow, or dirt can degrade the performance of LiDAR and vision-based localization systems. Poor lighting or dynamic changes in the environment (e.g., moving vehicles or crowds) further reduce reliability. | One major challenge is the "limited accuracy and reliability of GNSS (Global Navigation Satellite Systems)" such as GPS, especially in urban canyons, tunnels, or areas with dense foliage. Buildings can block or reflect satellite signals, leading to multi-path errors or complete signal loss. While techniques like Real-Time Kinematic (RTK) correction and augmentation via ground stations improve accuracy, these solutions can be expensive, infrastructure-dependent, and still prone to failure in GNSS-denied environments. To compensate, AVs often combine GPS with "sensor-based localization," including LiDAR, cameras, and IMUs (inertial measurement units), which enable map-based and dead-reckoning approaches. Sensor-based dead reckoning using IMUs and odometry can help bridge short GNSS outages, but "drift accumulates over time," and errors can compound, especially during sharp turns, vibrations, or tire slippage. Finally, "map-based localization" depends on the availability of high-definition (HD) maps that include detailed features like lane markings, curbs, and traffic signs. These maps are costly to build and maintain, and they can become outdated quickly due to road changes, construction, or temporary obstructions—leading to mislocalization. |
| |
| Path planning in autonomous vehicles is a complex and safety-critical task that involves determining the vehicle's trajectory from its current position to a desired destination while avoiding obstacles, complying with traffic rules, and ensuring passenger comfort. One of the most significant challenges in this area is dealing with dynamic and unpredictable environments. The behavior of other road users—such as pedestrians, cyclists, and human drivers—can be erratic, requiring the planner to continuously adapt in real time. Predicting these agents' intentions is inherently uncertain and often leads to either overly cautious or unsafe behavior if misjudged. Real-time responsiveness is another major constraint. Path planning must be executed with low latency while factoring in a wide range of considerations including traffic laws, road geometry, sensor data, and vehicle dynamics. This requires balancing optimality, safety, and computational efficiency within strict time limits. Additionally, the planner must account for the vehicle’s physical constraints such as turning radius, acceleration, and braking limits, especially in complex maneuvers like unprotected turns or obstacle avoidance. Another persistent challenge is operating with incomplete or noisy information. Sensor occlusion, poor weather, or localization drift can obscure critical details such as road markings, traffic signs, or nearby objects. Planners must therefore make decisions under uncertainty, which adds complexity and risk. Moreover, the vehicle must navigate complex and often-changing road topologies—like roundabouts, construction zones, or temporary detours—where map data may be outdated or ambiguous. Finally, the need for continuous replanning introduces issues of robustness and comfort. The path planning system must frequently adjust trajectories to respond to new inputs, but abrupt changes can degrade ride quality or destabilize the vehicle. All of this must be done while maintaining rigorous safety guarantees, ensuring that every planned path can be verified as collision-free and legally compliant. Developing a system that meets these demands across diverse environments and edge cases remains one of the toughest challenges in achieving fully autonomous driving. | Path planning in autonomous vehicles is a complex and safety-critical task that involves determining the vehicle's trajectory from its current position to a desired destination while avoiding obstacles, complying with traffic rules, and ensuring passenger comfort. One of the most significant challenges in this area is dealing with dynamic and unpredictable environments. The behavior of other road users—such as pedestrians, cyclists, and human drivers—can be erratic, requiring the planner to continuously adapt in real time. Predicting these agents' intentions is inherently uncertain and often leads to either overly cautious or unsafe behavior if misjudged. Real-time responsiveness is another major constraint. Path planning must be executed with low latency while factoring in a wide range of considerations including traffic laws, road geometry, sensor data, and vehicle dynamics. This requires balancing optimality, safety, and computational efficiency within strict time limits. Additionally, the planner must account for the vehicle’s physical constraints such as turning radius, acceleration, and braking limits, especially in complex maneuvers like unprotected turns or obstacle avoidance. Another persistent challenge is operating with incomplete or noisy information. Sensor occlusion, poor weather, or localization drift can obscure critical details such as road markings, traffic signs, or nearby objects. Planners must therefore make decisions under uncertainty, which adds complexity and risk. Moreover, the vehicle must navigate complex and often-changing road topologies—like roundabouts, construction zones, or temporary detours—where map data may be outdated or ambiguous. Finally, the need for continuous replanning introduces issues of robustness and comfort. The path planning system must frequently adjust trajectories to respond to new inputs, but abrupt changes can degrade ride quality or destabilize the vehicle. All of this must be done while maintaining rigorous safety guarantees, ensuring that every planned path can be verified as collision-free and legally compliant. Developing a system that meets these demands across diverse environments and edge cases remains one of the toughest challenges in achieving fully autonomous driving. |
| | |
| While somewhat effective, AV stacks are quite a bit less effective then a human who can navigate the world spending only about a 100 watts of power. There are a number of places where humans/machine autonomy differ. These include: | |
| - Focus: Humans have the idea of focus and peripheral vision...whereas AVs monitor all directions all the time. This has implications on power, data, and computation | |
| - Movement based Perception: Humans use movement as a key signature for identification. In contrast, current perception engines effectively try to work on static photos. | |
| - Perception based recognition: Humans use an expectation of the future movement of objects to limit computation. This technique has advantages in computation, but is not currently used in AVs. | |
| | |
| | |
| |
| |
| {{:en:safeav:avt:validation.jpg?600|}} | {{:en:safeav:avt:validation.jpg?600|}} |
| A major bottleneck remains the inability to fully validate AI behavior, with a need for more rigorous methods to assess completeness, generate targeted test cases, and bound system behavior. Advancements in explainable AI, digital twins, and formal methods are seen as promising paths forward. Additionally, current systems lack scalable abstraction hierarchies—hindering the ability to generalize component-level validation to system-level assurance. Borrowing principles from scalable semiconductor design could aid in building such structures. To build trust with users and regulators, the industry must also adopt a "progressive safety framework," clearly showing continuous improvement, regression checks during over-the-air (OTA) updates, and lessons learned from real-world failures. | A major bottleneck remains the inability to fully validate AI behavior, with a need for more rigorous methods to assess completeness, generate targeted test cases, and bound system behavior. Advancements in explainable AI, digital twins, and formal methods are seen as promising paths forward. Additionally, current systems lack scalable abstraction hierarchies—hindering the ability to generalize component-level validation to system-level assurance. To build trust with users and regulators, the industry must also adopt a "progressive safety framework," clearly showing continuous improvement, regression checks during over-the-air (OTA) updates, and lessons learned from real-world failures. |
| |
| In terms of "V\&V test apparatuses," both virtual and physical tools are emphasized. Virtual environments will play a key role in supporting evolving V\&V methodologies, necessitating ongoing work from standards bodies like ASAM. Physical test tracks must evolve to not only replicate real-world scenarios efficiently but also validate the accuracy of their virtual counterparts—envisioned through a “movie set” model that can quickly stage complex scenarios. Another emerging concern is "electromagnetic interference (EMI)," especially due to the widespread use of active sensors. Traditional static EMI testing methods are insufficient, and there is a need for dynamic, programmable EMI testing environments tailored to cyber-physical systems. | In terms of "V&V test apparatuses," both virtual and physical tools are emphasized. Virtual environments will play a key role in supporting evolving V&V methodologies, necessitating ongoing work from standards bodies like ASAM. Physical test tracks must evolve to not only replicate real-world scenarios efficiently but also validate the accuracy of their virtual counterparts—envisioned through a “movie set” model that can quickly stage complex scenarios. Another emerging concern is "electromagnetic interference (EMI)," especially due to the widespread use of active sensors. Traditional static EMI testing methods are insufficient, and there is a need for dynamic, programmable EMI testing environments tailored to cyber-physical systems. |
| |
| Finally, a rising concern is around "cybersecurity** in autonomous systems. These systems introduce systemic vulnerabilities that span from hardware to software, necessitating government-level oversight. Key sensor modalities like LiDAR, GPS, and radar are susceptible to spoofing, and detecting such threats is an urgent research priority. The V&V process itself must evolve to minimize exposure to adversarial attacks, effectively treating security as an intrinsic constraint within system validation, not an afterthought. | Finally, a rising concern is around cybersecurity in autonomous systems. These systems introduce systemic vulnerabilities that span from hardware to software, necessitating government-level oversight. Key sensor modalities like LiDAR, GPS, and radar are susceptible to spoofing, and detecting such threats is an urgent research priority. The V&V process itself must evolve to minimize exposure to adversarial attacks, effectively treating security as an intrinsic constraint within system validation, not an afterthought. |
| |
| | |
| {{:en:safeav:avt:waymo-robotaxi.jpg?600|}} | {{:en:safeav:avt:waymo-robotaxi.jpg?600|}} |
| |
| Robo-taxis, or autonomous ride-hailing vehicles, represent a promising use case for autonomous vehicle (AV) technology, with the potential to transform urban mobility by offering on-demand, driverless transportation. Key use models include urban ride-hailing in city centers, first- and last-mile transit to connect riders with public transportation, airport and hotel shuttle services in geofenced areas, and mobility on closed campuses like universities or corporate parks. These models aim to increase vehicle utilization, reduce transportation costs, and offer greater convenience, particularly in environments where human-driver costs are a major factor. | Robo-taxis, or autonomous ride-hailing vehicles, represent a promising use case for autonomous vehicle (AV) technology, with the potential to transform urban mobility by offering on-demand, driverless transportation. Key use models include urban ride-hailing in city centers, first- and last-mile transit to connect riders with public transportation, airport and hotel shuttle services in geofenced areas, and mobility on closed campuses like universities or corporate parks. These models aim to increase vehicle utilization, reduce transportation costs, and offer greater convenience, particularly in environments where human-driver costs are a major factor. However, the business challenges are substantial. The development and deployment of robo-taxi fleets require enormous capital investment in hardware, software, testing, and infrastructure. Operational costs remain high, particularly in the early stages when human safety drivers, detailed maps, and limited deployment zones are still necessary. Regulatory uncertainty also hampers scalability, with different jurisdictions applying inconsistent safety, insurance, and operational standards. This makes expansion slow and costly. |
| | |
| However, the business challenges are substantial. The development and deployment of robo-taxi fleets require enormous capital investment in hardware, software, testing, and infrastructure. Operational costs remain high, particularly in the early stages when human safety drivers, detailed maps, and limited deployment zones are still necessary. Regulatory uncertainty also hampers scalability, with different jurisdictions applying inconsistent safety, insurance, and operational standards. This makes expansion slow and costly. | |
| |
| In addition, consumer trust in autonomous systems remains fragile. High-profile incidents have raised safety concerns, and many riders may be hesitant to use driverless vehicles, especially in unfamiliar or emergency situations. Infrastructure constraints—such as poor road markings or limited connectivity—further limit the environments in which robo-taxis can operate reliably. Meanwhile, the path to profitability is challenged by competitive fare pricing, fleet maintenance logistics, and integration with broader transportation networks. Overall, while robo-taxis offer significant long-term promise, their success hinges on overcoming a complex mix of technological, regulatory, and business barriers. | In addition, consumer trust in autonomous systems remains fragile. High-profile incidents have raised safety concerns, and many riders may be hesitant to use driverless vehicles, especially in unfamiliar or emergency situations. Infrastructure constraints—such as poor road markings or limited connectivity—further limit the environments in which robo-taxis can operate reliably. Meanwhile, the path to profitability is challenged by competitive fare pricing, fleet maintenance logistics, and integration with broader transportation networks. Overall, while robo-taxis offer significant long-term promise, their success hinges on overcoming a complex mix of technological, regulatory, and business barriers. |
| |
| |
| The evolving economics of the semiconductor industry pose a significant challenge for low-volume markets, where custom chip development is often not cost-effective. As a result, autonomous and safety-critical systems must increasingly rely on Commercial Off-The-Shelf (COTS) components, making it essential to develop methodologies that can ensure security, reliability, and performance using these standardized parts. This shift places greater emphasis on designing systems that are resilient and adaptable, even without custom silicon. | The evolving economics of the semiconductor industry pose a significant challenge for low-volume markets, where custom chip development is often not cost-effective. As a result, autonomous and safety-critical systems must increasingly rely on Commercial Off-The-Shelf (COTS) components, making it essential to develop methodologies that can ensure security, reliability, and performance using these standardized parts. This shift places greater emphasis on designing systems that are resilient and adaptable, even without custom silicon. Additionally, traditional concerns like field maintainability, lifetime cost, and design-for-supply-chain practices—common in mechanical and industrial engineering—must now be applied to electronics and embedded systems. As electronic components dominate modern products, a more holistic design approach is needed to manage downstream supply chain implications. The trend toward software-defined vehicles reflects this need, promoting deeper integration between hardware and software suppliers. To further enhance supply chain resilience, there's a push to standardize around a smaller set of high-volume chips and embrace flexible, programmable hardware fabrics that integrate digital, analog, and software elements. This architecture shift is key to mitigating supply disruptions and maintaining long-term system viability. Finally, "maintainability" also implies the availability of in-field repair facilities which must be upgraded to handle autonomy. |
| | |
| Additionally, traditional concerns like field maintainability, lifetime cost, and design-for-supply-chain practices—common in mechanical and industrial engineering—must now be applied to electronics and embedded systems. As electronic components dominate modern products, a more holistic design approach is needed to manage downstream supply chain implications. The trend toward software-defined vehicles reflects this need, promoting deeper integration between hardware and software suppliers. To further enhance supply chain resilience, there's a push to standardize around a smaller set of high-volume chips and embrace flexible, programmable hardware fabrics that integrate digital, analog, and software elements. This architecture shift is key to mitigating supply disruptions and maintaining long-term system viability. | |
| | |
| | |
| |
