1st IJCAI Workshop on Safe Physical AI

Large language models are rapidly being integrated into robotic systems, promising robots that can interpret natural-language commands, reason about open-world tasks, and plan actions with unprecedented flexibility. Yet this same flexibility introduces a new and underappreciated attack surface: the safety guardrails that prevent an LLM from producing harmful text do not prevent an LLM-controlled robot from taking harmful physical actions. In this talk, I will show that LLM-enabled robots are alarmingly easy to jailbreak. Building on adversarial prompting techniques, our RoboPAIR algorithm achieves the first successful jailbreaks of LLM-controlled robots, eliciting unsafe and physically dangerous behaviors across simulated, white-box, and real-world platforms—often with complete success. These results demonstrate that aligning the language model is not equivalent to ensuring the robot it controls is safe. I will then turn to defenses, presenting RoboGuard, a runtime guardrail that grounds high-level safety specifications in contextual reasoning and formal verification to constrain robot behavior. RoboGuard reduces the execution of unsafe plans from over 90% to under 3% while preserving the robot’s ability to complete legitimate tasks. I will close by arguing that the safety of physical AI demands context-aware, action-level guarantees that go beyond text-level alignment, and outline open challenges for building robots we can trust to act in the world.

Robot foundation models (FMs) are rapidly becoming accessible: with modest compute, anyone can download a pretrained model and deploy it on a robot in a home or business “zero-shot”. But deploying a robot FM today often feels like operating a black box: you inherit a model’s capabilities but also its biases and failure modes without fully understanding where they came from. This makes it difficult to predict what a robot can and cannot do and whether it will behave safely and reliably in novel situations. In this talk, I will discuss my group’s work on controlling robot foundation models that we did not train ourselves. Specifically, I will describe a spectrum of control mechanisms that operate on a model’s inputs (e.g., natural language), internals (e.g., the generative process), and outputs (e.g., safety filters) to steer behavior toward safe, aligned, and high-performing outcomes while preserving the broad capabilities acquired during pretraining. Throughout the talk, I will present examples spanning both visuomotor policies and world models, with applications in robotic manipulation.

While Deep RL tackles long-horizon tasks by representation learning, these latent spaces are not interpretable and are hard to use in a safe manner by robots. Cyber-physical systems on the other hand use formal verification to verify safety-critical physical systems controlled by software. Because RL relies on trial and error, it can cause catastrophic failures during training and deployment. Formal verification acts as a mathematical guardrail to prevent these failures. However, classical integration approaches do not enable the robot to learn representations that are both interpretable and verifiable.

In this talk, we will introduce two methods combining deep reinforcement learning and formal verification. STAR is a deep hierarchical reinforcement learning algorithm that uses reachability analysis to devise a latent space grounded in the environment, both useful for more efficient learning by reinforcement learning and interpretable for interaction with humans. Reversely, we also use deep reinforcement learning as a framework to learn temporal rules describing long-horizon activities. These neurosymbolic methods combine the trial-and-error paradigm in continuous settings with formal verification to devise an interpretable representation for human verification and to be used for safe decision-making.

Physical AI systems increasingly generate plausible actions from demonstrations, learned policies, and foundation models. Yet an action that is statistically likely or task-relevant is not necessarily safe in the physical situation at hand. This talk presents the AICOR framework for reason-warranted physical action, which conceptualizes robot manipulation as the closed-loop solution of body motion problems within coupled robot–world systems. Before acting, the robot evaluates candidate actions against an explicit, task-relative model of the current situation, including their grounds, predicted effects and side effects, validity conditions, uncertainty, and safety and recovery margins.

During execution, the robot compares predicted and observed body–world evolution and responds to deviations through regulation, repair, withdrawal, or renewed deliberation. Semantic digital twins, executable action models, and episodic records make these processes operational, inspectable, and empirically assessable. Examples from everyday manipulation illustrate why successful execution alone is insufficient evidence of safe competence. Safe Physical AI therefore requires an architectural transition: from systems that merely generate likely actions to systems that can warrant, bound, monitor, revise, and account for their physical commitments.

AI in its various forms affects us on a daily basis. AI is used both in science and in everyday life for pattern recognition, classification, prediction and decision support, and the use of Generative AI for communication, information and the production of new content has skyrocketed since 2022. Moreover, these various AI systems are also increasing embedded in robotic systems with the aim to increase their performance.

Yet AI systems also pose various epistemic and ethical challenges, related to accuracy, bias and discrimination; a lack of transparency and accountability; sustainability and numerous other concerns. Thus, while we seem to rely on AI on a daily basis, the question remains whether we can trust it – and whether we should. I will argue that we can trust AI systems, if and only if we conceive them as socio-technical systems, but that we should trust it if and only if they are trustworthy. In my talk, I will delineate some epistemic and ethical requirements for trustworthy AI systems, focusing on particular on the role of deception, and end with some considerations on the implications for the design of AI systems.