Talks

Two questions: do our benchmarks test what a model has never seen, and do our experts teach what the student can perceive? Fail2Drive answers the first with 100 paired CARLA routes containing 30 unseen assets and 70 novel scenarios, revealing a large generalization gap for state-of-the-art policies such as TransFuser++, SimLingo, and PlanT 2.0, including memorized obstacle avoidance and ignored novel obstacles. LEAD (CVPR 2026) answers the second by identifying state and intent asymmetries between privileged experts and sensor-based students: experts react to invisible hazards, exploit unsafe margins, and receive sparse target points. Aligning the expert to what the student can see and providing denser navigation targets sharply reduces infractions and gives TransFuser v6 state-of-the-art results without architectural changes.

A survey of where synthetic data helps autonomous intelligence: imitation learning on skewed real-world data cannot cover the long tail and requires data and compute far beyond academic budgets. After comparing graphics simulators, log-replay simulators, and generative world models, the talk presents generative digital twins based on 3D Gaussian splatting (HUGSIM and World Engine), where RL post-training in re-simulated safety-critical scenarios reduces collisions by 45% and replaces the equivalent of 14 times more pre-training data. For neural world models, GenAD, Vista, and ReSim show that mixing real data with simulated out-of-distribution actions lets the model follow non-expert commands and predict rewards. The final part covers evaluation with NAVSIM v1 and v2, whose pseudo-simulation scores correlate far better with closed-loop performance than open-loop metrics, and closes with the AlpaSim challenge.

A case for looking inside today's black-box generation and reconstruction models. Trajectory Forcing (ECCV 2026) restructures a latent diffusion model in DINOv2 space so that every intermediate state is an interpretable coarse-to-fine image, obtained through a hierarchy of segmentations that can each be produced with a single mean-flow step and edited in shape or appearance. TTT3R revisits the recurrent CUT3R reconstruction model from a test-time training perspective: its state update is a delta rule with a hard-wired learning rate of one, and replacing it with a per-token rate derived from match confidence, a two-line code change, gives much more stable camera trajectories on sequences of thousands of frames at the same 20 FPS and constant memory.

Structure remains important in the era of end-to-end learning, both in the representation and in the computation. The first part presents PrITTI (CVPR 2026), a latent diffusion model that generates 3D semantic urban scenes as compact primitives (polylines for ground surfaces, cuboids and ellipsoids for objects) rather than voxels, enabling editing, inpainting, outpainting, and extrapolation, together with early results on turning these scenes into multi-dataset sensor simulations with a controllable video model. The second part covers structure in computation: ReSplat (ECCV 2026) learns recurrent Gaussian splatting by feeding rendering errors back into the network, and Learn2Splat extends this with a learned optimizer that keeps improving over thousands of iterations where Adam and ReSplat degrade. Real data teaches AI how to drive; simulation teaches it not to crash.

The current frontier of vision-language-action driving models is inefficient: reasoning traces are poorly grounded in actions, non-camera sensors are second-class inputs, and training requires around 80,000 hours of curated data and more than 60,000 GPU hours, far beyond academic reach. The talk proposes three open ingredients for progress: open datasets, open tooling, and open simulators. Concrete examples are 123D, which unifies more than 3,300 hours of public driving data across heterogeneous formats, sensor rigs, and coordinate conventions; the integration of 123D maps into the PufferDrive behavior simulator; and the AlpaSim challenge, a photorealistic closed-loop benchmark built on 3D Gaussian splatting reconstructions with results to be presented at NeurIPS 2026.

Is monocular 3D reconstruction ready for robotics, where reconstructed meshes must feed physics simulators for planning and learning? Together with collaborators from Oxford and MPI-IS, five robotics requirements were defined (accuracy, no interpenetration, physical stability, occlusion handling, and speed) and current object- and scene-level single-view methods were evaluated on the YCB-Video and Aria Digital Twin datasets. No method meets all requirements: errors are two to six times above manipulation tolerance, more than half of the reconstructions collide, only a third of planned grasps succeed, and most scenes are not physically stable. The second part presents PointDiT (ICML 2026), a pixel-space diffusion transformer for point map estimation that removes the VAE, uses a single loss, and trains only on synthetic data while producing sharper occlusion boundaries than regression baselines. The talk argues for benchmarks that score reconstructions by downstream utility in closed-loop simulation rather than by Chamfer distance.

The overloaded term world model is separated into two families, and the talk explains what each contributes to efficient driving policies. Video action models replace the text reasoning trace of a VLA with a latent, object-aware prediction of how the scene evolves; a 0.5B-parameter model trained on 2,000 hours of public data and post-trained with reinforcement learning matches a VLA of similar size at a fraction of the cost. Action video models predict future world states for arbitrary, including non-expert, actions and power World Engine, where 3D Gaussian splatting reconstructions serve as the interface for RL post-training, making interactive data about 14 times more valuable than passively collected demonstrations and enabling over 200 km of intervention-free driving in Shanghai. The talk closes with latent world models that learn the encoder end to end and combine real and simulated data to both learn representations and simulate.

Today's vision-language-action driving models, trained on around 80,000 hours of curated data and tens of thousands of GPU hours, are too data- and compute-hungry for autonomous driving to be democratized. The talk presents the open-source tooling KE:SAI is building to change this: 123D, a library that unifies the most used public driving datasets (nuScenes, Waymo, Argoverse, KITTI-360, nuPlan, PhysicalAI-AV, and CARLA) into a common format with more than 3,300 hours of data. 123D maps can be loaded into the PufferDrive simulator for fast self-play reinforcement learning, and training on a mixture of datasets generalizes better than any single one. Finally, the AlpaSim challenge is announced, a photorealistic closed-loop benchmark built on 3D Gaussian splatting reconstructions with large held-out test scenarios.

This opening keynote connects causal representation learning with world models and digital twins. Starting from spurious correlations in medical imaging and in multimodal language models, which confidently misjudge exaggerated versions of well-known optical illusions, it argues that models trained purely on correlations neither predict the effect of interventions nor transfer across distributions. The same idea becomes a tool for science: half-sibling regression removes instrument effects in exoplanet searches, and simulation-based inference enables fast gravitational-wave astronomy. The second part covers structural causal models and the independence of mechanisms, and how these principles lead to causal representations that can act as world models for thinking in an imagined space and as causal digital twins of complex systems.

How machine learning is changing the scientific process, and why current AI systems, which are built on correlations, still fail in ways that matter for science. Spurious correlations affect not only medical image classifiers but also frontier language models, which confidently misjudge exaggerated versions of well-known optical illusions. Turning the same idea into a tool, half-sibling regression explains away instrument effects in Kepler light curves and led to the discovery of the habitable-zone exoplanet K2-18b, and neural inference handles gravitational-wave events and binary neutron star mergers. The talk closes with a causal perspective on language models, the case for a CERN-like European institution for AI, and the argument that the key question is not whether AI is intelligent but whether we are intelligent in how we use it.

An introduction to reinforcement learning from the perspective of optimizing non-differentiable objectives, showing that cross-entropy classification can be derived as policy-gradient maximization of accuracy. This view is then applied to planning in autonomous driving by turning the CARLA driving score, route completion times a penalty factor, directly into a reward. This simple reward scales better with large PPO batch sizes than hand-designed reward shaping, which tends to get stuck in local minima. The resulting planner outperforms recent imitation learning and diffusion-based planners on two benchmarks while using a 2M-parameter network with roughly ten times less inference compute.