What the future objective actually sees
Where the present-frame loss is read from, what the future-frame target is made of on nuScenes and ONCE, and which of those observations can carry an inference about published results.
Three objects, kept apart
Before any of this is readable, three things have to stay separate.
- The paper’s description. What the method is stated to compute.
- The released implementation we inspected. What one public checkout computes when it is run. It is the thing a scaffold gets built on, and the only thing that can be examined directly.
- Our controlled scaffold. A fork in which the routing and the target construction are fixed and can be switched deliberately. Every experiment on the following pages runs on this.
Conflating the first two turns an implementation observation into a claim about a result. Conflating the second and third turns a control into a reproduction. Neither is intended here, and the observations below are the reason the third object exists at all.
Where the present-frame loss is read from
The paper describes the present frame as reconstructed from the encoder’s own features, with the latent transformation standing between the encoder and the future frame only. In the implementation we inspected, both losses read the transformed features.
As the paper describes it
The present frame is rendered from the encoder's own features. The latent transformation stands between the encoder and the future frame only.
The encoder feeds the present-frame range loss directly, and also feeds the latent transformation, which feeds the future-frame range loss.
As the released implementation routes it
In the code we inspected, both losses read the transformed features. The encoder reaches neither loss without passing through the discarded head.
The encoder feeds the latent transformation, and the transformation feeds both the present-frame and the future-frame range loss.
Three things to keep apart. The paper's description, the behaviour of the released code, and the arms compared in this study are not the same object. Rather than guess which one the published numbers were produced with, the experiments fix a single scaffold and change one part of it at a time, including variants that restore the routing as described. Nothing here establishes what any unreleased training run did.
The semantic difference is not subtle, and it is not about accuracy. It is about which module the gradient of the present-frame loss reaches first.
Under the described routing, the present-frame loss trains the encoder directly. The encoder is asked to be render-ready on its own, and the transformation exists only to serve the future target. A control that switches the future loss off then also removes the transformation’s entire reason to exist.
Under the released routing, every loss reaches the encoder through the transformation. The transformation is in the gradient path at all times, including when no future target is present, and the encoder is never asked to render anything by itself. A control that switches the future loss off leaves a fully trained head in place.
The second arrangement is the one in which a head-versus-future question is worth asking, since it is the one where the head survives the ablation. Which of the two graphs any particular published number was produced with is not something this study can determine, and it does not try to.
What the future-frame target is made of
Two further observations concern the future target specifically. Both were found by running the released configuration and inspecting the quantity the future loss was computed against.
nuScenes
Almost no future rays survive the filter
A bar representing the 6144 rays sampled for the future frame, of which roughly four pass the validity filter.
In the released configuration we ran, the validity mask applied to the future frame leaves on the order of four usable rays out of the 6144 sampled. The future term is present in the loss, and contributes almost no gradient. The sliver above is drawn wide enough to be visible; at true scale it would be a fraction of a pixel.
ONCE
The future frame lands in the wrong place
Schematic of a future scan overlaid on the present one: rotated and offset as inspected, closely overlapping once the pose handling is corrected.
On ONCE the two frames are related through recorded pose. In the path we inspected the future scan arrives rotated and displaced relative to the present one, so the future target asks the model to reproduce geometry that is not where it is claimed to be. Correcting the pose handling brings the two frames into visibly better agreement. The panels above are a schematic of that failure mode, not a rendered sample.
Why this is here. Neither observation is a result, and neither is offered as an explanation of anyone's published numbers. They are the reason this study stops reading the future-prediction objective off an existing implementation and instead builds a scaffold where the future term can be switched on and off deliberately.
nuScenes: the target is filtered away
The renderer rejects rays whose target lies outside the volume it can represent. On the released nuScenes configuration we ran, the future frame is transformed into a position where that filter removes almost everything: of the 6144 rays sampled for the future frame, on the order of four survive. The future term is present in the loss and contributes close to no gradient.
The check is a count, not an inference. The rays are sampled, the mask is applied, and what remains is counted.
ONCE: the target survives, in the wrong place
On ONCE the failure is the opposite shape. The future scan stays inside the volume, so the filter keeps it, but it arrives rotated and displaced relative to the present frame. The model is then asked to reproduce geometry that is not where the target claims it is.
Two independent conventions are involved. The ego translation is not applied, so the future scan is not moved forward by the distance the vehicle travelled. The rotation is read under a different component ordering than the one it was stored with, so the scan is also turned. Each would be enough on its own.
Bar chart of the distance between the present scan and the future scan mapped into the present frame: 11.9 m with no ego-motion compensation, 11.5 m as inspected, and 0.69 m once both pose conventions are applied consistently.
Show the numbers
| Handling of the recorded pose | This pair | All evaluated pairs |
|---|---|---|
| No compensation | 11.9 m | n/a |
| As inspected | 11.5 m | 2.37 m |
| Corrected | 0.69 m | 0.63 m |
| Static-scene floor | n/a | ≈ 0.3 m |
The severity of this depends on what the rays land on, which is why it is worth stating carefully. Most returns in a driving scene lie on surfaces running parallel to the direction of travel: road, kerbs, building fronts. A target displaced along that direction still lands on the same surface, so the depth it asks for is wrong by much less than the displacement. Structures running across the direction of travel are the ones that get metre-scale wrong targets. The effective objective under this handling behaves like a blurred reconstruction task, not like noise, which is consistent with the fact that it trains stably.
Ray origins under augmentation
One more item belongs with the other two, because it also concerns the geometry the loss is computed on rather than the loss itself. Training augmentation rotates and translates the target points, and in the path we inspected the corresponding ray origins are not transformed with them. Points and rays then describe two different scenes. This affects both datasets and the present-frame term as well as the future one, and it is corrected in the scaffold used for every experiment on the following pages.
How this was checked, and what it does not establish
Each observation is a measurement on a running configuration, not a reading of source text: a count of surviving rays, a nearest-neighbour distance between two point clouds, a comparison of transformed and untransformed origins. Each can be repeated by running the configuration and looking at the same quantity.
The consequence for the experiments is direct. An ablation that removes future supervision is informative only if future supervision was doing something to begin with. So the comparison is rebuilt on a scaffold where the target construction is fixed, the routing is explicit, and the future term can be switched on and off with nothing else moving.
Next Controlled comparisons and detector results sets out the arms, the protocols and what the detector does.