InfiniteWorld: A VideoQA Benchmark for Physical
Law Understanding in Parallel Physics Worlds

The instrument is the empty image

A video of a stone falling upward is not merely unusual. It is impossible — and a system that understands gravity should be able to say so after watching it. Recognising that a stone fell is a question about what happened; recognising that it could not have is a question about what the world permits. The second question is the one physical understanding is supposed to answer, and it is almost never asked directly, because natural video only ever contains events that did in fact occur.

So we build the counterfactual instead of looking for it. A scene is a simulator configuration: an environment, a rigid body, an initial state, and a set of physical parameters. Holding everything but the parameters fixed and re-running the simulator yields the same scene under a different law of nature. Each clip therefore carries something natural video cannot supply — a certificate of which law generated it, and whether that law is the one our world obeys.

The measurement that matters is not accuracy but the gap between accuracy and what the same model scores with the frames removed. A model can answer a physics video question without watching the video: the phrasing of a question about inverted gravity tends to reveal that something is wrong, and “these are all possible” is a defensible answer to almost any such question. Every question here is therefore run in three conditions — frames plus text, byte-identical text with the frames removed, and the same frames in randomised temporal order — compared per item with McNemar's test rather than in aggregate.

What the controls show

Using the video is a matter of scale, not of the paradigm

Across the frontier panel, three of five models are significantly more accurate with frames than without on the paired law-identification set. The other two are informative in themselves: Claude Sonnet 5 is flat, but its blind accuracy is the highest in the panel — it answers well from the text alone, so the frames add nothing — while Llama‑4‑Maverick is significantly worse with the video. Restricting every arm to the identical 384-item intersection leaves the pattern intact.

The escape option absorbs the answer

On the witnessed-possibility set, local models call almost every clip possible — Qwen2.5‑VL‑3B on 99.6% of clips, 7B on 88.6%, InternVL on 81.4%. The consequence is near-perfect accuracy on the half of the set that is possible and exactly zero on the half that is not. Remove the frames and every yes-rate goes to 100%: the verdict is set by language, and the image is a small perturbation on top of it. A model that understands nothing can post 0.99 on half a benchmark; an evaluation without the blind arm would report that number.

Forcing a commitment recovers little

Strip the “all possible” option and the three local models climb from near zero to roughly a third correct — but none separates from its own blind control (χ² = 2.73 / 2.09 / 0.00). The prior was not masking a competent percept; what remains once it is removed is about what the text alone supports.

Reframing helps; adding pixels does not

Asked whether a witnessed process could occur rather than which law produced it, three of four models gain significantly from the video (Δ = +0.063 / +0.080 / +0.082), while the same models gain nothing from naming laws. Two controls rule out the obvious alternative — that the anomalies are simply hard to see. Quadrupling the frames from 4 to 24 leaves accuracy flat (0.478 → 0.498 → 0.500), which is not the signature of evidence starvation. And agreement between the engine and photorealistic renderings of the same clip is high while accuracy is not: Qwen‑7B gives the same answer to both substrates at κ = 0.88 while getting roughly three quarters of both wrong. A model that read physics off pixels should be perturbed by repainting them; one reciting a prior should not, and is not. Cross-substrate consistency, often reported as robustness, here measures only how little the image mattered.

The dataset

Twelve law families over six environments and eighteen objects, each clip rendered twice and gated by tracking before any question is written.

Twelve families cover six single-factor alterations (inverted gravity, scaled gravity, restitution above unity, scaled friction, removed air dissipation, inverted buoyancy), five two-factor combinations, and a factorial cascade. Every family is paired with an Earth-normal control clip so that “something looks off” is not a constant winning answer. Clips render at 1280×704, median 49 frames, and each is repainted photorealistically to give a second substrate for the same trajectory — appearance and physics on independent axes.

Certification is what separates this from prompting a generator for implausible frames. A configuration that specifies inverted gravity does not guarantee the clip shows anything rising: the object may be occluded, barely move, or leave frame. Before any question is generated, a tracker recovers the trajectory and the clip must clear three gates — enough of the object stays visible, the tracked path matches ground truth on the manipulated axis, and (for anomalies with a closed-form kinematic signature) that signature fits. Of 1,680 candidates, 1,468 pass — an 87.4% survival rate. Only then does an answer count as the model's failure rather than an unseeable clip.

Certified sim ↔ photoreal pairs

Left is the engine; right is the photorealistic repaint of the same certified trajectory. Substrate agreement is measured on pairs like these.

Every question is asked three ways

The blind arm receives byte-identical text with the frames removed. The stem already says the physics may differ from Earth's, which is why the blind floor is the load-bearing control rather than an afterthought.

A near-miss of our own

The frontier sighted arms lost about 39% of rows to missing renders, and the loss was far from uniform — scaled friction lost 57.6% and scaled gravity 55.9%, buoyancy only 4.6%. Because each model lost the same clips, scoring on the rows each model happened to have — rather than on their intersection — produced a pooled result that was significant in the opposite direction. The intersection subset is balanced in family and control fraction (0.482 against a nominal 0.500), and the per-model conclusions hold on it, which is why every arm comparison we report is restricted to matched items. We keep this in the paper because the same discipline that reversed our own error is the one the benchmark exists to enforce.

BibTeX

@inproceedings{infiniteworld2026,
  title     = {InfiniteWorld: A VideoQA Benchmark for Physical Law
               Understanding in Parallel Physics Worlds},
  author    = {Anonymous},
  booktitle = {Under review at the International Conference on
               Learning Representations (ICLR)},
  year      = {2026},
  note      = {Under double-blind review}
}