InfiniteWorld: A VideoQA Benchmark for Physical
Law Understanding in Parallel Physics Worlds

Motivation and protocol

A video of a stone falling upward is not merely unusual; it is impossible, and a system that understands gravity should be able to state as much after observing it. Recognising that a stone fell concerns what occurred; recognising that it could not have concerns what the world permits. The second question is the one physical understanding is supposed to answer, and it is almost never asked directly, because natural video only ever contains events that did in fact occur.

We therefore construct the counterfactual rather than searching for it. A scene is a simulator configuration: an environment, a rigid body, an initial state, and a set of physical parameters. Holding everything but the parameters fixed and re-running the simulator yields the same scene under a different law of nature. Each clip therefore carries something natural video cannot supply — a certificate of which law generated it, and whether that law is the one our world obeys.

The quantity of interest is not accuracy itself but its margin over the accuracy the same model attains with the frames removed. A model can answer a physics video question without consulting the video: the phrasing of a question about inverted gravity tends to disclose that something is anomalous, and “these are all possible” remains defensible for almost any such item. Each question is therefore evaluated under three conditions — frames with text, byte-identical text with the frames withheld, and the same frames in randomised temporal order — and the conditions are compared per item using McNemar's test rather than in aggregate.

Results

Visual gains are scale-dependent

Across the frontier panel, three of five models are significantly more accurate with frames than without on the paired law-identification set. The other two are informative in themselves: Claude Sonnet 5 shows no effect, yet attains the highest blind accuracy in the panel, indicating that it answers largely from the text and that the frames contribute no additional signal; Llama‑4‑Maverick is significantly less accurate when given the video. Restricting every condition to the identical 384-item intersection leaves the pattern unchanged.

The escape option absorbs the response distribution

On the witnessed-possibility set, local models call almost every clip possible — Qwen2.5‑VL‑3B on 99.6% of clips, 7B on 88.6%, InternVL on 81.4%. The consequence is near-perfect accuracy on the half of the set that is possible and exactly zero on the half that is not. Remove the frames and every yes-rate goes to 100%: the verdict is determined by language, and the image acts as a small perturbation upon it. A model with no demonstrable physical understanding can thus attain 0.99 on half the benchmark, and an evaluation lacking a blind condition would report that figure as its result.

Forced choice recovers little of the deficit

With the “all possible” option removed, the three local models rise from near zero to approximately one third correct, yet none separates from its own blind control (χ² = 2.73 / 2.09 / 0.00). The prior was therefore not concealing an otherwise competent percept: what remains after its removal approximates what the text alone supports.

Possibility framing yields gains that frame budget does not

Asked whether a witnessed process could occur rather than which law produced it, three of four models gain significantly from the video (Δ = +0.063 / +0.080 / +0.082), whereas the same models show no gain when asked to name the governing law. Two controls exclude the natural alternative explanation, that the anomalies are difficult to perceive. Quadrupling the frame budget from 4 to 24 leaves accuracy unchanged (0.478 → 0.498 → 0.500), which is inconsistent with evidence starvation. Agreement between the engine and photorealistic renderings of a clip is high while accuracy is not: Qwen‑7B returns the same answer for both substrates at κ = 0.88 while answering approximately three quarters of each incorrectly. A model reading physics from pixels should be perturbed by repainting them, whereas a model reproducing a prior should not be, and is not. Cross-substrate consistency, commonly reported as robustness, therefore measures only the limited extent to which the image informed the answer.

Benchmark construction

Twelve law families across six environments and eighteen objects, with each clip rendered in two substrates and gated by tracking certification before any question is generated.

Twelve families cover six single-factor alterations (inverted gravity, scaled gravity, restitution above unity, scaled friction, removed air dissipation, inverted buoyancy), five two-factor combinations, and a factorial cascade. Every family is paired with an Earth-normal control clip, so that reporting an apparent anomaly is not a uniformly correct strategy. Clips render at 1280×704, median 49 frames, and each is repainted photorealistically to give a second substrate for the same trajectory — appearance and physics on independent axes.

Certification distinguishes this construction from prompting a generative model for implausible footage. A configuration specifying inverted gravity does not guarantee a clip in which anything visibly rises: the object may be occluded, may move imperceptibly, or may leave the frame. Before any question is generated, a tracker recovers the object trajectory and the clip must satisfy three criteria — a sufficient tracked-visible fraction, agreement between the tracked and ground-truth trajectory on the manipulated axis, and, for anomalies possessing a closed-form kinematic signature, a fit to that signature. Of 1,680 candidates, 1,468 pass, a survival rate of 87.4%. Only under this condition does an incorrect answer constitute a model failure rather than an unwitnessable question.

Certified engine–photorealistic pairs

Left, the engine rendering; right, the photorealistic repaint of the same certified trajectory. Substrate agreement is measured on pairs of this form.

Question conditions

The blind condition receives byte-identical text with the frames withheld. Because the stem itself indicates that the physics may depart from Earth-normal behaviour, the blind floor constitutes the primary control rather than a supplementary one.

Coverage imbalance and matched-item analysis

The frontier sighted conditions lost approximately 39% of rows to missing renders, and the loss was markedly non-uniform: scaled friction lost 57.6% and scaled gravity 55.9%, against 4.6% for buoyancy. Because the losses fell on the same clips for every model, scoring each condition on the rows it happened to retain, rather than on their intersection, produced a pooled result that was significant in the opposite direction. The intersection subset is balanced in family and in control fraction (0.482 against a nominal 0.500), and the per-model conclusions hold on it; every comparison reported here is consequently restricted to matched items. We retain this account in the paper because the methodological discipline that corrected our own error is the same one the benchmark is designed to enforce.

BibTeX

@inproceedings{infiniteworld2027,
  title     = {InfiniteWorld: A VideoQA Benchmark for Physical Law
               Understanding in Parallel Physics Worlds},
  author    = {Anonymous},
  booktitle = {Under review at the International Conference on
               Learning Representations (ICLR)},
  year      = {2027},
  note      = {Under double-blind review}
}