Senses Wide Shut

NeurIPS 2026

Senses Wide Shut

A Representation–Action Gap in Omnimodal LLMs

  • Nguyen Quang Trung1,2*
  • Yiming Gao1,2*
  • Fanyi Pu1,2
  • Kaichen Zhang1,2
  • Shuo Sun3
  • Ziwei Liu1,2,
  • 1Nanyang Technological University
  • 2LMMs-Lab Team
  • 3Johns Hopkins University

* Equal contribution   Corresponding author

TL;DR

We build IMAVB, a benchmark of 500 movie clips in which each question either describes the clip correctly or changes one visual or audio detail. Linear probes recover the false premise from the hidden states of omnimodal LLMs with up to 86% accuracy, yet seven of eight open-source models reject it in at most 16.2% of vision cases and 6.6% of audio cases. The models encode the mismatch, but they rarely act on it.

movie clips
500
questions
2,000
hours of video
20.7
models evaluated
9

Overview video

The paper in a short video

Abstract

When an omnimodal large language model accepts a question whose textual premise contradicts what it actually sees or hears, does the failure lie in perception or in action? Recent omnimodal models are positioned as perception-grounded agents that jointly process video, audio, and text, yet a basic form of grounding remains untested: catching a textual claim that conflicts with the model's own sensory input. We introduce IMAVB, a curated benchmark of 500 multi-minute movie clips with a 2×2 design crossing target modality (vision, audio) and premise condition (standard, misleading), which lets us measure conflict detection separately from ordinary multimodal comprehension. Across eight open-source omnimodal LLMs and Gemini 3.1 Pro, we document a Representation–Action Gap: hidden states encode premise–perception mismatches reliably for vision and more weakly for audio, even when the same models almost never reject the false claim in their outputs.

Behaviorally, models fall into two failure modes: under-rejection, in which they answer misleading questions as if the false premise were true; and over-rejection, in which they reject more often but also reject standard questions, sacrificing ordinary comprehension accuracy. The gap is modality-asymmetric (models reject false audio premises far less often than false visual ones) and prompt-resistant across seven variants. As an initial diagnostic intervention, a probe-guided logit adjustment (PGLA) re-injects the encoded mismatch signal into decoding and consistently improves rejection behavior at a bounded cost to standard accuracy. Together, these results suggest the bottleneck for omnimodal grounding lies in acting on the encoded mismatch signal rather than in encoding it.

Overview figure. The model receives a scene in which a cat chases a red ball while soft guitar music plays, and its internal representation holds both facts. Given a prompt that says a dog chases the ball, or that a loud drum beat plays, the correct action is to reject the premise, but the model answers the question instead.
Overview of the Representation–Action Gap on IMAVB.

The benchmark

One clip, four questions

IMAVB holds 500 movie clips of 60 to 300 seconds (median 140 s, 20.7 hours in total), and we keep their video and audio unedited. Every clip has four questions in a 2×2 design: the question targets vision or audio, and its premise is either correct or misleading. A misleading question copies its standard twin and changes exactly one premise detail. Options E (“The visual detail in the question is incorrect”) and F (“The audio detail in the question is incorrect”) appear in every prompt, so a model that notices the conflict can always report it.

Ten frames from a movie scene: a man in a dark suit talks to a younger man in a maroon polo shirt, and a woman in a red dress stands in the doorway behind them.
The example clip from the paper. The boxes mark the woman in the red dress. Audio cue: two men talk normally to each other, over a gentle, melancholic string underscore.
Standard premise Misleading premise
Vision

When the man in the dark suit faces the younger man wearing a maroon polo shirt, what colour is the dress of the woman in the doorway?

A. Black   B. Blue   C. White   D. Red

Answer: D

When the man in the dark suit faces the younger man wearing a blue polo shirt, what colour is the dress of the woman in the doorway?

A. Black   B. Blue   C. White   D. Red

Answer: E, the visual detail in the question is incorrect

Audio

As the suited man pleads while a delicate string melody swells, what does he say after “Who cried for the little boy?”

A. I will cry for him   B. He is lost forever   C. It hurts so much   D. He cries inside me

Answer: D

As the suited man pleads while a loud drum beat swells, what does he say after “Who cried for the little boy?”

A. I will cry for him   B. He is lost forever   C. It hurts so much   D. He cries inside me

Answer: F, the audio detail in the question is incorrect

Every question lists the same six options A to F. We show only A to D here for space.

From movie clips to verified questions

The annotation pipeline: movie clips from three sources are cut into 10-second segments, captioned separately for vision, audio and both, enhanced under a trust hierarchy, merged into one narrative, turned into four questions per clip, and verified by hand.
The three-pass annotation pipeline and the 2×2 question design.
  1. Pass 1, captions. We cut each clip into 10-second segments and caption each one three times: vision with GPT-4o at 1 fps, audio with Qwen3-Omni-30B-A3B-Captioner, and both together with Qwen3-Omni-30B-A3B-Thinking.
  2. Pass 2, enhancement. Qwen3.5-27B fuses the three captions of each segment under the trust hierarchy Omni > Vision > Audio, so audio-only captions cannot over-infer visual events.
  3. Pass 3, narrative. Qwen3.5-27B merges all segments into one timestamped narrative per clip, and it then writes the four questions, which span 8 question categories and 18 misleading subcategories.
  4. Verification. We checked all 2,000 items by hand while watching each clip with audio.

Results

Two failure modes

Seven of the eight open-source models under-reject: they answer 40–75% of standard questions correctly, but they reject at most 16.2% of false visual premises and 6.6% of false audio premises, and four of them never reject a false audio premise under fixed option order. Qwen3-Omni and Gemini 3.1 Pro instead over-reject: they reject more often, but they lose 15–25 points of standard accuracy.

modelstd_vstd_amis_vmis_aBal
open-source omnimodal LLMs
OLA71.071.66.80.037.4
OmniVinci75.471.46.60.038.4
Qwen2.5-Omni64.469.016.00.637.5
MiniCPM-o 2.656.654.29.06.631.6
Uni-MoE-2.0-Omni74.869.09.00.038.2
Baichuan-Omni-1.566.066.813.80.636.8
Video-SALMONN-269.866.616.20.038.2
Qwen3-Omni40.646.672.823.645.9
proprietary model
Gemini 3.1 Pro50.253.894.048.661.6

Baseline accuracy in percent under fixed option order. std_v and std_a are standard vision and audio questions, mis_v and mis_a are misleading ones, and Bal is the balanced accuracy, ½((std_v + std_a)/2 + (mis_v + mis_a)/2). Bold marks the best open-source value of each column. Gemini 3.1 Pro runs through its API on this experiment only. The paper also reports shuffled option orders, seven prompt variants and a length analysis, and none of them closes the gap.

Cross-modal interference

modelA→VV→Areading
Qwen2.5-Omni+6.2+1.0audio interferes
Video-SALMONN-2+6.4+0.0audio interferes
Baichuan-Omni-1.5+5.4+0.6audio interferes
OmniVinci+1.2+0.0minimal
OLA−0.6+0.0no interference
Uni-MoE-2.0-Omni−1.8+0.4no interference
MiniCPM-o 2.6−4.4−5.8AV synergistic
Qwen3-Omni+2.0+14.4video interferes

We run each open-source model with one modality removed and measure the change in misleading accuracy, in points. A→V removes the audio and reads mis_v, and V→A removes the video and reads mis_a. A positive value means that the removed modality interfered with detection.

The effect depends on the architecture. Audio interferes with visual detection for three models, while OLA and Uni-MoE-2.0-Omni show no interference. MiniCPM-o 2.6 is the only model for which joint input helps, and for Qwen3-Omni the video interferes with audio detection. No single cross-modal explanation therefore fits all models.

The Representation–Action Gap

The hidden states encode the mismatch

At each layer we train a linear probe on the last-token hidden state to separate misleading questions from standard ones, with folds grouped by video. For six of the models, the vision probes reach 83–86%, while the same six models reject a false visual premise in at most 16.2% of cases. To control for wording, we project out the features that predict the label from the question text alone and retrain the probes on the residual.

modelprobe accuracyresidualized probebehavioural rejection
visionaudiovisionaudiomis_vmis_a
OLA84.077.879.067.76.80.0
OmniVinci84.478.878.966.16.60.0
Qwen2.5-Omni86.075.677.767.016.00.6
MiniCPM-o 2.683.278.678.769.59.06.6
Uni-MoE-2.0-Omni84.476.376.765.69.00.0
Baichuan-Omni-1.5†99.398.799.8100.013.80.6
Video-SALMONN-283.577.180.765.816.20.0
Qwen3-Omni76.564.979.865.072.823.6
text-only baselines
TF-IDF73.471.4––––
SBERT66.857.3––––

Probe accuracy at the peak layer, probe accuracy after the text-predictive features are projected out, and the behavioural rejection rate, all in percent. † We leave Baichuan-Omni-1.5 out of aggregate statistics, because its signal peaks near ceiling at layer 2.

Text control

After residualization, the probes stay above the SBERT baseline for every model and modality and above the TF-IDF baseline for vision. For audio, several 7B models fall below TF-IDF, so the audio evidence is weaker.

Logit lens

For OLA, Qwen2.5-Omni and Uni-MoE-2.0-Omni the correct answer is readable mid-stack on standard questions and fades on misleading ones. For three other models the signal never aligns with the unembedding, although the probes still decode it.

Runner-up

For the seven under-rejecting models, the correct reject option is the second choice on 23.0% of misleading items, against 16.7% for a uniform ranking and 15.2% on the standard twins. A small part of the signal reaches the output logits.

A diagnostic intervention

Probe-guided logit adjustment

To test whether the encoded signal is useful, we feed it back into decoding. PGLA reads the hidden state that the model already computes, and we evaluate it with 5-fold cross-validation.

1

Probe

A two-layer MLP reads the hidden state at the peak layer and predicts the probability that the question is misleading. It sees only a binary label and no modality information.

2

Gate

The probe confidence passes through a sigmoid gate, so the adjustment is large when the probe is confident and smaller when it is not.

3

Boost

The gate scales a boost on the logits of the reject options E and F, which grows with the gap to the best content option, and a small term corrects the positional bias between E and F.

modelbaseline BalStdMisBalΔBal
Uni-MoE-2.0-Omni38.254.759.357.0+18.8
MiniCPM-o 2.631.645.552.849.2+17.6
OLA37.458.751.254.9+17.5
Video-SALMONN-238.246.264.155.1+16.9
Qwen2.5-Omni37.555.551.353.4+15.9
OmniVinci38.453.852.253.0+14.6
Qwen3-Omni45.946.568.857.7+11.8
Baichuan-Omni-1.536.852.035.043.5+6.7
mean38.051.654.353.0+15.0

PGLA results in percent, averaged over the five test folds. Std and Mis are the mean standard and misleading accuracy after the best configuration, and ΔBal is the gain in balanced accuracy over the baseline.

PGLA raises balanced accuracy for all eight models, by 15.0 points on average. When we also bound the loss in standard accuracy, the mean gain is 9.9 points at a 3-point budget and 5.4 points at a 1-point budget. For seven models PGLA moves the decision boundary toward rejection rather than improving perceptual grounding, so we treat it as evidence that the encoded signal is actionable, not as a fix.

Citation

BibTeX

bibtex
@inproceedings{nguyen2026senses,
  title     = {Senses Wide Shut: A Representation--Action Gap in Omnimodal {LLM}s},
  author    = {Nguyen Quang, Trung and Gao, Yiming and Pu, Fanyi and Zhang, Kaichen and Sun, Shuo and Liu, Ziwei},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026},
  eprint    = {2605.13737},
  archivePrefix = {arXiv}
}

Acknowledgements

This study is supported by the Ministry of Education, Singapore, under its MOE AcRF Tier 2 (MOE-T2EP20223-0002). This research is also supported by cash and in-kind funding from NTU S-Lab and industry partner(s). We wish to acknowledge the support of Nanyang Technological University through URECA Undergraduate Research Programme.