VLA-Trace

Diagnosing Vision-Language-Action Models through Representation and Behavior Tracing

Haoyuan Shi^1,2,†, Xiancong Ren^1,†, Yingji Zhang^1,3,†, Qinfan Zhang^1,4,†,
Jiayu Hu¹, Haozhe Shan^1,5, Han Dong^1,6, Jinpeng Lu^1,2, Yinda Chen^1,2,
Yi Zhang¹, Yong Dai^1,‡, Xiaozhu Ju^1,✉

¹ X-Humanoid ² University of Science and Technology of China ³ University of Manchester ⁴ Beihang University ⁵ Fudan University ⁶ University of New South Wales

† Core contributors · ‡ Project leader · ✉ Corresponding author

Paper GitHub ✂️ BibTeX

Motivation

VLA models can act — but why do they act?

High success rates reveal what VLA models do, but not how they actually work.VLA-Trace bridges this gap through representation, control, and behavior tracing.

The black-box gap

Success rate alone cannot explain failures.

A policy may successfully imitate trajectories while still ignoring subtle language edits, relying on gripper geometry, or exploiting background layout. VLA-Trace links internal representations to causal pathways and then to rollout behavior.

Q1 Representation 🧠 Does robot adaptation reorganize pretrained multimodal representations?

Q2 Control 🔀 Which modality actually drives action generation?

Q3 Behavior 🤖 Do internal pathways explain grounded behavior?

Three Questions Behind VLA Behavior

Build an evidence chain, not an isolated probe.

Existing analyses often stop at hidden-state probing or input perturbation. VLA-Trace connects three levels:

Representation Shifts What is preserved or reorganized?

Causal Pathways What pathway is necessary for action?

Behavioral Probes What does the robot actually rely on?

Method

A progressive evidence chain

VLA-Trace follows three model stages — C0 pretrained VLM, C1 pretrained VLA, and C2 task-finetuned VLA — and applies complementary probes from latent geometry to closed-loop rollout behavior.

Representation Shifts

Cross-modal & checkpoint-drift CKA

Measure image–text alignment and whether visual, textual, and joint subspaces are preserved or reorganized across C0, C1, and C2.

Causal Pathways

Attention Knockout

Block selected token pathways during prefill or generation to test whether vision, language, structural prompt tokens, or action paths are necessary.

Behavioral Probes

Rollout probes

Use attention IoU, patch masking, and input editing to expose grounding, shortcut dependence, and fine-grained semantic controllability.

Stage	Probe	Question	Benchmarks	Models
S1	Cross Modal CKA	How strongly do image and text representations align?	LIBEROCOCO	π0.5OpenVLA
S1	Checkpoint Drift CKA	Which representation subspaces are preserved or rewritten?	LIBERO	π0.5OpenVLA
S2	Attention Knockout	Which modality routes are causally required?	LIBEROCALVINRoboTwin2.0	π0.5OpenVLAOpenVLA-OFT
S3	Attention IoU And Patterns	Do action-conditioned heatmaps overlap with task-relevant regions?	LIBERO	π0.5OpenVLA
S3	Visual Patch Masking & Input Editing	Which visual cues or semantic edits actually change behavior?	LIBEROCALVINRoboTwin2.0SimplerEnv	π0.5OpenVLAOpenVLA-OFTX-VLA

Results carousel

Browse the experimental evidence as slides

The paper contains many diagnostic figures, so this page now uses carousels for fast browsing. Use the arrows, dots, thumbnails, or keyboard arrow keys; click any image to zoom in.

Representation Shifts → Finding 1~2

S1 · CKA

Cross-modal representation dynamics

Figure 2: Layer-wise image–text CKA across datasets and training stages. The panels compare cross-modal alignment for π0.5 and OpenVLA. Each curve denotes one checkpoint: C0, the pretrained VLM; C1, the pretrained VLA; and C2, the task-finetuned VLA.

Back to Finding 1~2 All findings

S1 · Drift CKA

Checkpoint-drift representation preservation

Figure 3: Stage-wise checkpoint-drift CKA for π0.5 and OpenVLA. Each cell reports the mean matched-layer CKA for vision, text, and joint representations.

Back to Finding 1~2 All findings

Causal Pathways → Finding 3~4

S2 · Knockout

All-layer attention knockout experiment

Table 6: All-layer attention knockout results across LIBERO suites. Each cell reports success rate (%) and the absolute change relative to the suite-specific all-layer baseline in parentheses. For π0.5, Prefill blocks V–L attention; for OpenVLA, Prefill removes image access. Comb. combines the corresponding prefill intervention with the specified generation-stage intervention.

Back to Finding 3~4 All findings

S2 · Knockout

OpenVLA-OFT LIBERO layerwise experiment

Figure 11: Layer-wise knockout results for OpenVLA-OFT on LIBERO-10, Goal, Object, and Spatial. Each point reports the success rate under a 3-layer knockout window centered at the indicated layer.

Back to Finding 3~4 All findings

S2 · Knockout

OpenVLA-OFT LIBERO layerwise experiment

Figure 12: Layer-wise knockout results for OpenVLA-OFT on LIBERO-10, Goal, Object, and Spatial. Each point reports the success rate under a 5-layer knockout window centered at the indicated layer.

Back to Finding 3~4 All findings

S2 · Knockout

OpenVLA-OFT LIBERO layerwise experiment

Figure 13: Layer-wise knockout results for OpenVLA-OFT on LIBERO-10, Goal, Object, and Spatial. Each point reports the success rate under a 7-layer knockout window centered at the indicated layer.

Back to Finding 3~4 All findings

S2 · Knockout

OpenVLA LIBERO layerwise experiment

Figure 9: Layer-wise knockout results for OpenVLA on LIBERO-10, Goal, Object, and Spatial. Each point reports the success rate under a 3-layer knockout window centered at the indicated layer.

Back to Finding 3~4 All findings

S2 · Knockout

OpenVLA LIBERO layerwise experiment

Figure 10: Layer-wise knockout results for OpenVLA on LIBERO-10, Goal, Object, and Spatial. Each point reports the success rate under a 7-layer knockout window centered at the indicated layer.

Back to Finding 3~4 All findings

S2 · Knockout

pi05 LIBERO

Figure 6: Layer-wise knockout results for π0.5 on LIBERO-10, Goal, Object, and Spatial. Each point reports the success rate under a 3-layer knockout window centered at the indicated layer.

Back to Finding 3~4 All findings

S2 · Knockout

pi05 LIBERO

Figure 7: Layer-wise knockout results for π0.5 on LIBERO-10, Goal, Object, and Spatial. Each point reports the success rate under a 5-layer knockout window centered at the indicated layer.

Back to Finding 3~4 All findings

S2 · Knockout

pi05 LIBERO

Figure 8: Layer-wise knockout results for π0.5 on LIBERO-10, Goal, Object, and Spatial. Each point reports the success rate under a 7-layer knockout window centered at the indicated layer.

Back to Finding 3~4 All findings

S2 · Knockout

OpenVLA-OFT RoboTwin

Figure 14: Layer-wise knockout results for OpenVLA-OFT on RoboTwin tasks. Each row corresponds to one RoboTwin task, and each point reports the success rate under a 1-layer knockout window at the indicated layer.

Back to Finding 3~4 All findings

S2 · Knockout

OpenVLA-OFT RoboTwin

Figure 15: Layer-wise knockout results for OpenVLA-OFT on RoboTwin tasks. Each row corresponds to one RoboTwin task, and each point reports the success rate under a 3-layer knockout window at the indicated layer.

Back to Finding 3~4 All findings

Behavioral Probes → Finding 5~7

Attention IoU and mass over rollout phases

S3 · Attention IoU And Patterns

Attention IOU experiment

Figure 22: OpenVLA attention IoU and mass on LIBERO-10. (a) Object IoU dynamically shifts between the first (Phase 1) and second (Phase 2) instruction subgoals. (b) Attention mass allocation over robot and object regions. These results indicate that VLA policies successfully generate visually grounded trajectories by tracking task-relevant objects over time.

Back to Finding 5~7 All findings

S3 · Attention IoU And Patterns

Attention IOU Schematic

Figure 16: Schematic illustration of attention-IoU computation. We project action-conditioned attention onto the image patch grid, keep the top 10% highest-attention patches, and compute IoU against a simulator-derived target mask. The same mask is also used to compute continuous attention mass and peak-hit rate.

Back to Finding 5~7 All findings

S3 · Action to Image Attention

Action-to-Image Attention

Figure 17: Qualitative action-to-image attention visualizations across rollout stages. We compare pretrained and fine-tuned π0.5 and OpenVLA models across early, middle, and final execution stages. Finetuning shifts action attention from diffuse or background-biased regions toward task-relevant robot-object interaction regions, indicating stronger visually grounded action generation during manipulation.

Back to Finding 5~7 All findings

S3 · Text to Image Attention

Text-to-Image Attention to Pi05

Figure 18: Token-wise text-to-image attention for pretrained and fine-tuned π0.5 across execution steps. Columns correspond to instruction tokens, and rows compare checkpoints. Each heatmap shows how the selected text token attends to image patches.

Back to Finding 5~7 All findings

S3 · Text to Image Attention

Text-to-Image Attention to OpenVLA

Figure 19: Token-wise text-to-image attention for pretrained and fine-tuned OpenVLA across execution steps. Columns correspond to instruction tokens, and rows compare checkpoints. Each heatmap shows how the selected text token attends to image patches.

Back to Finding 5~7 All findings

S3 · PATCH MASKING

LIBERO-10 Masking Performance

Table 7: Success rates (%) across tasks and masking strategies on LIBERO-10 for π0.5, OpenVLA, OpenVLA-OFT and X-VLA. The instruction is provided in Tab. 15.

Back to Finding 5~7 All findings

S3 · PATCH MASKING

LIBERO-Object Masking Performance

Table 8: Success rates (%) across tasks and masking strategies on LIBERO-Object for π0.5, OpenVLA, OpenVLAOFT, and X-VLA. The instruction is provided in Tab. 15.

Back to Finding 5~7 All findings

S3 · PATCH MASKING

LIBERO-Spatial Masking Performance

Table 9: Success rates (%) across tasks and masking strategies on LIBERO-Spatial for π0.5, OpenVLA, OpenVLAOFT, and X-VLA. The instruction is provided in Tab. 15.

Back to Finding 5~7 All findings

S3 · PATCH MASKING

LIBERO-Goal Masking Performance

Table 10: Success rates (%) across tasks and masking strategies on LIBERO-Goal for π0.5, OpenVLA, OpenVLAOFT, X-VLA. The instruction is provided in Tab. 15.

Back to Finding 5~7 All findings

S3 · PATCH MASKING

Simpler Environment Masking

Table 11: Performance of X-VLA under different masking strategies on Simpler.

Back to Finding 5~7 All findings

S3 · Grounding

LIBERO-10 Attention Grounding

Table 13: LIBERO-10 attention grounding for gripper, robot arm, and object regions. All values are pooled over task-step samples. Gripper is the union of robot masks whose instance name contains “gripper”; Robot Arm is the union of the remaining robot masks; Object is the current subtask focus union. Full is recomputed over the full rollout using the union of both subtask focus sets.

Back to Finding 5~7 All findings

S3 · INPUT EDITING

LIBERO-10 Text Editing

Table 5: Success rate (%) comparison between original and edited prompts.

Back to Finding 5~7 All findings

Key findings

What VLA-Trace reveals

Our experimental findings suggest that the conclusion is not simply that “vision matters” or “language matters.” Instead, different VLA architectures preserve, route, and use multimodal information in distinct ways.

Representation trace

Causal pathway trace

Behavior trace

Distinct adaptation dynamics

π0.5 and OpenVLA exhibit different modality-specific representation changes during VLA finetuning.

View S1 evidence ↗

Related evidence: S1 · CKA.

Different routing strategies

π0.5 is dominated by a concentrated visual-to-action pathway, while OpenVLA relies more broadly on visual grounding and prompt-region access.

View S1 evidence ↗

Related evidence: S1 · Drift CKA.

Layer dependency is not uniform

Some dependencies concentrate in narrow bottlenecks; others are distributed across broader layer ranges. The appendix extends this with windowed sweeps.

View S2 evidence ↗

Related evidence: S2 · Knockout.

Grounding is spatial and temporal

Action attention overlaps with robot-object interaction regions and shifts across the two phases of long-horizon LIBERO-10 tasks.

View S2 evidence ↗

Related evidence: S2 · Knockout.

Visual anchors are causal

Target-object masking causes large drops, showing that attended target objects are not merely visualized but causally required for execution.

View S3 + table ↗

Related evidence: S3 · Visual Patch Masking.

Shortcut dependence is model-specific

Gripper, robot body, and background perturbations expose different architecture-specific reliance patterns.

View S3 + table ↗

Related evidence: S3 · Shortcut Probes.

Semantic control remains partial

Edited instructions do not always redirect behavior, suggesting a gap between visual grounding and compositional language-conditioned control.

View S3 + table ↗

Related evidence: S3 · Input Editing.

Outlook

What this suggests for future VLA design

Representation

Preserve semantic priors

VLA training should balance adapting to robot control with preserving visual, language, and cross-modal semantic pathways.

Reasoning

Enable reliable VLA routing

Action generation depends on how visual and language information is routed through the model, not merely on its availability.

Semantics

Train for counterfactual control

Semantic-edit objectives, object-swap interventions, and contrastive behavior supervision could make subtle instruction changes causally actionable.

Citation

If you find this work useful, please cite the paper.

@misc{shi2026vlatracediagnosingvisionlanguageactionmodels,
  title={VLA-Trace: Diagnosing Vision-Language-Action Models through Representation and Behavior Tracing}, 
  author={Haoyuan Shi and Xiancong Ren and Yingji Zhang and Qinfan Zhang and Jiayu Hu and Haozhe Shan and Han Dong and Jinpeng Lu and Yinda Chen and Yi Zhang and Yong Dai and Xiaozhu Ju},
  year={2026},
  eprint={2605.30117},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2605.30117},
}