Grounded in observations
Memory writes come from visual-language observations.
T²Mem:
Learning Test-Time Memory for Robotics
Recall a hidden object.
Count repeated actions.
Recall a demonstrated sequence.
Task demonstrations from RoboMME. Red border: demonstration phase.
56.83%
Mean success on RoboMME
+38.90pp
Over the memory-free π0.5 baseline
16tasks
Across four memory categories
Abstract
Memory-dependent robotic manipulation requires policies to use information that is no longer available in the current observation. Retaining history alone is insufficient: memory must preserve information that supports future actions. One challenge is whether a memory-free foundation model can learn to retain and use historical information from action demonstrations alone, without external memory support. We introduce T²Mem, a framework that develops this capability within a pretrained vision-language-action policy, without external reasoning models or memory-specific annotations. T²Mem uses test-time training to encode observation history into compact fast weights through online self-supervised updates, avoiding repeated processing of the full history. An observation-grounded interface extracts vision-language information for memory formation and supplies retrieved context to the action expert. Action supervision shapes what the memory learns to retain and use, while alternating memory-policy learning gives each component a fixed counterpart during optimization. Across 16 RoboMME tasks, T²Mem improves average success from 17.93% to 56.83% over the memory-free base policy and outperforms the recurrent-memory methods reported in the benchmark, while controlled profiling indicates at least 3x inference speedup over explicit methods.
01 / THE METHOD
Observation-Grounded Memory
Observe → read memory → act → update memory.
Observation + instruction↓Vision-language backbone
Current scene features
OBSERVATION-GROUNDEDMemory interfaceObservation features
read ←write →
EPISODE STATEFast weights W
SELF-SUPERVISED WRITEL = mean[(fW(K) − V)²]
↓Action expert↓History-conditioned action
Illustrative features · Blue: observation · Violet: retrieved history · Orange: memory update
Adaptive self-supervision
Write strength adapts to new observations.
Independent episode state
Fast weights update online and reset each episode.
The memory interface extracts compact vision-language features, retrieves relevant history from fast weights, and supplies both to the action expert. The original backbone-to-action pathway preserves access to the current scene.
Memory is grounded in observations, without direct access to action or proprioception tokens. At inference, only fast weights update; each episode starts from the learned initialization.
Inside Test-Time Training
Read with a forward pass. Write with a self-supervised gradient.
READ · Retrieve memoryProject the interface → Qₜ
Rₜ = fWₜ(Uₜ θQ)Read: retrieve history without changing W.
Schematic network and feature colors. Read → Write → Repeat · Only fast weights update during inference.
Read: project interface features into a query and pass it through the fast-weight network to retrieve context.
Write: project the same features into keys and values, then update fast weights to predict values from keys. Adaptive write strength helps preserve earlier associations. Reading happens before writing.
02 / LEARNING
Alternating Memory–Policy Learning
Learn what to remember. Then learn how to use it.
Action supervision updates the policy
MemorySlow parameters
PolicySlow parameters
Action lossExpert demonstrationsSupervision
Memory inactive
Blue: slow-parameter learning · Orange: episode-local fast-weight updates
↻ Alternate 2A ↔ 2B. Fast weights update in both.
Stage 1: adapt the policy without memory. Stage 2A: freeze the policy and train memory. Stage 2B: freeze memory’s slow parameters and train the policy to use it.
Alternate 2A and 2B using action demonstrations alone. Episode-local fast weights keep updating in both phases; no memory-specific labels are needed.
03 / RESULTS
Results on RoboMME
T²Mem raises average success from 17.93% to 56.83% across 16 tasks, outperforming the reported recurrent-memory baselines. Gains are strongest in cue retrieval and event counting; tracking swaps and precise insertion remain harder.
Ablation Studies
Architecture and learning: observation-grounded memory and alternating updates outperform their ablated counterparts on VideoUnmask and MoveCube.
Adaptive writing: early-memory reconstruction error falls by 18.8%–46.9% over 300 subsequent frames, indicating less interference with earlier associations.
04 / INSIDE THE MEMORY
Does the Policy Use Its Memory?
Same scene. Different histories. Different choices.
Same observation. Different histories.
VideoUnmask · counterfactual expert demonstrations
Front viewWrist view
Front viewWrist view
Same instruction: “Watch the video carefully, then pick up the container hiding the blue cube.” The container layout stays fixed; the earlier color locations change, so the robot must grasp a different container. Both clips include the full grasp.
Expert demonstrations illustrating the counterfactual task, not T²Mem policy rollouts.
Change only the memory: Correct uses the same episode’s history; Conflicting uses a donor episode with a different target; Empty uses the learned initialization without demonstration history. Instruction, goal, and diffusion seed stay fixed.
Across 50 pairs × both directions × 3 seeds, these conditions yield 300/300, 1/300, and 101/300 successes. Wrong history is more damaging than no episode history, showing that the policy uses memory content.
Timing matters: eight writes before occlusion achieve 76.67% success, versus 20.67% after occlusion—and match all 16 demonstration writes.
Inference Efficiency
Compact fast weights avoid reprocessing the full history. Under controlled profiling, T²Mem is 3.0× faster than FrameSamp+Modul and 69.2× faster than MemER.
Citation
@misc{liu2026t2memlearningtesttimememory,
title={T$^2$Mem: Learning Test-Time Memory for Robotics},
author={Yize Liu and Huang Huang and Yining Hong and Zijian Du and Zhi Cao and Li Fei-Fei and Jiajun Wu},
year={2026},
eprint={2609.36720},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2609.36720},
}