Traffic signs from 0.4% of the input 90.4% top-1 ›
How little does a model need to see?
LookThere uses reinforcement learning to pick which patches to compute. A small selector looks at a 154 px preview. A large extractor reads only the patches it picks.
Vision transformers typically treat every image token as equally important, yet for most tasks in computer vision only a fraction are needed. Adaptive computation methods accelerate inference by choosing which tokens to process, but existing methods struggle at extreme sparsity and require heuristics that may not generalize like token diversity and attention scores. We address these limitations with LookThere, achieving a new pareto frontier in performance-compute trade-offs through an end-to-end reinforcement learning framework that jointly trains a shallow input selector and a deep representation extractor. The selector learns where to look and the extractor learns what to see, together saving computation by selecting only what is worth processing for a given task without relying on auxiliary signals. We show that LookThere only selects the task-specific input, excelling at sparse recognition in high-resolution settings (traffic signs, billiards), and maintaining accuracy with as little as 0.2% of the input. It generalizes across tasks and models, including global recognition (ImageNet classification), local recognition (ADE20K segmentation), zero-shot classification (by distillation), and regression (counting). Across all settings, LookThere surpasses state-of-the-art selection to provide a general and scalable framework for specialized and efficient adaptive computation.
Interesting is not the same as important. Attention finds what stands out. The task needs something else.
Here are the top 10% of patches by DINOv2 attention and by LookThere. DINOv2 looks at the leaves around the bird. On the street it looks at cars and lamps instead of the sign. LookThere learns to look at what the task needs.
Birdsnap
DINOv2 attentiontop 10%
LookTheretop 10%
Traffic signs
DINOv2 attentiontop 10%
LookTheretop 10%
Select first, then compute. The selector decides before the deep layers run. Dropped patches cost nothing.
input
selector
policy
extractor
Two modelsA small selector looks at a low-resolution copy of the image. A large extractor reads only the patches it keeps at full resolution.
Coarse to fineThe policy decides on a 6×6 grid, then 18×18, then 37×37. If it skips a region, it skips everything inside it.
LookThereLookWhere
Any backboneImageNet top-1 at about 90% sparsity.
Traffic signsTop-1 accuracy (%) vs sparsity (%) · 1008 px
LookThereLookWhereDense
BilliardsTop-1 accuracy (%) vs sparsity (%) · 1008 px
LookThereLookWhereDense
Birdsnap, 518 px: accuracy against inference GFLOPs
The question decides where it looks. Same table. Different task. Different patches.
Billiard Balls asks three questions about the same image. LookThere learns a different selection for each one. LookWhere keeps the same patches for every question.
5 of 5,184 patches
1 / 4
One selector. Any budget you ask for. Tell it how many patches to keep.
Each budget level has its own learned setting. Here LookWhere keeps the same number of patches. On these ImageNet images, LookThere puts its patches on the object. LookWhere spends many of them along the edges of the image. LookThere matches LookWhere overall and does better at small budgets.
LookThere65 of 1,369 patches
LookWhere65 of 1,369 patches
Sparse inference in a few lines. Weights and training code for every dataset.
Load a checkpoint. The selector samples a mask. The extractor runs on the kept patches.
1Warm upThe extractor learns the task from all patches.
2Fine-tune with RLThe selector tries 8 masks per image. GRPO rewards the masks that are accurate and sparse.
3EvaluateSample a mask, or keep each patch with p above 0.5.
inference.pytrain.sh
from modeling import Extractor, Selector, mask_to_padded_indices
ckpt = torch.load("billiards/max-left-right/sparsity90/model.pt")
selector = Selector(cfg["img_size"], cfg["grid_stages"])
extractor = Extractor(cfg["img_size"], num_classes)
with torch.no_grad():
logits, prefix = selector(image) # (1, num_patches)
keep, _, _ = selector.sample(logits, cfg["grid_stages"], temp)
idx, pad = mask_to_padded_indices(keep)
out = extractor(image, prefix, idx, pad) # kept patches only
pred = extractor.head(out["cls"]).argmax(-1)
Cite. If LookThere helps your work.
@inproceedings{rammohan2026lookthere,
title = {LookThere! Sparse Vision by Reinforced Selection},
author = {Rammohan, Sreehari and Yassin, Yousef and Fuller, Anthony
and Wen, Junfeng and Vondrick, Carl and Shelhamer, Evan},
booktitle = {Advances in Neural Information Processing Systems},
year = {2026}
}