✨ TL;DR
This paper shows that applying Semantic Tube Prediction (STP) at reasoning step boundaries instead of random token positions dramatically improves multi-step latent prediction in LLMs (168x vs 4x improvement), revealing that sampling position is critical for geometric regularization of reasoning trajectories.
Semantic Tube Prediction (STP) is a technique that regularizes LLM hidden-state trajectories toward locally linear geodesics during fine-tuning to improve data efficiency. The original STP approach samples random token sub-spans, but it remains unclear whether the choice of sampling position affects the semantic structure and geometric properties of multi-step reasoning trajectories. Understanding how sampling strategy impacts the geometric regularization of reasoning paths is important for optimizing LLM training and improving reasoning capabilities.
The researchers modified the STP approach by applying it at consecutive semantic reasoning step boundaries rather than random token positions. They evaluated this step-boundary STP against frozen baselines and random-token STP using ProcessBench (3,400 samples), measuring multi-step latent prediction accuracy. To probe the geometric properties of the resulting trajectories, they used both linear extrapolation and a learned 3-layer MLP predictor to analyze the latent manifold structure. They also investigated the tradeoff between language modeling loss and geometric purity by training models with and without the language modeling objective.