Researchers introduce NCP-ArchPreview, a latent-space language model that extends standard autoregressive pretraining by adding a Next Concept Prediction objective alongside next-token prediction, forecasting discrete concepts that span multiple tokens. The model builds a discrete concept vocabulary using product quantization over hidden states and trains concept-level and token-level objectives jointly. At 8.9 billion parameters trained on 5.73 trillion tokens, it matches the loss of a 7-billion-parameter baseline using only 51.3% as many tokens and outperforms OLMo-3-7B by 2.45 points on downstream tasks, while its learned latent space also supports lightweight domain adaptation.
