Large Language Models (LLMs) have demonstrated remarkable capabilities across numerous tasks, yet principled explanations for their underlying mechanisms and several phenomena, such as scaling laws, hallucinations, and related behaviors, remain elusive. In this work, we revisit the classical relationship between compression and prediction, grounded in Kolmogorov complexity and Shannon information theory, to provide deeper insights into LLM behaviors. By leveraging the Kolmogorov Structure Function and interpreting LLM compression as a two-part coding process, we offer a detailed view of how LLMs acquire and store information across increasing model and data scales—from pervasive syntactic patterns to progressively rarer knowledge elements. Motivated by this theoretical perspective and natural assumptions inspired by Heap’s and Zipf’s laws, we introduce a simplified yet representative hierarchical data-generation framework called the Syntax-Knowledge model. Under the Bayesian setting, we show that prediction and compression within this model naturally lead to diverse learning and scaling behaviors of LLMs. In particular, our theoretical analysis offers intuitive and principled explanations for both data and model scaling laws, the dynamics of knowledge acquisition during training and fine-tuning, factual knowledge hallucinations in LLMs. The experimental results validate our theoretical predictions.

Transformer 是当前大模型的核心架构，在自然语言处理、计算机视觉等领域展现出卓越性能，但其内在机理的理解仍较为有限。本报告从统计物理能量视角构建新框架，发现自注意力机制本质是最小化 Helmholtz 自由能的过程，由此为其提供了物理层面的原理解释。在此基础上，进一步从能量视角设计了新型 Transformer 架构，且实验验证了该架构的有效性。本研究为 Transformer 机理解读提供新视角，亦为其新架构设计提供新思路。

近年来，扩散模型的采样速度与生成质量显著提升，在多个领域均表现出优异性能。然而，针对扩散模型的理论分析与设计空间仍有待深入探索。本报告从扩散模型的两大核心部分采样过程与训练动态入手，重点分析如何提升模型的采样与训练效率。在采样复杂度方面，本报告系统梳理了不同模型的特性，明确阐释了基于流的模型为何相较以往方法具有更优表现；在训练动态方面，本报告结合实际数据特征进行建模，以降低估计误差并改善优化景观。

极小极大优化问题（又称鞍点计算或Nash均衡点计算）是数值优化、博弈论等领域的基础问题之一。该问题的最优一阶复杂度早已被研究透彻：1976年Korpelevich提出了外梯度算法并证明其具有 O(1/T) 的收敛率，20004年Nemirovski进一步推广了外梯度法并且证明匹配的下界。然而，高阶算法（牛顿法以及张量法）求解该问题的复杂度仍然是未知的。 此前，最好已知的二阶算法是由Monteiro和Svaiter在2012年提出的牛顿外梯度法，具有 O(1/T^{3/2}) 的收敛率。近五年内，大量工作使用不同的算法得到了相同的结果，并且将算法推广到p阶张量形式，具有 O(1/T^{(p-1)/2}) 收敛率。 很多研究者认为这已经是该问题的最优复杂度，并且给出了看似匹配的下界。 在本次报告，我会讨论已知的下界为什么可以被打破。并且介绍我们近期的工作，提出一个p阶算法打破普遍推测的下界，首次达到O(1/T^{(3p+1)/4}) 的收敛率。我们结果取p=2，可以得到一个快速牛顿法，具有O(1/T^{7/4})的收敛率，该结果已经发表于COLT'25，并且获得最佳学生论文奖。

本次报告将介绍对大型语言模型(LLMs) Hessian阵的分析及其对算法设计的影响。 首先，我们基于神经网络的Hessian的近似分块对角阵的观察，解释了 SGD 在 Transformer 上不如Adam的原因: (i) Transformer 是“异质性的”:不同参数块之间的 Hessian 谱差异显著; (ii) 异质性阻碍了 SGD: SGD 在存在块异质性的问题上表现不佳。受此发现启发，我们引入了 Adam-mini，它为每个块中的所有权重分配一个单一的二次动量项，相比 Adam 节省了 35-50%的优化器内存，在 8B 规模的语言模型表现出色。其次，我们初步分析了神经网络Hessian 阵为何具有分块对角的特性。由于分块对角是K-FAC,Shampoo,muon等优化器设计的出发点之一，对该性质的分析可帮助更好的理解这些优化器和设计更好的优化器。我们研究了两层神经网络MSE和cross-entropy两种损失函数下的Hessian阵，利用随机矩阵理论证明随机初始神经网络的 Hessian为近似块对角阵。一个意外的发现是，这个结论成立的一个条件是输出类别数需趋于无穷大。

随着大语言模型编程能力的持续提升，面向编程辅助的智能体系统迅速发展，并在实际开发中展现出显著成效。Cursor、Claude Code、Base44 等代表性工具已吸引大量开发者用户。然而，当前系统仍面临两大关键挑战：其一，难以高效生成具备完整架构与高一致性的超大规模软件系统；其二，交互模式多依赖“黑盒式”对话，用户在生成过程的细节控制上缺乏透明性与干预能力，代码的调整与优化往往依赖模型的迭代重生成，而非结构化引导。本报告尝试从范畴论与拓扑斯理论的角度出发，探索如何将这些数学基础应用于辅助编程系统的架构与交互设计，以突破现有瓶颈。我们提出的框架在类型编译器的帮助下，能够支持一次性并行生成数万行、并以较高概率一次编译通过的软件系统。更重要的是，该框架为用户（包括非专业编程背景者）提供了可解释、可干预的生成路径，实现前后端代码的协同构建与细粒度控制，从而在提升生成效率的同时增强系统的透明性与可控性。

While Reinforcement Learning (RL) and Fine-Tuning demonstrably enhance Large Language Model (LLM) capabilities, particularly in reasoning and task adaptation, the underlying mechanisms remain poorly understood. This talk integrates insights from two complementary studies to advance mechanistic interpretability. First, we dissect Reinforcement Learning with Verifiable Rewards (RLVR), revealing its core benefit lies in optimizing the selection of existing high-success-rate reasoning patterns, with theoretical convergence analyses showing distinct dynamics for strong versus weak initial models (mitigated by prior supervised fine-tuning). Second, we employ circuit analysis to interpret fine-tuning mechanisms, uncovering that circuits undergo significant edge changes rather than merely adding components, contrasting prior findings. Leveraging this, we develop a circuit-aware LoRA method, improving performance over standard LoRA by 2.46%. Furthermore, we explore combining circuits for compositional tasks. Together, these studies provide novel theoretical and empirical insights: RL enhances reasoning primarily through pattern selection, while fine-tuning fundamentally rewires circuit connections. This deeper understanding informs the design of more effective and interpretable adaptation strategies for LLMs.

In this talk, I will present our theoretical and practical advances in reinforcement learning (RL) and Reinforcement Learning from Human Feedback (RLHF). We introduce the Generalized Eluder Coefficient (GEC), a novel complexity measure that characterizes the fundamental statistical complexity in general decision-making problems. Our research demonstrates that GEC effectively captures the essential learning difficulty in various RL settings and enables the development of sample-efficient algorithmic frameworks. We extend our analysis to RLHF by introducing a preference-based variant of GEC, which addresses the sample complexity challenges in learning from human feedback. This theoretical advancement provides insights into the fundamental limits of RLHF algorithms. Building on these findings, we propose iterative Direct Preference Optimization (DPO) and its optimistic variant, which have demonstrated practical success in real-world applications. Our works establish a unified theoretical framework encompassing both RL and RLHF, offering a comprehensive understanding of their fundamental principles. This unified perspective not only advances theoretical understanding but also provides practical insights for developing more efficient learning algorithms.

模型即服务（MaaS）是模型决策时代重要的商业模式之一。MaaS分为三步：（1）模型消费者向模型供应商业提出需求；（2）模型供应商从相关的数据供应商中获取相关数据；（3）模型供应商使用数据供应商的数据构建人工智能模型，并用于数据消费者提供的商业场景中进行商业决策。在MaaS的场景下解决如何数据要素估值并公平的分配给数据供应商是重要的科学问题。基于合作博弈理论，把数据供应商作为合作博弈的玩家，把合作中获取的商业价值使用经典的Shapley值进行分配是公平合理的数据要素估值与分配的解决途径之一。但是，面对复杂的数据要素交易场景，基于经典的合作博弈进行数据要素估值与分配仍然无法应用于实际的数据要素交易市场。具体挑战为：（i）应用场景的不确定性；（ii）交易场景的高复杂性；（iii）计算成本的高耗费性。本研究将试图针对上述挑战寻找一定的解决途径。

We introduce AdamS, a simple yet effective alternative to Adam for large language model (LLM) pretraining and post-training. By leveraging a novel denominator, i.e., the root of weighted sum of squares of the momentum and the current gradient, AdamS eliminates the need for second-moment estimates. Hence, AdamS is efficient, matching the memory and compute footprint of SGD with momentum while delivering superior optimization performance. Moreover, AdamS is easy to adopt: it can directly inherit hyperparameters of AdamW, and is entirely model-agnostic, integrating seamlessly into existing pipelines without modifications to optimizer APIs or architectures. The motivation behind AdamS stems from the observed $(L_0, L_1)$ smoothness properties in transformer objectives, where local smoothness is governed by gradient magnitudes that can be further approximated by momentum magnitudes. We establish rigorous theoretical convergence guarantees and provide practical guidelines for hyperparameter selection. Empirically, AdamS demonstrates strong performance in various tasks, including pre-training runs on GPT-2 and Llama2 (up to 13B parameters) and reinforcement learning in post-training regimes. With its efficiency, simplicity, and theoretical grounding, AdamS stands as a compelling alternative to existing optimizers.

自回归模型串行生成的特点限制了其推理速度。近期，扩散大语言模型（Diffusion LLM）通过多步去噪的方式展现出了并行生成的潜力，并在精度上展现了媲美乃至超越自回归模型的效果。本报告主要讨论扩散大语言模型的推理加速方法，包括其缓存策略、解码策略、投机解码策略、以及变长生成方法等。同时，介绍扩散语言模型的一些潜在应用场景。

We discover the emergence of basins in the loss landscape of large language models. As model scale increases, LLMs become progressively more resilient to random perturbations in the parameter space, giving rise to expansive stability regions where models exhibit nearly identical performance, but outside of which their capabilities collapse. We observe that pre-training creates a basic capability basin, and subsequent alignment fine-tuning forms specific capability basins (e.g., safety, math, coding) within this basin. Thus, we argue that benign fine-tuning confined to the basin should preserve prior capabilities. Besides, we also analyze the loss landscape for worst-case directions, which is consistently sharp and detrimental. We find that adversarial fine-tuning moves along the nearly worst-case directions, thus rapidly degrading model capabilities. Finally, we provide a theoretical analysis demonstrating that the basin size bounds the performance degradation of any fine-tuning, including the adversarial ones, while also guaranteeing the model robustness w.r.t. input perturbations, suggesting the benefit of enlarging basins.