Seeding Word Embeddings with Heat Diffusion on a Known Hierarchy用热扩散初始化词嵌入

Jiawen Kang · October 20262026 年 10 月Back

Most text models start by encoding each token as a one-hot vector over the vocabulary. Viewed as a distribution, it has zero entropy: one token takes all the mass and the others get none. At the other end, if every token were given the uniform distribution, the entropy would be at its maximum, but all tokens would look the same. Sometimes we already know which words are related and which are not. We can use this knowledge as a prior and give each word a representation somewhere in between.

词与词在语义上有远有近。“书桌”和“台灯”同属家具，“书桌”和“印度”则几乎没有关联。然而，大多数文本模型从 one-hot 向量或随机初始化的嵌入（embedding）出发，任意两个词之间的距离都相同，哪些词彼此相关，只能依靠训练从数据中发现。

TrajLift does this by letting part of the mass flow to related words. We build a graph from the semantic relations between the words and let each word’s one-hot vector spread over it like heat, first to its closest neighbors and then to more distant ones. Given enough time, the heat evens out and every word ends up with the uniform distribution. We stop before then, while related words have similar embeddings and unrelated ones still do not.

TrajLift 从我们对这些词已有的认识出发。我们把词之间的语义关系写成一张图（graph），让每个词的嵌入像热量一样在图上扩散：先到达最相近的词，再到达较远的词。这样得到的嵌入在训练开始之前，就已经反映了语义相似性。

The diffusion time controls how far the heat spreads. Because it is continuous, we can also adjust it word by word so that the embedding carries timing. For example, when a word is followed by a longer pause, its heat spreads a little further toward the uniform end. In this post we look at some properties of these embeddings and use them to encode word recall trajectories.

热量扩散的范围由一个连续的量控制，即扩散时间（diffusion time）。逐词调整这个量，同一机制还能发挥第二个作用：记录一个词在序列中的位置，以及序列在它前后如何展开。两者结合，就得到一种语义与位置增强的文本嵌入（semantic- and position-enhanced text embedding），我们在单词序列分类任务上验证了它的表征能力。

We work with the 16-word list of the Hong Kong List Learning Test (HKLLT), a verbal memory test in which people study the words and recall as many as they can in any order. Each token in a recall sequence is one of these words. All figures are interactive and computed live in your browser.

本文分三步介绍这一构造，并以一个语义层级事先已知的 16 词词表为例。这些词来自香港词语学习测验（Hong Kong List Learning Test，HKLLT），一项言语记忆测验。文中所有图表均可交互，数值在浏览器中实时计算。

1. Building the semantic graph用图写下词之间的语义关系
2. Heat diffusion creates semantic similarity热扩散带来语义相似性
3. Encoding timing through diffusion time通过扩散时间编码位置与时序动态
4. Putting it together整体流程

1. Building the semantic graph1. 用图写下词之间的语义关系

To represent relations between words, we use a weighted graph where each word is a node and the weight of an edge says how closely two words are related. The weights come from the natural hierarchy of our 16 words, which fall into four categories (relatives, countries, vegetables and furniture) that group into an abstract and a concrete domain. To turn this hierarchy into a distance, we ask how far up the hierarchy two words must go before they fall into the same group. For example, desk and lamp meet in the furniture category (distance 1), desk and onion meet in the concrete domain (distance 2), and desk and India meet only at the top (distance 3). This is the ultrametric distance $d(i,j)$, and the edge weight decays with it:

要让嵌入反映哪些词相近，首先要把“相近”写成算法能够使用的形式。带权图正好可以做到：每个词是一个节点，边的权重表示两个词的关联程度。

这 16 个词本身带有一个自然的层级：它们分为四个类别（亲属、国家、蔬菜和家具），这些类别又组成两个领域，即抽象与具体。为了把层级转化为距离，我们看两个词需要沿层级向上走几层才会落入同一组。“书桌”和“台灯”向上一层就在“家具”中相遇，距离为 1；“书桌”和“洋葱”要到“具体”这一领域才相遇，距离为 2；“书桌”和“印度”只在最顶层相遇，距离为 3。这就是超度量距离 $d(i,j)$，边权随它递减：

With $\rho = 10$ as in the paper, words in the same category are joined with weight 1, words in the same domain with weight 0.1, and words in different domains with weight 0.01. Figure 1 shows the graph, and Figure 2 shows the same weights as a matrix. Sorted by the hierarchy, the matrix has small dark blocks for the categories nested inside larger, lighter blocks for the domains.

取论文中的 $\rho = 10$，同一类别的词之间边权为 1，同一领域的词之间为 0.1，不同领域的词之间为 0.01。图 1 是由此得到的语义图，点击任意一个词可以查看它与其他词的连接。图 2 用矩阵展示同样的边权：按层级排序后，矩阵呈现嵌套的块状结构，深色的小块是类别，外围浅一些的大块是领域。

The ratio $\rho$ sets how sharply the hierarchy is drawn. When $\rho$ is close to 1, all edges have similar weights and the levels blur together. As $\rho$ grows, categories and domains stand out more clearly. The slider below changes $\rho$ for every figure on the page.

参数 $\rho$ 决定层级被刻画得有多分明。$\rho$ 接近 1 时，所有边的权重相差不大，各个层级混在一起；$\rho$ 越大，类别和领域就越清晰。下方滑块调整的 $\rho$ 作用于全文所有图表。

Because the hierarchy is fixed in advance, the graph is the same for every sequence and has no learned parameters. It records what we know about the words.

由于层级是事先给定的，这张图对所有序列都相同，也不含需要学习的参数，它记录的是我们对这些词已有的认识。下一步，是让每个词的嵌入用上这张图。

2. Heat diffusion creates semantic similarity2. 热扩散带来语义相似性

We now want each word’s embedding to reflect the relations in the graph. In the one-hot vector of desk, for instance, lamp and India both get nothing; we would like lamp to get a fair share and India almost none.

先看一个词 $v$ 最简单的嵌入，即 one-hot 向量 $x = e_v$。它只标识这个词本身，与其他所有词的余弦相似度都为零，因此“书桌”和“台灯”的距离与它和“印度”的距离完全相同。我们希望得到的，是这样一个“书桌”的嵌入：包含较多“台灯”“镜子”“衣柜”的成分，少量蔬菜类词的成分，几乎不含国家类词的成分。

Heat diffusion on the graph gives us this. For a word $v$ with one-hot vector $x = e_v$, we place one unit of heat on $v$ and let it flow along the edges for a time $\tau$. Heat moves from warmer nodes to cooler ones and moves faster along heavier edges. Formally, this flow is described by the graph Laplacian $L$, which measures how much the value at each node differs from a weighted average of its neighbors. We use the normalized form $L = I - D^{-1/2} W D^{-1/2}$, where $D$ holds the weighted degrees. The heat distribution at time $\tau$ is

图上的热扩散（heat diffusion）恰好能得到这样的嵌入。在 $v$ 上放一份热量，让它沿着边流动一段时间 $\tau$。热量从温度高的节点流向温度低的节点，边权越大，流得越快。描述这一过程的是图拉普拉斯矩阵（graph Laplacian）$L$，它衡量每个节点上的值与其邻居加权平均之间的差异。我们使用归一化形式 $L = I - D^{-1/2} W D^{-1/2}$，其中 $D$ 为加权度矩阵。时刻 $\tau$ 的热量分布为

and we use $z(\tau)$ as the embedding of $v$. Since category edges are the heaviest, heat reaches the other words in the same category first, then the rest of the domain, and only later the other domain (Figure 3). If we diffuse every word and compare the results, the cosine similarity matrix starts as the identity, develops four category blocks, merges them into two domain blocks and finally becomes uniform (Figure 4). Drag the slider or press play to watch this happen.

我们把 $z(\tau)$ 作为 $v$ 的嵌入。由于类别内部的边权最大，热量会先到达同一类别的其他词，然后是同一领域的词，最后才到达另一个领域（图 3）。对每个词分别进行扩散并比较结果，余弦相似度矩阵从单位矩阵开始，先出现四个类别块，再合并为两个领域块，最终趋于均匀（图 4）。拖动滑块或点击“播放”可以观察这一过程；向下滚动时，控制栏会固定在屏幕顶部。

The order in which the levels appear comes from the frequencies of the graph. The eigenvectors of $L$ play the role of Fourier modes here: an eigenvector with a small eigenvalue varies slowly and takes similar values across a whole domain or category, while one with a large eigenvalue changes sharply between neighboring words. For our graph, the 16 eigenvalues take only four distinct values (Figure 5): a constant mode at $\lambda = 0$, one mode that separates the two domains, two modes that separate the categories within each domain, and twelve modes that distinguish the words within each category. So each level of the hierarchy gets its own frequency band. The bands are separated by clear gaps, which widen as $\rho$ grows.

各层级为什么会按这个顺序、而且如此分明地出现？答案在于图的“频率”。$L$ 的特征向量相当于图上的傅里叶基：特征值小的特征向量变化缓慢，在整个领域或类别内取值相近；特征值大的特征向量则在相邻的词之间剧烈变化。对于我们的图，16 个特征值只取四个不同的值（图 5），对应四组特征向量：1 个常数特征向量（$\lambda = 0$），1 个区分两个领域的特征向量，2 个区分各领域内类别的特征向量，以及 12 个区分类别内各个词的特征向量。层级的每一层都占据一个独立的频带，相邻频带之间有明显的间隔，$\rho$ 越大，间隔越宽。

Heat diffusion removes these bands one at a time. The heat kernel multiplies each eigencomponent by $e^{-\tau\lambda}$, so it acts as a low-pass filter whose cutoff moves down as $\tau$ grows (Figure 6): the word band fades first, followed by the category band and then the domain band. Each time a band fades, the distinctions it carries disappear from the embedding, and words that differ only in those distinctions become similar. This is why the average similarity of word pairs in Figure 7 rises in order of their hierarchical distance.

热扩散逐个作用于这些频带。热核（heat kernel）把每个特征分量乘以 $e^{-\tau\lambda}$，相当于一个低通滤波器，其截止频率随 $\tau$ 增大而降低（图 6）。词级频带的特征值最大，最先衰减，其次是类别频带，最后是领域频带。每当一个频带衰减，它所承载的差异就从嵌入中消失，仅在这些差异上不同的词也随之变得相似。图 7 展示了这一结果：词对的平均相似度按层级距离由近到远依次上升。

The diffusion time therefore decides which level of the hierarchy the embedding resolves. Since our hierarchy has three levels, TrajLift uses three diffusion times, one for each, called micro, meso and macro. Specifically, the micro scale keeps the differences between individual words, the meso scale keeps the categories apart, and the macro scale keeps only the two domains apart. The three diffusion times are initialized as $\tau_s = -\ln\alpha_s / \lambda_{\max}$ with $\alpha_s = 0.85,\ 0.25,\ 0.002$ and then learned during training. With these initial values, the word band retains 85%, 25% and 0.2% of its amplitude, respectively. The buttons in the control bar jump to the three initial diffusion times.

因此，一个连续参数就决定了嵌入分辨到层级的哪一层。TrajLift 使用三个扩散时间，分别称为微观（micro）、中观（meso）和宏观（macro），初始化为 $\tau_s = -\ln\alpha_s / \lambda_{\max}$，其中 $\alpha_s = 0.85,\ 0.25,\ 0.002$，并在训练中学习。微观尺度保留单个词之间的差异，中观尺度区分各个类别，宏观尺度只区分两个领域。控制栏中的三个按钮可以直接跳转到这三个取值。借助这三个视角，每个词在模型见到任何序列之前，就已经拥有了在三种分辨率下反映其语义邻域的嵌入。

3. Encoding timing through diffusion time3. 通过扩散时间编码位置与时序动态

In Section 2, each scale uses a fixed diffusion time chosen to resolve one level of the hierarchy. Changing $\tau$ moves the filter’s cutoff, and as long as the cutoff stays within the same gap between bands (Figure 5), the scale still resolves the same level. Each scale therefore has some room in $\tau$ that does not change the level it resolves. Because $\tau$ is continuous, this room can carry a continuous signal attached to each word.

In a recall trajectory, the pauses between words are part of the data, but the embeddings so far ignore them: desk followed by a one-second pause gets the same embedding as desk followed by a ten-second pause. We use this room to encode the pauses. The longer the pause after a word, the longer its diffusion time at every scale, and the further its heat spreads. Since the extra heat flows along the edges of the graph, the same pause has a different effect depending on where the word sits. At the micro scale, the extra heat after desk goes mainly to the other furniture, and after India, mainly to the other countries.

到目前为止，嵌入只取决于它表示的是哪个词。但在序列中，每个 token 还携带着词本身之外的信息：它出现在什么位置，以及序列在它前后如何展开，例如时间戳、持续时长或与下一个 token 之间的停顿。扩散时间 $\tau$ 是一个连续参数，自然可以作为把这类信息写入嵌入的入口。问题在于：逐个 token 调整 $\tau$，是否会同时破坏 $\tau$ 原本用来刻画的语义结构？

In general, let $s_k$ be any continuous scalar attached to the token at position $k$. We shift the diffusion time at each scale by

由此得到一个通用的做法。设 $s_k$ 是附着在第 $k$ 个 token 上的任意连续标量，我们按下式调整每个尺度的扩散时间：

where $\varphi$ is a monotone compression and $\gamma_s \ge 0$ sets the modulation strength. The value $s_k$ could be the token’s position, a timestamp, a duration or the pause after it. We bound $\gamma_s$ so that $\tilde\tau$ stays within a factor $r$ of $\tau_s$ for typical signals.

其中 $\varphi$ 是单调的压缩函数，$\gamma_s \ge 0$ 控制调制强度。下标 $k$ 只用来标识 token，真正写入嵌入的是连续值 $s_k$，它可以是位置坐标、时间戳，也可以是刻画序列演化过程的量。我们对 $\gamma_s$ 加以约束，使得对于常见的信号取值，$\tilde\tau$ 不超过 $\tau_s$ 的 $r$ 倍。图 8 把 $r = 3$ 时各尺度的调制范围，与各层级被分辨的区域放在一起比较。

To check that the semantic structure stays intact, we expand the embedding to first order in the shift $\Delta\tau = \tilde\tau - \tau_s$:

谱结构给出了答案。第 2 节中我们看到，层级的各个层次占据相互分离的频带，相邻频带之间存在谱间隔（spectral gap），如图 5 所示。一个尺度能分辨某一层级，是因为它的滤波器保留了直到该层级的频带，并抑制了更高频的频带。改变 $\tau$ 会移动这一截止位置。只要截止位置仍处于同一个谱间隔内，该尺度分辨的层级就保持不变，改变的只是保留频带内部的衰减程度。做一阶展开：

The zero-order term is the structural embedding, which the modulation leaves unchanged. The modulation enters only through the first-order term, which is proportional to $\Delta\tau$ and passes through the Laplacian, so the change follows the edges of the graph. The semantic structure stays in place, and the new information comes in as a small correction to it.

零阶项就是结构嵌入，不受调制影响。调制只通过一阶项进入，该项与 $\Delta\tau$ 成正比，并经过拉普拉斯矩阵作用。换言之，这一变化沿着图的边发生，表现为更多的热量从这个词流向它的邻居。语义结构因此保持不变，新的信息叠加在它之上。

The bound on $\gamma_s$ is there to keep each cutoff inside its gap. Figure 8 shows the resulting ranges for $r = 3$ against the regions where each level is resolved.

With $r = 3$ and the default $\rho = 10$, each scale’s range stays inside its own region.

取 $r = 3$ 时，每个尺度的调制范围都落在各自的区域内。在这一范围内改变 $\tilde\tau$，改变的是热量扩散的程度，而不是该尺度所分辨的层级。

For the HKLLT recall task, $s_k$ is the latency $\delta_k$ in seconds between consecutive recalled words, compressed with $\varphi(\delta) = \log(1+\delta)$, and $\gamma_s$ is initialized so that $\tilde\tau$ reaches $3\tau_s$ at the 90th-percentile latency. Longer pauses, up to a cap of 20 seconds, produce the dashed extensions in Figure 8. In this setting the diffusion time carries only the timing, and the order of the words is left to the sequence encoder in Section 4. Figure 9 shows a recall sequence. Select a word to see its embedding, change the latency at that position with the slider and switch scales to compare the three channels.

以 HKLLT 回忆任务为例。受试者在学习词表后尽可能多地回忆其中的词，顺序不限，相邻两次回忆之间的停顿反映了记忆检索的过程。此时 $s_k$ 取相邻两次回忆之间的间隔 $\delta_k$（单位为秒），用 $\varphi(\delta) = \log(1+\delta)$ 压缩；$\gamma_s$ 的初始化使间隔达到第 90 百分位数时 $\tilde\tau$ 恰为 $3\tau_s$。更长的停顿（上限为 20 秒）对应图 8 中的虚线延伸。词的先后顺序由第 4 节的序列编码器处理，扩散时间只承载时间信息。图 9 给出了一条示例回忆序列。选择其中一个词可以查看它的嵌入，用滑块调整该位置的间隔，切换尺度可以比较三个通道。

For each word, TrajLift keeps both the modulated embedding $z_{\text{mod}}$ and the residual $\Delta z$. The residual contains only the change caused by the pause, so the model sees the timing both on its own and mixed into the embedding.

间隔越长，扩散时间越长，就有更多热量离开源词、流向它的邻居。TrajLift 同时保留调制后的嵌入 $z_{\text{mod}}$ 和残差 $\Delta z$，后者作为专门承载位置与时序动态（position and dynamics）信息的通道。

The residual grows smoothly and monotonically with the latency (Figure 10). The three scales also respond to different parts of the spectrum. To see why, note that to first order the residual multiplies each eigencomponent by $\lambda e^{-\tau_s \lambda}$. The response is a band-pass filter: it vanishes at $\lambda = 0$ and as $\lambda \to \infty$, and peaks at $\lambda^\star = 1/\tau_s$ (Figure 11). Each scale has its own $\tau_s$, so the peaks fall at different places along the spectrum.

残差随信号平滑、单调地增大（图 10），因此连续的信号对应于嵌入的连续变化。此外，三个尺度对应于谱的不同部分。在一阶近似下，残差对每个特征分量施加的响应为 $\lambda e^{-\tau_s \lambda}$。这是一个带通滤波器：它在 $\lambda = 0$ 和 $\lambda \to \infty$ 时趋于零，在 $\lambda^\star = 1/\tau_s$ 处取得峰值（图 11）。由于各尺度的 $\tau_s$ 不同，峰值落在谱的不同位置。

Decomposing the residual energy $\|\Delta z\|^2$ by eigenspace shows this more directly (Figure 12). The coarser the scale, the more of its residual energy lies in the low-frequency domain and category bands. Combined with the next word in the sequence, the three residuals can help the model tell a long pause before a category switch from a long pause within a category.

将残差能量 $\|\Delta z\|^2$ 按特征子空间分解，可以更直接地看到这种分工（图 12）：尺度越粗，残差能量越集中于低频的领域频带和类别频带。因此在回忆任务的例子中，类别内部的停顿与跨类别切换时的延迟会被不同的残差通道分别捕获。

At each of the three scales, every position in a sequence now has an embedding of the word’s meaning and a residual that records its timing.

至此，序列中的每个位置在三个尺度上各有两个向量：表达词义的嵌入，以及记录位置与时序动态的残差。剩下的工作是把它们组合起来。

4. Putting it together4. 整体流程

At each position, we normalize the embedding and the residual at each scale and concatenate them into $\left[\mathrm{norm}(z^{(s)}_{\text{mod}}) \,\|\, \mathrm{norm}(\Delta z^{(s)})\right] \in \mathbb{R}^{2N}$. The three scales are then merged from fine to coarse. The micro scale serves as the backbone, and the meso and macro scales add context through learned gates that decide dimension by dimension how much of the coarser view to let in:

在每个位置上，我们把每个尺度的嵌入和残差分别归一化，再拼接为 $\left[\mathrm{norm}(z^{(s)}_{\text{mod}}) \,\|\, \mathrm{norm}(\Delta z^{(s)})\right] \in \mathbb{R}^{2N}$。三个尺度随后由细到粗依次融合：微观尺度作为主干，中观和宏观尺度通过学习得到的门控补充上下文，门控逐维决定引入多少较粗尺度的信息：

The fused sequence is projected, encoded by a two-layer LSTM and classified by a linear layer, all trained end to end with cross-entropy.

融合后的序列经过线性投影，由两层 LSTM 编码，最后经线性层完成分类，整个模型以交叉熵损失端到端训练。

To test whether combining structure and timing helps, we construct a synthetic benchmark called ST-XOR. Its two classes have the same mix of within-group and between-group transitions and the same mean latency, so only the interaction between structure and timing can tell them apart. On the three-level variant every baseline stays near chance, while TrajLift reaches 98.7% accuracy. On real data, we apply TrajLift to HKLLT delayed-recall sequences for Alzheimer’s screening. It outperforms clinical measures as well as sequence and graph baselines, reaching 80.5% and 81.0% accuracy on Trials 4 and 5.

把结构与时间结合起来是否真的有用？我们构造的合成基准 ST-XOR 正是为检验这一点而设计的。它的两类序列具有相同的边类型分布和相同的平均间隔，只有结构与时间的交互能够区分它们。在三层层级的版本上，所有基线方法的准确率都接近随机水平，而 TrajLift 达到 98.7%。在真实数据上，我们把 TrajLift 用于阿尔茨海默病筛查中的 HKLLT 延迟回忆序列，它同样优于临床指标以及序列模型和图模型基线，在第 4 次和第 5 次试验上的准确率分别为 80.5% 和 81.0%。

Summary总结

TrajLift turns prior knowledge about words into an embedding in three steps. We record the semantic hierarchy as a graph. Heat diffusion on the graph then gives each word an embedding that reflects its semantic neighborhood from the start, and the diffusion time sets the level of detail. Adjusting the diffusion time token by token adds the pauses between words to the same embedding, within a range where each scale still resolves the same level. The paper gives the full method, the spectral analysis and the experiments.

TrajLift 用三步把关于词的先验知识转化为嵌入。首先，用一张图记录语义层级。其次，在图上进行热扩散，使每个词的嵌入从一开始就反映其语义邻域，并由一个连续的扩散时间决定细节的层次。最后，逐个 token 调整这一时间，在语义结构保持不变的范围内，把位置与时序动态写入同一个嵌入。完整的方法、谱分析和实验见论文。

2026 © Jiawen Kang.