https://akiyangdev.github.io/llm-inference-notes/SGLang、Ascend 与 DeepSeek 推理工程文章zh-CNhttps://akiyangdev.github.io/llm-inference-notes/articles/performance/aisbench-whole-native-tpot-itl/TPOT 10.4ms、ITL 58.6ms为何同时成立？沿固定AISBench源码解释请求时间线、事件加权、whole/native选样边界、并发封顶与两种输出token/s。https://akiyangdev.github.io/llm-inference-notes/articles/performance/aisbench-whole-native-tpot-itl/Sat, 24 Oct 2026 11:59:50 +0800https://akiyangdev.github.io/llm-inference-notes/articles/distributed/ascend-dynamic-eplb-weight-migration/从逻辑专家与物理槽位出发，沿真实Ascend适配解释W8A8/NZ权重暂存、HCCL通信等待、逐层映射提交和Graph地址约束，结合功能验证与正负性能结果分析动态EPLB的成本。https://akiyangdev.github.io/llm-inference-notes/articles/distributed/ascend-dynamic-eplb-weight-migration/Sat, 24 Oct 2026 11:57:30 +0800https://akiyangdev.github.io/llm-inference-notes/articles/performance/sglang-stream-acceptance-cache-timing/8条请求已带投机开关，SSE却没有sglext。从字段产生、父类解析和writer落盘走到原计时规则，解释metadata-only事件、接受率分母与whole/native窗口。https://akiyangdev.github.io/llm-inference-notes/articles/performance/sglang-stream-acceptance-cache-timing/Sat, 24 Oct 2026 11:45:18 +0800https://akiyangdev.github.io/llm-inference-notes/articles/performance/128k-prefix-cache-remaining-costs/沿真实128K/1K请求区分复用前缀、新Query和逻辑上下文，解释DeepSeek-V4的SWA/C4/C128历史访问、PD状态交接，以及90%命中之后仍然存在的生成与排队成本。https://akiyangdev.github.io/llm-inference-notes/articles/performance/128k-prefix-cache-remaining-costs/Sat, 24 Oct 2026 11:42:32 +0800https://akiyangdev.github.io/llm-inference-notes/articles/performance/aisbench-sglang-first-request-to-report/从一条请求接通SGLang，到独立配置、JSONL、N/C/r和原生稳态后处理。附小样例，讲清retry的版本语义、实际token长度和原始结果怎样保留。https://akiyangdev.github.io/llm-inference-notes/articles/performance/aisbench-sglang-first-request-to-report/Sat, 24 Oct 2026 11:23:08 +0800https://akiyangdev.github.io/llm-inference-notes/articles/performance/ascend-910c-cp2-vs-cp4-prefill-throughput/复盘CP4/DP4切到CP2/DP8的N480同负载负例：单rank行数、page预算、EP16通信轮次与请求驻留时间，解释TPOT改善却没有吞吐收益。https://akiyangdev.github.io/llm-inference-notes/articles/performance/ascend-910c-cp2-vs-cp4-prefill-throughput/Sat, 24 Oct 2026 11:14:20 +0800https://akiyangdev.github.io/llm-inference-notes/articles/performance/expert-count-export-hccl-gloo/160 条请求已生成完，存专家计数时服务却崩了。沿错误栈追到设备通信，再看 CPU/Gloo 导出怎样保住统计含义。https://akiyangdev.github.io/llm-inference-notes/articles/performance/expert-count-export-hccl-gloo/Sat, 24 Oct 2026 10:58:45 +0800https://akiyangdev.github.io/llm-inference-notes/articles/performance/npu-timeline-gaps-and-recoverable-speedup/NPU 时间线约两成空白，为什么不能当成可回收的加速空间？先算清区间覆盖，再检查等待、跨 Rank 依赖和测量范围。https://akiyangdev.github.io/llm-inference-notes/articles/performance/npu-timeline-gaps-and-recoverable-speedup/Sat, 24 Oct 2026 10:56:12 +0800https://akiyangdev.github.io/llm-inference-notes/articles/performance/128k-prefix-cache-benchmark-calibration/输入长度正确、请求全部成功，缓存却没达到九成。记录公共前缀、预热与逐条检查怎样把压测场景校准。https://akiyangdev.github.io/llm-inference-notes/articles/performance/128k-prefix-cache-benchmark-calibration/Sat, 24 Oct 2026 10:43:21 +0800https://akiyangdev.github.io/llm-inference-notes/articles/performance/deepseek-v4-prefill-chunk-size-selection/把 chunk 从 8192 增到 8320，分块少了，吞吐却略降。结合页对齐、并行布局和通信轮次解释为什么。https://akiyangdev.github.io/llm-inference-notes/articles/performance/deepseek-v4-prefill-chunk-size-selection/Sat, 24 Oct 2026 10:40:31 +0800https://akiyangdev.github.io/llm-inference-notes/articles/performance/deepseek-v4-indexcache-freq2-cross-layer-topk-reuse/跨层复用 TopK 后，哪些计算省下了，哪些状态还得更新？对照实际执行路径和同负载实验，看 freq2 的收益与代价。https://akiyangdev.github.io/llm-inference-notes/articles/performance/deepseek-v4-indexcache-freq2-cross-layer-topk-reuse/Sat, 24 Oct 2026 10:26:35 +0800https://akiyangdev.github.io/llm-inference-notes/articles/performance/ascend-910c-prefill-scenario-static-eplb/专家布局怎样从路由计数拟合出来？沿一次 P 端静态方案解释放置与分发，并说明 14.18% 的组合收益该怎样归因。https://akiyangdev.github.io/llm-inference-notes/articles/performance/ascend-910c-prefill-scenario-static-eplb/Sat, 24 Oct 2026 10:11:05 +0800https://akiyangdev.github.io/llm-inference-notes/articles/performance/decode-fast-throughput-limited-by-prefill/平均 TPOT 已经低至 8.7ms，吞吐仍没跟上。一个 4-token 尾块，把排查方向从 Decode 带回了 Prefill 合批。https://akiyangdev.github.io/llm-inference-notes/articles/performance/decode-fast-throughput-limited-by-prefill/Sat, 24 Oct 2026 10:04:14 +0800https://akiyangdev.github.io/llm-inference-notes/articles/performance/ascend-910c-inference-tuning-negative-results-and-stop-rules/TPOT 降了，吞吐却没涨；分块少了，执行反而慢了。沿几项负例，判断原假设漏了什么、何时值得重试。https://akiyangdev.github.io/llm-inference-notes/articles/performance/ascend-910c-inference-tuning-negative-results-and-stop-rules/Sat, 24 Oct 2026 09:53:42 +0800https://akiyangdev.github.io/llm-inference-notes/articles/performance/ascend-910c-inference-tuning-effective-optimizations-and-costs/哪些改动带来了吞吐，哪些只增加了实验成本？放在一起看 P 端合批、IndexCache、静态布局和 Compact 的真实对照。https://akiyangdev.github.io/llm-inference-notes/articles/performance/ascend-910c-inference-tuning-effective-optimizations-and-costs/Sat, 24 Oct 2026 09:43:15 +0800https://akiyangdev.github.io/llm-inference-notes/articles/performance/ai-driven-inference-autotuning-decision-loop/把吞吐目标交给 AI 后，下一轮实验该怎么选？结合真实调优，讨论比较条件、候选依据和停止投入的时机。https://akiyangdev.github.io/llm-inference-notes/articles/performance/ai-driven-inference-autotuning-decision-loop/Wed, 21 Oct 2026 19:50:15 +0800https://akiyangdev.github.io/llm-inference-notes/articles/performance/ai-driven-inference-autotuning-execution-and-evidence/640 条请求已成功，统计程序却退出非零。怎样接着处理原数据，并让配置、进程和结果始终对得上？https://akiyangdev.github.io/llm-inference-notes/articles/performance/ai-driven-inference-autotuning-execution-and-evidence/Wed, 21 Oct 2026 19:50:15 +0800https://akiyangdev.github.io/llm-inference-notes/articles/performance/inference-tuning-workflow-postmortem/正式请求只跑了约三分钟，准备和修统计却耗时更久。沿几次真实返工，看连续调优的时间花在了哪里。https://akiyangdev.github.io/llm-inference-notes/articles/performance/inference-tuning-workflow-postmortem/Wed, 21 Oct 2026 17:46:11 +0800https://akiyangdev.github.io/llm-inference-notes/articles/performance/ascend-950dt-single-node-pd-optimization/一次 Ascend 950DT 单机长输入调优记录：解释两端配置、缓存与并发如何一起影响结果，避免把总收益归给一个开关。https://akiyangdev.github.io/llm-inference-notes/articles/performance/ascend-950dt-single-node-pd-optimization/Wed, 14 Oct 2026 18:05:21 +0800https://akiyangdev.github.io/llm-inference-notes/articles/performance/deepseek-v4-pro-32k-prefix-cache-concurrency-tuning/先把缓存命中率校准，再比较并发。记录一次 32K/1K 调优中，吞吐、TPOT 与首 Token 等待之间的取舍。https://akiyangdev.github.io/llm-inference-notes/articles/performance/deepseek-v4-pro-32k-prefix-cache-concurrency-tuning/Sat, 10 Oct 2026 15:55:39 +0800https://akiyangdev.github.io/llm-inference-notes/articles/sglang/gsm8k-accuracy-debugging-from-200-to-full-evaluation/200 题得分未过线，扩大到 1314 题后却通过了。沿配置对照和逐题结果，分开检查评测覆盖与输出稳定性。https://akiyangdev.github.io/llm-inference-notes/articles/sglang/gsm8k-accuracy-debugging-from-200-to-full-evaluation/Sat, 10 Oct 2026 15:25:26 +0800https://akiyangdev.github.io/llm-inference-notes/articles/pr-reviews/sglang-pr-39120-tensor-view-storage-cache/缓存里只有 0.8 MiB 的小 Tensor，为什么可能占住约 7.8 MiB？解释 View 如何引用整块存储，以及补丁为什么选择复制。https://akiyangdev.github.io/llm-inference-notes/articles/pr-reviews/sglang-pr-39120-tensor-view-storage-cache/Mon, 21 Sep 2026 20:53:14 +0800https://akiyangdev.github.io/llm-inference-notes/articles/pr-reviews/sglang-pr-39871-attention-parallel-widths/TP16 / DP2 的 Attention 实际宽度是 8，日志却打印 16。这个小补丁说明了原始配置值和运行时派生值为什么要分清。https://akiyangdev.github.io/llm-inference-notes/articles/pr-reviews/sglang-pr-39871-attention-parallel-widths/Mon, 21 Sep 2026 20:10:06 +0800https://akiyangdev.github.io/llm-inference-notes/articles/distributed/why-large-models-need-multi-card-tp-dp-ep/每个 Token 只激活部分专家，为什么仍需要放下整套权重？先算设备内存，再区分 TP、DP 和 EP 各自拆分什么。https://akiyangdev.github.io/llm-inference-notes/articles/distributed/why-large-models-need-multi-card-tp-dp-ep/Mon, 21 Sep 2026 20:01:10 +0800https://akiyangdev.github.io/llm-inference-notes/articles/fundamentals/sglang-decode-source-reading-primer/刚生成的 Token，下一轮怎样变成模型输入？用三条请求的小批次串起长度、KV 写入、Attention、MoE 和采样。https://akiyangdev.github.io/llm-inference-notes/articles/fundamentals/sglang-decode-source-reading-primer/Mon, 21 Sep 2026 19:43:45 +0800https://akiyangdev.github.io/llm-inference-notes/articles/fundamentals/prefill-vs-decode-performance/同一份权重，为什么处理长输入和逐枚生成的性能不同？从矩阵形状讲到权重复用、专家 Token 数和每轮固定开销。https://akiyangdev.github.io/llm-inference-notes/articles/fundamentals/prefill-vs-decode-performance/Mon, 21 Sep 2026 17:33:23 +0800https://akiyangdev.github.io/llm-inference-notes/articles/speculative-decoding/deepseek-v4-dspark-adaptive-verify-scheduler/验证预算有限时，每条请求都分配相同长度合适吗？解释 DSpark 的候选预算分配，以及 Ascend 固定版本仍使用 static 的原因。https://akiyangdev.github.io/llm-inference-notes/articles/speculative-decoding/deepseek-v4-dspark-adaptive-verify-scheduler/Mon, 21 Sep 2026 16:40:15 +0800https://akiyangdev.github.io/llm-inference-notes/articles/performance/deepseek-v4-w8a8-ascend-910c-performance/INT8 矩阵乘法省下的时间，能否覆盖动态量化等额外开销？结合小 M、专家批次和 NZ 布局分析该怎样测量。https://akiyangdev.github.io/llm-inference-notes/articles/performance/deepseek-v4-w8a8-ascend-910c-performance/Mon, 21 Sep 2026 15:58:22 +0800https://akiyangdev.github.io/llm-inference-notes/articles/ascend/deepseek-v4-w8a8-ascend-910c/W8A8 权重加载后，激活在哪里量化，矩阵乘法又交给谁？分别沿 Dense 与 MoE 追到 Ascend 算子入口。https://akiyangdev.github.io/llm-inference-notes/articles/ascend/deepseek-v4-w8a8-ascend-910c/Mon, 21 Sep 2026 15:12:48 +0800https://akiyangdev.github.io/llm-inference-notes/articles/distributed/sglang-collectives-deepseek-v4/该求和、拼接，还是把 Token 送给另一位专家？先看四类通信的小张量例子，再对照 SGLang 的 Attention 与 MoE。https://akiyangdev.github.io/llm-inference-notes/articles/distributed/sglang-collectives-deepseek-v4/Mon, 21 Sep 2026 13:00:33 +0800https://akiyangdev.github.io/llm-inference-notes/articles/distributed/sglang-rank-group-deepseek-v4/同一个进程为什么有好几个 Rank 编号？用 TP16 / DP8 展开它在 Attention、请求分组和 MoE 中的不同位置。https://akiyangdev.github.io/llm-inference-notes/articles/distributed/sglang-rank-group-deepseek-v4/Mon, 21 Sep 2026 11:51:13 +0800https://akiyangdev.github.io/llm-inference-notes/articles/performance/deepseek-v4-speculative-decoding-ascend-profiling/DSpark 慢在哪一段，怎样从计时字段追到 NPU 时间线？说明阶段边界、通信等待和 Profiling 对照方法。https://akiyangdev.github.io/llm-inference-notes/articles/performance/deepseek-v4-speculative-decoding-ascend-profiling/Mon, 21 Sep 2026 11:19:11 +0800https://akiyangdev.github.io/llm-inference-notes/articles/performance/deepseek-v4-speculative-ascend-performance-critical-path-analysis/开了 Graph 和多流，下一步为什么仍在等？沿 EAGLE/NextN 标出候选、缓存和通信依赖，再说明怎样与实测时间线对应。https://akiyangdev.github.io/llm-inference-notes/articles/performance/deepseek-v4-speculative-ascend-performance-critical-path-analysis/Mon, 21 Sep 2026 11:06:39 +0800https://akiyangdev.github.io/llm-inference-notes/articles/fundamentals/ai-infra-working-glossary/读源码时遇到陌生名词，可以按正在解决的问题查：输入表示、模型计算、缓存调度、多卡通信和 Ascend 执行。https://akiyangdev.github.io/llm-inference-notes/articles/fundamentals/ai-infra-working-glossary/Mon, 21 Sep 2026 10:51:43 +0800https://akiyangdev.github.io/llm-inference-notes/articles/performance/deepseek-v4-speculative-performance-analysis/接受长度变大，投机解码为什么仍可能变慢？把每轮有效输出与 Draft、Verify、通信和并发成本放在一起算。https://akiyangdev.github.io/llm-inference-notes/articles/performance/deepseek-v4-speculative-performance-analysis/Sun, 20 Sep 2026 23:14:47 +0800https://akiyangdev.github.io/llm-inference-notes/articles/speculative-decoding/sglang-speculative-decoding-scheduler-source-analysis/开启重叠执行后，CPU 记录和设备进度可能暂时不同。看请求长度、KV 预留和结果发布怎样保证下一轮接得上。https://akiyangdev.github.io/llm-inference-notes/articles/speculative-decoding/sglang-speculative-decoding-scheduler-source-analysis/Sun, 20 Sep 2026 23:10:35 +0800https://akiyangdev.github.io/llm-inference-notes/articles/speculative-decoding/deepseek-v4-accept-decision-source-analysis/Verify 算出了候选位置的预测，哪些 Token 最终能输出？区分几种接受规则，并解释行索引与新 bonus 的错位关系。https://akiyangdev.github.io/llm-inference-notes/articles/speculative-decoding/deepseek-v4-accept-decision-source-analysis/Sun, 20 Sep 2026 22:40:02 +0800https://akiyangdev.github.io/llm-inference-notes/articles/speculative-decoding/deepseek-v4-tree-attention-source-analysis/候选树排成 Tensor 后，一行怎样只读自己的祖先？用一棵小树解释位置和可见性；多分支部分属于通用 EAGLE 机制。https://akiyangdev.github.io/llm-inference-notes/articles/speculative-decoding/deepseek-v4-tree-attention-source-analysis/Sun, 20 Sep 2026 22:29:11 +0800https://akiyangdev.github.io/llm-inference-notes/articles/speculative-decoding/deepseek-v4-speculative-kv-commit-compact-rollback-source-analysis/候选算过了，不代表都能留在历史里。看接受路径怎样映射到正式 KV，以及被拒绝候选的空间何时整理、复用或释放。https://akiyangdev.github.io/llm-inference-notes/articles/speculative-decoding/deepseek-v4-speculative-kv-commit-compact-rollback-source-analysis/Sun, 20 Sep 2026 21:42:36 +0800https://akiyangdev.github.io/llm-inference-notes/articles/distributed/deepseek-v4-speculative-distributed-consistency-source-analysis/一张卡接受两枚候选，另一张接受三枚，会发生什么？区分模型分片通信与投机决策同步，追踪分歧怎样传到下一轮。https://akiyangdev.github.io/llm-inference-notes/articles/distributed/deepseek-v4-speculative-distributed-consistency-source-analysis/Sun, 20 Sep 2026 21:08:53 +0800https://akiyangdev.github.io/llm-inference-notes/articles/speculative-decoding/deepseek-v4-eagle-draft-tree-source-analysis/EAGLE 怎样把候选分支整理成一次 Verify 输入？跟着小树读索引和接受路径；DeepSeek-V4 固定版本仍限定 topk=1。https://akiyangdev.github.io/llm-inference-notes/articles/speculative-decoding/deepseek-v4-eagle-draft-tree-source-analysis/Sun, 20 Sep 2026 19:49:17 +0800https://akiyangdev.github.io/llm-inference-notes/articles/speculative-decoding/deepseek-v4-mtp-speculative-decoding-source-analysis/三枚 Draft 候选需要几次 NextN Forward？算清第一枚候选的来源、后续逐步预测和 Target 验证窗口的关系。https://akiyangdev.github.io/llm-inference-notes/articles/speculative-decoding/deepseek-v4-mtp-speculative-decoding-source-analysis/Sun, 20 Sep 2026 19:27:32 +0800https://akiyangdev.github.io/llm-inference-notes/articles/sglang/deepseek-v4-pd-disaggregation-source-analysis/P 端已经采样出首 Token，为什么它还没有自己的 KV？从这个交接点看 Decode 接管需要哪些缓存、资源和确认。https://akiyangdev.github.io/llm-inference-notes/articles/sglang/deepseek-v4-pd-disaggregation-source-analysis/Sun, 20 Sep 2026 19:06:22 +0800https://akiyangdev.github.io/llm-inference-notes/articles/speculative-decoding/deepseek-v4-speculative-decoding-source-analysis/Draft 猜 A/B/C，Target 接受 A/B 后生成 X，下一轮从哪里继续？沿一个例子走完 DSpark 的提议、验证、接受和提交。https://akiyangdev.github.io/llm-inference-notes/articles/speculative-decoding/deepseek-v4-speculative-decoding-source-analysis/Sun, 20 Sep 2026 18:52:38 +0800https://akiyangdev.github.io/llm-inference-notes/articles/distributed/deepseek-v4-dspark-parallel-layout-analysis/Target 能使用 TP32 / DP16，为什么 MoE Draft 启动时被拒绝？追踪 DSpark 的配置检查、Token 布局和决策同步。https://akiyangdev.github.io/llm-inference-notes/articles/distributed/deepseek-v4-dspark-parallel-layout-analysis/Sun, 20 Sep 2026 16:44:09 +0800https://akiyangdev.github.io/llm-inference-notes/articles/sglang/deepseek-v4-distributed-parallel-source-analysis/同一批进程在 Attention 和 MoE 中怎样改变分工？用 TP32 / DP16 的 Target 布局追踪 Token 行的转换与恢复。https://akiyangdev.github.io/llm-inference-notes/articles/sglang/deepseek-v4-distributed-parallel-source-analysis/Sun, 20 Sep 2026 16:32:31 +0800https://akiyangdev.github.io/llm-inference-notes/articles/sglang/deepseek-v4-moe-source-analysis/一个 Token 会被送到哪些专家，算完又怎样回到原位置？用四个 Rank、八个专家的例子解释路由、分发和合并。https://akiyangdev.github.io/llm-inference-notes/articles/sglang/deepseek-v4-moe-source-analysis/Sun, 20 Sep 2026 16:18:22 +0800https://akiyangdev.github.io/llm-inference-notes/articles/sglang/deepseek-v4-attention-source-analysis/多条请求打包进同一个 Tensor 后，Attention 怎样找到各自的历史？沿 Query、缓存写入和页表追踪一次 Decode。https://akiyangdev.github.io/llm-inference-notes/articles/sglang/deepseek-v4-attention-source-analysis/Sun, 20 Sep 2026 15:58:45 +0800https://akiyangdev.github.io/llm-inference-notes/articles/sglang/sglang-modelrunner-source-analysis/两条请求共有九个输入 Token，为什么最后只需两行 Logits？用这个例子读懂 ModelRunner 的输入、前向和采样交接。https://akiyangdev.github.io/llm-inference-notes/articles/sglang/sglang-modelrunner-source-analysis/Sun, 20 Sep 2026 15:45:51 +0800https://akiyangdev.github.io/llm-inference-notes/articles/sglang/sglang-ascend-request-lifecycle/一句聊天消息怎样变成 NPU 上的一轮计算？跟随一条请求，串起分词、排队、缓存、组批和持续生成。https://akiyangdev.github.io/llm-inference-notes/articles/sglang/sglang-ascend-request-lifecycle/Sun, 20 Sep 2026 15:38:39 +0900https://akiyangdev.github.io/llm-inference-notes/articles/sglang/sglang-scheduler-source-analysis/请求会持续很多轮，设备每轮只执行一个批次。看 Scheduler 怎样在前缀缓存、Token 预算和 KV 空间之间安排工作。https://akiyangdev.github.io/llm-inference-notes/articles/sglang/sglang-scheduler-source-analysis/Sun, 20 Sep 2026 15:32:54 +0800