PyTorch, vLLM, SGLang, Megatron-LM and more — open pull requests
Beyond the merged work above, I have open pull requests across the rest of the AI infrastructure stack. Each link goes to the upstream discussion; several carry a minimal reproducer, a measurement, and a regression test.
- PyTorch — #196787: fix NaN in higher-order gradients of
logaddexp/logaddexp2. The backward weight was written as1 / (1 + exp(other - self)), which overflows toinfonce the difference exceedslog(DBL_MAX), so the expression autograd differentiates for a double backward evaluates toinf / inf = nanfor finite inputs. The formula now usessigmoid, which is bounded. Fixes #196704. - vLLM — #56198: reject
int4_per_token_headfor non-power-of-two head sizes. The mode’s write path asserts a power-of-two dimension, but the option was accepted for every head size, so a model with head size 96 selected TRITON_ATTN and then aborted inside the engine-core process with an opaque message. Fixes #56197. - SGLang — #39315: throttle the fully-idle load snapshot publish. With
--sleep-on-idleoff, an idle scheduler republished an unchanged load snapshot on every pass: 10,470 publishes/s and ~0.36 CPU cores on a single-H200 server, measured before and after. - Megatron-LM — #7287: describe prepended-axis shards with the checkpointable path. The legacy PyTorch conversion enumerates the entire global shard grid — remote shards included — for every key with a prepended axis, so its cost scaled with total fragmentations rather than locally held shards. Closes #7275.
- TensorRT-LLM — #18904: make
BlockRadixTree.clear()terminate on a childless root.clear()shrank its queue only as a side effect of detaching a child, so aRootBlockthat was already childless made the outer loop re-select the same root forever. Closes #17908. - CUTLASS — #3613: skip cuBLAS verification when C and D element types differ.
cutlass_profileraborted the entire profiling session with SIGABRT when it reached an SM90 GEMM writing narrow FP8 output from an FP32C, which kills a full kernel sweep at the first such kernel. - LMCache — #5117: reject non-positive sizes in
AddressManager, where a zero-byte request corrupted the explicit free list — inserting an uncoalescable zero-sized entry whileget_free_size()kept reporting the heap as free — andbatched_allocate(0, n)raisedZeroDivisionError; and #5125: enforce lazy%-format logging (ruff G004), which stops new f-string logging from landing in already-migrated files and ends the stream of one-file migration pull requests. - Miles (ROCm / FSDP) — #3266 non-scalar rewards aborting the on-policy-distillation rollout in the zero-std metrics, #3267 resolving the SGLang Triton attention kernel across layouts after its module relocation, #3268 applying the Qwen3 precision contract in the AMD true-on-policy example (measured on MI300X / gfx942), and #3269 a verify step that spoke only CUDA while the hardware table listed AMD.
- Cordis — #148: apply updates issued while a fiber is loading. A restart issued during load toggled the dependency epoch through INACTIVE and back, so the in-flight reload cleared inertia without applying the new configuration; a generation counter fixes the lost update. Fixes #34.
Bug reports
I file issues the way I would want to receive them: a minimal reproducer, the mechanism, and where I stop short of knowing. Thirteen so far, including DeepSpeed #8262 and #8263 — the deadlock and the partial policy injection fixed by the merged pull requests above — plus verl #7624, vLLM #56197, Transformers #48670 (ASR feature extractors do not convert batched multi-channel input to mono), and Miles #3265, #3270, #3271.