We are pleased to announce the release of zml/llmd 20261112.0. This week, we are bringing DSpine, more speculative decoding options, faster DeepSeek prefill, Apple M1 and M2 support, and more control over sampling.

DSpine speculative decoding

This release brings support for Tencent’s DSpine drafters:

• Qwen3.8-27B-DSpine-b16, paired with Qwen3.8-27B
• Qwen3-8B-DSpine-b16, paired with Qwen3-8B

DSpine extends DFlash by passing information between adjacent draft positions as it refines its proposals. You can use it through the same --speculative-drafter-model flag:

llmd --model=hf://Qwen/Qwen3.8-27B \
    --speculative-drafter-model=hf://tencent/Qwen3.8-27B-DSpine-b16 \
    --batch-size=1

More speculative decoding support

Last week, we brought Ripple to zml/llmd. This week, Ripple gets support for DFlash, DFlash2, DSpark and DSpine through the shared speculative decoding implementation.

We also added support for the DSpark Qwen3-4B block7 checkpoint family, which we tested on GH200.

We also added --speculative-synthetic-acceptance-rates for performance experiments. It lets you supply a comma-separated list of acceptance probabilities, one per draft position, with values between 0 and 1. For example, --speculative-synthetic-acceptance-rates=0.95,0.85,0.65,0.35 works with four draft positions. This lets us measure how speculative decoding performs at different acceptance rates without depending on the particular prompt or drafter’s accuracy. It replaces normal draft verification with synthetic acceptance, so it is a benchmarking tool, not a setting for generating correct model output.

Sampling controls

We now resolve sampling parameters for each sequence. Request parameters take priority over CLI settings, followed by model recommendations where available, and finally zml/llmd’s defaults.

You can set --temperature, --top-k, --top-p and --min-p on the command line. Chat completion requests also accept top_p, alongside temperature, so individual requests can override those server settings.

llmd --model=hf://Qwen/Qwen3.8-27B \
    --temperature=0.7 --top-k=40 --top-p=0.9 --min-p=0.05

The resolved parameters travel with each sequence, including when requests with different sampling settings share a batch.

Faster DeepSeek prefill

The hill climb continues. DeepSeek-V4.1 now uses a fused MegaMoE kernel for supported CuTe MXFP4 configurations. It pipelines the expert computations and combines the routed and shared expert contributions inside the kernel.

For compiled token batches above 1,024 tokens, we select it automatically. Smaller batches keep the existing CuTe path, which performed better at those sizes.

Measured on 2×GB300, time to first token improved by roughly 12% on long prompts. The X axis shows prompt length in tokens; the Y axis shows time to first token in milliseconds:

Apple M1 and M2 support

Fixed a Metal attention kernel launch configuration that broke decode on M1 and M2. The paged attention kernel could launch up to 512 threads per threadgroup, while those GPUs accept at most 384.

The kernel now uses at most 256 threads per threadgroup. We also split the work according to the GPU core count and the number of rows, so it can spread across the GPU without exceeding the threadgroup limit.

Models with attention head dimensions of 32 and 64 now use the vector decode kernel as well. The fix addresses the M1/M2 launch limit.

Expect slightly more performance in general.

Under the hood

ZML now exposes Exe.memoryStats(), which returns the compiled executable’s memory statistics through PJRT. This gives us another way to inspect the memory requirements of a compiled model.

We also made the mesh and input/output partitioning explicit at the shared MoE boundary, so sharding choices are supplied by the caller rather than inferred inside each backend.

Fixes and improvements

A few more fixes made it into this release:

• Tokenizer loading now includes missing added tokens from tokenizer_config.json.
• The text decode buffer is sized from the tokenizer, fixing cases with unusually long tokens in models such as Laguna, GLM 5.3 and Mistral Large 4.
• Corrected Llama’s memory requirements when asynchronous scheduling is enabled.
• Fixed a Llama crash when speculative decoding is disabled.
• Fixed the Triton MLA path and improved automatic inline attention backend selection.
• Metal now enables GPU memory preallocation in zml/llmd, matching the intended allocator configuration, fixing a bug where --gpu-memory-fraction=0.5 was needed for get the right performance.
• Fixed slot indexing in the StableHLO paged attention backend for sequences that are not aligned to 16 tokens.
• ZML’s direct-memory writer now handles all input slices and repeated fill patterns correctly, including a case where a partially written fill could loop indefinitely.

Try it today

NVIDIA

docker run -p 8000:8000 --shm-size=256GB --gpus=all -e HF_TOKEN -it zmlai/llmd:cuda \
    --model=hf://Qwen/Qwen3.8-27B \
    --speculative-drafter-model=hf://tencent/Qwen3.8-27B-DSpine-b16 --batch-size=1

AMD

docker run -p 8000:8000 --shm-size=256GB --device=/dev/kfd --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:rocm \
    --model=hf://Qwen/Qwen3.8-27B \
    --speculative-drafter-model=hf://tencent/Qwen3.8-27B-DSpine-b16 --batch-size=1

Intel

docker run -p 8000:8000 --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:oneapi \
    --model=hf://Qwen/Qwen3.8-27B

Google TPU

docker run --net=host --privileged -e HF_TOKEN -it zmlai/llmd:tpu \
    --model=hf://Qwen/Qwen3.8-27B

Apple Metal

brew install zml/zml/llmd
llmd --model=hf://Qwen/Qwen3.8-27B-FP8 --batch-size=1