Meta and the PyTorch team detailed how ExecuTorch now runs Muse Glimmer, an open-weight 30-billion-parameter model distilled for on-device agentic workflows, on NVIDIA GPUs and Apple silicon. The implementation exports directly from GGUF checkpoints, maps K-quant weights to backend-specific kernels using CUDA Triton or native Metal, and adds DFlash speculative decoding, CUDA-graph-captured decode steps, and support for 128K-plus token context with a mostly sliding-window KV cache. On an Apple M5 Pro, the setup reached 33.0 tokens per second with speculative decoding enabled, a 52.8% improvement over standard decoding with no quality loss, and the team has released prebuilt runtime artifacts for both platforms.
