Engineering
Netflix Details Its In-House LLM Serving Platform With Triton and vLLM
Netflix built an in-house LLM serving platform combining Triton and vLLM to handle inference across CPUs and GPUs within its…
Engineering
The State of Simulation for Physical AI: An Overview
As collecting real-world robotics data remains expensive and slow, developers increasingly rely on GPU-accelerated simulators to generate synthetic training data…
Engineering
An Inside Look at the Relay Market Powering Token Resellers and Fraud
An investigation detailed a marketplace, operating mainly out of China, for reselling discounted LLM API tokens using open-source proxy software…
Engineering
Hugging Face on AMD Instinct MI455X: First Transformers Results
Hugging Face reports successful testing of AMD’s new Instinct MI455X GPU, which offers 432 GB of high-bandwidth memory, more than…
Engineering
Beyond a Single Model: Building Mixture-of-Models Systems with vLLM Semantic Router
The vLLM Semantic Router project outlines its evolution from routing requests between fast and reasoning model paths toward coordinating full…
Engineering
Keeping vLLM Production Quality
vLLM describes the testing infrastructure it uses to maintain production quality, including a continuous integration system running 266 jobs across…
Engineering
TML Inkling on vLLM: Day-0 Support with Optimized Performance
vLLM announced day-0 support for Thinking Machines Lab’s Inkling, a large multimodal mixture-of-experts model that accepts text, image, and audio…
Engineering
How We Benchmark Deep Agents
LangChain explains how it evaluates its open-source Deep Agents framework using Harbor, an evaluation runner that requires specifying an agent,…
Engineering
Towards Automating Eval Engineering
LangChain introduces an Eval Engineering Skill that helps coding agents automatically build evaluations by analyzing repository context and production traces.…
Engineering
Building Governed Agents: A Framework for Cost, Control, and Compliance
LangChain outlines a governance framework for enterprises deploying AI agents, arguing that LLM gateways should act as a runtime control…
Engineering
Towards Demystifying the Creativity of Diffusion Models
Google Research explains how diffusion models generate novel images instead of memorizing training data. The research shows that this “creativity”…
Engineering
Harness Engineering for Self-Improvement
In this post, researcher Lilian Weng examines how harness engineering, the systems that orchestrate model execution, tool use, and workflow…