The Ai2 team detailed the architecture behind Shippy, a maritime AI agent, which they split into a system-prompt-defined soul, markdown-file skills for individual tasks, and a runtime config layer. To keep the agent reliable in high-stakes operations, they built a deterministic CLI interface between the agent and their Skylight API so it cannot construct malformed API calls directly, and isolated each user session in its own Kubernetes sandbox with credentials injected at provisioning time. Rather than using standard benchmarks, the team built a custom evaluation framework in which subject-matter experts and an LLM judge score agent responses against live data using weighted criteria.
