This technical report examines automated LLM evaluation systems, arguing the field has shifted from reference-based metrics like BLEU and ROUGE toward model-based LLM-as-judge approaches, and presents a layered taxonomy spanning output-quality dimensions, reference-based and reference-free scoring, and domain-specific evaluators for RAG, agents, code generation, and multimodal outputs. It proposes a six-layer implementation architecture: OpenTelemetry-instrumented tracing, versioned golden datasets from production traces, calibrated scorer abstractions mixing deterministic assertions with human-validated LLM judges, CI/CD-integrated eval runners reporting confidence intervals, inference-time safety guardrails, and continuous production evaluation with error-analysis feedback loops. The author advocates task-specific evaluators grounded in systematic error analysis and statistical rigor, including pass^k reliability metrics for agents, over generic off-the-shelf metrics.
