Independent release evidence

Agentic systems,
tested as systems.

Run reproducible black-box checks for contracts, evidence, approval gates, live boundaries, idempotency, and latency.

01 / LIVE EVALUATION

Choose a deployed system

Ready. The lab calls only the three allowlisted public deployments.

Technical case study 04Published by Ahmad Bukhari. Built and verified by AiXCEL Solutions.

03 / SCENARIO BRIEF

A live evaluator that proves it can fail.

Agent demos often validate only the happy path. This lab calls three allowlisted public systems, verifies seven contract and governance controls, repeats requests for idempotency, probes forbidden live mode, measures latency, and exposes evidence for every score.

03allowlisted deployments
07weighted checks
04baseline and fault scenarios
00user-supplied outbound URLs
Control failure

A permanent score of 100 proves nothing.

The original interface only showed a passing baseline. Missing evidence, approval bypass, and latency breach can now be injected explicitly and each lowers the score.

Security boundary

The evaluator cannot fetch arbitrary URLs.

Target names resolve through a code allowlist. This prevents the public service from becoming a server-side request-forgery tool.

Acceptance boundary

Public checks are not production certification.

Privacy, provider quality, semantic task success, and real-data mutations require authenticated project-specific evaluation.

04 / ARCHITECTURE

Probe, compare, score, and gate.

Evaluation Lab system context
System contextThe evaluator only reaches three fixed portfolio deployments.
Evaluation Lab check flow
Evaluation flowHealth, two replay runs, live boundary, weighted controls, and optional fault.
Evaluation Lab runtime and observability
Release gateCI, preview, live black-box checks, trace evidence, and artifact promotion.

05 / DECISIONS

Binary controls stay deterministic.

Black-box HTTP over unit tests alone

The public URL, headers, latency, and boundary behavior must work outside the repository.

Rules over LLM-as-judge

An approval bypass or missing evidence has one repeatable result. A model judge is reserved for calibrated qualitative output.

Hard allowlist over submitted URLs

The public evaluator has no reason to fetch arbitrary hosts.

Two replay calls over one

Idempotency is observable only when the same key is used more than once.

Standard library HTTP over another dependency

The bounded request surface is small enough for the platform runtime.

Vercel over queues and Kubernetes

Current checks finish synchronously. Scheduled suites and long model graders are the trigger for a queue.

06 / EVALUATION

Seven weighted checks

  • Availability
  • Typed run contract
  • Resolvable evidence
  • Human decision gate
  • Live-provider boundary
  • Idempotent replay
  • Bounded trace and latency

07 / OBSERVABILITY

Current monitoring path

  • X-Trace-ID and JSON logs
  • Per-probe latency and evidence
  • Vercel runtime logs
  • GitHub Actions regression tests
  • Score, grade, and fault label in the response

08 / NEXT GATE

Production evaluation layer

  • Durable run history and schedules
  • OpenTelemetry trace correlation
  • Langfuse or LangSmith qualitative suites
  • Sentry application errors
  • Human-labelled golden sets and drift alerts
Trace tagsserviceenvironmenttrace_idtarget_systemevaluation_scenariocheckstatuslatency_msscorefault_injectederror_type
Inspect repository ↗Open API contracts ↗