Case study · Part 2 · Observability & contracts

The Gateway That Learned to Say No

Degradation as a first-class contract

A fallback ladder keeps the AI answering, but it can’t tell you whether the answer is right. This part makes degradation an explicit contract: every response carries its tier and confidence in a type callers can’t ignore, prompts and routing live in Redis so a rollback is a pointer flip instead of a redeploy, cost comes from our own price table, and a fourth tier validates domain invariants after every “successful” call.

<5s

mean time to roll back a prompt or routing policy — a pointer flip in Redis, never a redeploy

4th

orthogonal tier: invariant validation that runs after a successful LLM call, because success ≠ correctness

“Availability is what keeps the door open. The contract is what keeps it honest.”

01 · The problem part 1 didn't solve

Success is not correctness

The ladder in Part 1 ensures a response always comes back. It says nothing about whether that response is right. During a Gemini degradation window, a well-formed but semantically wrong answer passed every tier as success — caught only because a downstream payroll validator rejected the number. Prompt edits required a redeploy; cost was attributed by provider-reported usage that silently lied across retries. And callers could ignore the tier entirely.

02 · The approach

Make degradation a contract

Every response carries tier, degraded and confidence as a discriminated union — the type system forbids ignoring them. Prompts and routing policies live in Redis and reload via pub/sub. Cost is computed from our own price table, not the provider's. A fourth orthogonal tier — invariant validation — runs after every "successful" call, because the ladder protects availability and invariants protect correctness.

Architecture

Response envelope · discriminated union

                    tier:
                    'retry' | 'rag' | 'floor'
                  
                    degraded:
                    boolean
                    // tier !== 'retry'
                  
                    confidence:
                    'high' | 'medium' | 'low'
                  
                    providerMeta: {
                    provider, model }
                  
                    invariants: {
                    passed: boolean; violated: string[] }

If degraded is false

Caller treats it as a normal high-quality answer.

If degraded is true

Caller must call acceptDegraded() or the response throws at runtime.

Figure 1 · Degradation can't be silently ignored. The type system and runtime both refuse.

Config plane · prompts and routing as data, not code

Rollback path · flip a pointer, not a deploy

Mean time to rollback: under 5 seconds. Every response logs (promptVersion, routingPolicy, provider, tier), so any bad output traces to a specific config tuple.

Figure 2 · Prompts, routing and tenant overrides are all data, reloaded via Redis pub/sub.

The fourth tier · invariant validation, orthogonal to the ladder

Key insight: a Tier-1 "success" that violates a domain invariant is silently downgraded to Tier 3 and flagged for review. Availability and correctness are separate concerns, enforced separately.

Figure 3 · The ladder protects uptime. Invariants protect truth. They never share a code path.

Envelope trace {{ latency }}

                      
                          {{ g.t }}
                          {{ g.level }}
                          {{ g.msg }}

Simulation · pick a scenario to watch how the contract responds to each caller.

Animated · response envelope — Retry → RAG → Floor, with invariant gate

Animated · config plane — prompt & routing version rollback via Redis pub/sub

Design guarantees

Implementation

LlmResponse.ts · TypeScript / NestJS

export type LlmTier = 'retry' | 'rag' | 'floor';
export type Confidence = 'high' | 'medium' | 'low';

export type LlmResponse<T> =
  | { tier: 'retry'; degraded: false; confidence: 'high'; content: T;
      providerMeta: ProviderMeta; invariants: InvariantReport }
  | {
      tier: 'rag' | 'floor';
      degraded: true;
      confidence: 'medium' | 'low';
      readonly _content: T;
      providerMeta: ProviderMeta;
      invariants: InvariantReport;
      acceptDegraded(): T;
    };

/**
 * Degraded responses do not expose `.content` at all. The payload is only
 * returned after `acceptDegraded()` runs the invariant check and audit log.
 */
export function acceptDegraded<T>(res: LlmResponse<T>): T {
  if (res.degraded) {
    if (!res.invariants.passed) {
      throw new DegradedInvariantViolation(res.invariants.violated);
    }
    logger.warn('degraded_accept', {
      tier: res.tier,
      confidence: res.confidence,
      violations: res.invariants.violated,
    });
    return res._content;
  }

  return res.content;
}

// Gateway side — invariant validation runs AFTER a Tier-1 success.
const raw = await this.dispatchPrimaryLLM(req, tenantContext);
const invariants = this.invariantEngine.check(raw, req.domainRules);

if (!invariants.passed) {
  // Tier 1 "succeeded" but violated a domain rule. Downgrade.
  return this.ruleEngine.extractHeuristicData(req.rawDocument, 'floor', invariants);
}

return { tier: 'retry', degraded: false, confidence: 'high', content: raw, providerMeta, invariants };

Want the contract on your platform?

Contact me on Gravatar ↗ Read part 1