Case study · Resilience architecture

Designing for AI Failure

RAG and BullMQ fallback architecture

Every AI feature fails eventually: the provider times out, rate-limits you, or just has a bad day. Most teams answer with a try/catch and an apology message. I built a three-tier fallback ladder instead — retry inside the queue, RAG fallback over the tenant’s own data, then a rule-based floor — so an AI outage never reaches the user as an error. Five AI-backed capabilities sit behind one Unified AI Service and inherit the same reliability pattern: one contract, one queue, one fallback ladder.

60%

cut in AI integration complexity from one routing layer over OpenAI, Gemini and Anthropic, with automatic failover

fallback tiers: queue retry, grounded RAG answer, rule-based floor

“The AI makes the system smart; the fallback ladder makes it something you can depend on during business hours, every day.”

01 · The problem

LLMs fail in production

Third-party LLM APIs hit unpredictable rate limits (HTTP 429), regional outages (HTTP 503), latency spikes of ten seconds or more, and strict token budgets. A platform that couples core operations directly to raw LLM calls stops the moment one upstream provider does, taking hiring pipelines and recruiter consoles with it.

02 · The approach

One service, one ladder

A single Unified AI Service, backed by idempotent BullMQ and Redis queues, runs every request down a strict three-tier ladder: automatic multi-provider retry and failover, then tenant-scoped RAG retrieval, then a deterministic rule-based floor. Users never see a 500 error.

Architecture

Recruiter console

ATS / HRMS

Careers portal

Partner APIs

↓ single provider contract · NestJS guard and ingestion controller

Unified AI Service

One entry point for every AI request, one provider contract and telemetry hub

↓ async offload

Async work queue · BullMQ + Redis

Idempotent, retry-safe workers with exponential backoff

↓ shared capabilities

Smart parsing

documents and OCR

Candidate scoring

explainable fit

Interview questions

role-aware

Hiring verdict

evidence-backed

Assistant

grounded chat

↓ every capability inherits the ladder

Reliability ladder

AI answer → confidence and latency check → RAG grounded fallback → rule-based floor

Grounded knowledge (RAG)

Vector retrieval over the tenant's own jobs and candidates

Model providers

Gemini · OpenAI · Anthropic, with runtime failover

Multi-tenant security: vaulted API secrets · tenant-scoped vector queries · audit logging · GDPR erasure cascade

Figure 1 · Every client talks to one service. Every heavy task runs on one queue.

Write path · indexing, async on the BullMQ worker pool

Read path · retrieval and grounded generation

If the model provider is down: the read path returns ranked snippets instead. No generation is needed, and answers stay grounded and cited.

Right to erasure. Deleting a candidate or tenant triggers an atomic cascade that purges the relational record, the MeiliSearch entry, raw S3/GCS payloads and every derived 768-dimension embedding.

Figure 2 · The read path degrades to grounded snippets, never an error.

Figure 3 · Retry stays inside the queue, RAG catches provider failure, rules catch empty retrieval.

Telemetry {{ latency }}

                      
                          {{ g.t }}
                          {{ g.level }}
                          {{ g.msg }}

Simulation · pick a failure scenario to watch the ladder step down.

Animated · LLM fallback chain — Retry → RAG fallback → Rule-Based Floor

Animated · multi-agent swarm — four agents, one BullMQ & Redis message bus

Design guarantees

Implementation

UnifiedAIService.ts · TypeScript / NestJS

export type LlmTier = 'retry' | 'rag' | 'floor';
export type Confidence = 'high' | 'medium' | 'low';

export type LlmResponse<T> =
  | { tier: 'retry'; degraded: false; confidence: 'high'; content: T;
      providerMeta: ProviderMeta; invariants: InvariantReport }
  | {
      tier: 'rag' | 'floor';
      degraded: true;
      confidence: 'medium' | 'low';
      readonly _content: T;
      providerMeta: ProviderMeta;
      invariants: InvariantReport;
      acceptDegraded(): T;
    };

/**
 * Degraded responses do not expose `.content` at all. The payload is only
 * returned after `acceptDegraded()` runs the invariant check and audit log.
 */
export function acceptDegraded<T>(res: LlmResponse<T>): T {
  if (res.degraded) {
    if (!res.invariants.passed) {
      throw new DegradedInvariantViolation(res.invariants.violated);
    }
    logger.warn('degraded_accept', {
      tier: res.tier,
      confidence: res.confidence,
      violations: res.invariants.violated,
    });
    return res._content;
  }

  return res.content;
}

// Gateway side — invariant validation runs AFTER a Tier-1 success.
const raw = await this.dispatchPrimaryLLM(req, tenantContext);
const invariants = this.invariantEngine.check(raw, req.domainRules);

if (!invariants.passed) {
  // Tier 1 "succeeded" but violated a domain rule. Downgrade.
  return this.ruleEngine.extractHeuristicData(req.rawDocument, 'floor', invariants);
}

return { tier: 'retry', degraded: false, confidence: 'high', content: raw, providerMeta, invariants };

Want the pattern on your platform?

Ready to build something resilient? Let’s connect ↗ See services Read part 2 →