Sphera AI Platform
PoC → Production Blueprint

• Company: Sphera — enterprise ESG, sustainability & operational-risk software (data-heavy, compliance-sensitive).
• Platform: Microsoft Azure is the approved cloud. AI PoC is built but cannot move to production.
• Current components: Claude (Anthropic), nginx, AI-DLC (AI-Driven Development Life Cycle), Hermes agents, Git, RAG, Pendo, Juno*, Jina (embeddings/rerankers), Databricks, MS Teams.
• Symptom: "Currently having code issue" — PoC code does not meet production bar (see Section 04 triage).
• *Juno assumed to be an internal AI assistant / orchestration tool — confirm vendor & role in the Assumptions Register (§02).

• Symptom: API keys in .py/.env committed to Git; keys rotate and break the PoC.
• Root cause: Fast PoC iteration, no secret hygiene.
• Fix: Azure Key Vault + Managed Identity (DefaultAzureCredential) — zero keys in code.

# ❌ PoC: key in code / .env
ANTHROPIC_API_KEY = "sk-ant-..."

# ✅ Prod: identity-based, no secret in code
from azure.identity import DefaultAzureCredential
from azure.keyvault.secrets import SecretClient

cred = DefaultAzureCredential()   # Managed Identity in prod, dev login locally
client = SecretClient(vault_url="https://sphera-ai.vault.azure.net/", credential=cred)
ANTHROPIC_API_KEY = client.get_secret("anthropic-api-key").value

• Symptom: Random 429/5xx from Anthropic under load; jobs die mid-run; users see errors.
• Root cause: Raw client.messages.create() with no retry, no backoff, no fallback model.
• Fix: Retry with exponential backoff + jitter, circuit breaker, and a model/route fallback chain.

from tenacity import retry, stop_after_attempt, wait_exponential_jitter, retry_if_exception_type
import httpx

@retry(
    stop=stop_after_attempt(5),
    wait=wait_exponential_jitter(initial=1, max=60),
    retry=retry_if_exception_type((httpx.HTTPStatusError, ConnectionError)),
    reraise=True,
)
def call_llm(model: str, messages: list[dict]) -> str:
    # ... invoke via AI Foundry gateway, fallback to Anthropic direct on timeout
    return response

# Circuit breaker: if error_rate > 10% in 60s → route to fallback model for 5 min

• Symptom: One giant notebook or main.py; nobody can test it; imports are a tangle.
• Root cause: PoC written as a linear experiment, never refactored.
• Fix: Package by layer: ingest/ · retrieval/ · llm/ · agents/ · api/ · eval/, typed interfaces, no business logic in notebooks.

sphera-ai/
├── pyproject.toml            # pinned deps (uv/poetry)
├── src/sphera_ai/
│   ├── config.py             # pydantic-settings (env-driven, validated)
│   ├── retrieval/            # chunking, embeddings, search
│   ├── llm/                  # gateway client, prompt templates, fallback
│   ├── agents/               # Hermes/tool orchestration, MCP registry
│   ├── api/                  # FastAPI routes (thin)
│   └── eval/                 # golden sets, scorers
└── tests/                    # unit + integration + eval (run in CI)

• Symptom: LLM returns malformed JSON → KeyError crashes; garbage in → hallucinated garbage out.
• Root cause: Free-form json.loads() on model output.
• Fix: Structured output (tool-use / Pydantic) + retry-on-parse-failure.

from pydantic import BaseModel, Field
from typing import Literal

class TradeSignal(BaseModel):
    action: Literal["buy", "sell", "hold"]
    confidence: float = Field(ge=0.0, le=1.0)
    reasoning: str

# model called with response_format / tool schema → parse with .model_validate()
# on ValidationError → one retry with "fix your output" prompt, else degrade to hold

• Symptom: RAG documents or user fields containing "ignore previous instructions"; model exfiltrates data.
• Root cause: No isolation between instructions, data, and user input; no output filtering.
• Fix: Delimiters + instruction hierarchy, input sanitization, output guardrails (PII/secret regex), least-privilege tool access, human approval for high-impact tool calls.

• Symptom: "It worked yesterday" — no logs, no traces, no token/cost accounting.
• Root cause: print() debugging in a serverless world.
• Fix: Structured logging + OpenTelemetry spans per request: prompt hash, model, tokens, latency, cost, retrieval ids.

import logging, time, opentelemetry.trace as trace

tracer = trace.get_tracer("sphera-ai")
logger = logging.getLogger("sphera_ai.llm")

with tracer.start_as_current_span("llm.call") as span:
    t0 = time.perf_counter()
    resp = call_llm(model, messages)
    span.set_attributes({
        "llm.model": model, "llm.prompt_hash": sha256(str(messages)),
        "llm.tokens_in": resp.usage.input_tokens, "llm.tokens_out": resp.usage.output_tokens,
        "llm.latency_ms": int((time.perf_counter()-t0)*1000),
        "llm.cost_usd": estimate_cost(model, resp.usage),
    })
logger.info("llm_call", extra={"model": model, "tokens": resp.usage})

• Symptom: Results change between runs; can't roll back a bad prompt; "which model was this?"
• Root cause: Prompts inline in code, model names hardcoded, no registry.
• Fix: Prompt templates as versioned artifacts in Git (or a prompt registry); model versions pinned in config; prompt/model deploy = release with rollback.

# config: pinned, environment-specific
model:
  primary: "claude-sonnet-4-6@2026-06-01"   # pinned via AI Foundry deployment
  fallback: "claude-haiku-4-5@2026-06-01"
  embedding: "jina-embeddings-v3@latest-validated"

prompt_version: "summary-v2-2026-08-01"

• Symptom: Every change is a gamble; regressions discovered by users.
• Root cause: "Evals are hard" → skipped.
• Fix: Golden-set evals in CI (50–200 curated Q/A with rubric scorers), LLM-as-judge with guardrail, regression gates on answer quality, latency, cost, hallucination rate. Start small; grow the set with Pendo-flagged bad answers.

• Symptom: Bill spikes; slow responses; users complain.
• Root cause: Huge contexts, no caching, no model tiering, no budgets.
• Fix: Prompt caching (Anthropic + AI Foundry support), model tiering by task (haiku for classification, sonnet for generation), token caps, response caching for identical queries, Azure budget alerts at 50/80/100%.

• Symptom: Anybody with the URL can query the API; internal data exposed.
• Root cause: PoC had no identity layer.
• Fix: Entra ID SSO (OAuth2 PKCE for SPA), APIM validates JWT, role-based authorization per capability (viewer/analyst/admin), per-tenant data isolation checks.

• ☐ Data residency — region pin (e.g., EU/US) enforced at storage & inference
• ☐ DPA with Anthropic (and Azure) covering sub-processors
• ☐ Zero-retention or short-retention model settings where applicable
• ☐ Audit log: who asked what, which model answered, what was retrieved
• ☐ Model cards & system cards for each deployed use case (Responsible AI)
• ☐ Incident response runbook for prompt-injection / data-exposure events
• ☐ Pen test + threat model before GA; quarterly red team of the RAG path

• Latency: per-model, per-feature; alert on P95 breach.
• Quality: online evals (LLM-as-judge on sampled traffic) + offline golden set each deploy.
• Cost: per use case, per tenant; budgets with 50/80/100% alerts; token caps per key.
• Drift: input distribution, retrieval hit-rate, "I don't know" rate, hallucination proxy (groundedness score).
• Feedback: Pendo thumbs up/down + comments → weekly triage → eval set growth → model/prompt improvement (AIDLC loop).

• Adaptive Cards for human-in-the-loop approvals (e.g., agent-proposed actions, sensitive queries).
• Bot notifications: eval regressions, cost spikes, drift alarms, weekly AI report.
• Prompt-injection or PII incident → Teams channel with severity + runbook link.
• Users ask in Teams → same governed pipeline (APIM → agents → AI Foundry). One platform, one audit trail.

1. Detect: error-rate / cost / groundedness alert fires (App Insights → Teams).
2. Triage: trace lookup by request id; classify (model, prompt, retrieval, infra).
3. Mitigate: feature flag off / rollback to previous prompt+model release (registry = instant).
4. Post-mortem: add regression to golden set; AIDLC gate re-run before re-promotion.

The printable quick reference (also standalone at /sphera-cheatsheet — download PDF, or use the print button):