Correct memories are not always transferable.

Documentation Map

Use this page as task-oriented library documentation. The default path is local and dependency-light; optional OpenAI and TiDB paths are called out where they apply.

Quickstart

pip install handovergap==1.0.1

handovergap demo

handovergap evaluate --compare

Core Concepts

HandoverGap is a gate that runs before a RAG answer, agent action, or operational decision. It checks whether the selected profile has enough context to safely use a retrieved memory.

CLI Reference

handovergap detect --scenario S001 --profile CS

handovergap evaluate --dataset holdout --stress-filling

handovergap retrieve-evidence \
  --scenario S001 \
  --profile CS \
  --slot communication_status \
  --mode hybrid

handovergap report --dataset all \
  --output reports/evaluation-latest.md

Command Index

Profile YAML Reference

Built-in profiles include CS, Engineer, and Sales. They are examples, not a fixed industry taxonomy. Use a YAML profile file when your team needs a different gate, such as incident response, legal review, renewal handoff, or an autonomous agent action check.

profiles:
  IncidentCommander:
    required_slots:
      - slot_name: blast_radius
        gap_type: blast_radius_gap
        description: Impacted users, systems, and time window are not explicit.
        question: Which users, systems, and time window are impacted?
        severity: HIGH
        high_risk: true
        retrieval_hints:
          preferred_source_types:
            - incident_note
            - status_update
            - monitoring_dashboard
          search_terms:
            - blast radius
            - impacted users
            - affected systems
      - slot_name: rollback_owner
        gap_type: rollback_owner_gap
        description: The person or team authorized to trigger rollback is not explicit.
        question: Who is authorized to trigger rollback?
        severity: HIGH
        high_risk: true
      - slot_name: customer_message
        gap_type: customer_message_gap
        description: Customer-safe incident wording is not available.
        question: What customer-facing message should be used?
        severity: MEDIUM

handovergap profiles validate \
  examples/profiles/incident_readiness.yml

handovergap detect \
  --scenario S001 \
  --profile IncidentCommander \
  --profile-file examples/profiles/incident_readiness.yml

What It Adds To RAG

Local And Live Demo

pip install "handovergap[demo]"
handovergap serve

pip install "handovergap[live]"
export OPENAI_API_KEY="..."
export TIDB_HOST="..."
export TIDB_USER="..."
export TIDB_PASSWORD="..."
handovergap serve

TiDB Audit Store

TiDB is used as more than a vector store. HandoverGap stores the decision path so a blocked answer can be explained across SQL rows, vector-backed evidence, JSON retrieval metadata, gaps, and questions.

handovergap audit-sql

The generated SQL joins transfer_assessments, memory_items, context_gaps, slot_fill_attempts, source_events, and clarification_questions. It answers: which required slot was missing, what evidence was checked, and what should be asked before the answer is used.

This is a 10-iteration TiDB Cloud validation result for the audit query path, not a load-test claim. The p95 includes cold or variable cloud latency.

Evaluation Guide

These are deterministic results from the bundled synthetic 20-scenario mini dataset. They demonstrate reproducible behavior, not production accuracy.

Live gpt-5-mini with the tuned gpt5_strict prompt reached tacit gap recall 1.00, unsafe transfer prevention 1.00, safe transfer allowance 1.00, and blocked precision 1.00 on the synthetic holdout protocol. This is model- and prompt-specific evidence, not a general accuracy guarantee. See results notes.

The adversarial split keeps recall low at 0.38 but reduces false clarification to 0.00 after evidence-backed slot reconciliation. The sanitized split uses field-realistic anonymized CRM, incident, runbook, and deal-review style notes without real company data.

Follow-up Retrieval Metrics

handovergap evaluate --retrieval-mode expand-before-ask measures whether a bounded second retrieval pass reduces missing slots or user-facing questions before asking a person. The default safety_policy=strict keeps unsafe_answer_rate as the guardrail by requiring evidence support for high-risk slots before answering.

LLM Judge Rubric

LLM-as-a-judge is supported as an evaluation aid, not as runtime detection. Inspect the packaged rubric with handovergap judge-rubric; it forbids gold gaps, unsafe labels, expected strings, and scenario-specific hints as judge inputs.

Python API

from handovergap import TransferabilityGate

gate = TransferabilityGate()
result = gate.check(
    memory="Use CSV for this release; API support is deferred.",
    profile="CS",
    task_context="Answer customer questions about the workaround.",
    evidence=["CSV workaround approved for the release."],
    provided_slots=["scope"],
    evidence_slots=["scope"],
)

print(result.transferability_status)
print(result.gaps)
print(result.questions)

from handovergap import TransferabilityGate

gate = TransferabilityGate.from_profile_file(
    "examples/profiles/incident_readiness.yml"
)
result = gate.check(
    memory="The checkout incident is mitigated by disabling the new queue worker.",
    profile="IncidentCommander",
    task_context="Decide whether customer-facing mitigation is complete.",
    evidence=["Rollback owner is on-call SRE. Blast radius is checkout only."],
    provided_slots=["rollback_owner"],
    evidence_slots=["blast_radius"],
)

for question in result.questions:
    print(question.question)

pip install "handovergap[tidb]"
handovergap schema --dialect tidb

Stable API Contract

The core API does not require OpenAI or TiDB. Those integrations are optional paths for semantic slot filling and audit persistence.

Evidence-To-Slot Mapping

Raw retrieved evidence is not automatically treated as complete context. Integrations should pass provided_slots for context explicit in the memory and evidence_slots for context supported by retrieved evidence.

from handovergap import TransferabilityGate, map_evidence_slots_by_keywords

evidence = [
    "Customer notice was sent on Monday with approved wording.",
    "Support can answer standard questions, but must not promise API dates.",
    "If CSV import fails, use the manual upload fallback and escalate in the support channel.",
]
slot_keywords = {
    "communication_status": ["notice was sent", "customer notice"],
    "authority": ["can answer", "must not promise"],
    "fallback_plan": ["fallback", "manual upload"],
    "escalation_path": ["escalate", "support channel"],
    "customer_facing_wording": ["approved wording"],
}

result = TransferabilityGate().check(
    memory="Use CSV for this release; API support is deferred.",
    profile="CS",
    task_context="Answer customer questions without overpromising.",
    evidence=evidence,
    provided_slots=["scope"],
    evidence_slots=map_evidence_slots_by_keywords(evidence, slot_keywords),
)

Manual review, deterministic rules, and optional LLM slot filling can all feed the same evidence_slots contract. The core API does not require an LLM.

Slot Filling Modes

HandoverGap is a slot-based readiness gate, not an LLM-only extractor. Use reviewed slots, deterministic rules, or optional LLM extraction depending on the trust boundary of your workflow.

handovergap evaluate --compare --slot-fill-mode user_provided
handovergap evaluate --slot-fill-mode optional_llm --model gpt-example --prompt-profile strict

User Dataset Evaluation

Bundled datasets are fictional. For adoption work, keep your anonymized operational data local, export an annotation template, import reviewed labels, then evaluate and report from that local file.

handovergap datasets export-template ./local/anonymized_scenarios.jsonl --output ./local/labels.csv
handovergap datasets import-labels ./local/anonymized_scenarios.jsonl \
  --labels ./local/labels.csv \
  --output ./local/reviewed_scenarios.jsonl
handovergap evaluate --dataset-file ./local/reviewed_scenarios.jsonl --compare
handovergap report --dataset-file ./local/reviewed_scenarios.jsonl --output ./local/evaluation.md

The annotation template includes ids, profile, task context, required slots, and label columns. It does not copy raw memory or evidence text into the CSV.

RAG Integration Pattern

Place HandoverGap after retrieval and before answer generation. Let your normal retriever collect candidate memory and evidence, then let the gate decide whether to answer, ask, or block.

== first retrieval ==
provided_slots=scope
evidence_slots=communication_status,authority,customer_facing_wording
status=blocked action=block
gaps=fallback_plan,escalation_path
questions:
- 想定外の場合の代替手段は何ですか？
- 問題が起きた場合のエスカレーション先は誰ですか？
safe_context=withheld

== after retrieving runbook evidence ==
provided_slots=scope
evidence_slots=communication_status,authority,fallback_plan,escalation_path,customer_facing_wording
status=transferable action=answer
gaps=none
questions=none
safe_context=available

OpenAI slot filling can replace deterministic keyword mapping, and TiDB can persist the route audit trail. Neither is required for the core runtime.

python examples/end_to_end_integration.py

python examples/langchain_gate.py

python examples/llamaindex_gate.py

Full copyable recipes are in docs/30_rag_integration_recipes.md.

from handovergap import TransferabilityGate

gate = TransferabilityGate()

retrieved_memory = retriever.search(user_question)
evidence = retriever.search_evidence(user_question)

result = gate.check(
    memory=retrieved_memory.text,
    profile="CS",
    task_context="Answer the customer's question without overpromising.",
    evidence=[item.text for item in evidence],
    provided_slots=retrieved_memory.slots,
    evidence_slots=[slot for item in evidence for slot in item.slots],
)

if result.transferability_status != "transferable":
    return {
        "status": result.transferability_status,
        "reason": [gap.description for gap in result.gaps],
        "questions": [q.question for q in result.questions],
    }

return llm.answer(user_question, context=retrieved_memory.text)

Product Routing Semantics

Retrieval Modes

Safety Policies

Deployment Modes

from handovergap import TransferabilityGate, route_transferability_result

result = TransferabilityGate().check(
    memory="Use CSV for this release; API support is deferred.",
    profile="CS",
    task_context="Answer customer questions without overpromising.",
    provided_slots=["scope"],
)

route = route_transferability_result(
    result,
    safe_context=result.memory,
    deployment_mode="shadow",
    retrieval_mode="expand_before_ask",
    safety_policy="strict",
)

return {
    "status": route.status,
    "action": route.action,
    "recommended_action": route.recommended_action,
    "deployment_mode": route.deployment_mode,
    "retrieval_mode": route.retrieval_mode,
    "safety_policy": route.safety_policy,
    "enforcement": route.enforcement,
    "should_interrupt": route.should_interrupt,
    "next_step": route.next_step,
    "retrieval_queries": [query.model_dump() for query in route.retrieval_queries],
    "reason": route.reason,
    "questions": route.questions,
    "safe_context": route.safe_context,
}

In hard mode, safe_context is only returned for transferable results. In shadow and soft modes, the original context can continue while the recommended interruption is logged. With expand_before_ask, missing slots produce retrieve_more and run_followup_retrieval before user-facing clarification. Retrieval queries include slot-level source preferences and search terms when the profile defines retrieval_hints. With default strict safety, high-risk slots such as authority, fallback plan, escalation path, and contract impact must be evidence-backed before answering.

handovergap detect --scenario S001 --profile CS --deployment-mode shadow
handovergap detect --scenario S001 --profile CS --deployment-mode soft
handovergap detect --scenario S001 --profile CS --deployment-mode hard
handovergap detect --scenario S001 --profile CS --retrieval-mode expand-before-ask
handovergap detect --scenario S001 --profile CS --safety-policy balanced

handovergap evaluate --retrieval-mode expand-before-ask
handovergap evaluate --retrieval-mode expand-before-ask --safety-policy balanced
handovergap judge-rubric

Actionable Errors

HandoverGap reports common configuration mistakes with fix-oriented messages while avoiding raw evidence payloads in exception text.

Privacy Defaults

Core commands run locally by default and do not call OpenAI, TiDB, Slack, GitHub, or other external services. Live OpenAI slot filling, TiDB persistence, and Streamlit live mode require explicit optional dependencies and credentials.

handovergap privacy-check

Keep user datasets, reviewed labels, generated reports, and .env files out of git. Remove real names, direct IDs, URLs, email addresses, payment details, and secrets before publishing examples.

Versioning Policy

HandoverGap 1.0.0 is the first stable public-contract release. Earlier 0.x.y releases were allowed to clean up API names, CLI names, profile schema, result fields, and TiDB schema details when the change improved product readiness. Breaking changes must be called out in the changelog and release notes.

Full policy and release checklist: docs/31_versioning_policy.md.

What This Page Helps Validate