ComplianceGPT Lab · AI Innovation & Diffusion REU 2026
Agentic Systems
When one prompt isn't enough — reflection, tool use, and multi-agent pipelines, illustrated with our own production code
Week 3
One prompt, one call
→
Today
Reflection · tools · pipelines
→
Week 4
Retrieval as a component
2
Learning Arc
By the end of today you can
Place any LLM system on the spectrum: single-shot → reflective → tool-calling agent → multi-agent pipeline
Explain exactly what extract_with_reflection() does, pass by pass
State why our pipeline is fixed-topology, not a free-roaming agent — and why that's a deliberate choice for a compliance system
Today's notebook
notebooks/teach_07_agentic.ipynb — build a minimal one-step reflective wrapper around a real GoldCoin extraction call and measure whether reflection fixes a known failure case.
3
Part 1 — What Makes a System "Agentic"?
A Spectrum, Not a Binary
"Agentic" gets used loosely. It's more useful as a spectrum of how much autonomy the system has over its own next step.
Level 0
Single-shot
One prompt in, one answer out. No self-checking.
Level 1
Reflection
The model re-examines its own prior output via another prompt. No external tool calls.
Level 2
Tool-calling (ReAct)
The model itself decides when to call an external function and reads the result before continuing.
Level 3
Multi-agent pipeline
Several specialized components (LLM or not) hand off to each other in a designed sequence.
Spoiler: ComplianceGPT sits at Level 1 internally (reflection) inside a Level 3 pipeline (extractor → verifier → explainer). It is not a Level 2 ReAct agent — and that's on purpose, coming up.
4
Part 2 — Case Study: Our Own Pipeline
Three Real Methods, Three Levels of Sophistication
All three live in connector/llm1_extractor.py, and you can call any of them from app/batch_runner.py with a flag.
Method
Flag
What it does
extract()
(default)
One LLM call. Scenario in, JSON facts out.
extract_with_rag()
--use-rag
BM25-retrieves relevant predicate vocabulary before the one extraction call.
extract_with_reflection()
--reflect
RAG extraction, then 3 more LLM passes that re-examine the first pass's own output.
Judicial check — only fires for §164.512(e)/(f) scenarios
Every pass is a fresh LLM call. Pass 2's prompt literally includes the words "already verified for roles/attribute/purpose" — each pass hands the next one its improved facts, in text.
6
Part 2 — Case Study: Our Own Pipeline
Important Distinction: This Is Reflection, Not Tool-Calling
What it is
Every pass is still just text in, text out
The model never calls the Souffle Datalog verifier mid-extraction
"Verification" here means: another LLM prompt re-reads the facts and second-guesses them
What it would take to be Level 2 (ReAct)
The model would decide, itself, "I should check this against the verifier now"
It would call a real function, get back real verifier output, and read it before answering
Our pipeline never does this — the sequence of passes is fixed by the engineer, not chosen by the model
Terminology check: some people call any multi-step LLM process "agentic." We'll be precise — reflection and tool-calling are genuinely different mechanisms with different failure modes.
7
Part 2 — Case Study: Our Own Pipeline
Does Reflection Actually Help? Depends on the Model. verified — RQ2 ablation
n=137, GoldCoin-HHS, same scenarios, same three configurations.
Configuration
Qwen2.5:72B
Gemma3:4B
Single-pass (no RAG)
92.7%
54.0%
+ BM25 RAG (no reflect)
84.7% (−8.0pp)
63.5% (+9.5pp)
+ Four-pass reflection
92.7% (+8.0pp)
94.2% (+30.7pp)
Oracle ceiling
100%
100%
For the small model, the full pipeline is the whole story — a +30.7pp gain. For the large model, raw retrieval hurts and reflection's real job is undoing that damage, not adding new capability.
8
Part 2 — Case Study: Our Own Pipeline
Why Would Retrieval Ever Make Things Worse?
A large, well-instruction-tuned model already extracts facts fairly reliably on its own (92.7%). BM25 then injects retrieved predicate candidates into the same single-shot call, with no chance to reconcile them.
The mechanism
Noisy or partially-matching retrieved vocabulary competes with the model's own read of the scenario in the same forward pass. For a model that didn't need the help, this is pure noise some of the time — accuracy drops to 84.7%.
The reflective passes then re-examine those same facts against the oracle criteria and recover the loss — accuracy lands back at 92.7%, exactly where single-pass started. Reflection's role, for a strong model, is damage control on a component (RAG) that helps weaker models but can mislead stronger ones.
9
Part 3 — True Tool-Calling Agents
ReAct: Reason + Act, in a Loop
The most common "real" agent pattern. The model alternates between reasoning in text and calling a tool, observing the tool's real output before deciding what to do next.
Thought
→
Action (call a tool)
→
Observation (real result)
→
Thought again...
Thought: I need to know if this disclosure requires authorization.
Action: query_verifier(facts)
Observation: missing predicate: obtained_authorization_164_508
Thought: I should ask a follow-up extraction pass specifically for that field.
Action: reextract_field("obtained_authorization_164_508")
Observation: false
Thought: Facts are now complete. Final verdict: DENIED.
10
Part 3 — True Tool-Calling Agents
What Would an Agentic ComplianceGPT Look Like?
Our current design (fixed pipeline)
Always exactly 4 passes, same order, every scenario
Engineer chose the checks in advance (CI-tuple, oracle, judicial)
The Souffle verifier only runs once, after extraction is finished
A hypothetical ReAct version
Model decides after each fact whether to query the verifier
Model decides which specific predicate to re-extract, if any
Number of LLM calls varies scenario to scenario
This is a real design decision, not an oversight — next slide.
11
Part 3 — True Tool-Calling Agents
Why a Compliance System Should Distrust Autonomy
Flexibility and predictability trade off directly. For a legal-compliance verifier, predictability usually wins.
Cost of letting the model decide
Two runs of the same scenario could take different paths
Harder to audit: "why did it call the verifier 3 times here and 0 times there?"
New failure mode: the model can also get the decision to call a tool wrong
Benefit of the fixed pipeline
Every scenario gets the identical, reviewable sequence of checks
The paper's "proof tree" claim — a faithful audit trail — depends on this determinism
Easier to reason about in a security/compliance threat model
General lesson: more autonomy is not automatically better engineering. Match the level of agency to how much you need to audit and reproduce the result.
12
Part 4 — Multi-Agent Pipelines
The Whole System Is a 3-Agent Pipeline — and One of Them Isn't an LLM
LLM1
Extractor — narrative → formal facts
→
Souffle Datalog
Symbolic verifier — deterministic, not an LLM
→
LLM2
Explainer — proof tree → plain-English rationale
Most "multi-agent" tutorials chain several LLMs together. We deliberately put a non-LLM, 100%-deterministic formal-methods engine in the middle. That's the actual reason the system's verdicts are auditable at all — the part that decides PERMIT/DENY never hallucinates, because it isn't a language model.
13
Where Everything We Covered Sits on the Spectrum
System
Level
Autonomy over next step?
extract()
0 — Single-shot
N/A
extract_with_rag()
0 — Single-shot + retrieval
No
extract_with_reflection()
1 — Reflection
No — passes are fixed in advance
Hypothetical ReAct extractor
2 — Tool-calling
Yes — model chooses when/what to query
Full ComplianceGPT pipeline
3 — Multi-agent
No, by design — fixed topology, one non-LLM component
14
Today's Notebook — teach_07_agentic.ipynb
Load one known-wrong GoldCoin scenario from your Week 2 batch results
Run it through extract() — confirm the wrong verdict, print the raw facts
Wrap it in a one-step reflection: re-prompt the model with its own facts plus "double check this against the oracle criteria"
Run the Souffle verifier on both fact sets — before and after reflection
Compare: did the one reflection pass change the verdict? Was it the right change?
15
Assignment — Reflective Wrapper Mini-Lab
Task
Take 15 scenarios from GoldCoin-HHS (your choice, but include the wrong prediction you used in Week 3's prompting exercise). Run each through:
Baseline: extract(), single pass
Your own one-step reflection wrapper (built in today's notebook)
Report: accuracy before vs. after, and for every case that flipped (right→wrong or wrong→right), one sentence on why.
This previews the kind of experiment several Week 3 project options (Prompt Engineering Study, Error Taxonomy) will scale up to 137 rows.
16
Where This Goes Next — Week 4: RAG
You've now seen retrieval (BM25) purely as a source of noise-or-signal inside a fixed pipeline. Week 4 asks the deeper question: what actually goes into the retrieval index, and how do you make it a reliable component rather than a coin flip?
Reflection (today)
→
Retrieval quality (Week 4)
→
Your project (Week 4–5)
17
Questions to Sit With
If reflection mostly undoes RAG's own damage on Qwen, would skipping RAG entirely and keeping only reflection do just as well? How would you test that with the CSVs you already have?
What's one place in your own project where a ReAct-style "let the model decide" design would be actively dangerous?
Is there a 4th level beyond multi-agent pipelines? What would it look like?
18
Recap
"Agentic" is a spectrum: single-shot → reflection → tool-calling → multi-agent
Our 4-pass extractor is reflection, not tool-calling — no live verifier call mid-loop
Reflection's benefit is model-dependent: capability gain for small models, damage control for large ones
Our full pipeline is a 3-agent, fixed-topology system with a deliberately non-LLM verifier at its center
Autonomy and auditability trade off — compliance systems should default to less autonomy, not more
19
One Prompt Was Never the Whole Story
Reflection, retrieval, and a deterministic verifier — three components, one designed pipeline, and now you've read the code that proves it.