Independent Build
No fixed calendar days for this one · Self-paced, suggested order below · Deliverable: a working Streamlit app + a short written comparison
Why This Exists
I'm going to be away for a few days. Rather than pausing, this is a chance to build something real, on your own, that pulls together everything you've learned so far: prompting strategies (Week 3), retrieval-augmented generation (Week 4), and agentic systems (Agentic Systems slides / teach_07_agentic.ipynb) — into one artifact you build end-to-end and fully own. Unlike the main ComplianceGPT pipeline, where the compliance-reasoning engine is a black box you call via API, everything in this project is yours: the interface, the retrieval, the agents, the prompts. Nothing here is off-limits to inspect or change.
It's also a genuinely useful research artifact. The main ComplianceGPT pipeline always pairs an LLM extraction layer with a symbolic verifier (Datalog/Soufflé) that makes the final call. This project deliberately removes that verifier and asks: how far does an LLM get on its own — with no augmentation, with retrieval, and with a team of specialized prompts — when it has to produce the verdict itself? Comparing your three frameworks side by side is a small, self-contained experiment in exactly the question this whole lab is built around.
What You're Building, In One Paragraph
A single Streamlit app where a user can ask a HIPAA compliance question (e.g. "Can a hospital share psychotherapy notes with a private investigator without the patient's consent?") and choose which of three reasoning frameworks answers it — Baseline (raw LLM), RAG (LLM + retrieved HIPAA text), or Agentic (a small pipeline of specialized prompts) — and how they want to interact with it: a single one-shot Q&A box, or a multi-turn chat like you'd use with a SOTA chatbot. Six usable combinations (3 frameworks × 2 modes) in one app.
The Core Design: 3 Frameworks × 2 Modes
Think of this as a grid. Every cell should be reachable from the same app — most naturally as a framework selector (tabs or a sidebar radio) crossed with a mode toggle (single vs. chat).
| Single Q&A | Chat | |
|---|---|---|
| Baseline | Ask once, get one verdict + explanation, no memory. | Same LLM, but conversation history is fed back in — user can ask "what about if it was a court order instead?" as a follow-up. |
| RAG | Question → retrieve top HIPAA sections → inject into prompt → one answer with citations. | Same retrieval step runs per turn (or per new topic); citations should still show up in a multi-turn conversation. |
| Agentic | Question flows through your agent pipeline once, final answer shown. | Hardest cell — optional stretch. If you get everything else working first, try making the agent pipeline conversational. |
Priority order if you're short on time: get all three frameworks working in Single Q&A mode first. Add Chat mode to Baseline and RAG next. Agentic + Chat together is the stretch cell — nice to have, not required.
Output Contract — Applies to All Three Frameworks
Every single answer your app produces — Baseline, RAG, or Agentic, Single Q&A or Chat — must surface two things as distinct, clearly-labeled fields, not buried inside a paragraph of prose:
- Verdict —
PERMITTED,DENIED, orUNKNOWNif the question genuinely doesn't give you enough to decide. Guessing to avoid sayingUNKNOWNis worse than saying it. - Citation(s) — the specific HIPAA section number(s) the verdict rests on (e.g.
§ 164.502(a)(1)(vi)), not just "HIPAA" or "the Privacy Rule." If more than one section is actually relevant to the answer, all of them must be listed — most real HIPAA determinations stack a general permission with one or more specific exceptions, and citing only the first one you find is exactly the kind of shortcut that produces a wrong-for-the-right-reasons answer.
A reasonable UI treatment: a small results block under the free-text explanation, e.g. Verdict: DENIED | Citations: § 164.502(a), § 164.512(f).
Watch for this specifically: Baseline has no retrieval step, so its citations can only come from whatever the model memorized during training — which means they're the most likely of the three to be confidently wrong (a real section number attached to the wrong rule, or a section number that doesn't exist at all). RAG's citations should trace directly to sections your retrieval step actually returned. Agentic's citations should trace to whichever agent did the retrieving. Checking whether each framework's citations are real and correctly applicable — not just present — is one of the more interesting things you'll find when you compare them.
The Three Frameworks
Baseline — Raw LLM Reasoning
No retrieval, no tools, no extra scaffolding. The question goes straight to the local model with a system prompt establishing it's answering HIPAA compliance questions. This measures what the model already "knows" from pretraining — exactly what you did by hand in Extraction Warm-Up Step 1, now wrapped in an app instead of a terminal.
Why it matters as a baseline: every improvement RAG or Agentic makes should be measurable against this. If Baseline already gets a question right, that's not evidence RAG or Agentic "work" — you need cases where Baseline is wrong and the other frameworks fix it (or vice versa).
RAG — Retrieval-Augmented Generation
Before answering, retrieve the most relevant sections of the actual HIPAA regulation text (Title 45 CFR Part 164) and inject them into the prompt as context — the "open book exam" version, same idea as Week 4. This is the framework where the eCFR data source (below) does the real work: your retrieval corpus needs to be built from real regulatory text, not the model's memory of it.
Minimum viable retrieval: BM25 or TF-IDF keyword search over your parsed HIPAA sections (rank_bm25 or sklearn.feature_extraction.text.TfidfVectorizer — both are what the main pipeline already uses, see Week 4). Embedding-based semantic search is a nice upgrade if you have time, not a requirement.
Show your citations. Whatever sections got retrieved for a given question should be visible in the UI (e.g., an expandable "Sources used" box under the answer, showing § numbers and the retrieved text). This is what makes RAG's answer auditable in a way Baseline's isn't.
Agentic — A Team of Specialized Prompts
Instead of one prompt doing everything, break the task into steps, each handled by a differently-instructed call to the model (an "agent" here just means: a prompt with one narrow job, possibly using a different, more specialized system prompt than the others). This is the same idea as teach_07_agentic.ipynb, applied to this app instead of the main pipeline.
A reasonable default split, if you want a starting point rather than designing from scratch:
- Extractor agent — reads the question and pulls out the Contextual Integrity fields (sender, receiver, subject, attribute, purpose) as structured JSON, same as the CI 5-tuple from Contextual Integrity and Step 2 of Extraction Warm-Up.
- Retriever agent — takes the extracted fields (not the raw question) and searches your HIPAA corpus — this is a great place to reuse your RAG framework's retrieval code rather than rewriting it.
- Verdict agent — given the extracted fields and retrieved text, decides PERMITTED / DENIED and writes the explanation.
You don't have to use exactly this split — a "classifier agent" that first routes the question to the right HIPAA subpart, or a "critic agent" that reviews the verdict agent's answer before it's shown, are both reasonable alternatives. The requirement is just that at least two agents with genuinely different responsibilities and different prompts are chained together — not one prompt wearing an "agent" label.
In the UI, consider showing the intermediate steps (what did the Extractor pull out? what did the Retriever find?) somewhere visible, even collapsed by default — it's the most interesting thing to look at when comparing this framework to the other two.
No Formal Verifier — On Purpose
The main ComplianceGPT pipeline you've been learning about always ends with a symbolic reasoning engine (Datalog/Soufflé) that makes the actual PERMITTED/DENIED call from extracted facts — the LLM never decides the verdict itself, it only extracts structured facts that feed the verifier. This project has no formal component at all. In every framework here — Baseline, RAG, and Agentic — the LLM itself produces the final verdict and explanation, with nothing symbolic checking its work. That's intentional: it's a smaller, fully self-contained system you can build without needing the verifier's internals, and it doubles as a natural experiment in how much the formal layer is actually buying the main pipeline. Keep that framing in mind for your written comparison at the end.
The Two Communication Modes
Single Q&A
One text input, one submit action, one answer. No memory between questions — each submission is independent. This is the simplest mode to build first and the easiest to evaluate against a fixed list of test questions.
Chat
A running conversation, like you'd use with a SOTA chatbot. Streamlit has native chat elements for exactly this — st.chat_input for the input box and st.chat_message to render each turn, with the history kept in st.session_state so it survives reruns.
Whatever you pass as "history" into your prompt should include prior turns, so a follow-up like "what if it was a court order instead?" actually resolves against the earlier question.
Scope: HIPAA Only, For Now
Build this for HIPAA (Title 45 CFR Part 164) only. Do not try to wire up GDPR, CCPA, SOX, GLBA, or COPPA in this pass — that's explicitly out of scope right now. What is in scope: writing your code so a second regulation could be added later without a rewrite. In practice that mostly means: don't hardcode "HIPAA" or "164" into your retrieval/agent logic — pass a regulation config (a name, a data source, a system prompt fragment) through instead, even though today only one value exists for it.
A caution for when you do extend this later
Not every regulation in the lab's scope lives in the eCFR the same way HIPAA does. HIPAA and GLBA are U.S. federal regulations and should be reachable through the same eCFR full-text API pattern (different title/part numbers). COPPA is FTC-enforced federal regulation, also in the eCFR (Title 16). But CCPA is California state law — it isn't in the federal eCFR at all — and GDPR is EU law, entirely outside U.S. federal sources. Those would need different official sources (California legislative text, EUR-Lex for GDPR) and a different fetcher than the one you're building now. This is exactly why the regulation config should carry a source type, not just assume "eCFR" works for everything.
Data Source: the eCFR Full-Text API
Yes — the XML endpoint you were given is a suitable RAG source. It's the authoritative, machine-readable full text of the regulation, maintained by the federal government, and it's exactly the same underlying text the main ComplianceGPT pipeline's own RAG strategy retrieves against (see Week 4 and the note in Foundations → Regulations). The one thing it needs before it's usable for retrieval is parsing — the raw XML has markup, nested cross-references, and citation footers mixed in with the actual rule text, so a cleaning step matters.
https://www.ecfr.gov/api/versioner/v1/full/2026-07-16/title-45.xml?part=164
I fetched this endpoint to check its shape before writing this page, so the structure below is verified, not guessed. Part 164 (Security & Privacy) comes back as 41 sections, nested like this:
The N attribute on each <DIV8> is the section number (e.g. "164.502") — that's your natural chunk key and citation. <HEAD> is the section title, and the sequence of <P> elements is the body text (they include nested <I> italics tags around defined terms — strip those, keep the text). One appendix (DIV9) also appears; you can skip it or include it, it's not operative rule text.
Build an ecfr_caller.py module
This is the piece the assignment specifically asks for — a dedicated caller that handles the eCFR side, not something bolted into your Streamlit file. It should:
- Fetch the XML from the endpoint above (Python
requests). - Cache it locally (e.g.
data/title45_part164.xml) — don't re-fetch from the API every time the app starts. A simple "does the file exist?" check is enough; you don't need a real cache-invalidation policy for this. - Parse it with
xml.etree.ElementTree, walking every<DIV8 TYPE="SECTION">, pulling outN, the<HEAD>text, and the joined<P>text (with tags stripped). - Save the parsed result as a clean local corpus — a CSV or JSON with one row per section:
section, heading, text. This file, not the raw XML, is what your RAG and Agentic retrieval actually load and search.
Suggested File Layout
Suggested Build Order
No fixed dates on this one, but tackle it in roughly this order — each step produces something you can actually run, which makes it much easier to debug than writing everything then testing once at the end.
Get a bare Streamlit app running (streamlit run app.py) with a framework selector (even if only "Baseline" works) and Single Q&A mode. Wire it to your local Ollama model with a HIPAA-focused system prompt. You should be able to ask a question and get an answer before doing anything else.
ecfr_caller.py + CorpusBuild and test the eCFR fetcher/parser in isolation — run it from a plain Python script or notebook first, before wiring it into Streamlit. Confirm you get 41 sections out, spot-check a few against the real regulation text (§164.502 is a good one to check against the snippet above).
Do this before Step 3, so you have real test cases ready as soon as RAG exists. See "Build Your Own Test Set" below for the full instructions.
Add BM25 or TF-IDF search over hipaa_sections.csv. Wire retrieval into a new prompt that injects the top-k sections before asking. Add a "Sources used" display under the answer. Compare a few questions against Baseline — do the citations actually look relevant?
Chain at least two agents (suggested: Extractor → Retriever → Verdict, reusing Step 3's retrieval code). Show the intermediate output somewhere in the UI. This is usually the step that takes the longest — budget extra time here.
Add st.chat_input / st.chat_message + st.session_state history to Baseline and RAG. Test that a genuine follow-up question ("what if it was a court order instead?") actually uses the earlier turn's context.
Run the same set of test questions (below, or your own) through all three frameworks in Single Q&A mode. Note where they agree, where they disagree, and your best guess at why.
Quick Sanity Check
Two real cases from the lab's own data, already used in Extraction Warm-Up — use these first, before your FAQ test set exists, just to confirm all three frameworks are wired up and producing a verdict + citation at all:
Both of these are genuinely non-obvious (see the reveal boxes in Extraction Warm-Up for why) — good test cases precisely because a naive Baseline answer is likely to get at least one wrong.
Build Your Own Test Set — from the HIPAA FAQ
The two cases above are a smoke test, not a real evaluation — two examples can't tell you much about where each framework tends to succeed or fail. The actual test set you'll use for the written comparison is one you build yourself from HHS's official HIPAA FAQ.
Source: HHS Office for Civil Rights — HIPAA FAQ
HHS maintains an official FAQ database at hhs.gov/hipaa/for-professionals/faq/, organized by topic (uses and disclosures, minimum necessary, right of access, and more). Many individual answers cite specific 45 CFR §164 sections directly — those are the ones you want. (Note: hhs.gov blocks automated fetches, so you'll be browsing this by hand, not scraping it — that's fine, this is meant to be a small, manually curated set, not a scrape.)
- Browse the FAQ by topic. For each FAQ you use, find one whose answer names a specific section (e.g. "as permitted by § 164.512(f)...").
- Turn it into a compliance question in the same style as the sanity-check cases above — a short scenario or direct question with a clear PERMITTED/DENIED/UNKNOWN answer.
- Record the expected verdict and every section the FAQ's own answer cites — this is your ground truth for the Output Contract above.
- Keep the source FAQ URL for each row, so anyone (including you, later) can check your ground truth against the original.
Target size: 15–20 questions is enough to see real patterns without becoming a huge manual effort. Save it as data/hipaa_faq_testset.csv with columns: question, expected_verdict, expected_citations, source_faq_url.
Deliverable Checklist
What "done" looks like
1. Working Streamlit app — runs via streamlit run app.py with no manual setup beyond installing requirements. Framework selector + mode toggle both functional.
2. All three frameworks working in Single Q&A mode. Chat mode working for at least Baseline and RAG (Agentic + Chat is a stretch goal, not required).
3. ecfr_caller.py — actually fetches and caches the real eCFR XML, and parses it into a clean local corpus file. Not a hand-written stand-in for the corpus.
4. Output contract enforced everywhere — every answer, in all three frameworks, shows a distinct verdict field and a citation field listing all applicable sections, not prose-only answers and not just the first section found.
5. Agentic pipeline with ≥2 distinct agents chained together, with intermediate output visible somewhere in the UI.
6. data/hipaa_faq_testset.csv — 15–20 questions built from the real HHS HIPAA FAQ, each with expected verdict, expected citation(s), and source URL.
7. Short written comparison (½–1 page) — run your full FAQ test set through all three frameworks. Where did they agree? Disagree? Were citations correct and complete, or did any framework (especially Baseline) cite a wrong or invented section? Which framework got closest to your ground truth, and why do you think that was? Tie this back to the "no formal verifier" framing above — what do you think the symbolic verifier in the main pipeline is actually buying, based on what you saw?
Stretch Goal — Multi-Regulation, If Everything Else Is Solid
Only attempt this after the checklist above is fully working. Add a second regulation (GLBA or COPPA are the easiest — both reachable via the same eCFR pattern, just a different title/part number).
Parity requirement — no shortcuts: whatever depth you built for HIPAA, the second regulation needs the same depth, not a token version. That means: its own real caller/parser (not a hand-typed stub), its own full section corpus, the same verdict + full-citation output contract, and its own 15–20 question test set built from that regulation's own official FAQ or guidance (e.g. the FTC's COPPA FAQ). If HIPAA got a real dataset and the second regulation gets three made-up examples, this isn't done — it's testing whether your regulation config abstraction from the "Scope" section actually held up, and a shallow second regulation won't tell you that.