Student Projects

Research Question

How do Gemma3:4B, Llama3.1:8B, and Qwen2.5:7B compare on field-level extraction accuracy for HIPAA compliance scenarios — and can a formal error taxonomy, combined with targeted prompt interventions, reduce the most common failure types?

Background

Prior work compares models on verdict accuracy alone (correct/wrong). But two models can reach the wrong verdict for completely different reasons — one hallucinated a court order, the other misidentified the sender. You can't fix what you haven't named. This project builds the first formal taxonomy of LLM extraction errors in legal compliance and tests whether knowing the error type predicts which prompt fix resolves it.

Method

• Run all 3 models on the same 50-scenario subset (diverse HIPAA exception categories)
• Compare each model's extracted JSON against gold annotations field by field; compute per-field accuracy
• Collect all wrong extractions across all 3 models; inductively build a taxonomy:
  • Read cases, notice patterns, group into categories, write definitions, re-annotate until stable
  • Category candidates: role error, purpose OOV, oracle hallucination, oracle under-extraction, format failure, direction flip
• Validate taxonomy: compute inter-annotator agreement on a 20-case subset
• For each error type: design a targeted prompt fix; measure whether it reduces that error type's rate on a held-out set
• Analyze: which error types are model-specific? Which are universal across all three?

Deliverable

3-model field-level accuracy comparison table + formal error taxonomy (definitions, examples, inter-annotator agreement) + per-type prompt fix results. 2-page write-up. Poster.

Engine connection: Priscilla runs all 3 models' JSON outputs through Soufflé locally to add verdict-level accuracy as the headline number.

Research Question

Can adversarial perturbations to a HIPAA scenario narrative flip the LLM's extraction of oracle predicates — and which predicates are most vulnerable to which attack types?

Background

Oracle predicates (e.g. has_court_order, is_required_by_law) are boolean flags that enable HIPAA exceptions. If an adversarial narrative can flip one from False to True without a corresponding legal fact, the entire pipeline is exploitable. The attack happens at the NLP layer — no engine access needed to study it.

Method

• Write 30 adversarial scenarios in 3 categories (10 each):
  • P1 — Injection: narrative implies an enabling legal fact that doesn't actually exist (e.g. "acting pursuant to what may be a court order")
  • P2 — Stripping: narrative omits an enabling fact that is present in the original
  • R — Rhetorical framing: legally suggestive language without a concrete enabling fact
• Run each adversarial scenario through the extraction model; compare oracle predicates to original
• Attack Success Rate (ASR) = % of scenarios where the target predicate flipped
• Analyze: which predicates are most vulnerable? Which attack type is most effective?

Deliverable

30-scenario adversarial dataset with labels + ASR table by attack type × predicate. 2-page write-up. Poster.

Engine connection: Priscilla pipes your adversarial JSON outputs through Soufflé to show verdict-level impact.

Research Question

Which prompting strategy — zero-shot, few-shot, chain-of-thought, role-play, or schema-constrained — produces the highest field-level extraction accuracy for HIPAA compliance scenarios?

Background

Prompt engineering is the fastest lever for improving LLM extraction without retraining. ComplianceGPT uses a multi-rule schema-constrained prompt. Whether chain-of-thought or few-shot examples improve extraction accuracy specifically (not just verdict accuracy) in the legal domain has not been studied.

Method

• Define 5 prompt variants: zero-shot, 2-shot examples, chain-of-thought ("think step by step before filling each field"), role-play ("you are a HIPAA compliance analyst"), schema-constrained (JSON schema in system prompt with field definitions)
• Run all 5 on 30 scenarios with Gemma3:4B
• Measure field-level extraction accuracy per prompt type vs. gold annotations
• Analyze: does CoT help more on oracle predicates? Do examples help more on rare HIPAA exceptions?

Deliverable

5-way field-level accuracy comparison + breakdown by field type and HIPAA category. 2-page write-up. Poster.

Engine connection: Priscilla runs best-prompt outputs through Soufflé to confirm field-level gains translate to verdict-level gains.

Research Question

Are the natural language explanations generated by the LLM for a HIPAA compliance decision accurate, faithful to the underlying legal logic, and useful to non-expert users?

Background

ComplianceGPT generates a natural language explanation alongside each verdict. This explanation is LLM-generated — it is not guaranteed to correctly reflect the formal reasoning that produced the verdict. Evaluating and improving explanation quality is a purely NLP task: you annotate, score, and improve LLM-generated text against a rubric.

Method

• Use pre-computed verdict results (provided by Priscilla) as ground truth for 50 scenarios
• Prompt the LLM to generate an explanation for each verdict given the scenario
• Develop a 4-dimension rubric: correctness (factually accurate?), faithfulness (aligned with actual verdict logic?), usefulness (would a non-expert understand?), completeness (cites the right HIPAA provision?)
• Score all 50 explanations on the rubric (1–5 scale per dimension)
• Test: does a revised LLM prompt improve rubric scores?

Deliverable

Explanation quality rubric + 50-item annotated dataset + before/after scores with improved prompt. 2-page write-up. Poster.

Research Question

Given a user question about health or financial data, can an LLM reliably identify which regulations apply — and does it get the right extraction under each applicable regulation when prompted separately?

Background

A question like "Can I delete my medical records?" simultaneously touches HIPAA, GDPR (EU residents), CCPA (California), COPPA (minors), and SOX (financial records at public institutions). No existing system maps a question to its full regulatory footprint before answering. This is a pure NLP classification and extraction task — no engine needed for the classification step.

Example Overlap Questions

• "Can I request my medical records?" — HIPAA §164.524, GDPR Art. 15, CCPA §1798.100
• "Can I delete my records?" — HIPAA (limited), GDPR (right to erasure), CCPA, COPPA (if minor), SOX §802
• "Can a bank share my health insurance data?" — HIPAA, GLBA, CCPA
• "Can a school share my child's health records?" — HIPAA, COPPA, FERPA

Method

• Create 20 overlap questions spanning 2–4 regulations each
• Manually annotate ground truth: which regulations apply, and what each requires
• Task 1 — Applicability classification: prompt LLM to output which regulations apply to each question; measure recall (does it catch all applicable regulations?)
• Task 2 — Cross-regulation extraction: for each applicable regulation, prompt the LLM to extract the relevant CI schema fields using that regulation's schema; measure field-level accuracy per regulation
• Analyze: which regulations are most often missed? Where do extracted fields conflict across regulations?

Deliverable

20-question multi-regulation dataset + applicability labels + per-regulation extraction accuracy table. 2-page write-up. Poster.

Engine connection: Priscilla runs each regulation's extracted JSON through the corresponding local engine to produce verdict-level cross-regulation comparison.

Research Question

Can parameter-efficient fine-tuning (LoRA) on the GoldCoin extraction task substantially improve a small model's field-level accuracy over prompting alone — and does improvement on oracle predicate extraction specifically translate to verdict accuracy gains when connected to the engine?

Background

Prompting is fast but has a ceiling — the model was not trained on legal compliance extraction. Fine-tuning adapts the model's weights to the task. LoRA (Low-Rank Adaptation) makes fine-tuning feasible on a single GPU with a model as small as 4B parameters. This project creates a task-specific model that Priscilla can permanently integrate into the pipeline.

Method

• Dataset preparation: from the 137 GoldCoin cases, create (scenario, gold_json) training pairs; split into 110 train / 27 test
• Baseline: run Gemma3:4B or Llama3.1:8B with the current prompt on the 27-case test set; record field-level accuracy per field
• Fine-tuning: use HuggingFace PEFT + LoRA to fine-tune on the 110 training pairs (supervised: input = scenario, target = gold JSON string)
• Evaluation: run fine-tuned model on the 27-case test set; compare field-level accuracy before vs. after fine-tuning
• Analysis: which fields improved most? Did oracle predicate accuracy improve? What does the fine-tuned model still get wrong?

Tools

transformers, peft, bitsandbytes (4-bit quantization for memory efficiency), datasets. All installable via pip. Works on any GPU with ≥8GB VRAM — university HPC, Google Colab Pro, or a shared lab GPU.

Deliverable

Fine-tuned LoRA adapter weights + before/after field-level accuracy table + error analysis of remaining failures. 2-page write-up. Poster.

Engine connection: Priscilla loads your fine-tuned adapter locally, runs the test set through the full pipeline (extraction → Soufflé → verdict), and computes the verdict-level accuracy improvement — the headline number for your paper.