aucourt-ingest/plan.md

# AuCourtIngest — Implementation Plan

## Audit Notes (from research)

- **High Court URL is wrong in spec**: `/cases/recent-judgments` → 404. Real page is `/cases-and-judgments`. The crawl strategy will need to be discovered empirically — the page structure is unknown.
- **AustLII blocks direct access** (403). AusLaw MCP is the only viable path to AustLII content. Remove any "AustLII direct" assumptions.
- **QLD Judgments**: no documented API. The search endpoint at `/caselaw-search/query?queryStringSearchText={term}` exists but behaviour is undocumented. Treat as HTML scraping, not API calls.
- **AusLaw MCP**: repo exists and is active. Tools confirmed: search, legislation, OCR. But it's an MCP server — needs to be running as a separate process. Integration pattern differs from direct HTTP sources.
- **Python version**: spec says 3.12, Rog has 3.13. Use 3.13.
- **No VIC source defined**: spec architecture diagram mentions "VICCatalogue" but section 3 only has 5 sources (FedCourt, HighCourt, NSW, QLD, AusLaw MCP). VIC is only mentioned as a Phase 2 backfill item via AusLaw MCP targeted search. Not a direct source adapter.

---

## Build Order

The spec's 6-phase structure is fine conceptually but has dependency issues. The plan below fixes ordering and adds missing work the spec assumes but doesn't call out.

### Stage 1: Skeleton + Data Models (no external deps)

Goal: runnable `python -m aucourt_ingest --help` that does nothing yet.

- [ ] Create `aucourt_ingest/__init__.py` package
- [ ] Create `models.py` — all dataclasses: `RawDocument`, `CaseMeta`, `Chunk`, `JurorPersona`, `JurorContext`, `FetchQueueItem`, `SourceState`
- [ ] Create `config.py` — load `config.toml`, expose typed config objects (source configs, rate limits, API keys, paths)
- [ ] Create `main.py` — argparse entry point with `bootstrap`, `watch`, `backfill`, `audit` subcommands
- [ ] Create `config.toml` — all source definitions, rate limits, storage paths, API key placeholders
- [ ] Create `pyproject.toml` — dependencies, scripts entry point
- [ ] Create `data/` directories (docs/, raw/)
- [ ] Verify: `python -m aucourt_ingest --help` runs, all imports work

### Stage 2: MetaDB + Utils (no external deps except aiosqlite)

Goal: SQLite is initialised, can track documents through pipeline states.

- [ ] Create `storage/meta_db.py` — init schema (documents, fetch_queue, source_state tables), CRUD methods
- [ ] Create `utils/mnc_parser.py` — MNC regex parser + validator
- [ ] Create `utils/rate_limiter.py` — async token bucket per source_id
- [ ] Create `utils/retry.py` — configurable exponential backoff decorator
- [ ] Tests: `test_mnc_parser.py` — parse valid/invalid MNCs, edge cases (missing brackets, non-standard courts)
- [ ] Tests: `test_meta_db.py` — init, insert doc, update status transitions, queue operations, source state
- [ ] Tests: `test_rate_limiter.py` — basic burst behavior, per-source isolation

### Stage 3: Source Layer (one source first, then extend)

Goal: Fetch a real document from one court and store it as raw HTML.

**Start with NSW Caselaw** — it has the clearest pagination, HTML-only (no DOCX parsing needed), and the `?filter=criminal` endpoint works.

- [ ] Create `sources/base.py` — `BaseSource` ABC: `discover()`, `fetch(url)`, `build_queue()`. Enforce rate limit. Log to MetaDB. Handle retry per exception type.
- [ ] Create `sources/nsw_caselaw.py` — browse pagination, extract MNC + URL per row, fetch HTML
- [ ] Create `storage/doc_store.py` — save raw downloads to `data/raw/{source_id}/{year}/{doc_id}.html`
- [ ] Wire: `main.py bootstrap --source nsw_caselaw --limit 3` fetches 3 real criminal cases
- [ ] Verify: raw HTML files on disk, MetaDB shows `fetch_status=fetched`

Then extend to other sources:

- [ ] Create `sources/fedcourt.py` — RSS poll (`/rss/fca-judgments` is confirmed working), extract DOCX links from RSS items, download DOCX. Bootstrap pagination via `/digital-law-library/judgments/search` endpoint (needs empirical testing).
- [ ] Create `sources/highcourt.py` — crawl from `/cases-and-judgments` (NOT `/cases/recent-judgments` which 404s). Page structure is unknown — needs manual inspection first. Apply PRIORITY_KEYWORDS filter post-fetch.
- [ ] Create `sources/qld_judgments.py` — HTML scraping at `/caselaw-search/query?queryStringSearchText={term}`. No documented API. May need to reverse-engineer search parameters for criminal filter.
- [ ] Create `sources/auslaw_mcp.py` — MCP client wrapper. NOT a direct HTTP source — spawns/requires auslaw-mcp server process. Use `mcp-python-sdk`. Handles: fetch by citation, OCR for scanned PDFs, targeted search.

### Stage 4: Processing Pipeline (format-agnostic)

Goal: Raw document → structured metadata + chunks + embeddings. Works regardless of source.

- [ ] Create `processing/doc_parser.py` — dispatch to format handler. HTML (BeautifulSoup, strip nav/header/footer), DOCX (python-docx), PDF (pdfminer.six text extraction). Output: `RawDocument`.
- [ ] Create `processing/meta_extractor.py` — call Claude Haiku with the extraction prompt from spec. Parse JSON response into `CaseMeta`. Handle malformed LLM output gracefully (retry once with "fix this JSON").
- [ ] Create `processing/outcome_parser.py` — separate verdict extraction pass. Resolution order: headnote → orders section → appeal cross-ref → external exoneration list. Load exoneration lookup table from `data/exoneration_sources.json` (to be curated manually).
- [ ] Create `processing/chunk_engine.py` — use Claude Haiku to identify structural boundaries, then split at boundaries within token budgets. Output: list of `Chunk` with type, sequence, speaker.
- [ ] Create `processing/embed_engine.py` — batch embed chunks via OpenAI text-embedding-3-small, store in ChromaDB collection `au_cases_{source_id}`.
- [ ] Create `storage/vector_index.py` — ChromaDB wrapper: `embed_and_store(chunks)`, `query(text, chunk_types, doc_ids, top_k)`
- [ ] Tests: `test_doc_parser.py` — fixture HTML (NSW-style), DOCX, PDF text. Verify clean extraction, char counts, no nav/footer leakage.
- [ ] Tests: `test_meta_extractor.py` — mock Haiku response, verify CaseMeta parsing, verify null handling for missing fields.
- [ ] Tests: `test_outcome_parser.py` — fixture with explicit headnote, fixture with orders-only, fixture requiring appeal cross-ref.
- [ ] Tests: `test_chunk_engine.py` — mock boundary detection, verify token budget enforcement, verify sequence ordering.

### Stage 5: Graph Builder

Goal: Build property graph from CaseMeta + Chunks. Enable juror queries.

- [ ] Set up Neo4j (Docker container on Lappy or local)
- [ ] Create `storage/graph_db.py` — Neo4j driver wrapper: `create_node(label, properties)`, `create_relationship(from, to, type, props)`, `query(cypher)`
- [ ] Create `processing/graph_builder.py` — from CaseMeta: create Case, Charge, Judge, Witness, Exhibit, Ruling, Timeline nodes and relationships. From Chunks: create Chunk nodes, FOLLOWS edges. Pairwise similarity: CORROBORATES (>0.85) and CONTRADICTS (<0.25 same topic).
- [ ] Tests: `test_graph_builder.py` — create case with known meta, verify graph shape (node counts, relationship types, no dangling references).
- [ ] Verify: small test case → Neo4j has correct node/edge counts

### Stage 6: Juror Interface

Goal: Persona-driven subgraph queries work end-to-end.

- [ ] Create `jury/personas.py` — `JurorPersona` dataclass + `PERSONA_TRAVERSAL` dict (6 personas from spec)
- [ ] Create `jury/subgraph_query.py` — `get_juror_context(case_mnc, persona, max_tokens)`. Traverse from persona anchor nodes, follow specified edge types, collect chunks up to token budget. **MUST exclude RULED_INADMISSIBLE edges at traversal level.**
- [ ] Tests: `test_subgraph_query.py` — fixture graph with inadmissible evidence. Verify every persona excludes it. Verify token budget is respected. Verify persona-specific node selection (nurse sees medical exhibits, not financial).
- [ ] Tests: `test_inadmissibility_wall.py` — dedicated test that RULED_INADMISSIBLE exclusion cannot be bypassed regardless of persona config.

### Stage 7: Orchestrator + Watch Mode

Goal: Autonomous continuous operation.

- [ ] Create `orchestrator.py` — main loop: poll sources → enqueue fetch_queue → dispatch workers (source→parse→extract→chunk→embed→graph) → update MetaDB at each stage transition
- [ ] Create `utils/telegram.py` — alert sender: source_degraded, daily_summary, milestone
- [ ] Wire: `python -m aucourt_ingest watch` runs APScheduler cron jobs (6h poll per spec)
- [ ] Wire: `python -m aucourt_ingest backfill --source fedcourt --from 1995 --to 2009 --format pdf`
- [ ] Wire: `python -m aucourt_ingest audit` re-processes all documents in MetaDB with current schema
- [ ] Integration test: full pipeline (NSW Caselaw → 3 docs → end to end → graph queryable)

### Stage 8: Bootstrap Runs

- [ ] Bootstrap FedCourt: RSS 2010-2026 criminal filter (~small test first, then full)
- [ ] Bootstrap High Court: crawl 2000-2026 criminal appeals
- [ ] Bootstrap NSW Caselaw: browse criminal 2010-2026
- [ ] Validate corpus stats against spec targets (~5,000 cases for Phase 1)
- [ ] Set up Telegram alerts
- [ ] Enable watch mode on all 3 primary sources

---

## Key Risks

| Risk | Impact | Mitigation |
|------|--------|------------|
| High Court page structure unknown | Blocks source adapter | Manual inspection first, write adapter from real HTML |
| QLD search endpoint undocumented | Fragile scraping | Build defensive parser, expect breaking changes |
| Haiku extraction quality varies | Bad metadata | Validation layer, manual_review queue for low-confidence |
| Neo4j on Docker adds infra dep | Slows dev | Could start with NetworkX in-memory for testing |
| 5,000 cases × Haiku calls = cost | Burn rate | Estimate before bootstrap, consider batching |
| AusLaw MCP server as external dep | Availability | Document start-up procedure, health check |
| Court sites may block scrapers | Source degradation | Polite headers, rate limits, robots.txt respect, Telegram alerts |

## Decisions Needed Before Stage 3

1. **Graph backend for dev**: Neo4j Docker vs NetworkX in-memory for testing. Neo4j is spec target but adds Docker dependency. Consider: use NetworkX for unit tests, Neo4j for integration.
2. **Email AustLII now?** — Long lead time. Low effort. Do it during Stage 1 skeleton work.
3. **Contact email in User-Agent** — spec has placeholder `your@email.com`. Need real one.
4. **Exoneration data curation** — RMIT register, Innocence Project lists need manual collection into `data/exoneration_sources.json`. When?