Tax & AuditISO 27001 / FBR / IRS Audit

Extract Multi-Column Tax Audit PDFs with a Local LLM: Table-Accurate Pipeline

Turn multi-column ledgers and audit schedules into validated JSON with Docling and a local LLM, reconciled against document totals and kept in ISO scope.

Published

Architecture

Recommended model

Qwen3-32B (Q8_0)

Parser / tooling

Docling (TableFormer table structure)

VRAM required

41.5 GB

Q8_0, 32k tokens

Compliance standard

ISO 27001 / FBR / IRS Audit

Ledgers, trial balances and multi-column audit schedules are parsed with Docling, whose table-structure model reconstructs rows and columns before any LLM sees the data. Qwen3-32B at 8-bit precision then normalizes headings, maps line items to your chart of accounts and returns JSON that is cross-checked against the document's own totals. Running at Q8_0 trades extra VRAM for fewer numeric transcription errors, and all processing stays inside your ISO 27001 scope.

  1. 01

    Ingest

    Audit evidence PDFs are hashed on arrival to preserve chain of custody.

  2. 02

    Parse

    Docling detects tables and reconstructs multi-column layouts into cell-level JSON.

  3. 03

    Normalize

    Qwen3-32B maps line items to the chart of accounts using a strict JSON schema.

  4. 04

    Reconcile

    A deterministic script checks extracted totals against document totals and flags mismatches.

Performance & privacy profile

Pre-lab estimates for Qwen3-32B (32B), Q8_0 (8-bit), 32k tokens. Not measured results.
MetricValueBasis
VRAM required41.5 GBWeights + FP16 KV cache + 1 GB runtime overhead
Decode speed~22–33 tok/sMemory-bandwidth estimate on 2× RTX 5090 (64GB), single user
Table extraction accuracyPending lab testWill be measured on our reference document set
Privacy score8/10Editorial rubric, see breakdown below
  • On-prem inference, no third-party AI processor(implemented)+3
  • No outbound network route from AI hosts(implemented)+2
  • Encryption at rest for documents and indexes(not included)+1
  • Per-request audit logging(implemented)+2
  • Role- or matter-based access control(implemented)+1
  • Data minimization before storage (redaction or hashing)(not included)+1

Deployment config

Starting point only. Pin image versions and review network policy before production.

# docker-compose.yml: table extraction for audit evidence.
# Requires 2 GPUs (or one 48 GB+ card) for Qwen3-32B at Q8_0 with 32k context.
services:
  ollama:
    image: ollama/ollama:latest  # pin a specific tag in production
    volumes:
      - ollama:/root/.ollama  # pre-pulled: qwen3:32b-q8_0
    environment:
      - OLLAMA_CONTEXT_LENGTH=32768
    networks: [audit-internal]
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

  docling:
    image: quay.io/docling-project/docling-serve:latest
    networks: [audit-internal]

networks:
  audit-internal:
    internal: true

volumes:
  ollama:

Size it for your hardware

Local LLM VRAM Calculator

2026 models

Total VRAM required

41.5GB

Weights
32.5 GB
KV cache
8.0 GB
Overhead
1.0 GB

Recommended local hardware

  • 2× RTX 5090 (64GB)~22–33 tok/s
  • Mac mini M4 Pro 64GB~3–5 tok/s

Estimated single-user decode speed from memory bandwidth. Real throughput depends on runtime, drivers and prompt length.

Insufficient Local VRAM? Deploy a SOC 2 / HIPAA GPU Pod on RunPod ($0.89/hr) →

Frequently asked questions

Why not let the LLM read the PDF tables directly?

Multi-column layouts and merged header cells are where LLMs most often misplace numbers. A dedicated table-structure model reconstructs the grid first, so the LLM only has to label and normalize cells instead of inferring the layout. A deterministic reconciliation step then compares extracted totals with the document's own totals, so errors are caught rather than silently passed downstream.

Can this pipeline be used for FBR or IRS audit evidence?

The pipeline helps prepare and analyze evidence; it doesn't change your obligations to the Federal Board of Revenue (Pakistan) or the IRS. Keep the original documents, record the hash of every file processed, and store extracted outputs alongside their sources so an auditor can trace each figure back to the page it came from. Have a qualified professional review figures before they are filed or relied on.

How does this fit into an ISO 27001 information security management system?

Treat the pipeline as an in-scope information system: add it to your asset inventory, apply your access-control and logging policies, and include it in your risk assessment. Because inference runs locally, you avoid adding an AI provider as a supplier, which simplifies supplier-relationship controls and keeps client tax data inside your existing security boundary.

More blueprints