Private AI for Law Firms, Clinics and Offices: Document Chat

Private document AI is a chat assistant on a box inside the office that answers questions from the firm's own files, so documents and questions never leave the building. It suits law firms, clinics and accountants whose clients or regulators object to cloud AI. It does not make anyone compliant by itself: GDPR, Singapore's and Malaysia's PDPA and HIPAA still apply, and as the installer you may become a data processor or business associate.

This is the office guide in the edge AI series. The software is covered in the local LLM stack.

Why do offices want AI that stays in the building?

Because their data is the product. A law firm's precedents, a clinic's patient notes and an accountant's client files are the reason clients trust them.

The surveys point the same way. In an OECD survey of more than 5,000 SMEs in seven countries, 52.5% of those not using generative AI cited concerns about what happens to the information fed into the models. Cisco's 2024 privacy benchmark found that 27% of organisations had banned generative AI at least temporarily, 48% admitted entering non-public company information into such tools, and 91% believed their data would be inherently safer stored in their own country or region.

Regulators and professional bodies are writing it down. Singapore's Ministry of Law, in its March 2026 guide for using generative AI in the legal sector, ranks protection in increasing order from free tools to enterprise tools to in-house systems, and warns that cloud data "may reside on servers outside Singapore. This could conflict with clients' data residency requirements." The Law Society of England and Wales advises firms to "establish whether the generative AI tool is a closed system within your firm's boundaries".

What does a private document chat box do?

Staff open a browser page on the office network, log in and ask questions such as "What notice period do our standard leases use?" or "What is our protocol for a patient on warfarin before an extraction?" The system:

1. searches the firm's indexed documents for the relevant passages;
2. passes those passages and the question to a model running on the box;
3. answers with links to the source documents, so staff can check.

The usual stack is Ollama for the model and Open WebUI or AnythingLLM for logins, document upload and search. A Mac mini with 64GB of memory is a sensible starting box for a small office. How many people it serves at once depends on the model and the length of the documents, so test it with the client's real load during the pilot. Details and licences are in the local LLM stack and the box in local AI hardware.

Which jobs work well, and which do not?

Local models get things wrong, as cloud models do. Design for checking: answers must show their sources, and the firm's policy must say a qualified person signs off anything that goes to a client or patient.

What do the data protection laws say?

This is general information, not legal advice. Tell the client in writing that compliance is their responsibility and that they should check with their own adviser.

GDPR (EU) and UK GDPR. Penalties reach EUR 20 million or 4% of worldwide annual turnover, whichever is higher, for the most serious breaches. A DPIA is required for large-scale processing of special categories of data, which includes health data. If you can access the documents while maintaining the box, you are likely a processor and need a data processing agreement with the client.

Singapore PDPA. The maximum financial penalty is 10% of annual turnover in Singapore for organisations with turnover above S$10 million, or S$1 million in any other case, in force since 1 October 2022.

Malaysia PDPA. The 2024 amendment, Act A1727, raised the maximum fine from RM300,000 to RM1 million, requires data controllers and data processors to appoint data protection officers, and requires breach notification to the Commissioner. According to a Nishimura and Asahi briefing, the processor security duty took effect on 1 April 2025, and the DPO, breach notification and data portability rules on 1 June 2025. As an installer who maintains the box, you may count as a data processor under these rules.

HIPAA (US healthcare). HHS says covered entities generally must have contracts with their business associates, and that a business associate is directly liable under the HIPAA rules. Its cloud guidance adds that a provider storing only encrypted data is still a business associate even without the key. A local box removes the cloud AI vendor from that chain, but if you can reach patient data during maintenance, expect the clinic to ask you to sign a BAA. The Security Rule still requires administrative, physical and technical safeguards for the box itself.

Lawyers' professional rules. The ABA's Formal Opinion 512 (July 2024) reminds lawyers of the duty of confidentiality under Model Rule 1.6. The ABA's own summary says the opinion recommends informed client consent before using client confidences in generative AI tools, and that boilerplate consent in an engagement letter is not adequate. The opinion is not binding, and state bars set their own rules.

How do you set up the box so the firm can defend it?

1. Network. The box sits on the office network with no inbound access from the internet. Remote maintenance goes through a VPN the client controls and can switch off.
2. Accounts. Single sign-on or individual accounts, never a shared login. Groups by department so HR files stay with HR.
3. Encryption. Full-disk encryption on the box. Encrypted backups the client holds.
4. Logs. Keep a log of who asked what, and agree with the client how long to keep it.
5. Model updates. No automatic updates. Test a new model on a copy, then switch with the client's sign-off.
6. Paperwork. A data processing agreement or BAA if you can reach the data, a one-page description of the system, and a written handover of admin passwords.

How do you price an office install?

A typical office project is a document audit, a pilot on a test folder, the full install and a monthly retainer for updates and support. Upwork's guide to hiring AI consultants lists a readiness audit at $1,500 to $4,000 and a proof-of-concept build at $8,000 to $20,000, which gives a sense of what clients see elsewhere. A small office box sits at the low end of that range. Run the client's payback in the local AI vs cloud calculator, and see how to price local AI installs for the setup fee, hardware margin and retainer.