An AI document classifier takes an incoming document — a PDF, scanned image, or text file — and assigns it to a category: invoice, contract, receipt, CV, medical form, ID document, or whatever categories your use case requires. This eliminates the manual sorting step in document workflows and enables downstream automation to route each document to the appropriate processing pipeline. Building a reliable classifier is simpler than most teams expect, especially with modern LLMs that classify accurately from a prompt description alone without any training data.
Two Approaches: LLM Prompting vs Embedding Classification
There are two fundamentally different approaches to building an AI document classifier. LLM-based classification uses a language model to read the document text and output a category based on a prompt that defines the categories. This requires no training data, works immediately, handles novel document types gracefully, and produces high accuracy on clearly distinct categories. It costs an LLM API call per document and is slightly slower than the embedding approach. Embedding-based classification generates a vector embedding of the document, trains a lightweight classifier (logistic regression or SVM) on a labelled dataset of embeddings, and classifies new documents by comparing their embeddings to the trained model. This requires labelled training data (typically 50-200 examples per category), runs extremely fast once trained, costs only the embedding generation, and works well for large volumes of similar documents.
Figure 1 — LLM vs embedding classification: when to use each
Building an LLM-Based Classifier
The LLM-based classifier requires three components: text extraction (converting the document to plain text), a classification prompt, and output parsing. The prompt defines each category with a one-to-two sentence description, instructs the model to return only the category name and a confidence score as JSON, and includes the document text. Example category definitions for a business document classifier: “INVOICE — A bill from a supplier requesting payment for goods or services delivered. Contains line items, amounts, and payment terms. RECEIPT — Proof of a payment already made. CONTRACT — A legally binding agreement between two or more parties. QUOTE — A non-binding estimate of cost for goods or services.” The model reads the document text and returns the matching category. GPT-4o-mini handles this reliably for most document types; Qwen2.5 7B via Ollama handles it locally with slightly lower accuracy on ambiguous cases.
Handling Confidence and Ambiguous Documents
Request a confidence score (0-100) alongside the classification and route low-confidence classifications (below 80, for example) to a human review queue rather than the automated downstream pipeline. Documents that fall below threshold are flagged in your workflow, a human classifies them correctly, and you log the correct label. Over time this log becomes a training dataset for an embedding classifier if you want to upgrade to the faster approach. For genuinely ambiguous documents — a document that could be either a quote or an invoice depending on interpretation — add an “AMBIGUOUS” class and route those explicitly to human review. The classifier’s job is to be reliably right on the clear cases and honest about uncertainty on the hard ones, not to force a confident answer on every input.
Building an Embedding-Based Classifier
Once you have 50+ labelled examples per category, the embedding classifier offers better throughput at lower per-document cost. Generate embeddings for each labelled document using an embedding model (OpenAI’s text-embedding-3-small, or nomic-embed-text via Ollama locally). Train a logistic regression or SVM classifier on the embedding vectors using scikit-learn — fit the classifier on 80% of labelled examples, evaluate on the remaining 20%. The trained classifier is a small model file you load at inference time. For new documents, generate the embedding and call classifier.predict() to get the category and classifier.predict_proba() for the confidence score. This pipeline runs thousands of classifications per minute on a single CPU, compared to the LLM approach which processes a few documents per minute per API call.
Integrating into n8n or Dify
For the LLM-based classifier, integrate into n8n by using the Ollama or OpenAI node with your classification prompt, followed by a JSON Parse node and a Switch node that routes documents to different subsequent nodes based on the category output. For a Python embedding classifier, expose it as a FastAPI endpoint running alongside your other services and call it from n8n via the HTTP Request node. In Dify, create a Workflow app where the first node is a document file input, the second is an LLM node with the classification prompt, and subsequent branches handle each category. The Dify approach requires no external infrastructure and is the fastest path to a working classifier with a web UI for manual overrides.
Practical Example: Classifying Incoming Email Attachments
A practical small business use case: classify PDF attachments from incoming emails as invoices, contracts, or other, then route invoices to the invoice processing workflow and contracts to a legal review folder. In n8n: an Email Trigger watches for new emails with attachments; a PDF Extract node converts the attachment to text; an Ollama LLM node classifies it; a Switch node routes invoices to the invoice extraction workflow and contracts to a Google Drive folder. This runs automatically for every incoming email attachment, requires no human sorting, and feeds structured data directly into the accounting system for invoices while flagging contracts for human attention. The total build time for this workflow in n8n is two to four hours, and it handles dozens of emails per day without any manual intervention after setup.
Frequently Asked Questions
- How accurate are LLM-based document classifiers?
- For clearly distinct document types (invoice vs contract vs CV), accuracy exceeds 95% with a well-written prompt. For closely related categories (purchase order vs invoice vs quote), accuracy drops to 85-90% and an AMBIGUOUS routing class becomes important. Always measure accuracy on a held-out test set of your actual documents before deploying to production.
- Can I classify handwritten documents?
- With an OCR step first. Run the handwritten document through an OCR tool (Tesseract for simple handwriting, Google Document AI or AWS Textract for better accuracy on complex layouts) to produce text, then classify the text. Handwriting recognition accuracy varies widely by handwriting quality and the classifier inherits any OCR errors.
- How many categories can an LLM classifier handle?
- Up to around 15-20 well-defined categories before accuracy starts to degrade from category confusion. For more categories, use a hierarchical approach: first classify into broad groups, then sub-classify within each group with a second LLM call.
Last updated: October 2026
Evaluating Classifier Performance Over Time
A classifier that is accurate today may degrade as document types evolve. Track two metrics on an ongoing basis: overall accuracy (what percentage of classifications are correct, measured by periodic sampling and human verification) and per-class recall (what percentage of invoices are correctly identified as invoices, what percentage of contracts as contracts). Class recall is more informative than overall accuracy for imbalanced datasets where one class is rare — a classifier that always outputs INVOICE looks 90% accurate if 90% of documents are invoices, but has 0% recall on all other classes. Review classifier performance monthly, add new training examples for classes with falling recall, and update the prompt descriptions when new document subtypes appear that the classifier is mishandling. A document classifier is infrastructure that improves over time with maintenance rather than a one-time deployment.
Multi-Label Classification
Some documents belong to more than one category simultaneously. A purchase order might also be a contract. An email might be both a customer complaint and an invoice query. Multi-label classification handles this by returning a list of matching categories with individual confidence scores rather than a single winning class. Implement this in the LLM prompt by asking the model to return all applicable categories: ‘Return a JSON array of objects, each with category and confidence. Include all categories that apply to this document.’ In the downstream routing, check the returned list and trigger all applicable workflows rather than only the top-scoring one. Multi-label classification is more complex to build and validate, but is the correct approach for document sets where category membership is not mutually exclusive. For most business document classifiers, the categories are mutually exclusive (a document is either an invoice or a contract, not both) and single-label classification is simpler and sufficient.
Keeping Labels Consistent Over Time
A classifier is only as consistent as its category definitions. Write a short labelling guide that lists each category, a clear definition, and two or three tricky edge cases with the correct label, and use that same text in your prompt. Keep a golden set of documents with agreed labels and re-run it after every prompt or model change. When two reviewers disagree on a document, treat it as a signal that the definitions are ambiguous, and refine the guide rather than just picking a winner. Version your category list, because adding or merging categories later silently changes the meaning of historical results.