RAG vs NLP vs Raw LLM Calls: A Startup's Guide to Choosing the Right AI Architecture

Most “AI-powered” products start the same way: send text to an LLM, parse the response, ship it. No custom models. No fine-tuning. No MLOps infrastructure. That’s usually the right call, at least at first.

Which architecture fits depends on your stage: raw LLM calls for speed, RAG for document Q&A, classical NLP for cost efficiency at scale. Each has different tradeoffs, and most teams end up using all three eventually.

This post breaks down when each approach makes sense, how to blend them, and why starting simple and optimizing later beats trying to build it right the first time.

The Three Approaches

Here are the three approaches, defined plainly:

Raw LLM Calls: Send your data + a prompt to a frontier model, get structured output back.

response = openai.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": f"Extract metrics from: {document_text}"}]
)

RAG (Retrieval-Augmented Generation): Store your data as embeddings, retrieve relevant chunks, send those chunks + user query to an LLM.

relevant_docs = vector_db.similarity_search(query, k=5)
context = "\n".join([doc.content for doc in relevant_docs])
response = llm.generate(f"Based on this context:\n{context}\n\nAnswer: {query}")

Classical NLP: Use pre-trained models (BERT, FinBERT, spaCy) or rule-based systems to extract entities, classify text, or parse structure without calling external APIs.

doc = nlp(text)
entities = [(ent.text, ent.label_) for ent in doc.ents]

Each has different cost curves, accuracy profiles, and operational complexity.

The Honest Comparison

*LLM accuracy varies significantly by task complexity. Recent benchmarks show 60-80% on contract extraction (Vellum), 96%+ on medical data extraction (Nature Digital Medicine), and 82% on finance tasks (AIMutiple). GPT-5.2 achieves 98% accuracy on 256k context retrieval tasks.

LLMs trade money and vendor dependency for speed and simplicity. NLP trades development time for control and cost efficiency.

Quick Decision Guide

START HERE: What are you building?
│
├─► "Users ask questions about documents"
│   └─► RAG (with LLM fallback for complex queries)
│
├─► "Extract specific fields from documents"
│   │
│   ├─► Less than 10k docs/month?
│   │   └─► Raw LLM calls (optimize later)
│   │
│   ├─► More than 10k docs/month?
│   │   │
│   │   ├─► Structured format (tables, forms)?
│   │   │   └─► Hybrid: NLP for tables + LLM for edge cases
│   │   │
│   │   └─► Unstructured format (free text, mixed layouts)?
│   │       └─► LLM with confidence scoring + human review
│   │
│   └─► Need real-time (<500ms)?
│       └─► NLP only (no API calls in hot path)
│
└─► "Classify or categorize documents"
    │
    ├─► Few categories, lots of examples?
    │   └─► Fine-tuned classifier (BERT, DistilBERT)
    │
    └─► Many categories, few examples?
        └─► LLM with few-shot prompting

What This Looks Like in Practice

This matches what I found building an extraction system for financial and ESG data. We needed five years of financial data and five years of ESG data for 16,500 companies, across multiple countries and languages, and we had to extract it and fine-tune on it. The architecture that got to production fastest was prompt-based.

A typical ESG (Environmental, Social, and Governance) extraction pipeline extracts hundreds of indicators per document (emissions, water usage, board composition, and so on), at 80-90% accuracy on first-pass extraction, for $0.02-0.05 per document depending on model and document length, with 30-60 seconds of latency per document. At 1,000 documents/month, that’s $30-50. At 100,000 documents/month, it’s $3,000-5,000, and that’s when the math changes.

A hybrid system routes content by type instead: tables, often 70-80% of the structured data, go through NLP extraction at about $0.001/doc; free-form text (15-20%) goes through NER models at about $0.002/doc; charts and visuals (5-10%) go through LLM extraction at about $0.03/doc; anything low-confidence falls back to LLM verification.

The savings at scale are real, often 60-75% cost reduction with 3-4x faster processing. And unlike pure LLM approaches, accuracy can improve over time as corrections feed back into the NLP models.

Prompt-based extraction is still usually the right starting point, and there’s a clear path to optimization once scale demands it.

When to Use Raw LLM Calls

Raw LLM calls make sense in a few situations:

• You’re validating the product. If you don’t know whether users want extracted financial metrics, don’t spend eight weeks building an NLP pipeline. Spend three days on an LLM wrapper and find out.
• The task is hard for rule-based systems. Free-form document understanding, nuanced classification, anything requiring “reasoning.”
• Volume is low. At 1,000 documents/month and $0.03/doc you’re paying $30/month, and the engineering time to build an NLP system costs more than that.
• Accuracy matters more than cost. For high-stakes extractions like financial filings or legal documents, LLM accuracy often beats custom NLP without extensive training data.

The typical LLM extraction pattern looks like this:

EXTRACTION_PROMPT = """
You are an expert data extractor. Extract the following from the document:
- Total Revenue (in millions USD)
- Net Income (in millions USD)
- Total Assets (in millions USD)

Return as JSON: {"revenue": ..., "net_income": ..., "total_assets": ...}
If a value is not found, use null.

Document:
{document_text}
"""

def extract_financials(document_text: str) -> dict:
    response = gemini.generate_content(
        EXTRACTION_PROMPT.format(document_text=document_text)
    )
    return json.loads(response.text)

That’s it. That’s the “AI-powered extraction engine.” And it works surprisingly well.

The Hidden Costs of LLM-Only

The cost table doesn’t show everything. LLMs are non-deterministic: run the same prompt on the same document twice and you can get different results. Temperature settings help, but you can’t guarantee identical outputs, which is a real problem for data pipelines that need reproducibility. You can’t write unit tests that assert exact outputs, and you can’t diff yesterday’s extraction against today’s to see what changed.

There’s also no learning loop. Every extraction is independent, and if you correct an error, the model doesn’t learn from it, so you’re paying for the same mistakes forever. NLP systems can ingest corrections as training data; LLMs just forget.

Debugging is a black box. When extraction fails, you can’t easily tell why: the prompt, the document format, token limits, or a model update that changed behavior. I’ve spent hours debugging extractions only to discover the model was updated and now interprets a phrase differently.

There’s a latency floor too. Even fast models (Gemini Flash, GPT-4-turbo) take 1-5 seconds per call. For real-time applications that’s brutal, and for batch processing it means 30-60 seconds per document instead of the 100ms you’d get from local NLP.

And your “AI feature” is actually OpenAI’s uptime. Rate limits and outages become your problem. Model deprecations force a scramble. Pricing changes shift your unit economics overnight. Model updates also happen without warning, so an extraction that worked fine for six months can start returning different field names or misinterpreting units, and without logging and monitoring you won’t notice until a customer complains.

I still start with LLMs. I just treat LLM-based systems as prototypes that may need hardening later.

When to Use RAG

RAG is the middle ground: smarter than raw prompts, simpler than full NLP pipelines. It fits when users ask questions over a corpus (chatbots, search, “ask your documents” is what RAG is built for), when context matters more than extraction (RAG is better at finding relevant information and synthesizing it than at pulling specific fields), when you have a growing knowledge base (new documents get embedded and become searchable immediately, no retraining needed), and when accuracy needs to beat a raw LLM call (grounding responses in retrieved context cuts down hallucination).

The typical RAG pattern looks like this:

# Indexing (once per document)
def index_document(doc_id: str, text: str):
    chunks = split_into_chunks(text, chunk_size=500)
    for i, chunk in enumerate(chunks):
        embedding = embed_model.encode(chunk)
        vector_db.upsert(
            id=f"{doc_id}_{i}",
            vector=embedding,
            metadata={"doc_id": doc_id, "text": chunk}
        )

# Querying
def answer_question(query: str, user_id: str = None) -> str:
    query_embedding = embed_model.encode(query)

    # Retrieve relevant chunks
    results = vector_db.search(
        query_embedding,
        top_k=5,
        filter={"user_id": user_id} if user_id else None
    )

    context = "\n---\n".join([r.metadata["text"] for r in results])

    # Generate answer with context
    response = llm.generate(f"""
    Based on the following documents:
    {context}

    Answer this question: {query}

    If the answer isn't in the documents, say so.
    """)

    return response

RAG Architecture Decisions

Embedding model choice matters. OpenAI’s text-embedding-3-small is cheap and good, but for domain-specific content (finance, legal, medical) it’s worth considering fine-tuned embeddings or models like FinBERT. Chunk size is a tradeoff: smaller chunks (200-500 tokens) give more precise retrieval but less context per chunk, while larger chunks (1000-2000 tokens) give more context but risk pulling in irrelevant material. Hybrid search often wins, combining vector similarity with keyword search (BM25) catches queries that are better served by exact matches.

# Hybrid search with reciprocal rank fusion
def hybrid_search(query: str, top_k: int = 5):
    # Vector search
    vector_results = vector_db.search(embed(query), top_k=20)

    # Keyword search
    keyword_results = bm25_index.search(query, top_k=20)

    # Combine with RRF
    scores = {}
    for rank, result in enumerate(vector_results):
        scores[result.id] = scores.get(result.id, 0) + 1 / (60 + rank)
    for rank, result in enumerate(keyword_results):
        scores[result.id] = scores.get(result.id, 0) + 1 / (60 + rank)

    return sorted(scores.items(), key=lambda x: -x[1])[:top_k]

The Hidden Costs of RAG

RAG has its own hidden costs. Changing embedding models means re-embedding everything, which gets expensive at scale. Important information split across chunk boundaries may never get retrieved together. RAG reduces hallucination but doesn’t eliminate LLM costs: you’re paying for embeddings and generation. Documents change, so keeping embeddings in sync with source documents takes real infrastructure. And users don’t always know what to ask: RAG assumes they can formulate good queries, but in practice people ask vague questions like “what are the environmental metrics?” when they need specific Scope 1 emissions data, and without structured extraction running alongside RAG, you’re dependent on query quality, which is often the weakest link.

This is why hybrid approaches work: RAG for conversational access, structured extraction for making sure you capture everything regardless of what users ask.

When to Use Classical NLP

NLP earns its complexity once you’ve validated the product and need to scale: the LLM prototype works, users like it, and now you’re processing 100k documents/month and the costs don’t work anymore. It fits structured, repetitive extraction (tables, forms, SEC filings, anything with a predictable format), cases that need sub-second latency (NLP models run locally in milliseconds, no API round-trip), cases where accuracy should improve over time (labeled corrections let you fine-tune and accuracy compounds), and cases where you need explainability (NLP can show exactly what was matched and why; LLMs can’t).

NLP is the wrong call when:

• You’re still figuring out what to extract. It needs a schema upfront. If users keep asking for new fields, you’ll be constantly rewriting extraction rules, where an LLM handles ad-hoc requests without code changes.
• Documents are highly variable. Handwritten notes, informal reports, or wildly inconsistent layouts frustrate rule-based systems in ways LLMs handle better.
• You don’t have training data. Fine-tuned NER models need hundreds of labeled examples. Without them, your NLP accuracy will lag a well-prompted LLM.
• The ROI doesn’t justify the engineering. An NLP pipeline takes weeks. If you’re processing 5,000 documents/month at $150 in LLM costs, saving $100/month doesn’t justify a four-week investment. Do the math first.
• Edge cases reach users. NLP fails hard, returning nothing or garbage. LLMs fail soft and usually return something plausible, which matters when no result is worse than an approximate one.

The sweet spot for NLP is high-volume, structured, repetitive extraction where you’ve already validated the product. Everything else should probably stay on LLMs until scale forces the conversation.

The hybrid pattern I use looks like this:

Document → Layout Analysis → Content Routing
                               ↓
              ┌────────────────┼────────────────┐
              ▼                ▼                ▼
         TABLES            TEXT             CHARTS
     (NLP: Camelot)    (NLP: NER)      (LLM: Vision)
       85-95% acc       80-90% acc        75-95% acc
        $0.001          $0.002           $0.030
              │                │                │
              └────────────────┼────────────────┘
                               ▼
                      Confidence Scoring
                               ↓
              ┌────────────────┼────────────────┐
              ▼                ▼                ▼
          High (>0.9)    Medium (0.7-0.9)   Low (<0.7)
           Accept         LLM Verify        Human Review

Table extraction accuracy depends on document quality and tool choice. Benchmarks show Camelot at 72-73% on complex layouts, while deep learning approaches (Table-Transformer, TableFormer) achieve 93%+ (arXiv PDF Parsing Study). For clean, well-structured tables, rule-based tools perform excellently.

Most financial/ESG data lives in tables, and tables are where NLP shines:

class TableExtractor:
    def __init__(self):
        self.detector = TableTransformer()  # Microsoft's model
        self.parser = CamelotParser()
        self.matcher = IndicatorMatcher(load_indicators())

    def extract(self, pdf_path: str) -> List[Extraction]:
        # 1. Detect tables
        tables = self.detector.detect(pdf_path)

        results = []
        for table in tables:
            # 2. Parse table structure
            df = self.parser.parse(table)

            # 3. Classify table type
            stmt_type = self.classify_statement(df)

            # 4. Extract and match metrics
            for row in df.iterrows():
                metric = self.extract_metric(row, stmt_type)
                if metric:
                    matched = self.matcher.match(metric)
                    matched.confidence = self.calculate_confidence(matched)
                    results.append(matched)

        return results

    def classify_statement(self, df) -> str:
        """Detect: Balance Sheet, Income Statement, Cash Flow"""
        headers = ' '.join(df.columns).lower()
        if 'assets' in headers or 'liabilities' in headers:
            return 'balance_sheet'
        elif 'revenue' in headers or 'income' in headers:
            return 'income_statement'
        elif 'cash' in headers and 'operating' in headers:
            return 'cash_flow'
        return 'unknown'

The Cost Math

Here’s the comparison at different scales:

At 1,000 documents a month (seed stage), just pay for the LLM. RAG’s savings are marginal for the complexity it adds, and hybrid NLP’s engineering cost exceeds what it saves. At 10,000 documents a month (Series A), LLM cost starts to matter: RAG is worth it for chat features, and hybrid NLP breaks even in about six months. At 100,000 documents a month (Series B+), LLM-only is painful, RAG is better but still high, and hybrid NLP is the clear winner.

The crossover point is typically 10-50k documents/month. Below that, LLMs win on simplicity. Above that, NLP wins on cost.

The Evolution Path

Here’s how I recommend startups evolve their “AI” stack:

Stage 1: Raw LLM (Day 1 - Product-Market Fit)

# This is your entire AI pipeline
def extract(document: str) -> dict:
    return json.loads(gpt4(PROMPT + document))

Ship it, learn what users actually need, and don’t optimize yet.

Stage 2: RAG (Month 3-6 - Chat/Search Features)

When users ask “can I search my documents?” or “can I ask questions?”, add RAG:

# Add pgvector to your existing Postgres
ALTER TABLE documents ADD COLUMN embedding vector(1536);

# Index on insert
def save_document(doc):
    doc.embedding = embed(doc.content)
    db.save(doc)

# Search
def search(query):
    return db.query("""
        SELECT * FROM documents
        ORDER BY embedding <-> %s
        LIMIT 5
    """, embed(query))

Use your existing database. Don’t add Pinecone yet.

Stage 3: Hybrid NLP (Month 12+ - Scale)

When LLM costs exceed engineering costs, invest in NLP. The LLM calls don’t go away, they just get rarer.

The 6-stage hybrid pipeline:

1. Document Classification
   └─ What type of document is this? (report, filing, form, etc.)
   └─ Tool: Simple classifier or rule-based detection

2. Layout Analysis
   └─ Where are the tables, text blocks, charts, headers?
   └─ Tool: LayoutLMv3, pdfplumber, or PyMuPDF

3. Content Routing
   └─ Route each content block to the right extractor
   └─ Tables → NLP, Text → NER, Charts → LLM

4. Parallel Extraction
   └─ Each extractor runs independently
   └─ NLP extractions return confidence scores

5. Confidence-Based Verification
   └─ High confidence (>0.9): Accept
   └─ Medium (0.7-0.9): LLM verification
   └─ Low (<0.7): Human review queue

6. Feedback Loop
   └─ Log every extraction with source location
   └─ Corrections become training data for NLP models

Start with tables. In most structured documents (financial reports, sustainability disclosures, regulatory filings), 70-80% of the data you want lives in tables. Tables are deterministic: same input, same output. NLP table extraction is a solved problem.

class HybridExtractor:
    def extract(self, document) -> List[Extraction]:
        results = []

        # NLP for tables (cheap, fast, deterministic)
        for table in detect_tables(document):
            extractions = self.table_extractor.extract(table)
            for e in extractions:
                e.source = "nlp_table"
                e.confidence = self.calculate_confidence(e)
            results.extend(extractions)

        # NLP for text entities (cheap, fast, okay accuracy)
        for text_block in detect_text(document):
            extractions = self.ner_model.extract(text_block)
            for e in extractions:
                e.source = "nlp_ner"
            results.extend(extractions)

        # LLM for everything else (expensive, slow, good accuracy)
        for chart in detect_charts(document):
            extractions = self.llm_extractor.extract(chart)
            for e in extractions:
                e.source = "llm"
            results.extend(extractions)

        # LLM verification for medium-confidence NLP results
        for result in results:
            if result.source.startswith("nlp") and 0.7 < result.confidence < 0.9:
                result = self.llm_verify(result)
                result.source = f"{result.source}_verified"

        return results

Track extraction accuracy by source (NLP vs LLM), confidence score calibration (are your 0.9 predictions actually 90% correct), LLM fallback rate (what percentage still needs LLM), and time-to-extraction by document type.

Build the feedback loop early. Every extraction should be logged with the source document location (page, bounding box), the extraction method used, a confidence score, and any user corrections. Corrections are training data: the more users correct, the better your NLP models get. This is the learning loop that LLM-only systems lack.

The Technology Stack

What I’d use today for each approach:

Raw LLM

• Model: Gemini 2.5 Flash (fast, cheap) or Claude Sonnet 4.5 (best reasoning). For complex extraction, GPT-5 or Claude Opus 4.5.
• Structured output: Instructor library for Pydantic validation
• Cost tracking: LangSmith or custom logging

RAG

• Vector DB: pgvector (if you have Postgres) or Qdrant (if you don’t)
• Embeddings: OpenAI text-embedding-3-small or Cohere embed-v3
• Chunking: LangChain’s RecursiveCharacterTextSplitter
• Orchestration: LlamaIndex or raw code (frameworks add complexity)

Classical NLP

• Table extraction: Camelot, pdfplumber, or Table-Transformer
• Layout analysis: LayoutLMv3 (Microsoft, open-source)
• NER: spaCy with domain-specific fine-tuning
• Financial NER: FinBERT
• OCR (if needed): DocTR (Apache 2.0)

Fine-Tuning, Open-Source Models, and Vision Models

Two questions come up often: should we fine-tune a model, and what about open-source models?

Usually, no, don’t fine-tune. It makes sense when you have thousands of labeled examples, the task is narrow and well-defined, prompt engineering has hit a ceiling, or you need to cut per-request costs at very high volume. For most startups, prompt engineering gets you 80-90% of the way there, and fine-tuning is a month of work to squeeze out the last 5-10%, time better spent shipping features. It does make sense for classification tasks with clear categories, domain-specific entity extraction where generic models struggle (legal clauses, medical terms), or cost optimization at 1M+ requests/month where small per-request savings compound.

Open-source models (Llama, Mistral, Qwen) are viable for cost reduction at scale (self-hosted Llama 3.1 70B can match GPT-4 quality for many tasks at a fraction of the cost, if you have the infrastructure), for data privacy requirements some enterprises can’t get around, and for latency-sensitive applications where local inference beats an API round-trip.

The trade-offs are real: infrastructure complexity (GPU servers, model serving like vLLM or TGI, monitoring, scaling, which is a full-time job on its own), a quality gap on complex tasks (frontier models still lead on nuanced reasoning and multi-step extraction), and rapid obsolescence (the model you self-host today gets outperformed by a new release in 3-6 months, while API providers handle upgrades for you).

My recommendation is to start with APIs (OpenAI, Anthropic, Google), and once you’re spending $5,000+/month on LLM calls with engineering capacity for infrastructure, evaluate self-hosting. For most startups, that day is further away than you think.

Multimodal models (GPT-4o, Gemini 2.0 Flash, Claude’s vision capabilities) have also made OCR and chart extraction viable without traditional pipelines. They’re good at charts and graphs, pulling data points from bar charts, line graphs, and pie charts in a single API call where before this needed a computer vision pipeline; at scanned documents, handwritten notes, faxes, and low-quality scans where traditional OCR struggles; and at mixed layouts with embedded images, tables, and text.

Current benchmarks (late 2025) put GPT-4o and Qwen2.5-VL at roughly 75% accuracy on complex OCR benchmarks (OmniAI Benchmark), with Qwen2.5-VL hitting 96.4% on DocVQA, close to the human-level 98.1% (OCRBench v2). For simple, clean documents, multimodal LLMs now match traditional OCR; for complex layouts (rotated text, overlapping elements), they still struggle.

The practical tradeoff is cost and speed: vision models are slower and more expensive than text-only calls. A chart extraction that costs $0.01 with vision might cost $0.001 with traditional table parsing, but if you’re processing charts, which NLP can’t handle, vision models are often the only option short of building custom computer vision. I use vision models for content NLP genuinely can’t handle: charts, diagrams, handwritten notes. For clean, machine-generated PDFs with selectable text, traditional text extraction plus NLP is still faster and cheaper.

Start with the wrapper

Every successful AI product I’ve seen started the same way: ship an LLM wrapper, learn what users actually need, then optimize the parts that matter. The custom models come later, when the unit economics force the issue.

The best optimization I made on the extraction product wasn’t a custom model at all. Chat messages needed to be matched to company names, and that started out as a vector index lookup plus an AI call. We replaced the vector index with fuzzy search over the company names. Now only about 1 in 40 messages gets sent upstream to an LLM to work out intent. That cut our dependency on AI and the cost of every message, and it only became obvious once the wrapper had shown us what users were actually typing.

Resources

LLM APIs & Structured Output:

• OpenAI API - GPT-4, embeddings, structured outputs
• Google Gemini - Gemini Flash for cost-effective extraction
• Anthropic Claude - Strong reasoning, good for complex documents
• Instructor - Pydantic validation for LLM outputs

Vector Databases & RAG:

• pgvector - Vector similarity for Postgres (start here)
• Qdrant - Purpose-built vector DB if you need more
• LlamaIndex - RAG framework (useful, but adds complexity)

NLP & Document Processing:

• Camelot - Table extraction from PDFs
• pdfplumber - PDF text and table extraction
• Table-Transformer - Microsoft’s table detection model
• LayoutLMv3 - Document layout understanding
• spaCy - Industrial-strength NLP
• FinBERT - Financial sentiment and NER
• DocTR - OCR (Apache 2.0 license)

Benchmarks & Research:

• LM Council Benchmarks - December 2025 comparisons of GPT-5.2, Claude 4.5, Gemini 3.0
• OmniAI OCR Benchmark - 1,000 document OCR comparison across multimodal models
• Finance LLM Benchmarks - GPT-5, Gemini 2.5 Pro accuracy on financial extraction
• OCRBench v2 - Academic benchmark for visual text localization

Further Reading:

• Patterns for Building LLM-based Systems - Eugene Yan’s excellent overview
• What We Learned from a Year of Building with LLMs - O’Reilly practical lessons
• GPT-5.2 vs Gemini 3.0 vs Claude 4.5 - Detailed benchmark comparison

For API architecture that scales, see building scalable APIs with Go.