Building a RAG System Over Live Salesforce Data

Why RAG Over Salesforce?

Salesforce holds critical business data: accounts, cases, opportunities, contacts. But that data lives in a database, not in your LLM's training set. When your support team asks "What's the status of this customer's recent issues?", you need to query live Salesforce data and ground the answer in what you actually find.

That's RAG: Retrieval-Augmented Generation. You retrieve relevant records from Salesforce, feed them to an LLM with a question, and get an answer backed by real data. No hallucinations. No stale information.

This post walks through the architecture we built at Insolla, the OAuth integration dance, vector search setup, and deployment to production.

The Architecture

Three layers: data retrieval, embedding, and generation.

OAuth 2.0 JWT Flow for Salesforce

Salesforce doesn't let you query as "the app." You need a user, or you use JWT to impersonate one. JWT is cleaner for service-to-service calls.

Tokens expire in 5 minutes. We cache them in Redis to avoid re-issuing on every query. The bottleneck is the Salesforce API rate limit (15 calls/second per user), not the token handshake.

Building the Vector Index

Your Salesforce records need to be searchable by semantic similarity. We embed Account summaries, open Case descriptions, and Opportunity details, then store them in Pinecone with metadata tags.

# Embed a Salesforce Account
account_text = f"Account: {name}, Industry: {industry}, Revenue: {annual_revenue}"

embedding = openai.Embedding.create(
  model="text-embedding-3-small",
  input=account_text
)

pinecone.upsert([
  (f"account-{id}", embedding, {
    "account_id": id, "record_type": "Account"
  })
])

Query time: user asks "What's our biggest customer in healthcare?" We embed the question, search Pinecone for the top 5 Accounts, fetch their details from Salesforce, and pass them to the LLM.

Grounding Answers in Source Data

The magic: every answer cites which Salesforce record it came from. We use OpenAI's structured outputs (JSON schema validation) to enforce citations.

response_schema = {
  "type": "object",
  "properties": {
    "answer": {"type": "string"},
    "citations": {
      "type": "array",
      "items": {"id": "string"}
    }
  }
}

The LLM must return an answer object with a list of source Salesforce IDs. No hallucinated citations. No answers without sources.

Production Considerations

• •Rate limiting: Salesforce throttles queries. Queue requests and retry with exponential backoff.
• •Stale embeddings: When a Salesforce record changes, update its embedding immediately (webhook-driven or batch nightly).
• •Data access control: A user querying Salesforce should only see records they have access to. Enforce Salesforce's permission model in your retrieval layer.
• •Cost: Embedding every record + vector search + LLM inference adds up. Monitor usage and set spending limits.

See It in Action

We built this for Insolla's support team. Try the live demo to ask questions over a sample Salesforce org.

For a deep-dive case study (architecture diagram, performance metrics, lessons learned), see the Salesforce Intelligence case study.