Tutorial: Build a RAG pipeline with LangChain, OpenAI and Pinecone

This tutorial contains everything you need to build production-ready Retrieval Augmented Generation (RAG) pipelines on your own data.

Whether you're working with a corporate knowledge base, personal blog, or ticketing system, you'll learn how to create an AI-powered chat interface that provides accurate answers with citations.

Complete Example Code

The full source code for this tutorial is available in the companion repository on GitHub. This repository contains a complete, working example that you can clone and run locally.

Try It Yourself

See the complete working demo at /chat. This tutorial walks you through building this exact same experience:

Table of contents

• System Architecture
• Phase 1: Data processing
  • Step 1: Load and configure the data processing notebook
  • Step 2: Clone the data source
  • Step 3: Install dependencies
  • Step 4: Loading blog posts into memory
  • Step 5: Loading your Open AI and Pinecone API keys into the environment
  • Step 6: Creating a Pinecone index
  • Step 7: Creating a vectorstore with LangChain
  • Understanding Document Chunking
• Phase 2: Application development
  • Step 1. Create the supporting services
    • The context service
    • The embeddings service
    • The Pinecone service
  • Step 2. Create the chat API
  • Step 3. Create the user interface
• Phase 3: Deployment
• Additional Resources

System Architecture

How does the RAG pipeline work? (Click to expand)

Let's understand the complete system we'll be creating:

This is a Retrieval Augmented Generation (RAG) pipeline that allows users to chat with your content. Here's how it works:

1. When a user asks a question, their query is converted to a vector (embedding)
2. This vector is used to search your Pinecone database for similar content
3. The most relevant content is retrieved and injected into the LLM's prompt, the LLM generates a response based on your content, and the response is streamed back to the user along with citations

Phase 1: Data processing

What are the main steps we'll follow? (Click to expand)

We'll build this system in the following order:

1. Data Processing: First, we'll process your content (blog posts, documentation, etc.) into a format suitable for vector search. We'll use a Jupyter Notebook for this phase.

2. Vector Database Creation: We'll convert your processed content into embeddings and store them in Pinecone, creating a searchable knowledge base.

3. Knowledge Base Testing: We'll verify our setup by running semantic search queries against the vector database to ensure we get relevant results.

4. Backend Development: We'll build the Next.js API that accepts user queries, converts queries to embeddings, retrieves relevant content from Pinecone, provides context to the LLM, and streams the response back to the user

5. Frontend Implementation: Finally, we'll create the chat interface that accepts user input, makes API calls to our backend, displays streaming responses, and shows related content and citations

Step 1: Load and configure the data processing notebook

I've created a Jupyter Notebook that handles all the data preprocessing and vector database creation.

This notebook is designed to be easy to understand and customizable - you can swap out my example site with your own content source.

1. First, open the notebook in Google Colab with this direct link:

2. Configure your API keys in Colab's secrets manager:

• Obtain your OpenAI API key from here
• Obtain your Pinecone API key from here

Click the key icon in the left sidebar to add your secrets.

Name your OpenAI API key secret OPENAI_API_KEY your Pinecone API key secret PINECONE_API_KEY.

Ensure the Notebook access toggles are enabled for both secrets. This grants your notebook access to the secrets.

Now that you've configured your secrets, we're ready to step through the notebook, understanding and executing each cell.

Step 2: Clone the data source

The next cell clones the open source companion example site which contains the blog posts. Run it to pull down the site, which you can then view in the content sidebar: