AnythingLLM RAG Setup

1 Setting up RAG AI with AnythingLLM

Updated: Mar 22, 2026.a

I'm using AnythingLLM v1.10 on Windows 11.

This web page does not use external images, all images are embedded in the HTML, so if you want to download and print this page, you just download the HTML page. All the images will be in there.

A RAG AI allows you to add your own documents to the AI and the AI will answer questions using only your documents.

2 Steps

Download and install AnythingLLM for your OS. https://anythingllm.com/

AnythingLLM uses the "workspace" concept. That means you can set up a different LLM (remote or local) to use with each workspace. When you start ALLM the workspaces you have created are listed on the left pane.

My current workspaces

In the image above, to the right of the Search box, click + to make a new workspace. Let's call it "MyTestRag".

Move your mouse over the new workspace "MyTestRag". You will see an upload icon, and gear icon. The gear icon is for settings. Click the gear to change the settings for this workspace.

Now go to the Chat tab. Under the Chat Mode section click "Query". That is what is used for RAG. I don't know why they call it "Query" but you are querying your own documents.

Choose query mode for this workspace

Now click the Back button in the upper left of the setting window to save the settings. Under your workspace "MyTestRag" you will see a box "default". I call this a "chat trail". You can make multiple chat trails for each workspace. This is like clicking "New chat" on Gemini or ChatGPT.

Now use the upload button next to the workspace name to upload your documents. You will get this window:

This is the My Documents area, these are not in your AI RAG workspace yet.

At the bottom of the Upload window is a text box for the website URL. Enter the URL then click "Fetch website". All your documents will be saved in the My Documents area, they are not in this workspace yet.

In the document list that shows uploaded documents, check to select the documents you want to move to your workspace. Click "Move to workspace" button. Now your RAG AI will use those documents. Scroll down this popup window to see the actual documents in this workspace.

Select at least one URL to move to your workspace.

Scroll down this popup window some more and click Save and Embed button to save the documents you moved to your workspace.

Now save and embed the data.

Now go back to the workspace page. Select your workspace, now in the right pane chat with your documents where it says "Send a message".

That's it! You're done.