Customizing your node

We want to enable you to create and run your own agent service node. You probably won't be using the default Phi-3 LLM and Paris guidebook knowledge base. So let's show you how to customize your node!

Pre-set configurations​

The gaianet/config.json file hold all node configuration options, like the LLM settings, vector collection for the knowledge base and prompts. You can either edit this file directly to use your models and vector collections, or you can select a different config.json when you initialize the node.

Simply pass in a URL to the config.json file in your gaianet init command, the URL to the config.json must point to the actual text file.

We have several pre-set config.json files to choose from in this repo. For example, the following command initializes a GaiaNet node with a Llama 3 8B model:

gaianet init --config https://raw.githubusercontent.com/GaiaNet-AI/node-configs/main/llama-3-8b-instruct/config.json

The config subcommand​

Even after initializing the node, you are able to make changes to its configuration by editing the config.json file. However, it is a best practice to use the gaianet CLI in order to make the changes as it is safer and easier.

After any changes have been made to the node configuration, thegaianet init command MUST be run.

gaianet config list command shows the config.json fields that are able to be changed.

Selecting an LLM​

Huggingface has over 10,000 fine-tuned, open-source LLMs to choose from. Each of them have different sizes (the larger models are more capable yet more expensive to run), unique capabilities (for example, some can support large context length, are uncensored, or excel in math), domain expertise (like coding or medicine), and/or styles (some respond in code, can speak like a pirate, etc).

Making changes to the model file, prompt template, and model context length parameters will allow you to replace your node's default LLM with an alternative fine-tuned model. The parameters vary depending on the model, but they can be found on the gaianet Huggingface organization's model cards.

For example, the following command changes the LLM to a Llama 3 8B model:

gaianet config \
  --chat-url https://huggingface.co/gaianet/Llama-3-8B-Instruct-GGUF/resolve/main/Meta-Llama-3-8B-Instruct-Q5_K_M.gguf \
  --chat-ctx-size 4096 \
  --prompt-template llama-3-chat 

The llama 3 8B model requires at least 16GB of RAM.

If none of the published fine-tuned models are perfect for your use case, you can also fine-tune your own LLM by following these guides. Your GaiaNet node can run your own fine-tuned models.

The --chat-url argument could point to a local file under $HOME/gaianet instead of a public URL. That allows you to use a privately trained or fine-tuned LLM model file.

Selecting a knowledge base​

A key feature of GaiaNet is that users can create and deploy proprietary knowledge base on the node to supplement the LLM. Each knowledge base is a snapshot file for a vector collection. You can use ready-made knowledge bases, but we encourage you to create your own knowledge base . You will need to do the following:

1. specify the URL to the vector collection (i.e., the snapshot or snapshot.tar.gz file) in the snapshot option.

2. use the same embedding model that generated this vector collection.

3. modify the system_prompt to give the model background knowledge.

4. modify the rag_prompt to instruct the model to answer the question when context is retrieved from the vector collection.

The following example changes the knowledge base in the node from "Paris guidebook" to "London guidebook":

gaianet config \
  --snapshot https://huggingface.co/datasets/gaianet/london/resolve/main/london_768_nomic-embed-text-v1.5-f16.snapshot.tar.gz \
  --embedding-url https://huggingface.co/gaianet/Nomic-embed-text-v1.5-Embedding-GGUF/resolve/main/nomic-embed-text-v1.5.f16.gguf \
  --embedding-ctx-size 8192 \
  --system-prompt "You are a tour guide in London, UK. Please answer the question from a London visitor accurately." \
  --rag-prompt "The following text is the context for the user question.\n----------------\n"

The --snapshot could point to a local file under $HOME/gaianet instead of a public URL. That allows you to use a private vector collection snapshot.

Depending on the quality and size of the vectors, you might also need to change the qdrant- options to customize the retrieval behavior.

• qdrant-limit sets the max number of relevant context to add to the prompt. If your knowledge base consists of large sections of text (i.e., each book chapter is a vector), you should probably make this 1 or 2 to limit the prompt length to a reasonable size.

• qdrant-score-threshold is the min match "score" the knowledge content must meet in order to be considerred "relevant". This depends on the quality of the knowledge text and the embedding model. In general, this score should be over 0.5 to reduce irrelevant context in the prompt.

The embedding model encodes and transforms text into vectors so that the can be stored, searched and retrieved. For different context material, you might need a different embedding model to achieve the optimal performance. The MTEB leaderboard is a good place to see the performance benchmarks of embedding models. You can find many of them in the gaianet organization on Huggingface.

Customizing prompts​

In config.json, you can also customize the prompts. Prompts are often tailored for the fine-tuned LLM or the knowledge base to generate optimal responses from the node.

The --system-prompt option sets a system prompt. It provides the background and "personality" of the node. Each API request can set its own system prompt.

The --rag-prompt is the prompt to be appended after the system prompt (or user query). It introduces the RAG context retrieved from the vector database, which follows it.

The --rag-policy option specifies where the rag-prompt and context should go. By default, its value is system-message and it puts the context in the system prompt. But you could also set it to last-user-message, which puts the rag-prompt and context in front of the latest message from the user.

Next steps​

Remember to re-initialize and re-start the node after you make configuration changes.

# If the node is running:
gaianet stop

gaianet init
gaianet start