> For the complete documentation index, see [llms.txt](https://developer.gaianet.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.gaianet.ai/getting-started-as-a-developer/setting-up-your-node/troubleshooting-your-node.md).

# Troubleshooting your node

Below are some common issues that you may run into with your node, here's how to fix them!

## The system cannot find CUDA libraries[​](https://docs.gaianet.ai/node-guide/troubleshooting#the-system-cannot-find-cuda-libraries) <a href="#the-system-cannot-find-cuda-libraries" id="the-system-cannot-find-cuda-libraries"></a>

Sometimes, the CUDA toolkit is installed in a non-standard location. The error message here is often not able to find `libcu*12`. For example, you might have CUDA installed with your Python setup. The following command would install CUDA into Python's enviornment.

```
sudo apt install python3-pip -y
pip3 install --upgrade fschat accelerate autoawq vllm
```

The easiest way to fix is simply to link those non-standard CUDA libraries to the standard location, like this:

```
ln -s /usr/local/lib/python3.10/dist-packages/nvidia/cublas/lib/libcublas.so.12 /usr/lib/libcublas.so.12
ln -s /usr/local/lib/python3.10/dist-packages/nvidia/cuda_runtime/lib/libcudart.so.12 /usr/lib/libcudart.so.12
ln -s /usr/local/lib/python3.10/dist-packages/nvidia/cublas/lib/libcublasLt.so.12 /usr/lib/libcublasLt.so.12
```

## Failed to recover from collection snapshot on Windows WSL[​](https://docs.gaianet.ai/node-guide/troubleshooting#failed-to-recover-from-collection-snapshot-on-windows-wsl)

On Windows WSL, you could see this error while running `gaianet init:`

`Import the Qdrant collection snapshot ... The process may take a few minutes. Please wait ... * [Error] Failed to recover from the collection snapshot. {"status":{"error":"Service internal error: Tokio task join error: task 1242 panicked"},"time":0.697784244}`

When you look into the `~/gaianet/log/init-qdrant.log` file, you could see this line of error:

```
2024-05-20T07:24:52.900895Z ERROR qdrant::startup: Panic occurred in file /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cgroups-rs-0.3.4/src/memory.rs at line 587: called `Result::unwrap()` on an `Err` value: Error { kind: ReadFailed("/sys/fs/cgroup/memory.high"), cause: Some(Os { code: 2, kind: NotFound, message: "No such file or directory" }) }  
```

The solution is to disable the `autoMemoryReclaim` feature in WSL. Step to turn on/off this feature:

1. Edit `C:\Users<Your user name>.wslconfig`
2. Remove or comment out `autoMemoryReclaim` in `[experimental]` section.

   <figure><img src="https://1203875980-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGjhz2bnyiEt8FqZJwwjV%2Fuploads%2FhE1a6qXxF0hm58WPyoSh%2Fdisable_autoMemoryReclaim_wsl-af7a70038bb2a972c29c0e1b2bc44cf8.png?alt=media&amp;token=29ee3bf5-d7fa-4e27-92c8-445fe6d12894" alt=""><figcaption></figcaption></figure>

You can learn more about it [here](https://github.com/GaiaNet-AI/gaianet-node/issues/46), this was discovered by [RoggeOhta](https://github.com/RoggeOhta).

## Failed to start the node with an error message `Port 8080 is in use. Exit ...`[​](https://docs.gaianet.ai/node-guide/troubleshooting#failed-to-start-the-node-with-an-error-message-port-8080-is-in-use-exit-)

You may see the following error when you run `gaianet start`.

```
gaianet start
[+] Checking the config.json file ...

You already have a private key.
[+] Starting LlamaEdge API Server ...

    Port 8080 is in use. Exit ...
```

The solution is to run `gaianet stop` first to kill all processes, and then run `gaianet start` to start the node.

## Load library failed: libgomp.so.1: cannot open shared object file: No such file or directory[​](https://docs.gaianet.ai/node-guide/troubleshooting#load-library-failed-libgompso1-cannot-open-shared-object-file-no-such-file-or-directory)

On Windows WSL, you may see this error when running `curl -sSfL 'https://github.com/GaiaNet-AI/gaianet-node/releases/latest/download/install.sh' | bash`

```
* Generate node IS
[2024-07-02 17:50:55.175] [error] loading failed: invalid path, Code: 0x20
[2024-07-02 17:50:55.175] [error]   Load library failed: libgomp.so.1: cannot open shared object file: No such file or directory
[2024-07-02 17:50:55.176] [error] loading failed: invalid path, Code: 0x20
[2024-07-02 17:50:55.176] [error]   Load library failed: libgomp.so.1: cannot open shared object file: No such file or directory
```

The error is caused by the lack of `libgomp.so.1`, a library that should be automatically installed on Ubuntu by default.

To solve this, you must install the `libgomp.so.1` library:

```
sudo apt-get update
sudo apt-get install libgomp1
```

If you're using CentOS, you can use:

```
yum install libgomp
```

This issue was fixed in `version 0.2.2`.

## Failed to remove the default collection[​](https://docs.gaianet.ai/node-guide/troubleshooting#failed-to-remove-the-default-collection) <a href="#failed-to-remove-the-default-collection" id="failed-to-remove-the-default-collection"></a>

```
Failed to remove the default collection. {"status":{"error":"Service internal error: No such file or directory (os error 2)"},"time":0.050924542}
```

It typically indicates that the Qdrant instance was not shut down properly before you try to init it again with a new snapshot. The solution is to stop the GaiaNet node first:

```
gaianet stop
```

Alternatively, you could mannually kill the processes from the terminal or in the OS's Actvity Monitor:

```
sudo pkill -9 qdrant
sudo pkill -9 wasmedge
sudo pkill -9 frpc
```

Then you can run `gaianet init` and then `gaianet start` again.

## File I/O error[​](https://docs.gaianet.ai/node-guide/troubleshooting#file-io-error) <a href="#file-io-error" id="file-io-error"></a>

```
    * Import the Qdrant collection snapshot ...
      The process may take a few minutes. Please wait ...
    * [Error] Failed to recover from the collection snapshot. An error occurred processing field `snapshot`: File I/O error: Operation not permitted (os error 1) 
```

It typically indicates that the Qdrant instance was not shut down properly before you try to init it again with a new snapshot. The solution is to stop the GaiaNet node first:

```
gaianet stop
```

Alternatively, you could mannually kill the processes from the terminal or in the OS's Actvity Monitor.

```
sudo pkill -9 qdrant
sudo pkill -9 wasmedge
sudo pkill -9 frpc
```

Then you can run `gaianet init` and then `gaianet start` again.

## The "Failed to open the file" Error[​](https://docs.gaianet.ai/node-guide/troubleshooting#the-failed-to-open-the-file-error) <a href="#the-failed-to-open-the-file-error" id="the-failed-to-open-the-file-error"></a>

```
Warning: Failed to open the file 
Warning: https://huggingface.co/datasets/max-id/gaianet-qdrant-snapshot/resolve
Warning: /main/consensus/consensus.snapshot: No such file or directory
curl: (23) Failure writing output to destination
```

The reason for this type of error is a mis-configured `config.json` file. The solution is to delete the comments in `config.json` and re-run the `gaianet init` command.

## The "Too many open files" Error on macOS[​](https://docs.gaianet.ai/node-guide/troubleshooting#the-too-many-open-files-error-on-macos) <a href="#the-too-many-open-files-error-on-macos" id="the-too-many-open-files-error-on-macos"></a>

When running `gaianet init` to initialize a new node on macOS, you may encounter an error related to snapshot recovery if your snapshot contains a large amount of text. The error message may be the following:

```
 * [Error] Failed to recover from the collection snapshot. {"status":{"error":"Service internal error: Too many open files (os error 24)"},"time":1.574064833}
    * [Error] Failed to recover from the collection snapshot. {"status":{"error":"Service internal error: Too many open files (os error 24)"},"time":1.574064833}
```

This issue is caused by the default file descriptor (FD) limit on macOS, which is set to a relatively low value of 256.

To resolve this issue, you can increase the default FD limit on your system. To do so, run the following command:

```
ulimit -n 10000
```

This will temporarily set the FD limit to 10,000. Next, use `gaianet init` and `gaianet start` commands in the SAME terminal.

## Permission denied when use the installer script to install WasmEdge[​](https://docs.gaianet.ai/node-guide/troubleshooting#permission-denied-when-use-the-installer-script-to-install-wasmedge) <a href="#permission-denied-when-use-the-installer-script-to-install-wasmedge" id="permission-denied-when-use-the-installer-script-to-install-wasmedge"></a>

When running `curl -sSfL 'https://github.com/GaiaNet-AI/gaianet-node/releases/latest/download/install.sh' | bash` to install GaiaNet node software, you may meet the permisson denied error especially installing the WasmEdge runtime.

<figure><img src="https://1203875980-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGjhz2bnyiEt8FqZJwwjV%2Fuploads%2FAf06BGeoH7moRMf75H6B%2Ftroubleshooting-01-38a832c2c9b16055602256ce4af65948.png?alt=media&amp;token=c3c2678e-53e5-4cda-97a1-96eacdee67ea" alt=""><figcaption></figcaption></figure>

This error is caused by the lack of `/tmp` write permission. You can use `--tmpdir` to specify where you want to install the WasmEdge runtime. Please note, you will need to have written permission to the `<YOUR_PATH>` folder.

```
curl -sSfL 'https://github.com/GaiaNet-AI/gaianet-node/releases/latest/download/install.sh' | bash -s -- --tmpdir YOUR_PATH
```

This problem is fixed in `version 0.2.3`.