| --- |
| license: llama3 |
| --- |
| |
| Quantized Llama 3 8B Instruct to Q40 format supported by [Distributed Llama](https://github.com/b4rtaz/distributed-llama). |
|
|
| ## License |
|
|
| Before download this repository please accept [Llama 3 Community License](https://llama.meta.com/llama3/license/). |
|
|
| ## How to run |
|
|
| 1. Clone this repository. |
| 2. Clone Distributed Llama: |
| ```sh |
| git clone https://github.com/b4rtaz/distributed-llama.git |
| ``` |
| 3. Build Distributed Llama: |
| ```sh |
| make main |
| ``` |
| 4. Run Distributed Llama: |
| ``` |
| ./main inference --prompt "Hello world" --steps 128 --weights-float-type q40 --buffer-float-type q80 --nthreads 4 --model path/to/dllama_meta-llama-3-8b_q40.bin --tokenizer path/to/dllama_meta-llama3-tokenizer.t |
| ``` |
|
|
| ### Chat Template |
|
|
| Please keep in mind this model expects the prompt to use the chat template of llama 3. |
