Instructions to use CohereLabs/c4ai-command-r-plus with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use CohereLabs/c4ai-command-r-plus with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="CohereLabs/c4ai-command-r-plus")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("CohereLabs/c4ai-command-r-plus")
model = AutoModelForCausalLM.from_pretrained("CohereLabs/c4ai-command-r-plus")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use CohereLabs/c4ai-command-r-plus with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "CohereLabs/c4ai-command-r-plus"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "CohereLabs/c4ai-command-r-plus",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/CohereLabs/c4ai-command-r-plus

SGLang

How to use CohereLabs/c4ai-command-r-plus with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "CohereLabs/c4ai-command-r-plus" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "CohereLabs/c4ai-command-r-plus",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "CohereLabs/c4ai-command-r-plus" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "CohereLabs/c4ai-command-r-plus",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use CohereLabs/c4ai-command-r-plus with Docker Model Runner:
```
docker model run hf.co/CohereLabs/c4ai-command-r-plus
```

It's down again

#57

by iNeverLearnedHowToRead - opened Jul 18, 2024

Discussion

iNeverLearnedHowToRead

Jul 18, 2024

Can't get any response from this model again. Just a heads up that it's down again.

shinshin369

Jul 18, 2024

I also can't get any response, since yesterday - the only thing I can see is the three dots indicator that later vanishes, leaving the response box completely empty.

shivalikasingh

Jul 18, 2024

Hey @iNeverLearnedHowToRead , @shinshin369

I'm assuming you're talking about the model being down on Hugging Chat ? Would be great if you can confirm.
I'll ping HF staff in the mean time and get this resolved.

shinshin369

Jul 18, 2024

@shivi
In my case, yes, I'm talking about the model. It was working fine today in the afternoon but now it is the same situation as I've mentioned previously - I see the three bouncing dots for a few minutes

iNeverLearnedHowToRead

Jul 18, 2024

•

edited Jul 18, 2024

I'm assuming you're talking about the model being down on Hugging Chat ? Would be great if you can confirm.

Yes, that was what I meant, and it's already working again :) thank you!

dsikdar

Jul 19, 2024

I am new to Hugging Face. I am trying to run this locally, but I get the following error message:

Do you know how I can get access to the model? I have already installed transformers jic.

shivalikasingh

Aug 1, 2024

Hi @dsikdar , did you accept the terms for using the model? It's important to accept the terms & then do hugging-face cli or notebook login using using access token. After that you should be able to access the model.

shivalikasingh changed discussion status to closed Aug 1, 2024

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment