Instructions to use OpenGVLab/InternVL3-2B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use OpenGVLab/InternVL3-2B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="OpenGVLab/InternVL3-2B", trust_remote_code=True) messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("OpenGVLab/InternVL3-2B", trust_remote_code=True, dtype="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use OpenGVLab/InternVL3-2B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "OpenGVLab/InternVL3-2B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "OpenGVLab/InternVL3-2B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/OpenGVLab/InternVL3-2B
- SGLang
How to use OpenGVLab/InternVL3-2B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "OpenGVLab/InternVL3-2B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "OpenGVLab/InternVL3-2B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "OpenGVLab/InternVL3-2B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "OpenGVLab/InternVL3-2B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use OpenGVLab/InternVL3-2B with Docker Model Runner:
docker model run hf.co/OpenGVLab/InternVL3-2B
fix chat_template
Browse filesInternVL3 chat template fails on system messages with OpenAI content format
| vLLM | `0.22.1` |
| PyTorch | `2.11.0+cu129` |
| Transformers | `5.10.2` |
run vllm serve:
```bash
vllm serve /InternVL3-2B \
--trust-remote-code \
--host 0.0.0.0 \
--port 8000 \
--served-model-name internvl3-2b
```
when i crul serve such as:
```bash
curl http://my_servers_ip:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "internvl3-2b",
"messages": [
{
"role": "system",
"content": "你是一个专业的导游。"
},
{
"role": "user",
"content": "请用三句话介绍一下杭州"
}
],
"temperature": 0.7,
"max_tokens": 256
}'
```
error such as:
can only concatenate str (not \"list\") to str
- tokenizer_config.json +1 -1
|
@@ -268,7 +268,7 @@
|
|
| 268 |
"<|video_pad|>"
|
| 269 |
],
|
| 270 |
"bos_token": null,
|
| 271 |
-
"chat_template": "{%- if messages[0]['role'] == 'system' %}{{- '<|im_start|>system\n'
|
| 272 |
"clean_up_tokenization_spaces": false,
|
| 273 |
"eos_token": "<|im_end|>",
|
| 274 |
"errors": "replace",
|
|
|
|
| 268 |
"<|video_pad|>"
|
| 269 |
],
|
| 270 |
"bos_token": null,
|
| 271 |
+
"chat_template": "{%- if messages[0]['role'] == 'system' %}{{- '<|im_start|>system\n' }}{% if messages[0]['content'] is string %}{{ messages[0]['content'] }}{% else %}{% for content in messages[0]['content'] %}{% if content['type'] == 'text' %}{{ content['text'] }}{% endif %}{% endfor %}{% endif %}{{- '<|im_end|>\n' }}{%- else %}{{- '<|im_start|>system\n\u4f60\u662f\u4e66\u751f\u00b7\u4e07\u8c61\uff0c\u82f1\u6587\u540d\u662fInternVL\uff0c\u662f\u7531\u4e0a\u6d77\u4eba\u5de5\u667a\u80fd\u5b9e\u9a8c\u5ba4\u3001\u6e05\u534e\u5927\u5b66\u53ca\u591a\u5bb6\u5408\u4f5c\u5355\u4f4d\u8054\u5408\u5f00\u53d1\u7684\u591a\u6a21\u6001\u5927\u8bed\u8a00\u6a21\u578b\u3002<|im_end|>\n' }}{%- endif %}{% for message in messages %}{%- if messages[0]['role'] != 'system' or not loop.first %}{{'<|im_start|>' + message['role'] + '\n'}}{% if message['content'] is string %}{{ message['content'] }}{% else %}{% for content in message['content'] %}{% if content['type'] == 'image' %}{{ '<image>\n' }}{% elif content['type'] == 'video' %}{{ '<video>\n' }}{% elif content['type'] == 'text' %}{{ content['text'] }}{% endif %}{% endfor %}{% endif %}{{'<|im_end|>\n'}}{%- endif %}{% endfor %}{% if add_generation_prompt %}{{'<|im_start|>assistant\n' }}{% endif %}",
|
| 272 |
"clean_up_tokenization_spaces": false,
|
| 273 |
"eos_token": "<|im_end|>",
|
| 274 |
"errors": "replace",
|