NVFP4
This is great, any chance of a NVFP4 quant for Blackwell GPU’s?
We are discussing this with NVIDIA
Please dont forget to make nvfp4 mlx quant for apple users. Thank you
Awesome
I've been trying every which way I can think of to quantize this to NVFP4 and so far no amount of hacking on llm-compressor or ModelOpt results in a good quant. The 3D MoE layers are not cooperating.
Please dont forget to make nvfp4 mlx quant for apple users. Thank you
You can use the int4 version for now, just in case you did not notice there is an int4 version
I've been trying every which way I can think of to quantize this to NVFP4 and so far no amount of hacking on llm-compressor or ModelOpt results in a good quant. The 3D MoE layers are not cooperating.
Ya, no luck here. I got a model to compile, but the output was garbage.
Please dont forget to make nvfp4 mlx quant for apple users. Thank you
You can use the int4 version for now, just in case you did not notice there is an int4 version
The int4 works great, however it's in gguf format and llama.cpp performance is terrible in prompt-processing and batching request. I would like to use it in VLLM but there is no way to load the gguf into it as it's not yet supported, and from the available quants, not a single one works on vllm or sglang for Ampere archs. There is a int4 gptq model out there but it produces garbage output on my system.