Comfy-Org

/

MOSS-Image

Like 556

Follow

Comfy Org 15.4k

What is low VRAM in this case? The model is already so small and int8 is going to be a lot faster for most GPUs. GGUFs aren't and won't be supported, but 4bit quant can be added if there's actual use for it.

4bit quants make it possible for people with 6gb vram to run it with TE offload to CPU

It would be helpful if GGUF versions like the ones from https://huggingface.co/realrebelai/MOSS-Image_GGUFs were included in this package. That way, people using lower-VRAM GPUs could run the model more easily.

Of course this would be helpful, they will not support this because they hate anything that isn't Int8 ConvRot 😂 (I'm being ironic, they just don't want to waste their time, and yes, they don't like GGUF too).

Fairly certain the int8 version runs faster and with higher quality even with 6GB, at least when the GPU has int8 support on hardware level. GGUF is a needlessly slow format for diffusion models.

a 5.41 GB file is not gonna fit into 6GB VRAM since the weights arent the only thing eating up VRAM. Thats exactly why int4 convrot would be useful.

a 5.41 GB file is not gonna fit into 6GB VRAM since the weights arent the only thing eating up VRAM. Thats exactly why int4 convrot would be useful.

It doesn't have to fit all at once, ComfyUI has very sophisticated automatic offloading system that takes care of it.

a4w8 may be useful for some systems, and it can be added, there's just one update we need to do first to make it perform on the level of int8 speed wise.

But GGUF would be more than 2x slower (if hardware supports int8, as most do) on activations, offloading a bit should have much lower performance hit than that.

It would be helpful if GGUF versions like the ones from https://huggingface.co/realrebelai/MOSS-Image_GGUFs were included in this package. That way, people using lower-VRAM GPUs could run the model more easily.

If you want to use ComfyUI, stop thinking about GGUF. And stop following this dimwit influencer. He really doesn't know what he's doing.

Unless you're using stable-diffusion.cpp, you should stop using GGUF altogether.

ComfyUI has some seriously advanced offloading and memory management, so there's no need to worry about GGUF. Just use natively supported quantization formats like int8, fp8, etc and you're good to go bro.

a 5.41 GB file is not gonna fit into 6GB VRAM since the weights arent the only thing eating up VRAM. Thats exactly why int4 convrot would be useful.

It doesn't have to fit all at once, ComfyUI has very sophisticated automatic offloading system that takes care of it.

a4w8 may be useful for some systems, and it can be added, there's just one update we need to do first to make it perform on the level of int8 speed wise.

But GGUF would be more than 2x slower (if hardware supports int8, as most do) on activations, offloading a bit should have much lower performance hit than that.

My hardware supports Int8 (standard), it runs really fast but Int8 ConvRot always performs slower than gguf for me. Even fp8 is faster than it. This is ridiculous, considering the fact that my hardware doesn't support fp8 and needs to emulate its maths. I'm not sure if I am missing something or if Int8 ConvRot doesn't work very well for non ada and blackwell GPUs at all...

a 5.41 GB file is not gonna fit into 6GB VRAM since the weights arent the only thing eating up VRAM. Thats exactly why int4 convrot would be useful.

It doesn't have to fit all at once, ComfyUI has very sophisticated automatic offloading system that takes care of it.

a4w8 may be useful for some systems, and it can be added, there's just one update we need to do first to make it perform on the level of int8 speed wise.

But GGUF would be more than 2x slower (if hardware supports int8, as most do) on activations, offloading a bit should have much lower performance hit than that.

My hardware supports Int8 (standard), it runs really fast but Int8 ConvRot always performs slower than gguf for me. Even fp8 is faster than it. This is ridiculous, considering the fact that my hardware doesn't support fp8 and needs to emulate its maths. I'm not sure if I am missing something or if Int8 ConvRot doesn't work very well for non ada and blackwell GPUs at all...

Which GPU is this? And do you have pytorch with cu130 minimum installed?

a 5.41 GB file is not gonna fit into 6GB VRAM since the weights arent the only thing eating up VRAM. Thats exactly why int4 convrot would be useful.

It doesn't have to fit all at once, ComfyUI has very sophisticated automatic offloading system that takes care of it.

a4w8 may be useful for some systems, and it can be added, there's just one update we need to do first to make it perform on the level of int8 speed wise.

But GGUF would be more than 2x slower (if hardware supports int8, as most do) on activations, offloading a bit should have much lower performance hit than that.

My hardware supports Int8 (standard), it runs really fast but Int8 ConvRot always performs slower than gguf for me. Even fp8 is faster than it. This is ridiculous, considering the fact that my hardware doesn't support fp8 and needs to emulate its maths. I'm not sure if I am missing something or if Int8 ConvRot doesn't work very well for non ada and blackwell GPUs at all...

Which GPU is this? And do you have pytorch with cu130 minimum installed?

My GPU is a T4, and yes, cuda 13.0 is already installed.

Some numbers for this: I benchmarked the MOSS-Image SFT files on three GPU generations (A10G ≈ RTX 3090, L40S ≈ RTX 4090, RTX PRO 6000 ≈ RTX 5090) with the same 12 prompts and seeds, the template's settings (1024², 25 steps) and the bf16 text encoder for every run.

LPIPS is the distance from the bf16 image with the same seed (lower is closer). int8_convrot was the fastest or close to it and the closest to bf16 on all three, including the Ampere card. I couldn't include a GGUF: unsloth's MOSS-Image GGUF doesn't load with ComfyUI-GGUF (unknown architecture moss_image), and I didn't test low-VRAM offloading.

All 216 images, the per-image data and the scripts: https://huggingface.co/datasets/anla1234-itachi/comfyui-quant-bench
(I ran the benchmark. The fp8_scaled file was made with my open-source converter, PyTorch Precision Converter: https://github.com/angelolamonaca/PyTorch-Precision-Converter)