Fix trtllm-serve command for TensorRT-LLM ToT Main
Browse filesThe TensorRT-LLM command in the Usage section fails on TensorRT-LLM 1.4.0rc0 before the model loads: `Error: No such option '--extra_visual_gen_options'`.
In 1.4.0rc0, `trtllm-serve` takes the visual-gen config via `--visual_gen_args`, and the Wan2.2 config is `examples/visual_gen/serve/configs/wan22.yml` (`wan.yml` does not exist there).
Changes:
- Use `--visual_gen_args ./examples/visual_gen/serve/configs/wan22.yml`
- Note that the command runs from the root of a TensorRT-LLM source checkout, since the config path is relative to it
- Link the TensorRT-LLM visual_gen serving README
The SGLang command is unchanged. The same fix for `nvidia/Wan2.2-T2V-A14B-Diffusers-FP8` is in a separate PR.
References (TensorRT-LLM v1.4.0rc0):
- `--visual_gen_args` option: https://github.com/NVIDIA/TensorRT-LLM/blob/cc63e66daf52644f1441e117bb7ba67d20566016/tensorrt_llm/commands/serve.py#L1323
- Serving README: https://github.com/NVIDIA/TensorRT-LLM/blob/cc63e66daf52644f1441e117bb7ba67d20566016/examples/visual_gen/serve/README.md
|
@@ -109,11 +109,11 @@ This model was obtained by quantizing the weights and activations of Wan2.2-T2V-
|
|
| 109 |
|
| 110 |
## Usage
|
| 111 |
|
| 112 |
-
To serve this checkpoint with [
|
| 113 |
|
| 114 |
```sh
|
| 115 |
-
# TRTLLM
|
| 116 |
-
trtllm-serve nvidia/Wan2.2-T2V-A14B-Diffusers-NVFP4 --
|
| 117 |
# SGLang
|
| 118 |
PROMPT='A cat and a dog baking a cake together in a cozy kitchen. The cat carefully measures flour while the dog stirs batter in a glass bowl, sunlight through the window, smooth cinematic camera motion.'
|
| 119 |
|
|
|
|
| 109 |
|
| 110 |
## Usage
|
| 111 |
|
| 112 |
+
To serve this checkpoint with [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/visual_gen/serve/README.md):
|
| 113 |
|
| 114 |
```sh
|
| 115 |
+
# TRTLLM (run from the root of the TensorRT-LLM source checkout)
|
| 116 |
+
trtllm-serve nvidia/Wan2.2-T2V-A14B-Diffusers-NVFP4 --visual_gen_args ./examples/visual_gen/serve/configs/wan22.yml
|
| 117 |
# SGLang
|
| 118 |
PROMPT='A cat and a dog baking a cake together in a cozy kitchen. The cat carefully measures flour while the dog stirs batter in a glass bowl, sunlight through the window, smooth cinematic camera motion.'
|
| 119 |
|