vllm support ?
Will this model be supported by vllm ?
Yes please+1
Nemotron ASR uses a FastConformer-RNNT architecture, which is fundamentally different from the autoregressive transformer decoder architecture that vLLM is designed to serve. vLLM's core optimizations (PagedAttention, continuous batching of KV-cache) target LLM-style token-by-token generation, not encoder-decoder transducer models.
For high-throughput serving of this model the options are:
NIM: The Nemotron ASR Streaming NIM handles batching, TensorRT optimization, and gRPC streaming out of the box. It sustains 275 concurrent streams on a single H100 with 80ms low latency.
NeMo + Triton: Deploy with NVIDIA Triton Inference Server for custom serving setups with dynamic batching.
ONNX / sherpa-onnx: Community members contributions
vLLM is not the right serving framework for transducer ASR models, but NIM and Triton provide the needed throughput optimizations for this architecture.
I have 2xNvidia 6000 pros and 1x5060ti
Can you please share a startup commands that I can use here for inferencing this locally? Does it expose an openai compatible endpoint?
Thanks !