Cloudflare Tunnel to expose LLMA with VLLM on a local GPU machine.
reddit / selfhosted

----------------------------------------------------------------------
Ok_Moose_110 · 2023-10-20 00:21

Hello,

For my project I am looking to host LLAMA on VLLM on a GPU machine
locally. And then expose this endpoint via Cloudflare tunnel. This
worked well with non-streaming generation on VLLM. However with stream
generation I am getting following error:

error cloudflared Failed to handle QUIC stream {“connIndex”:0,
“error”:“context canceled”,
“ip”:“[198.49.192.17](https://198.49.192.17)”}


Has anyone tried self hosting LLM with option of streaming tokens
back.