Headache with LocalAI reddit / selfhosted ---------------------------------------------------------------------- RudePiece · 2023-12-23 18:33 I am fairly new in the homelab/selfhosted space. And one of the projects I have been dying to run is LocalAI(running any model for now tbh). It has been nothing short of a headache, here is where I am currently stuck: Server specs: Dell R620, 384GB Ram, dual Xeon 2690 I cloned the repo and did a docker compose up to get it started, I then made the call to localhost/models/apply using the following: { "url": "github:go-skynet/model-gallery/gpt4all-j.yaml", "name": "gpt-3.5-turbo" } It worked fine and the job completed, files showing in /Models as expected. I run localhost:8080/models and get: { "object": "list", "data": [ { "id": "gpt-3.5-turbo", "object": "model" } ] } But as soon as I send the following to localhost /v1/chat/completions it throws me an error: { "model": "gpt-3.5-turbo.yml", "messages": [{"role": "user", "content": "How are you?"}], "temperature": 0.9 } response { "error": { "code": 500, "message": "could not load model -  all backends returned error: 29 errors occurred:\n\t* grpc service no t ready\n\t* could not load model: rpc error: code = Unavailable desc  = error reading from server: EOF\n\t* grpc service not ready\n\t* coul d not load model: rpc error: code = Unknown desc = failed loading mode l\n\t* could not load model: rpc error: code = Unknown desc = failed l oading model\n\t* could not load model: rpc error: code = Unknown desc  = failed loading model\n\t* could not load model: rpc error: code = U nknown desc = failed loading model\n\t* could not load model: rpc erro r: code = Unknown desc = failed loading model\n\t* could not load mode l: rpc error: code = Unknown desc = failed loading model\n\t* could no t load model: rpc error: code = Unknown desc = failed loading model\n\ t* could not load model: rpc error: code = Unknown desc = failed loadi ng model\n\t* could not load model: rpc error: code = Unknown desc = f ailed loading model\n\t* could not load model: rpc error: code = Unkno wn desc = failed loading model\n\t* could not load model: rpc error: c ode = Unavailable desc = error reading from server: EOF\n\t* grpc serv ice not ready\n\t* could not load model: rpc error: code = Unknown des c = stat /models/gpt-3.5- turbo.yml: no such file or directory\n\t* could not load model: rpc er ror: code = Unknown desc = unsupported model type /models/gpt-3.5- turbo.yml (should end with .onnx)\n\t* grpc process not found: /tmp/lo calai/backend_data/backend- assets/grpc/build/backend/python/transformers-musicgen/run.sh. some ba ckends(stablediffusion, tts) require LocalAI compiled with GO_TAGS\n\t * grpc process not found: /tmp/localai/backend_data/backend-assets/grp c/build/backend/python/sentencetransformers/run.sh. some backends(stab lediffusion, tts) require LocalAI compiled with GO_TAGS\n\t* grpc proc ess not found: /tmp/localai/backend_data/backend-assets/grpc/build/bac kend/python/vllm/run.sh. some backends(stablediffusion, tts) require L ocalAI compiled with GO_TAGS\n\t* grpc process not found: /tmp/localai /backend_data/backend-assets/grpc/build/backend/python/bark/run.sh. so me backends(stablediffusion, tts) require LocalAI compiled with GO_TAG S\n\t* grpc process not found: /tmp/localai/backend_data/backend- assets/grpc/build/backend/python/vall-e- x/run.sh. some backends(stablediffusion, tts) require LocalAI compiled  with GO_TAGS\n\t* grpc process not found: /tmp/localai/backend_data/b ackend-assets/grpc/build/backend/python/sentencetransformers/run.sh. s ome backends(stablediffusion, tts) require LocalAI compiled with GO_TA GS\n\t* grpc process not found: /tmp/localai/backend_data/backend-asse ts/grpc/build/backend/python/petals/run.sh. some backends(stablediffus ion, tts) require LocalAI compiled with GO_TAGS\n\t* grpc process not  found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/pyt hon/diffusers/run.sh. some backends(stablediffusion, tts) require Loca lAI compiled with GO_TAGS\n\t* grpc process not found: /tmp/localai/ba ckend_data/backend-assets/grpc/build/backend/python/exllama2/run.sh. s ome backends(stablediffusion, tts) require LocalAI compiled with GO_TA GS\n\t* grpc process not found: /tmp/localai/backend_data/backend-asse ts/grpc/build/backend/python/transformers/run.sh. some backends(stable diffusion, tts) require LocalAI compiled with GO_TAGS\n\t* grpc proces s not found: /tmp/localai/backend_data/backend-assets/grpc/build/backe nd/python/autogptq/run.sh. some backends(stablediffusion, tts) require  LocalAI compiled with GO_TAGS\n\t* grpc process not found: /tmp/local ai/backend_data/backend-assets/grpc/build/backend/python/exllama/run.s h. some backends(stablediffusion, tts) require LocalAI compiled with G O_TAGS\n\n", "type": "" } } If I remove the .yml in the call I get: { "error": { "code": 500, "message": "could not load model: rpc error: code = Unknown desc = failed loading model", "type": "" } } ​ ​ Pleeease can someone help me out with this. ---------------------------------------------------------------------- dapaxx · 2023-12-23 20:17 From my experience with LocalAI you have to restart the container after every config change. Did you do that? Apart from that, what’s in the logs? ---------------------------------------------------------------------- firewallzer0 · 2023-12-25 17:51 I am fairly new to using LocalAI myself, but I don't believe you have to pass the yml extension when calling the model. I just call the exact file name of the model I am trying to load. Hope this helps.