https://github.com/ggerganov/llama.cpp/pull/1998 Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Case Studies + Customer Stories + Resources * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this user All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. {{ message }} ggerganov / llama.cpp Public * Notifications * Fork 4.7k * Star 33.7k * Code * Issues 328 * Pull requests 82 * Discussions * Actions * Projects 4 * Wiki * Security * Insights More * Code * Issues * Pull requests * Discussions * Actions * Projects * Wiki * Security * Insights New issue Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community. Pick a username [ ] Email Address [ ] Password [ ] [ ] Sign up for GitHub By clicking "Sign up for GitHub", you agree to our terms of service and privacy statement. We'll occasionally send you account related emails. Already on GitHub? Sign in to your account Jump to bottom Simple webchat for server #1998 Merged Green-Sky merged 18 commits into ggerganov:master from tobi: server-simple-web Jul 4, 2023 Merged Simple webchat for server #1998 Green-Sky merged 18 commits into ggerganov:master from tobi: server-simple-web Jul 4, 2023 +3,416 -8 Conversation 66 Commits 18 Checks 22 Files changed 9 Conversation This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters Show hidden characters tobi Copy link Sponsor Collaborator @tobi tobi commented Jun 26, 2023 * edited I put together a simple web-chat that demonstrates how to use the SSE (ish) streaming in the server example. I also went ahead and served it from the root url, to make the server a bit more approachable. I tried to match the spirit of llama.cpp and used minimalistic js dependencies and went with the ozempic css style of ggml.ai. Initially I went for no-js dependencies but gave up and used a few minimal that i'm importing from js cdns instead of adding them here. Let me know if you agree with this approach. I needed microsoft's fetch-event-source for using event-source over POST (super disappointed that browsers don't support that, actually) and preact+htm for keeping my sanity with all this state,. The upshot is that everything is in one small html file. Speaking of- there is probably a better (and less fragile) way to include the server.html in the cpp binary, but it's been 25 years since I worked with cpp tooling. (updated screenshot) image Sorry, something went wrong. 36 FNsi, disdi, Senjai, antimora, sg3510, mlmabie, franciscogaluppo, thomaspaulmann, langley, sohooo, and 26 more reacted with thumbs up emoji [?] 13 ggerganov, koogle, vincentweisser, patw, jaymeh13, TheBlapse, norohind, fgblanch, bramses, starmorph, and 3 more reacted with heart emoji 18 Green-Sky, WangHaoranRobin, gianpaj, JackJollimore, 2187Nick, badcc, afloresescarcega, mlmabie, lucidrains, phelipetls, and 8 more reacted with rocket emoji All reactions * 36 reactions * [?] 13 reactions * 18 reactions @tobi tobi changed the title [DEL:Server simple web:DEL] [INS:Simple webchat for server:INS] Jun 26, 2023 ggerganov ggerganov approved these changes Jun 26, 2023 View reviewed changes Copy link Owner @ggerganov ggerganov left a comment There was a problem hiding this comment. Choose a reason for hiding this comment The reason will be displayed to describe this comment to others. Learn more. [Choose a reason] Hide comment Love this! Initially I went for no-js dependencies but gave up and used a few minimal that i'm importing from js cdns instead of adding them here. Let me know if you agree with this approach. I think it's good Speaking of- there is probably a better (and less fragile) way to include the server.html in the cpp binary, but it's been 25 years since I worked with cpp tooling. I guess it would be useful to specify the HTTP root path from the command line arguments instead of hard coding the path. But we can fix this later. Approving and letting the "server team" take a look and merge Sorry, something went wrong. 4 lin72h, vincentweisser, ignoramous, and I-NJ reacted with eyes emoji All reactions * 4 reactions @ggerganov ggerganov requested review from SlyEcho and Green-Sky June 26, 2023 07:51 @slaren Copy link Collaborator slaren commented Jun 26, 2023 I think this is a good idea, but the html file should be in the binary. This will not work with the automatic builds because they don't include the contents of the examples directory. All reactions Sorry, something went wrong. @IgnacioFDM Copy link IgnacioFDM commented Jun 26, 2023 * edited IMHO having the js dependencies locally would be better, so it works without an internet connection, and solves the risk of malicious js. 23 leeola, jminardi, ww9, patrakov, rictic, ZeroCool2u, willbr, subins2000, ibehnam, flbn, and 13 more reacted with thumbs up emoji All reactions * 23 reactions Sorry, something went wrong. @SlyEcho Copy link Sponsor Collaborator SlyEcho commented Jun 26, 2023 I have done something like this before, I was serving HTML files in a C HTTP server. There was a CMake option to either build them in or to read them from disk. Reading from files is useful for development because you don't need to rebuild and restart the server. But building them in requires creating a small program that can hexdump the file into a C array definition. Overall pretty complex and then we have the Makefile as well... using event-source over POST (super disappointed that browsers don't support that, actually) Maybe we could add an endpoint with GET and query parameters? All reactions Sorry, something went wrong. @Green-Sky Copy link Collaborator Green-Sky commented Jun 26, 2023 requires creating a small program that can hexdump the file into a C array definition. should be pretty simple, i don't touch Makefiles directly often, how bad are custom target? or we ship the generated. (more reading here https://thephd.dev/finally-embed-in-c23) All reactions Sorry, something went wrong. @SlyEcho Copy link Sponsor Collaborator SlyEcho commented Jun 26, 2023 should be pretty simple, i don't touch Makefiles directly often, how bad are custom target? I'd say Makefiles are a lot easier for this than CMake but it's just added complexity. @tobi, How hard would it be for you to jam the HTML file contents into the .cpp file? All reactions Sorry, something went wrong. @Green-Sky Copy link Collaborator Green-Sky commented Jun 26, 2023 * edited after some thinking, i realized that we have pure text file(s), which means we only need to pre and post-fix the file with raw string literal markers eg: echo "R\"htmlraw(" > html_build.cpp cat index.html >> html_build.cpp echo ")htmlraw\"" >> html_build.cpp in server.cpp: const char* html_str = #include "html_build.cpp" ; edit: resulting html_build.cpp: R"htmlraw()htmlraw" 2 tobi and ignoramous reacted with thumbs up emoji All reactions * 2 reactions Sorry, something went wrong. Green-Sky Green-Sky reviewed Jun 26, 2023 View reviewed changes examples/server/server.html Outdated Show resolved Hide resolved @Green-Sky Copy link Collaborator Green-Sky commented Jun 26, 2023 Ok, gave it a go (running it) and found an issue. when ever the window looses focus (switching windows), it restarts the current outputting promt. see screencap image (i switched a couple of times back and forth) All reactions Sorry, something went wrong. @tobi Copy link Sponsor Collaborator Author tobi commented Jun 26, 2023 @tobi, How hard would it be for you to jam the HTML file contents into the .cpp file? Simple enough. I was hoping to hear that there is some kind of #embed thing that works in all the cpp compilers that we care about. Crazy that it took till C23 to get that into the standard. I can just include it. I can also just embed one dependency js and call it a day. The best argument for keeping it in the html file is to allow people to hack on it easier. I think this could become a really good chatbot UX if we are welcome to contributors. It's got good bones All reactions Sorry, something went wrong. @SlyEcho Copy link Sponsor Collaborator SlyEcho commented Jun 26, 2023 #embed is not gonna work because it's too new. Yes, it will be harder to develop, but you can also run a simple web server like with Python while developing it. We can improve it later. All reactions Sorry, something went wrong. @howard0su Copy link Collaborator howard0su commented Jun 26, 2023 Check this cmake script: https://gist.github.com/sivachandran/3a0de157dccef822a230 I am also thinking if we should use the same tech to embed OpenCL kernels. The current approach which mixed kernel and normal C code will get into more maintenance headache. All reactions Sorry, something went wrong. @SlyEcho Copy link Sponsor Collaborator SlyEcho commented Jun 26, 2023 cmake script Cool but we also have to support pure Makefile. All reactions Sorry, something went wrong. @Green-Sky Copy link Collaborator Green-Sky commented Jun 26, 2023 i feel ignored we are not dealing with binary files here so my #1998 (comment) solution is simple 3 text file concats. pretty sure it wont get much simpler :) 1 tobi reacted with thumbs up emoji All reactions * 1 reaction Sorry, something went wrong. @SlyEcho Copy link Sponsor Collaborator SlyEcho commented Jun 26, 2023 3 text file concats Does it work in Windows? All reactions Sorry, something went wrong. @Green-Sky Copy link Collaborator Green-Sky commented Jun 26, 2023 3 text file concats Does it work in Windows? if you use make on windows, you likely also have some coreutils installed (echo and cat) cmake has built in functions for read/write/append file :) All reactions Sorry, something went wrong. @ggerganov Copy link Owner ggerganov commented Jun 26, 2023 For me, the greatest value of this example is that it demonstrates a minimalistic way of how to implement a basic HTML/JS client that communicates with the server using just a browser without having to install node or curl. How the client is served can be solved in many different ways, depending on the needs of the specific project. I recommend to merge the example as it is and potentially add improvements from master 5 slaren, jessejohnson, RahulVivekNair, HanClinto, and rictic reacted with thumbs up emoji All reactions * 5 reactions Sorry, something went wrong. @Green-Sky Copy link Collaborator Green-Sky commented Jun 26, 2023 For me, the greatest value of this example is that it demonstrates a minimalistic way of how to implement a basic HTML/ JS client that communicates with the server using just a browser without having to install node or curl. How the client is served can be solved in many different ways, depending on the needs of the specific project. I recommend to merge the example as it is and potentially add improvements from master Agree, except we should really not hard code the path to the html. we basically ship the server, and that would look funky. @tobi would it be too much to ask to implement the html root cli parameter for the server executable? or for the fasttrack, if the hardcoded .html file could not be loaded (!file.is_open()) to fall back to the previous html string? All reactions Sorry, something went wrong. @tobi Copy link Sponsor Collaborator Author tobi commented Jun 26, 2023 sure, i'll try to do that tonight. All reactions Sorry, something went wrong. @tobi Copy link Sponsor Collaborator Author tobi commented Jun 27, 2023 OK so I did basically all of those things. There is now a --path param that you can point to any directory and static files will be served from this. I also added a deps.sh which just bakes the index.html and index.js into .hpp file (as per @Green-Sky's suggestion). So really, you can launch ./server from the llama.cpp folder and it will use ./examples/server/public directory, copy the . /server file to tmp and it will just use the baked ones, or use --path to work on your own UX. The only downside is that we duplicate some files in git here, because of the baked .cpp files. But the deps are so small that it probably doesn't matter. It would be slightly cleaner to go and make deps.sh a build step in cmake and makefile, but... well... I ran out of courage. 2 IgnacioFDM and Green-Sky reacted with thumbs up emoji All reactions * 2 reactions Sorry, something went wrong. @tobi Copy link Sponsor Collaborator Author tobi commented Jun 27, 2023 @ggerganov server is in reasonably good shape overall. Maybe time for including it in the default build? All reactions Sorry, something went wrong. @IgnacioFDM Copy link IgnacioFDM commented Jun 27, 2023 I like the current approach, with the website embedded in the binary for simplicity, but also the option to serve from a directory, to improve iteration time and to allow user customization without recompiling. It also includes the js dependencies locally. I agree with merging this in its current state. Further improvements can be done in future PRs. All reactions Sorry, something went wrong. SlyEcho SlyEcho requested changes Jun 27, 2023 View reviewed changes Copy link Sponsor Collaborator @SlyEcho SlyEcho left a comment There was a problem hiding this comment. Choose a reason for hiding this comment The reason will be displayed to describe this comment to others. Learn more. [Choose a reason] Hide comment Actually, could you move the generated files next to the source files into the public folder. If we get more files it will keep it neater to keep to the same directory structure. Sorry, something went wrong. All reactions @ggerganov Copy link Owner ggerganov commented Jun 27, 2023 server is in reasonably good shape overall. Maybe time for including it in the default build? Yes, let's do that. Originally, I insisted to put it behind an option since it was bringing the boost library as a dependency, which is a very big burden. Now that the implementation is so self-contained and minimal, we should enable the build by default and maintain it long term 3 SlyEcho, IgnacioFDM, and Green-Sky reacted with thumbs up emoji All reactions * 3 reactions Sorry, something went wrong. Green-Sky Green-Sky requested changes Jun 27, 2023 View reviewed changes examples/server/index.html.cpp Outdated Show resolved Hide resolved examples/server/server.cpp Outdated Show resolved Hide resolved @Green-Sky Copy link Collaborator Green-Sky commented Jun 27, 2023 If you pull from master, the ci issues should go away. All reactions Sorry, something went wrong. @tobi Copy link Sponsor Collaborator Author tobi commented Jun 27, 2023 Actually, could you move the generated files next to the source files into the public folder. If we get more files it will keep it neater to keep to the same directory structure. that would make it possible to request the files at /index.html.cpp - still want that? All reactions Sorry, something went wrong. 24 hidden items Load more... @tobi Copy link Sponsor Collaborator Author tobi commented Jul 3, 2023 The second part I mentioned should be fixed, though. Otherwise the performance will suffer a lot. The important part here is the whitespace handling: due to the way the tokenizer works the model can return completions with a space added in the front. You can see what is happening if you dump the payload variable. I'll try to see if there is some easy fix. Really good catch. This is now fixed. 2 lin72h and JackJollimore reacted with thumbs up emoji All reactions * 2 reactions Sorry, something went wrong. @Green-Sky Copy link Collaborator Green-Sky commented Jul 3, 2023 i tried to reproduce it across windows, osx, chrome, and safari and couldn't. The new code does not have that issue orca-mini-3b.ggmlv3.q4_0.bin performing pretty good: --------------------------------------------------------------------- User: Tell me a long story about TeddyBear meeting a LLaMa llama: Once upon a time, in a small village near the mountains, there lived a little girl named Lily. She was known for her kindness and love towards all living beings. One day, while playing in the forest, she stumbled upon a teddy bear that had lost its way. The teddy bear was very sad and lonely, so Lily decided to take it home with her. As they spent more time together, Lily realized how much the teddy bear needed a friend like her. They would play games, share stories, and snuggle up together under the stars. Their friendship grew stronger every day, and soon the teddy bear became Lily's best companion. One day, as Lily was walking through the forest, she heard a strange sound coming from a nearby cave. Curious, she ventured inside and found a group of animals gathered around a fire. Among them was a llamma (a type of sheep) who had been lost and alone for days. Lily immediately knew what to do. She took the lllama back to her house, where she fed him some delicious soup and gave him a warm bed to sleep in. With the help of Lily and her kind heart, the lllama was saved and reunited with his friends. From then on, he was never alone again. And so, TeddyBear met LLaMa, and their friendship blossomed into a beautiful story that will be remembered forever. 3 tobi, ggerganov, and colinmegill reacted with thumbs up emoji All reactions * 3 reactions Sorry, something went wrong. Green-Sky Green-Sky approved these changes Jul 3, 2023 View reviewed changes Copy link Collaborator @Green-Sky Green-Sky left a comment There was a problem hiding this comment. Choose a reason for hiding this comment The reason will be displayed to describe this comment to others. Learn more. [Choose a reason] Hide comment Did not see anything wrong. Very nice addition to our collection. :) Sorry, something went wrong. 2 tobi and lin72h reacted with hooray emoji All reactions * 2 reactions SlyEcho SlyEcho approved these changes Jul 4, 2023 View reviewed changes Copy link Sponsor Collaborator @SlyEcho SlyEcho left a comment There was a problem hiding this comment. Choose a reason for hiding this comment The reason will be displayed to describe this comment to others. Learn more. [Choose a reason] Hide comment Yeah, it now looks good to me too. Sorry, something went wrong. All reactions @Green-Sky Copy link Collaborator Green-Sky commented Jul 4, 2023 @tobi i pushed a ci fix to master, if you could pull and rerun ci? All reactions Sorry, something went wrong. tobi and others added 18 commits July 4, 2023 09:14 @tobi expose simple web interface on root domain ... 627d3ba demonstrates how to use the stream option of generate. @tobi tighter b07b271 @ggerganov @tobi server : clear trailing whitespace c1cb0e1 @tobi minor aesthetic fixes e3fba85 @tobi embed index and add --path for choosing static dir 022bf2b @tobi newline police c8cedf5 @tobi switched to fprintf logging and to access_log a30d4b2 @tobi allow server to multithread ... 7a38956 because web browsers send a lot of garbage requests we want the server to multithread when serving 404s for favicon's etc. To avoid blowing up llama we just take a mutex when it's invoked. @tobi improvements 0f95689 @tobi revert log format changes e192f95 @tobi remove need for @microsoft/fetch-event-source dep (-7kb) 34fc3c7 @tobi let's try this with the xxd tool instead and see if msvc is happier w... ... dc7dd08 ...ith that @tobi enable server in Makefiles 8e1b04d @tobi add /completion.js file to make it easy to use the server from js dd1df3f @tobi slightly nicer css 98e612c @tobi rework state management into session, expose historyTemplate to settings fedce00 @tobi fix mobile, fix missing prompt cache eee6d69 @tobi basic response formatting c19daa4 @tobi tobi force-pushed the server-simple-web branch from b970292 to c19daa4 Compare July 4, 2023 13:21 Hide details View details @Green-Sky Green-Sky merged commit 7ee76e4 into ggerganov:master Jul 4, 2023 22 checks passed @jessejohnson jessejohnson mentioned this pull request Jul 4, 2023 Update Server Instructions For Web Front End #2103 Merged @rain-1 Copy link rain-1 commented Jul 5, 2023 Is it chat only or is there also a text completion UI? All reactions Sorry, something went wrong. @SlyEcho Copy link Sponsor Collaborator SlyEcho commented Jul 5, 2023 It is just chat for now. All reactions Sorry, something went wrong. @jarombouts Copy link jarombouts commented Jul 5, 2023 I tried to match the spirit of llama.cpp and used minimalistic js dependencies and went with the ozempic css style of ggml.ai. I have nothing of use to add to this discussion but just want to point out that your use of ozempic as an adjective is blowing my mind 5 tobi, Green-Sky, GeorgeIpsum, bkitano, and jchv reacted with laugh emoji All reactions * 5 reactions Sorry, something went wrong. @tobi tobi mentioned this pull request Jul 5, 2023 Expose generation timings from server & update completions.js #2116 Merged Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment Reviewers @developit developit developit left review comments @SlyEcho SlyEcho SlyEcho approved these changes @ggerganov ggerganov ggerganov approved these changes @Green-Sky Green-Sky Green-Sky approved these changes Assignees No one assigned Labels None yet Projects None yet Milestone No milestone Development Successfully merging this pull request may close these issues. None yet 10 participants @tobi @slaren @IgnacioFDM @SlyEcho @Green-Sky @howard0su @ggerganov @rain-1 @jarombouts @developit Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time.