[HN Gopher] Show HN: Use Code Llama as Drop-In Replacement for C...
___________________________________________________________________
Show HN: Use Code Llama as Drop-In Replacement for Copilot Chat
Hi HN, Code Llama was released, but we noticed a ton of questions
in the main thread about how/where to use it -- not just from an
API or the terminal, but _in your own codebase_ as a drop-in
replacement for Copilot Chat. Without this, developers don 't get
much utility from the model. This concern is also important
because benchmarks like HumanEval don't perfectly reflect the
quality of responses. There's likely to be a flurry of improvements
to coding models in the coming months, and rather than relying on
the benchmarks to evaluate them, the community will get better
feedback from people actually using the models. This means _real_
usage in _real_ , everyday workflows. We've worked to make this
possible with Continue (https://github.com/continuedev/continue)
and want to hear what you find to be the real capabilities of Code
Llama. Is it on-par with GPT-4, does it require fine-tuning, or
does it excel at certain tasks? If you'd like to try Code Llama
with Continue, it only takes a few steps to set up
(https://continue.dev/docs/walkthroughs/codellama), either locally
with Ollama, or through TogetherAI or Replicate's APIs.
Author : sestinj
Score : 154 points
Date : 2023-08-24 17:33 UTC (5 hours ago)
(HTM) web link (continue.dev)
(TXT) w3m dump (continue.dev)
| kristianp wrote:
| Are there any copilot-style plugins for Sublime Text? I still
| can't stomach the idea of vs code.
| dcre wrote:
| I don't see how things like Cursor[0] (a VS Code fork with a
| nicer Codex UI) have a moat or chance of profitability in the
| face of things like this. If you can run models locally or your
| employer can spin up their own instance for internal use on a
| decent server, why is anybody paying OpenAI? And that leaves
| Cursor with no cash flow to skim off of.
|
| [0]: https://www.cursor.so/
| int_19h wrote:
| You can't run anything even remotely close to GPT-4 locally.
|
| Large enterprises can run a private instance of it, but OpenAI
| is paid for that (indirectly via Microsoft).
| blackkettle wrote:
| Are you paying OpenAI? I'm not. Facebook's midterm game on
| "massive" AI seems to be "commoditize the competition" and ATM
| it looks like a good bet.
| dcre wrote:
| I am paying a couple bucks a month for GPT-4 API calls, but
| not for Copilot usage.
| lacoolj wrote:
| there's overhead to setting up your own local LLM instance (by
| that I mean both resource utilization and the setup process
| itself). so long as this continues (the setup portion at
| least), there will be people not willing to invest the time and
| rather pay cursor.
| zaptrem wrote:
| Their value proposition appears to be a fancy UI.
| Better/cheaper models would only improve their product and grow
| their addressable customer base.
| dcre wrote:
| Anybody can copy that UI once they figure out what works. No
| moat.
| mlsu wrote:
| I wouldn't discount good UI.
|
| I don't use Cursor, but consider that GPT-3 sat around for
| months and nobody was really talking about "AI doom" and
| "the lightcone" until OpenAI put a good chat UI layer over
| it.
|
| I would argue that actually capturing the value from these
| models lies _mostly_ in the UI: allowing the user to
| seamlessly and quickly extract useful information from the
| model.
| dcre wrote:
| Chat generally being a nice way to interact with the
| model is one thing, but if anything this proves my point:
| there are now dozens of little web UIs on GitHub that are
| as good as the official one or better. You put in an API
| key and you have fully recreated their UI. I even wrote a
| CLI[0] in 130 lines of Deno that I prefer over the web
| UI. People are trying to charge for them but I don't
| think any will succeed.
|
| [0]: https://gist.github.com/david-
| crespo/d9dbefe5a50c0f0da9ac3de...
| [deleted]
| say_it_as_it_is wrote:
| This would be great if I had a beefy machine with 32GB ram and 16
| cores. The alternatives presented are to use either Together or
| Standard api's to run llama-code. Both have a free tier but
| eventually you wind up paying. Has anyone calculated the
| financials and determined that the cost of these alternatives is
| less than copilot? Cost aside, there's a concern of quality.
|
| I should just buy a super computer..
| sestinj wrote:
| If you run Ollama locally, the cost is zero. With Replicate,
| you pay about $0.005 per second, so with an average of say 15
| seconds per request to generate the full response, and 25
| inputs per day, this comes out to ~$50/month. More expensive
| than classic Copilot, but also solving a different, potentially
| more valuable problem. Also likely that costs drop as we're
| seeing across the board. And at a minimum, the free tiers
| probably get you all the experimentation you need before
| figuring out which model is best :) (including Continue's free
| trial for GPT-4)
| freedomben wrote:
| Perfect reason to build a desktop IMHO :-)
|
| You'd be surprised how affordable it can be. Mine sits on a
| tailnet with my laptop and handles most of the workload for me.
| Especially while I'm at home, I barely even notice that my dev
| environment is not running "locally." Also doubles great as a
| host for audiobookshelf, jellyfin, archivebox, and more. I have
| virt manager set up too (client on laptop, virtmanager service
| running on desktop) so I can easily spin up all sorts of VMs. I
| love it. I spent about $2k but I bought top of the line.
| whimsicalism wrote:
| Yes, you likely will either have to pay a monthly fee or own
| compute yourself.
|
| Where did this expectation of free compute come from? Copilot
| costs too.
| ramesh31 wrote:
| >This would be great if I had a beefy machine with 32GB ram and
| 16 cores.
|
| Honestly this is a pretty standard workstation setup now. The
| days of 16GB being sufficient are pretty much over. RAM is
| cheap. And you can run the 7b model with well under that much.
| rtolsma wrote:
| When will tools/models like these start integrating with code
| servers and linters in IDEs instead of just yielding supercharged
| autocomplete?
| baggiponte wrote:
| The guy TJ De Vries works at Sourcegraph + contributes to
| neovim and he's building sg.nvim - a plug-in to hook
| Sourcegraph AND their code assistant Cody to neovim LSP.
| BaculumMeumEst wrote:
| surely steve yegge will not let this go unanswered
| _steve_yegge_ wrote:
| I guess I've been summoned.
|
| We're primarily focusing on VSCode, IntelliJ and Neovim for
| Cody. Of course I'll be working on an Emacs version, but
| that's kinda best-effort for now.
|
| As for the new crop of codegen models, they seem to be
| getting to parity with GPT/Claude/Bard-class models for
| code autocompletions, but not so much for other tasks.
|
| We're working on incorporating OSS models, but I'd be
| surprised if they're ready for prime-time this year. I
| think next year they'll be huge.
|
| Just my $0.02, take with a grain of salt. Shit moves fast.
| tjdevries wrote:
| I think "best-effort" would be a good tagline for emacs
| w0m wrote:
| I <3 neovim, but copilot has me in vscode more than i'd like.
| all of these tools are amazing; seriously excited about the
| future here.
| freedomben wrote:
| I use copilot in neovim[1]. It was remarkably simple to get
| installed. Highly recommend
|
| [1]: https://github.com/github/copilot.vim
| ramesh31 wrote:
| I just love this so much. It took what, maybe 6 hours for these
| model binaries to be publicly available? Never seen so much
| excitement in tech.
| Xiol32 wrote:
| Why is it necessary to sign up for two external services if this
| is running the LLM locally?
| daemonologist wrote:
| You can cobble together an OpenAI-esque server locally with
| llama.cpp: https://github.com/ggerganov/llama.cpp/issues/2766
| and the script here:
| https://www.reddit.com/r/LocalLLaMA/comments/15ak5k4/short_g...
| and use Continue's GGML Model option to query it:
| https://continue.dev/docs/customization#local-models-with-gg...
|
| (I've gotten this working, more or less, but don't have the
| hardware to make it practical so I can't give any feedback
| about Continue or the model.)
| mchiang wrote:
| No need to sign up for any service to use Continue with Ollama.
| This will do the inference all locally.
|
| I believe if you want to do the inference using Replicate or
| Together, you'll have to sign up for their services.
| Xiol32 wrote:
| I don't have a Mac.
|
| So it's either have a Mac or don't run locally if you want to
| use this service?
| mchiang wrote:
| Sorry about that. We are working on building the Windows
| and Linux versions. May I ask which specific OS you are on?
| sestinj wrote:
| ^exactly, Replicate or Together are only options that are
| potentially more convenient
| [deleted]
| [deleted]
| mchiang wrote:
| good to see code llama supported through Continue already.
|
| Are you seeing good results with code llama yet?
| sestinj wrote:
| It's definitely a knowledgable model. I'm seeing a bit of
| trouble with stopping at the right point and managing chat
| context, but this is all a matter of prompt engineering. Very
| promising, and exciting work to be done!
| jstummbillig wrote:
| > I'm seeing a bit of trouble with stopping at the right
| point and managing chat context, but this is all a matter of
| prompt engineering
|
| Can you explain what you mean by this, maybe provide some
| examples?
| sestinj wrote:
| Yeah, I asked it to write bubble sort in python, and it did
| so perfectly, but then decided to add a couple dozen
| parentheses to the end. I then followed up asking it to
| remove these, and it didn't seem to grasp that there was a
| conversation, as it just rewrote the same thing.
|
| All said, this was after like 5 minutes of playing with the
| 13B model, and was not using any kind of human/assistant
| formatting, hence the need for at least simple prompting to
| make that work (or fine-tuning if it isn't trained yet on
| conversational data)
| smith7018 wrote:
| I wish someone would make an IntelliJ/Android Studio plugin for
| Code Llama (or another local llm). I know both platforms have
| their own AI features but that involves sending my code to their
| servers which I'm really not a fan of.
| sestinj wrote:
| We've done work to set Continue up to be a JetBrains plugin--
| it's a matter of just a) figuring out the basics of putting a
| webview in a JetBrains plugin and b) implementing this class (h
| ttps://github.com/continuedev/continue/blob/main/extension/...)
| that communicates with the server. So long story short, we're
| planning on this soon and welcome PRs. It's a medium-sized
| task.
| javajosh wrote:
| Well, you might hold your nose for a few days while Copilot
| helps you write the IntelliJ plugin for Code Llama. Now _that_
| would be a good hacker news post...
___________________________________________________________________
(page generated 2023-08-24 23:01 UTC)