[HN Gopher] The lifecycle of a code AI completion
___________________________________________________________________
The lifecycle of a code AI completion
Author : tosh
Score : 86 points
Date : 2024-04-07 08:55 UTC (14 hours ago)
(HTM) web link (sourcegraph.com)
(TXT) w3m dump (sourcegraph.com)
| mayank wrote:
| Very interesting! I wonder to what extent this assumption is true
| in tying completions to traditional code autocomplete.
|
| > One of the biggest constraints on the retrieval implementation
| is latency
|
| If I'm getting a multi line block of code written automagically
| for me based on _comments_ and the like, I'd personally value
| quality over latency and be more than happy to wait on a spinner.
| And I'd also be happy to map separate shortcuts for when I'm
| prepared to do so (avoiding the need to detect my intent).
| ado__dev wrote:
| This is great feedback and something we are looking at in
| regards to Cody. We value developer choice and at the moment
| for Chat developers can choose between various LLM models
| (Claude 3 Opus, GPT 4-Turbo, Mixtral 8x7b) that offer different
| benefits.
|
| For autocomplete, at the moment we only support Starcoder
| because it has given us the best return on latency + quality,
| but we'd def love to support (and give users the choice to set
| an LLM of their choice, so if they prefer waiting longer for
| higher quality results, they should be able to)
|
| You can do that with our local Ollama support, but that's still
| experimental and YMMV. Here's how to set it up:
| https://sourcegraph.com/blog/local-code-completion-with-olla...
| ypeterholmes wrote:
| Fantastic article and impressive work by this company. They're
| basically wrapping LLM's with a working memory, and tying it to
| user input. And thus we step a little closer to AGI/ASI.
| ForHackernews wrote:
| > Congratulations, you just wrote a code completion AI!
|
| > In fact, this is pretty much how we started out with Cody
| autocomplete back in March!
|
| Am I wrong in thinking that there's only like 3(?) actual AI
| companies and everything else is just some frontend to
| ChatGPT/LLama/Claude?
|
| Is this sustainable? I guess the car industry is full of rebadged
| models with the same engines and chassis. It's just wild that we
| keep hearing about the AI boom as though there's a vibrant
| competitive ecosystem and not just Nvidia, a couple of software
| partners and then a sea of whiteboxers.
| airstrike wrote:
| you've got to differentiate between training, inference and
| hardware. they all benefit from the "AI boom" but at different
| levels and have varying levels of substitutability (Google
| tells me that's a real word)
| lozenge wrote:
| I mean... the article goes on to explain all their value add...
| which of course can be replicated, but it's not as if you can
| just grab an API key and do the same.
| ForHackernews wrote:
| But OpenAI could make this themselves in a few weeks, right?
| If they happen to decide to, then this company is done.
|
| That's what I don't get about all these AI startups. The core
| value of their business is an API call they don't own. It's
| like people who were selling ______ apps on iPhone when Apple
| added their own built-in _____ features to iOS, except the
| entire industry is like that.
| skybrian wrote:
| Well sure, a company with ample funding could in theory do
| anything. It seems sort of like asking why Google didn't
| beat GitHub on code hosting. They did try, but their focus
| was elsewhere, so they gave it up. [1] And OpenAI doesn't
| seem to be doing developer tools at all?
|
| GitHub and Microsoft are more likely candidates. GitHub
| Copilot is a competing product that could be improved if
| they chose to do so.
|
| [1] https://code.google.com/archive/
| ForHackernews wrote:
| I'm not really saying that small companies should never
| compete with big companies.
|
| I'm saying that a small company whose biggest competitor
| is also their key supplier is, if not doomed, at least
| operating on borrowed time. If my value proposition is
| reselling AWS hosting with an easy-to-use dashboard, I
| better pray to god Amazon never makes AWS simpler.
| ado__dev wrote:
| I feel like that can be said about any SaaS app. The
| reality is, mega-corps move very slowly and often can't
| deliver the same user experience as a start up. If
| everyone had the approach of "well I shouldn't make this
| app because if Google wanted to, they could do it in a
| few weeks" we wouldn't have 90% of the startups we see
| today.
| fragmede wrote:
| Aren't we all operating on borrowed time though? Even if
| some company exists for 3 years and is able to founders a
| small acquihire exit, what's wrong with borrowing some
| time from AWS/OpenAI? We can't all come up with PageRank
| and patent it and form a company around it.
| skybrian wrote:
| From the article, it sounds like they are using Claude?
| Maybe there's more than one possible supplier. There
| probably will be more.
|
| Edit: looks like the autocomplete provider is
| configurable in the VS Code plugin settings. Not sure
| what the default is.
| ado__dev wrote:
| Currently for chat we support Claude 2, Claude 3 Haiku,
| Sonnet, and Opus, GPT-4 Turbo, and Mixtral 8x7b.
|
| For autocomplete we're using Starcoder 16b.
|
| We also support local inference with any LLM via Ollama
| (this is experimental and only for Cody Free/Pro users).
| Enterprise customers can choose which LLMs they want.
| fluoridation wrote:
| Well, the same happened in the early days of the PC. There
| were all these companies that sold basic utilities like
| disk scanners, defragmenters, partitioners, antiviruses,
| etc. When operating systems started to get bigger and
| include these utilities by default, that industry dried up
| overnight. I don't think there's anything wrong in building
| an inherently ephemeral business that just seeks to fill a
| niche that exists just because someone else hasn't filled
| it yet.
| littlestymaar wrote:
| Don't forget text editors! And even though it didn't
| last, many people made very good money on these.
| phillipcarter wrote:
| The "core value" of many companies is a database service
| they don't own, but their businesses seem to do just fine.
| There's a few who get obviated over time (usually they are
| also slow to react), but they're a minority.
|
| I would also question the idea that OpenAI could build a
| code editing companion this robust in a fee weeks. It's
| been a long time since the models have been the limiting
| factor in these tools.
| ForHackernews wrote:
| Most companies have valuable data that they _do_ own
| stored in a commodity database they don 't. I know if
| Amazon started charging 10x for RDS, it would be painful
| but we could migrate in a few weeks.
| timrogers wrote:
| I think I'd argue that the models are still a limiting
| factor for these kinds of tools - but there's still also
| plenty of competitive space outside of models. The
| context you gather, the way you gather it and your
| prompting matters.
| tayo42 wrote:
| Which part do you think you couldn't do on your own with your
| own api key?
| ec109685 wrote:
| They've been thinking about the problem continuously for
| the past year, can run experiments across their user base
| and thus have a lot more context and insight than someone
| whipping up an implementation.
| tayo42 wrote:
| well they wrote a whole blog about how they do it... and
| im sure this isn't the only one on how to approach this.
| they also only have a 30% acceptance rate, which is only
| 10% over their initial attempt with just a prompt.
| freecodyx wrote:
| it make sense, in order to come up with a base model, you will
| need a lot of quality training data, tons of compute.
|
| the role of an AI startup is to come up with ideas, thus useful
| products. Most of existing products pre-AI, are also front ends
| to existing operations systems and exiting databases, because
| creating the whole stack does not make sense. At least we have
| state of the art open models that we can use freely
| Metricon wrote:
| For those who might not be aware of this, there is also an open
| source project on GitHub called "Twinny" which is an offline
| Visual Studio Code plugin equivalent to Copilot:
| https://github.com/rjmacarthy/twinny
|
| It can be used with a number of local model services. Currently
| for my setup on a NVIDIA 4090, I'm running both the base and
| instruct model for deepseek-coder 6.7b using 5_K_M Quantization
| GGUF files (for performance) through llama.cpp "server" where
| the base model is for completions and the instruct model for
| chat interactions.
|
| llama.cpp: https://github.com/ggerganov/llama.cpp/
|
| deepseek-coder 6.7b base GGUF files:
| https://huggingface.co/TheBloke/deepseek-coder-6.7B-base-GGU...
|
| deepseek-coder 6.7b instruct GGUF files:
| https://huggingface.co/TheBloke/deepseek-coder-6.7B-instruct...
| Lucasoato wrote:
| One thing I really miss is a standard way to block any copilot/ai
| code completion tool from reaching specific files. That's
| particularly important for .env files, containing sensitive info.
| We don't want leaking secrets outside our machine, imagine the
| risks if they become part of the next training dataset. That'd be
| really easy to standardize, it's just another .gitignore-like
| file.
| eddd-ddde wrote:
| Why not just use .gitignore itself? It's also unlikely they
| want to train on build files and such.
| myko wrote:
| That makes sense but not everyone uses git. A new file that
| you can point to your gitignore in would probably be a good
| idea.
| orhmeh09 wrote:
| If you do not use git, you might not be GitHub's target
| audience.
| fragmede wrote:
| That's clever, but GitHub is recognized as the way to do
| things, so I've seen a couple repos where the only
| commits are of binaries to download, which technically
| uses git, but I wouldn't really say that's using git.
| Atotalnoob wrote:
| You can purchase GitHub copilot and use it (even for
| corporations) and not use GitHub.
|
| My company buys copilot for all devs, but doesn't use GH
| or GHE
| timrogers wrote:
| GitHub employee here
|
| This does exist in GitHub Copilot - it's called content
| exclusions: https://docs.github.com/en/copilot/managing-github-
| copilot-i...
|
| I'm not sure if Cody has a similar feature, or if there's any
| move towards a standardised solution.
| ado__dev wrote:
| So Cody allows you multiple ways to manage this.
|
| In the Cody settings.json file you can disable autocomplete on
| entire languages/file types.
|
| Additionally, we recently rolled out a Cody Ignore file type
| where you can specify files/folders that Cody will not look at
| for context. This feature is still in experimental mode though.
| https://sourcegraph.com/docs/cody/capabilities/ignore-contex...
|
| With Cody, we have a relationship w/ both Anthropic and OpenAI
| to never use any data submitted via Cody users for training and
| data is not retained either.
| Jimmc414 wrote:
| I created a cli tool that copies a GitHub or local repo into a
| text file for llm ingestion. It only pulls the filetypes you
| specify.
|
| https://github.com/jimmc414/1filellm
| jameslevy wrote:
| These types of tools should exclude all files and directories
| in the .gitignore as standard operating procedure, unless those
| files are specifically included. Not just because of secrets,
| but also because these files are not considered to be part of
| the repository source code and it would be unusual to need to
| access them for most tasks.
| Exuma wrote:
| how does this compare to copilot? i would be willing to switch
| (or try it at least) if it can give better experience inside
| neovim
| phillipcarter wrote:
| Cody is very good. I recommend giving it a try.
|
| FWIW I believe Cody is much more actively developed than
| Copilot is these days, and so it has a more comprehensive
| feature set.
| ado__dev wrote:
| I wrote a blog post comparing Cody to Copilot a little while
| ago. Some of the stuff might be outdated now, but I think it
| still captures the essence of the differences between the two.
| Obviously I'm a little biased as I work for Sourcegraph, but I
| tried to be as fair as one could be. Happy to dive deeper into
| any details.
|
| https://sourcegraph.com/blog/copilot-vs-cody-why-context-mat...
|
| Our biggest differentiators are context, choice, and scale.
| We've been helping developers find and understand code for the
| last 10 years and are now applying a lot of that to Cody in
| regards to fetching the right context. When it comes choice, we
| support multiple LLMs and are always on the lookout in
| supporting the right LLM for the job. We recently rolled out
| Claude 3 Opus as well as Ollama support for offline/local
| inference.
|
| Cody also has a free tier where you can give it a try and
| compare for yourself, which is what I always recommend people
| do :)
|
| On Neovim, Cody actually does have experimental support for
| neovim: https://sourcegraph.com/docs/cody/clients/install-
| neovim. Not all features are supported as in VS Code though.
___________________________________________________________________
(page generated 2024-04-07 23:00 UTC)