[HN Gopher] Launch HN: CodeComplete (YC W23) - Copilot for Enter...
___________________________________________________________________
Launch HN: CodeComplete (YC W23) - Copilot for Enterprise
Hello HN! We're Max and Lydia, co-founders at CodeComplete AI
(https://codecomplete.ai), an AI-powered coding assistant for
enterprise companies. Many large companies can't use products like
GitHub Copilot because of the security and privacy risks, so we're
building a self-hosted version that's fine tuned to the company's
codebase. We love Copilot and believe that AI will change the way
developers work. Max wanted to use Copilot when he was an ML
engineer at Meta, but leadership blocked him because Copilot
requires sending company code to GitHub and OpenAI. We built
CodeComplete because lots of other companies are in the same boat,
and we want to offer a secure way for these companies to leverage
the latest AI-powered dev tools. To that end, our product is
really meant for large engineering teams at enterprise companies
who can't use GitHub Copilot. This generally means teams with more
than 200 developers that have strict practices against sending
their code or other IP externally. CodeComplete offers an
experience similar to Copilot; we serve AI code completions as
developers type in their IDEs. However, instead of sending private
code snippets to GitHub or OpenAI, we use a self-hosted LLM to
serve code completions. Another advantage with self-hosting is that
it's more straightforward to securely fine-tune to the company's
codebase. Copilot suggestions aren't always tailored to a company's
coding patterns or internal libraries, so this can help make our
completions more relevant and avoid adding tech debt. To serve
code completions, we start with open source foundation models and
augment them with additional (permissively-licensed) datasets. Our
models live behind your firewall, either in your cloud or on-
premises. For cloud deployments, we have terraform scripts that set
up our infrastructure and pull in our containers. On-prem
deployments are a bit more complicated; we work with the customer
to design a custom solution. Once everything's set up, we train on
your codebase and then start serving code completions. To use our
product, developers simply download our extension in their IDE (VS
Code currently supported, Jetbrains coming soon). After
authentication, the extensions provide in-line code completion
suggestions to developers as they type. Since we're a self-hosted
enterprise product, we don't have an online version you can just
try out, but here are two quick demos: (1): Python completion,
fine-tuned on a mock Twitter-like codebase:
https://youtu.be/YqkqtGY4qmc. (2) Java completion for
"leetcode"-style problems, like converting integers to roman
numerals: https://youtu.be/H4tGoFNC8oI. We take privacy and
security seriously. By default, our deployments only send back
heartbeat messages to our servers. Our product logs usage data and
code snippets to the company's own internal database so that they
can evaluate our performance and improve their models over time.
Companies have the option to share a subset of that data with us
(e.g. completion acceptance rate, model probabilities output,
latencies, etc), but we don't require it. We never see your code or
any other intellectual property. We charge based on seat licenses.
For enterprise companies, these contracts often demand custom
scoping and requirements. In general though, our pricing will be at
a premium to GitHub Copilot since there is significant technical
and operational overhead with offering a self-hosted product like
this. Having access to these types of tools would have saved us a
bunch of time in our previous jobs, so we're really excited to show
this to everyone. If you are having similar issues with security
and privacy at your current company, please reach out to us at
founders@codecomplete.ai! We'd love to hear your feedback.
Author : dingliqing53
Score : 88 points
Date : 2023-03-14 15:16 UTC (7 hours ago)
| sqs wrote:
| Congrats on the launch. This seems very compelling. For anyone
| asking "Why can't GitHub just do this?", (1) never underestimate
| an extremely smart and motivated team, and (2) competition is
| critical and valuable.
|
| Let's celebrate when teams build products to fill big needs,
| instead of dismissing things because of a potential threat from a
| big company. If we dismiss new things, then we'll just end up in
| a world where big companies get complacent and devs get less new
| stuff. (I know many GitHubbers, and they love seeing new stuff
| and are cheering for it because they can't do everything.)
| lumax15 wrote:
| Couldn't agree more :)
| digdugdirk wrote:
| Any thoughts on an individual "Here you go, but you're on your
| own" license? Something like this would be _massively_ beneficial
| to small IT teams who have the same inability to send code to
| OpenAI, but could use the benefit of a smart auto-complete style
| system like Copilot. They wouldn 't necessarily need the custom
| integration - even small benefits would be seen from an initial
| core model. Additional integrations could be sold as piecemeal
| consulting engagements after the fact if need be.
|
| Just a thought - there's a vast market out there for
| organizations with dev teams in the 1-10 person range.
| rileyphone wrote:
| The infra costs are probably minimum $1000/month so it doesn't
| make sense to target small teams with that overhead yet. I'd
| expect something open source to appear hear in the next few
| months, though it might be based on legally iffy LLaMa weights.
| dingliqing53 wrote:
| Definitely want to offer this in the future but not currently
| our ICP. Like rileyphone mentioned, the cost of hosting a model
| is substantial. Could you please elaborate on why the small IT
| teams can't send code to OpenAI? We'd love to think more about
| this!
| DeathArrow wrote:
| Have you benchmarked it against Copilot?
|
| Training data is only your own data, because that might not
| achieve a great accuracy?
|
| Does it work with any programming language?
| lumax15 wrote:
| We don't have any rigorous benchmarks against Copilot, but
| we're working on building an evaluation framework to do so.
| We've played a bunch with traditional academic metrics for
| codegen (e.g. pass @ k) and found they don't correlate super
| well with real-world performance. Also, want to mention that we
| are not competing directly with Copilot. A benchmark against
| Copilot is useful as we further improve our product, but our
| main value add here is not that we perform better than Copilot,
| but rather that we serve a customer segment that can't use
| Copilot. Would love to hear any thoughts you have.
|
| For training, we start with a capable open-source base model,
| augment it with a bunch of permissively-licensed repos, and
| then fine-tune on the customer codebase.
|
| We currently support C/C++, Go, Gosu, Java, Javascript, Python,
| Ruby, and Typescript, but we're continuously adding new
| languages.
| 8note wrote:
| What would a benchmark even look like?
|
| Does it make useful code? Does it make the same code?
|
| Or more strictly on something like latency and cost?
| aunch wrote:
| Codeium already has this enterprise option, with self-hosting,
| fine-tuning, and even things like search on top of autocomplete
| and way more IDEs: https://codeium.com/
|
| How are you different?
| dvngnt_ wrote:
| aren't you codeium?
| aunch wrote:
| Yes, thus the question :)
| apetuskey wrote:
| Excited to try this looks really good! Been looking for something
| like this for our Pynecone code base!
| teddyh wrote:
| > _Trained only on permissively-licensed repos to avoid legal
| risks_
|
| Do any of those "permissive" licences require attribution? Do
| they all grant a patent license?
| enono wrote:
| Always astonishes me how negative hacker news can be whenever
| people try to launch stuff. This is cool!
| primax wrote:
| Check out when Dropbox launched
|
| https://news.ycombinator.com/item?id=9224
| lumax15 wrote:
| Thanks enono :)
| PhysicalNomad wrote:
| For real, any of these threads is guaranteed to have a ton of
| nitpicks and comparisons with products that are way more
| mature. It's pretty sad to see specially in a so-called hacker
| community. Congrats on the launch!
| dang wrote:
| Negative reactions are quicker to arise and quicker to write
| out, which is a double whammy that causes them to appear more
| quickly and dominate the thread early. Slower, positive
| reactions eventually show up (usually!) and in the meantime we
| get the odd "contrarian dynamic" phenom where "I can't believe
| how negative these comments are!" reactions start popping up
| and get upvoted to the top. Internet is weird.
|
| https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...
| raincole wrote:
| The top comment usually goes like:
|
| negative comments -> some presumably "funny" witty joke ->
| positive/meaningful insight -> peopel complain about comment
| quality
| 1123581321 wrote:
| When I read these threads I mentally prepend "this is neat" to
| all comments. Perhaps I'm too optimistic, but I think nearly
| any engagement with a new startup or idea is appreciation on
| some level.
| duxup wrote:
| I usually prepend that to my critical comments ;)
|
| If I'm responding in a critical manner it usually is because
| I think something is useful/ there.
| ninjaa wrote:
| Congrats Lydia & Max! Great demo videos
| lumax15 wrote:
| Thanks ninjaa!
| alsodumb wrote:
| GitHub and other companies like Amazon have the advantage of the
| scale in terms of dataset. What's the guarantee that the pre
| trained model you have that you'll fine tune on a company's code
| base is as good as say Copilot? It makes it even hard to evaluate
| when you don't have a demo to try - it's not that hard to setup a
| pipeline to run your model in cloud and send invites to potential
| customers if you want to.
| sebzim4500 wrote:
| Wasn't copilot only trained on public GitHub repos? Presumably
| there's nothing stopping anyone from training on them.
| dingliqing53 wrote:
| Yea, there are tons of public data to train on. Copilot's
| under fire right now: https://githubcopilotlitigation.com/,
| but we make sure we only train on permissively licenses
| because some companies are sensitive to the IP issues here.
| swatcoder wrote:
| It doesn't make sense to treat Copilot as a competitive
| benchmark where Copilot isn't even an option for policy
| reasons, which is what this product is targeting.
|
| Does it provide net positive support to developers? Is that
| support worth the licensing and maintenance costs? Those would
| indeed benefit from a demo, but it's also hard to demo
| something whose value hinges on fine-tuning.
|
| More likely, they'll need to sell low-risk, supported
| integrations that help an org stand up the project on their own
| system and evaluate its quality in situ. It's a whole different
| model than retail SaaS services like Copilot.
|
| This is an exciting announcement for a product meant to fill a
| gaping hole. The orgs that need it will just have to see how
| well it works for them.
| dingliqing53 wrote:
| Thank you!! You've hit the nail on the head and our thoughts
| exactly. One anecdote is we offer video demos and one
| potential customer just said, "I don't need to see it. I know
| what Copilot does." We do offer pilots for customers after
| fine-tuning on their codebase!
| ninjaa wrote:
| Lol if Swatcoder is available they could be great at devrel
| lumax15 wrote:
| Let us know, swatcoder!
| swatcoder wrote:
| Hah. That's not where I'm investing my attention these
| days, but good luck!
| blintz wrote:
| Love the idea of running this stuff on-prem / in house. Just not
| being beholden to random updates in the rules or logging policies
| seems like a big value add. +1 to the other commenter who
| mentioned they'd love to see a version for smaller teams - there
| might be more privacy-oriented teams out there than you'd expect.
| I think even a version that can run in public clouds would be
| meaningful.
|
| Have you gotten feedback from folks using it on how it compares
| to Copilot in terms of usefulness? What's the 'order of
| magnitude' difference?
| bastardoperator wrote:
| Actual Copilot for Enterprise:
|
| https://github.blog/2023-02-14-github-copilot-for-business-i...
| lumax15 wrote:
| This is good for some business teams, but it still doesn't
| offer self-hosting or fine-tuning.
| doctoboggan wrote:
| Can you let us know which "open source foundation model" you use?
| anxtyinmgmt wrote:
| So if Microsoft/Github ever offers an on-prem version, what will
| be the advantage of using your product?
| jacobsenscott wrote:
| At that point you would need to compete on service and price.
| They'll also get the first mover lock in advantage. Once a
| large company deploys your product to 5000 devs you are pretty
| much in for life. /me waves to jira.
| dingliqing53 wrote:
| Another thing is that since Microsoft/Github are working with
| OpenAI's closed source Codex model, we think it's unlikely
| they'll offer something on-prem anytime soon since they would
| have to reveal the model weights, and thus risk a leak (ex.
| Meta's LLaMA model weights got leaked within a week)
| carlosdp wrote:
| That is a... very big assumption that I wouldn't put money
| on.
|
| It's become pretty clear that models aren't a moat. If
| everyone has Codex-class capability (which is already
| happening), there's no real risk to them deploying on-prem,
| because the model itself is a commodity.
| sebzim4500 wrote:
| Sounds like they'll finetune the model on your existing
| codebase. Even if Microsoft/Github offers this functionality,
| it's the kind of thing where it may be worth paying a premium
| to have good customer support telling you exactly how to make
| use of this.
|
| It's like how there are plenty of people paying for DB software
| which is 'worse' than mainstream free ones, but they really
| like the fact they can call experts who will tell them what
| they are doing wrong with query optimization, etc.
| DeathArrow wrote:
| >It's like how there are plenty of people paying for DB
| software which is 'worse' than mainstream free ones, but they
| really like the fact they can call experts who will tell them
| what they are doing wrong with query optimization, etc.
|
| And for us, poor peasants, there's always going to be
| stackoverflow.
| dingliqing53 wrote:
| Yes! Thanks for elaborating for us! :)
| saurabh20n wrote:
| Congrats on the launch. I think you should share some technical
| details for a more substantial pitch. You are using the OSS
| BigCode effort and "The Stack" [1, 2] (as you say in another
| comment), which is great.
|
| A few questions that might help an enterprise customer: How big
| is your base model? Where did you find more datasets (maybe just
| a hint would be sufficient)? Are you using SantaCoder [3]?
| Anything you can say about your fine-tuning that makes it
| special? Totally on board with you that HumanEval/MBPP are not
| great benchmarks for real world, and do you have a suggested
| alternative to help me see the value?
|
| The calculus for an enterprise customer might be: "We could fine
| tune a 6B model on our internal code and internal benchmarks (say
| with a month of work, a few thousand in compute, 2 people on
| task), but I'd rather buy an off-the-shelf solution like
| codecomplete.ai. They give us XYZ benefits." Articulate the XYZ
| for a technical decision maker who will be your target audience.
|
| * [1] https://huggingface.co/datasets/bigcode/the-stack
|
| * [2] https://arxiv.org/abs/2211.15533
|
| * [3] https://huggingface.co/bigcode/santacoder
| lumax15 wrote:
| Great questions. We want to keep some of our technical details
| closer to the chest, so I won't go into the specific
| technologies we're using here.
|
| I will expand a bit on fine-tuning. It's really hard to get
| this right, and the iteration speed is slow. Of course these
| companies can build their own, but we want to save them a lot
| of headache.
|
| So far, we haven't found any off-the-shelf open source base
| model that works super well for code completions. We've
| augmented models with a huge amount of data in order to see our
| current performance, and we ran into a lot of pain along the
| way.
| kanyethegreat wrote:
| > Permissively Licensed: Trained only on permissively-licensed
| repos to avoid legal risks
|
| You're fine-tuning the model. What model are you fine-tuning? I
| can't imagine you trained your own LLM from scratch, so how can
| you possibly guarantee the core model wasn't trained on non-
| permissively licensed code?
| lumax15 wrote:
| We're starting off with an open source base model that was
| trained on The Stack, a dataset containing only permissively-
| licensed code, and we're further augmenting it with additional
| repositories with MIT, BSD, or Apache Licenses.
| mtlynch wrote:
| How are you complying with the attribution requirement for
| MIT and Apache?
|
| From the MIT License[0]:
|
| > _The above copyright notice and this permission notice
| shall be included in all copies or substantial portions of
| the Software._
|
| From the Apache License[1]:
|
| > _You must give any other recipients of the Work or
| Derivative Works a copy of this License; and You must cause
| any modified files to carry prominent notices stating that
| You changed the files; and You must retain, in the Source
| form of any Derivative Works that You distribute, all
| copyright, patent, trademark, and attribution notices from
| the Source form of the Work, excluding those notices that do
| not pertain to any part of the Derivative Works; and_
|
| [0] https://opensource.org/license/mit/
|
| [1] https://opensource.org/license/apache-2-0/
| 8note wrote:
| This seems really cool for a company that has enough surface area
| of software where chances are somebody has already built the
| thing you need and instead of building a copy yourself, you can
| integrate with the existing system
| clbrmbr wrote:
| Will you train on accepted/regected and other responses from devs
| rather than just the final code, so that clients lose something
| when moving to a similar tool which will pop up tomorrow?
| dingliqing53 wrote:
| Yes! We definitely have that on our product roadmap, though we
| think of it more in terms of improving model performance vs.
| deterring clients from switching to a different tool. Since
| we're working with large enterprise codebases and deploying
| models to behind their firewalls, there's (hopefully) a
| nontrivial switching cost. Thanks for pointing it out!
| nikaspran wrote:
| How does this compare with https://www.tabnine.com/enterprise,
| which is also self-hosted, trained on permissively licensed
| repositories and supports training on private repos?
___________________________________________________________________
(page generated 2023-03-14 23:02 UTC)