▲ 422 points
back
79 comments
Bravo! I'm so glad this is born. I'm curious about its model, SalesForce CodeGen. Does it train on all the public repos on GitHub? Does Copilot have access to private repos that CodeGen cannot access?
Also, it would be really cool if I can personalize FauxPilot by feeding it with all my repos on GitHub. Sometimes I just need to reimplement a function I've written before but it's really hard to find where my old code is.
It is possible to fine-tune CodeGen using Huggingface Transformers! Then you'd be able to fine-tune it on your own code and use the resulting model. However, training is more expensive -- you'd need an A6000 or better to train the 6B model. Something like the following should work:
deepspeed --num_gpus 1 --num_nodes 1 run_clm.py --model_name_or_path=Salesforce/codegen-6B-multi --per_device_train_batch_size=1 --learning_rate 2e-5 --num_train_epochs 1 --output_dir=./codegen-6B-finetuned --dataset_name your_dataset --tokenizer_name Salesforce/codegen-6B-multi --block_size 2048 --gradient_accumulation_steps 32 --do_train --fp16 --overwrite_output_dir --deepspeed ds_config.json
It might be doable to set this up on an AWS machine with a beefy GPU or two. I haven't tried it yet though.
Once you have a model trained in Huggingface Transformers you'd be able to convert it using this script:
https://github.com/moyix/fauxpilot/blob/main/converter/huggi...
I train models 24/7 right now and PLEASE do not use AWS for it. You're going to pay out of your backside for it.
Better alternatives: Google Colab, Paperspace Gradient, Lambdalabs Cloud, Vultr GPU instances
Colab will give you a T4, K80, V100 or P100 (alternatively their own TPUs) for free - $50 for 24h uninterrupted background jobs, Gradient will give you free A6000s and sometimes even free A100s for a $40 subscription for 6 hours (repeatable ad infinitum), Lambdalabs gives you a RTX 6000 for 0.50/hour and A6000 for 0.80/hour and Vultr GPU will give you 1/7th of an A100 for 0.37/hour
Thank you for sharing the command for finetuning! Is it possible to share your ds_config.json? I tried to finetune the 2B model on A100 (40GB) using your command, but got a CUDA out of memory error. The ds_config I used was the one from huggingface (https://github.com/huggingface/transformers/blob/main/tests/...).
> I just need to reimplement a function I've written before
There was a project here some time back that allowed to call function using its hash. This avoided replication. It was entirely different paradigm to resolve dependencies.
Anyone remembers its name? It was controversial here but it was interesting nonetheless.
A comment below mentions those details are in the paper:
Copilot only accesses the private repositories for the user it is authenticated with.
I'm fine with them accessing public repositories, as long as they respect the license. Which they probably aren't for private?
On a related note, Google apparently treats their free customers in a way where all their data is mixed in for the algorithm, but apparently not the data from accounts who use Google Suite (paid business accounts), instead they have their own recommendations built for them on a per-user basis, then has the public algo mixed in.
For you and all my sibling comments here. Are you guys forgetting GitHub IS Microsoft? Do you guys actually believe Microsoft is respecting privacy on repositories? Did you read the EULA after the acquisition? C'mon, let's be serious!
Tangential, maybe someone here knows; how complicated would be to implement an autocompleter that just understands syntax + patterns and can offer me suggestions but based on my own stuff? Like what is the simplest version of Copilot that doesn't require huge amounts of training but just does a decent job at recognizing tokens and trying to fill in the structure based on some input code. e.g: `app.get('/',|` <- at this point I should get an express.js handler autocomplete like `(req, res, next) => {}`, maybe with multiple choices grepped from my own data ranked by occurences. Is this too extreme it needs a multibillion parameter AI model to achieve? Does anything like this exist? Like an auto-snippet thing but contextual and token replacing.
To do this in a way that’s actually useful is hard. Microsoft has had Intellisense, JetBrains have autocomplete in Intellij and other products. Both have big teams and decades of work put into them, and still and great, hence the ML approach being tried now.
That's exactly the thing I would be looking for, too.
Especially since I'm working in proprietary niche language, it would be so great to have my own code to be the corpus of the model.
It could be so easy: stuff your codebase into a model generator and out comes some sensible autocomplete. Well, at least I hope we will see something like this soon.
The plugin TabNine does what you're looking for. I've been using it enjoyably for about two years.
For sane, modern languages like C# and Java, autocomplete has worked somewhere between adequate and amazingly well for a long time already.
I think it used to be kind of usable also in VB6 also, at least that is how I remember it.
Golang and Typescript seems to have good support too now.
But for just the thing you mention above you could go a long way with live templates or what it is called in different IDEs, or just text templates with variables like the Rails guys used to use in their demos.
The latest version of Sublime Text works kind of like that. It know the tokens that most commonly appear together in the folders you currently have open in the sidebar and prioritizes the suggestions popup based on that. Seems to work pretty well for me.
It's a relatively new feature - within the last year I think.
Next step is to train a model exclusively on leaks of proprietary Microsoft source code. Fair use, right Microsoft?
It probably wouldn't produce good code though :D
Man that would be so ironic.
And if they want to sue for that, they will be shooting themseleves in the foot by proving their Fair-use argument isn't real.
Awesome work, I made a similar project which is making a cost effective and privacy focused alternative to OpenAI text generation that can be switched to in a one line/easy way because it's API compatible https://text-generator.io/blog/over-10x-openai-cost-savings-... also works for generating code too so would be excited for someone to try that out too.
Does it work with natural language to sql?
What's the license of the project? There is no license file in the repo
Fixed! Didn't expect this to hit HN so fast :)
This is awesome!
How is the quality vs. GitHub Copilot for Python or JavaScript?
The largest Python model (codegen-16B-mono) should be pretty competitive! The biggest thing that it lacks is the ability to make use of context after the cursor. Unfortunately, that capability will require retraining the models:
This is excellent. Please do suck the wind from Microsoft's exploitative sails here.
Copilot primarily exists as a way for Microsoft to end-run around the GPL.
I wonder what code corpus the SalesForce CodeGen model (which this uses) was trained on.
From the paper[1]:
> The family of CODEGEN models is trained sequentially on three datasets: THEPILE, BIGQUERY, and BIGPYTHON.
> The natural language dataset THEPILE is an 825.18 GiB English text corpus collected by Gao et al. (2020) for language modeling. The dataset is constructed from 22 diverse high-quality subsets, one of which is programming language data collected from GitHub repositories with >100 stars that constitute 7.6% of the dataset.
> The multi-lingual dataset BIGQUERY is a subset of Google’s publicly available BigQuery dataset, which consists of code in multiple programming languages such as C, Python, Ruby. For the multi-lingual training, the following 6 programming languages are chosen: C, C++, Go, Java, JavaScript, and Python.
> The mono-lingual dataset BIGPYTHON contains a large amount of data in the programming language, Python. We have compiled public, non-personal information from GitHub consisting of permissively licensed Python code in October 2021.
With model sizes starting at 2GB, wouldn't the model be hopelessly overdetermined for small codebases?
The models aren't customized for individual codebases; all of them were trained on most of GitHub.
I would love to build a free and oss clone of GitHub using things like this. Kudos to moyix
It might be interesting to look into existing projects like Gitea or SourceHut. Gitea specifically seems to have a similar look and feel as GitHub.
This is awesome and I'm looking forward to seeing more from this.
Maybe this is a dumb question, what would be the most common use-case for this?
Any company that are not allowed to share code with third parties either by internal or customer policy. This should be a lot of companies.
Well, do you prefer services over inspectable code you can run yourself? I don't. In fact, I strongly dislike having to rely on services.
man I finally got a 3080 and. ow you're telling me I need a second?!
haha seriously though, I'm kinda sad the model size is 2GB then straight to 13GB.
In theory the 2B model should be somewhere in between, whenever it gets fixed.
You probably don't need physical GPUs but you can rent them in some cloud for a few days.
you can get it running with resizable bar
Such a clever name :)
This is awesome! Will be trying it out. Hope you're doing well :D
I don't understand the point of dedicating so many resources just so that you can have a Clipper add-on second-guess your code and make bad suggestions.
Copilot is basically copying-from-stack-overflow on steroids. Maybe it should be called Copy-a-lot.
Whatever. Regardless of copying mechanisms I expect my fellow developers to understand every suggestion made by such tools.
Copilot has given me great utility and increased my productivity a lot. I constantly get surprised and amazed by the suggestions. A self-hosted version would be awesome.
You have to start somewhere and the AI suggestions are already a big time saver.
It will not end there. It began with Python when they removed "offensive" words such as "kill". Then it was GitHub then Git itself who removed the word "master".
Now GitHub / Microsoft, who are producing tools that integrate themselves more and more into the programmers' workflow, will now more opportunities to enforce this kind of fringe ideology.
Dystopian predictions; the following words will be replaced:
Parent / Child Inheritance Class Binary Invalid