back
▲ 627 points

Show HN: Tabby – A self-hosted GitHub Copilot

github.com
by wsxiaoys·3y ago·126 comments·view on hn ↗
I would like to introduce Tabby, which is a self-hosted alternative to GitHub Copilot that you can integrate into your hardware. While GitHub Copilot has made coding more efficient and less time-consuming by assisting developers with suggestions and completing code, it raises concerns around privacy and security.

Tabby is in its early stages, and we are excited to receive feedback from the community.

Its Github repository is located here: https://github.com/TabbyML/tabby.

We have also deployed the latest docker image to Huggingface for a live demo: https://huggingface.co/spaces/TabbyML/tabby.

Tabby is built on top of the popular Hugging Face Transformers / Triton FasterTransformer backend and is designed to be self-hosted, providing you with complete control over your data and privacy. In Tabby's next feature iteration, you can fine-tune the model to meet your project requirements.

126 comments
Copilot has so far been pretty useful to me as a "sometimes smarter intellisense". It'll frequently correctly guess the arguments I want for a function (and their types), and every once in a while I'll type `a.map(` and it'll auto-fill the transformation code I was planning on writing.

The simpler the task I'm trying to do, the better chance it has of being correct, but that's also the part where I feel I get the most benefit from it, because I already thoroughly understand exactly what I'm writing, why I'm writing it, and what it needs to look like, and Copilot sometimes saves me the 5-30s it takes to write it. Over a day, that adds up and I can move marginally faster.

It's definitely not a 100x improvement (or even a 10x improvement), but I'm glad to have it.

If this works as well, locally, to escape the privacy issue, I'll be thrilled. Checking it out.

Yes, I went into all of this thinking the tools need to handle complex code in order to really be useful. But that's the code I want to be most careful about, so it's the code I really want to architect myself, and understand thoroughly.

Making the simpler code much faster to develop leaves me a lot more time and focus for the complex work.

These tools really are amazing, even with their limitations.

Copilot also helps a lot with verbose, RSI-inducing languages like C++. On days where my hands are cold or my hands are just not feeling as physically limber as usual, it's a big relief when it autocompletes a long series of symbols for me.

I know regular old LSP/Intellisense helps here but you are still often constrained to only autocompleting one token at a time, and have to type it at least partially in most cases.

Tried it in hobby projects: saved a lot of time in java, less in python. In my scala project i have something like an internal dsl, and it was not much of an improvement over intellisense. I just realized how much i think and overthink, while typing. When it completed the entire block, I’d have to stop and think over it regardless.

Local models are a different game though. I want not an autocomplete, i want to tell it “go do this, similarly how we did it there” and it does it. Basically, a high level dsl, that magically appears after a model looks at your codebase.

Having been using Copilot for a few weeks I would substantially agree with this.

However, just yesterday in a module with quite a lot of context and types it autocompleted a function I was pondering for converting between different schema languages.

It was subtly wrong but the basic structure was correct. I had wanted to implement this function for a while but, having not really thought it through, imagined it would be much more complicated and would have required reading the ABI spec for the target. I never would have attempted it unless it had shown me the way and that it was relatively simple.

This is probably a piece of code that saves significant manual code. Copilot doesn't need to nudge me towards writing that many high leverage pieces of code to start becoming a big multiplier.

> to escape the privacy issue

Genuine question, do you not use GitHub for things other than copilot? It seems to me either the privacy issues of copilot are overblown or the privacy issues of GitHub itself are underblown, because they both end up with basically the same data.

Do you worry that if you use it too much you will no longer be able to bang out those simple tasks without it? It seems to me like it will make you weaker at some skills. Like how I take around 30 seconds to read a mechanical clockface.
I was working on a personal project last night, and I wanted to extract out a title between title=" and S0, I asked bing and it instantly gave me back a regex with the proper look ahead and look behind grouping.

Now theoretically, I know about those things, but even I'll admit I would have messed up the syntax and probably spent 15m on regex 101 tweaking things. The ai assist is great if you take it as a suggestion of something you should look at, rather than blindly trust it

Yeah, I think it's pretty good at this. I've also done "write a test like the one above but for Bar instead of Foo" and it got the right answer. It would be better to not duplicate code like that in most cases, but sometimes it's OK.

I recently needed a script to grab all my github repos from the github api and produce a CSV with the columns URL, name, description, and it could not do that at all. Just hallucinated libraries and APIs that did not exist.

I completely agree. Lately, I've been doing lots of green-field code development, and it shines in that area because there can often be lots and lots of boilerplate. I think Copilot makes me perhaps 20-30% more productive in that phase.

That's a lot less than 100x, but it's way more than nothing! I don't think I'd want to give it up.

i feel the same way, but the `a.map(` autofill is hit or miss for me. it'll interrupt my train of thought and it'll take me a minute to remember what i was trying to do, then i'll have to review the code to make sure it's right. and often there's a better way to do it, so i end up refactoring.
I was frustrated with it after a week and about quit it but am developing a more nuanced perspective.

Tldr; Can't imagine using it with a language that doesn't have great type checking to catch its BS, and they really need to tune how it interacts with traditional intelligence.

Been using it in VSCode with C# and Typescript. It gets in the way of the normal intellisense forcing you to use a shortcut to get out and back to intellisense.

For me this was really getting in the way because working one line at a time when you know what you need isn't its strong suite.

Stubbing out methods or even classes, or offering a "possibly correct" solution in areas you are fuzzy about is a strong point. Even stubbing out objects, methods, and etc using your existing code as a reference point...

But possible is the key. It's a bullshitter so it could randomly invent parameters or interfaces that are plausible, but not real. It also trips up on casing regularly.

All this to say; you gotta review everything it does and be on the lookout. Without the tooling help of a typed lang like c# or typescript this would be much harder.

I thought I was reaching a productive flow with this but then I loaded my proj in Rider, which didn't have copilot installed, and banged out some code and man it was frictionless in comparison.

Feel like copilot should be opt in via a shortcut, not opt out as it is. They really need to work out how to reduce the friction of using intellisense when it's the best option which it often is. But the copilot creators seem dead set on it constantly back-seat driving.

If not for the wonderful product itself, take my upvote for the "hire" page. It's the first time I see a company presenting a formula where you can input your data and figure out how much you would get paid. Nice!

> https://tabbyml.notion.site/Compensation-Sheet-ad61218889ab4...

Thank you, I was inspired by the brilliant ideas presented in the employer handbooks of Posthog and GitLab, as acknowledged in the career page.
And 10 years to exercise ISOs after departure. Very nice.
How does this compare to fauxpilot - https://github.com/fauxpilot/fauxpilot? Fauxpilot also uses Triton with fastertransformers and GPT-J style models (codegen).
Linked my replies to others' questions: [1], [2]

[1] https://news.ycombinator.com/item?id=35471882

[2] https://news.ycombinator.com/item?id=35471390

I hope these can be of help and answer your questions as well!

I'm confused about the premise here. The power of self-hosting such a thing is presumably that you would be able to train it on your own company's codebase as a corpus of examples to help other people in the company know how to navigate the specifics of your codebase.

But there's nothing in the introductory materials about how to train this thing.

Many of these that I've seen are more focused on keeping company data private than on training the model(s) on company data.

Which is pretty entertaining given how many of these companies host their code on Github already.

Looking at https://d1muf25xaso8hp.cloudfront.net/https%3A%2F%2Fs3.amazo... it looks like it's figures out you're using javascript and trains itself on all the libraries you install. I'm guessing there's a way to ask it to look at your specific code as well, but haven't dug into it yet.
The power of self-hosting this is for organizations that don't want to send their code to third parties. The code generation happens via in context learning using large language models, so there is no training these things. The biggest benefit I'd say is privacy.
I think that this is a really cool project.

Personally, I am working at a financial institution which is regulated, and the legal team + cyber are still evaluating if there would be any problems that could arise with CoPilot.

The way the Europe is heading towards AI regulation, Italy has already banned chatGPT. It seems likely that there would be a threat to Copilot as well in the Europe.

So these kind of solutions would make a ton of sense wrt organisations adopting this.

However as a developer who has used copilot before, here are my 2 cents:

The copilot makes a lot of sense, wrt auto completion and code generation, because it understands the current context of code.

It would be a friction for the developers if we are expecting them to use the user interface this project comes with.

It would be great for this project to go in the direction of the following:

- Ability for the developers to train the model with our custom projects, so it can give suggestions wrt to our style of coding

- Extensions for popular Editors like VIM, VS Code, IntelliJ etc

Happy to share more further info if needed :)

Out of curiosity, how do companies offering self-hosted/on-prem solutions monetize their offerings?

Do they rely on legal contracts to prevent customers from using the software for free or modifying it for their own purposes?

Is this serious? There's very little to go on in the repo to establish how well thought through this is.

I don't want to mark them down for poor language skills but the style of the comments on the TabbyML GitHub profile suggests a rather casual approach, and when combined with a lack of any serious documentation or even basic details beyond a sketched architecture diagram, I kind of wonder... Is there any particular context others can point to that I may be overlooking?

Just in case you didn't know that a project called Tabby exists (it was Terminus). It's a terminal (another one you could say). It's not my project, I'm just a user.

https://tabby.sh/

This looks exactly like what I need for a project I've been working on. How do you get your own code in to the model? Or is that the future fine-tuning step you're talking about?
Anyone know of a quick workaround for this?

    % docker run \
      -it --rm \
      -v ./data:/data \
      -v ./data/hf_cache:/home/app/.cache/huggingface \
      -p 5000:5000 \
      -e MODEL_NAME=TabbyML/J-350M \
      tabbyml/tabby
    Unable to find image 'tabbyml/tabby:latest' locally
    latest: Pulling from tabbyml/tabby
    docker: no matching manifest for linux/arm64/v8 in the manifest list entries.
    See 'docker run --help'.
I have an M2 Mac. I believe Docker is capable of running images compiled for different architectures using QEMU style workarounds, but is that something I can do with a one-liner or would I need to build a new image from scratch?

Previous experiments with Docker and QEMU: https://til.simonwillison.net/docker/emulate-s390x-with-qemu

Do any of the authors of these models disclose a clear list of codebases/discussion sites these models are trained on? Are they at the least avoiding stuff clearly GNU (or such) licensed? Is the hope that there is just no way to figure it out?
Trying the demo I got "value is not a valid enumeration member; permitted: 'unknown', 'python', 'javascript'". Trying some clearly identifiable Rust code with language set to 'unknown' I got a completion in Java back. The completion made sense, was properly indented and syntactically correct, it just was the wrong programming language.

Is this a limitation of the hosted demo or the chosen model, or do I simply have to wait a bit until my favorite niche language is supported?

What sort of resources do we need to run this, particularly VRAM? Also, how does this compare to Fauxpilot?
Great to have a self-hosted solution, both for data privacy and quality improvement potentials.
What LLM is this using? Or did you train your own?
After using Github Copilot for a couple of weeks, it doesn't seem to do much other than Stackoverflow/blog copy paste. Does anyone get much else from it when writing non-boilerplate, non-tutorial, and non-stackoverflow types of code?
Considering it's an alpha version I think a VSCode extension is the missing part
It's kinda funny that the same people knocking GPT-3 copilot are going to be losing their absolute minds when copilot X drops.

GPT-4 can autogenerate most code a business needs, just need a lone engineer to keep it in check.

Does this come pre-trained, or does it require training? What sources of data are you training on if so?

I restrict all usage of AI tools trained from publicly-sourced data because of an unknown copyright restriction, general unease, and lawsuits; however if this can be trained solely on my own codebases that are of clean providence, I can be 100% guaranteed against potential lawsuits.

Copilot is a cool tool, but super scary from a legal perspective. And even more heavily regulated industries (that I'm not in) would absolutely need their own firewalled version.

I'm glad to see projects like this.

I had a look at the demo (https://huggingface.co/spaces/TabbyML/tabby) and wasn't too impressed with the generated code for the default sample prompt (binary search) -- it recurses infinitely if the item is missing. It would be interesting to compare with Copilot's output. No idea how one would go about fixing this (other than manually add a correct binary search implementation to the training data, which feels like cheating).

Request to https://tabbyml-tabby.hf.space/v1/completions:

    {
      "language": "python",
      "prompt": "def binarySearch(arr, left, right, x):\n    mid = (left +"
    }
Response:

    {
      "id": "cmpl-...",
      "created": 1680867355,
      "choices": [
        {
          "index": 0,
          "text": " right) >> 1\n    if x < arr[mid]:\n        return binarySearch(arr, left, mid - 1, x)\n    elif x > arr[mid]:\n        return binarySearch(arr, mid + 1, right, x)\n    else:\n        return mid"
        }
      ]
    }
Formatted code:

    def binarySearch(arr, left, right, x):
        mid = (left + right) >> 1
        if x < arr[mid]:
            return binarySearch(arr, left, mid - 1, x)
        elif x > arr[mid]:
            return binarySearch(arr, mid + 1, right, x)
        else:
            return mid
Manually written test cases:

   arr = [1, 3, 5, 7]
   print(binarySearch(arr, 0, len(arr), 5)) # 2 (correct)
   print(binarySearch(arr, 0, len(arr), 4)) # RecursionError
Runnable demo:

https://tio.run/##lY/BCoJAEIbvPsUPXVw0yCyCKG@9QB3Fg@maC7bKuI...

This is awesome, and glad to see that SkyPilot is useful in distributing Tabby on any cloud! https://github.com/TabbyML/tabby/blob/main/deployment/skypil...
I like to use my brain as my copilot
Anyone dare to guess how fast would GH copilot be if it ran locally ?

My main problem with copilot is latency/speed - I would shell out for 4090 if it meant I could use local copilot model that's super fast/low latency/explores deep suggestions.

Very interesting to have a self-hosted version of Copilot. Will definitely try this out.
I hope someone comes with the AI-powered refactoring. I think it has more practical applications than Copilot, at least from my experience in C and C++ test runs of it.
Hi, will this work on AMD Gpu? I have plenty of VRAM available
Is AMD gpu support on the roadmap?

I'm interested in this stuff but I won't be buying Nvidia cards.

My assumption is that this would not be fast enough for practical use on M1/M2 Macbooks. Is that correct?
This exakt name is already used by a big open-source projekt, https://github.com/eugeny/tabby (50k stars) maybe consider changing the name for better SEO