Item — HN Reader

Open access for next 5 hours (Ternary-Bonsai-8B-Q2_0.gguf, running on RTX 3090) or until server crashes or the this spot instance gets taken away :) =>

https://uklkyvetsjf7qt-80.proxy.runpod.net

    ./build/bin/llama-server \
     -m ../Ternary-Bonsai-8B-Q2_0.gguf \
     -ngl 999 \
     --flash-attn on \
     --host 0.0.0.0 \
     --port 80 \
     --ctx-size 65500 \
     --batch-size 512 \
     --ubatch-size 512 \
     --parallel 5 \
     --cont-batching \
     --threads 8 \
     --threads-batch 8 \
     --cache-type-k q8_0 \
     --cache-type-v q8_0 \
     --log-colors on

# llama.cpp is forked one: https://github.com/PrismML-Eng/llama.cpp.git

# The server can serve 5 parallel request, with each request capped at around `13K` tokens...

# A bit of of benchmarks I did:

# 1. Input: 1001 tokens, ttfs: 0.3 second, outputs: 1618 tokens ~140t/s

# 2. Input: 9708 tokens, ttfs: 2.4 second, outputs: 2562 tokens at ~106t/s

# Vram usage was consistently at ~7GiB.

> https://huggingface.co/prism-ml/Ternary-Bonsai-8B-gguf/resol...

4 comments

Thanks a lot, I was about to clone their llama.cpp branch and do the same.

Some more interesting tidbits from my go-to tests:

* Fails the car wash test (basic logic seems to be weak in general)

* Fails simple watch face generation in html/css.

* Fails the "how many Rs in raspberry test" (not enough cross-token training data), but will funnily assume you may be talking about Indian Rupees and tell you a lot about raspberry prices in India without being asked. Possible Indian training data unbalance?

* Flat out refuses to talk about Tiananmen square when pushed directly - despite being from a US company. Again, perhaps they are exposed to some censored training data? Anyways, when slowly built up along the conversation by asking about locations and histories, it will eventually tell you about the massacre, so the censorship bias seems weak in general. Also has no problem immediately talking about anything Gaza/Israel/US or other sensitive topics.

* Happily tells you how to synthesize RDX with list of ingredients and chemical process step by step. At least it warns you that it is highly dangerous and legally controlled in the US.

The 1-bit Bonsai and Ternary Bonsai models are all based on the corresponding Qwen3 model: https://raw.githubusercontent.com/PrismML-Eng/Bonsai-demo/re... (page 4)

Thanks, already suspected as much. Also gives context to the other comment here that says it is basically equivalent in accuracy to Qwen3.5-4B. Essentially seems to be a very good quantization of that model, not a new BitNet.

It's a good-per-byte-but-not-in-absolute-terms quantization of Qwen3-8B that's comparable in accuracy to Qwen3.5-4B at 4-bit quantization (which makes the 4B model larger in terms of storage, though the lower number of parameters and hybrid attention give it a speed advantage if you're not bottlenecked on memory bandwidth for the model weights.)

I think the future of economically useful AI is to build efficient reasoners. A goal of model as oracle of truth or encylopedia for our world is orthagonal to a goal of a model that can reason about novel input. I think the focus on meme litmus tests is somewhat misguided. AI is not suited to be a spell checker, a news source, a history book, or an idependent developer. but if it can reason about a prompt and augment human effort then that is useful. the idea of an ai without human like world model matching or exceeding our own world model or store of facts is misguided in my opinion.

I think the future of economically useful AI is to build efficient reasoners. A goal of model as oracle of truth or encylopedia for our world is orthagonal to a goal of a model that can reason about novel input. I think the focus on meme litmus tests is somewhat misguided. AI is not suited to be a spell checker, a news source, a history book, or an idependent developer. but if it can reasom about a prompt and augment human effort then that is useful. the idea of an ai without human like world model matching or exceeding our own world model or store of facts is misguided in my opinion.

The future of AI I would imagine is that AI acts purely as logic within constraints

If you ask it who Ahmad Al Sharaa is, it outputs an answer that he was an ISIS commander (he wasn't, his group was rather opposed to them), and that he was killed in a US airstrike in 2017. It also refuses to answer direct questions about the cutoff date in its training dataset, but if you ask it who the president of syria is, it happily says that as of the training cutoff date of october 2023, bashar al-assad is the president. Same question about who the prime minister of canada is gets the same october 2023 response and says it's Trudeau.

Interestingly if you ask it "who is the dalai lama" it outputs an answer that certainly wouldn't be acceptable to post online in mainland china. It says:

The Dalai Lama is the spiritual and political leader of Tibet, and the title is held by the incarnate lama who is believed to be a reincarnation of the previous Dalai Lama. The Dalai Lama is a central figure in Tibetan Buddhism and holds a unique position in Tibetan society, both as a religious leader and as a symbol of Tibetan identity and independence. Key Points About the Dalai Lama: 1. Current Dalai Lama: Tenzin Gyatso (14th Dalai Lama)

    Born: 1935 in Amdo, Tibet (now part of China).
    Education: He studied at Ganden Monastery and later at Dunhuang University in China.
    Title: He is known as the 14th Dalai Lama, a title that has been passed down through generations of lamas who are believed to be reincarnations.
    Role: He is the spiritual leader of Tibetan Buddhism and the de facto leader of the Tibetan diaspora, particularly in India.

2. Historical Context

    The title of Dalai Lama has been held by Tibetan Buddhist lamas for centuries. It means "Ocean of Wisdom" in Tibetan.
    The 13th Dalai Lama (1876–1959) was a key figure in Tibetan history, known for his efforts to maintain Tibetan independence and promote modernization.
    He fled to India in 1950 after the Chinese Communist Party (CCP) took control of Tibet in 1951.

3. 14th Dalai Lama: Exile and Leadership

    The 14th Dalai Lama, Lama Tenzin Gyatso, was exiled to India in 1959 after the Tibetan Uprising.
    He has led the Tibetan government-in-exile in India since then, advocating for Tibetan independence and cultural preservation.
    He has also been a prominent voice for human rights and peace.

4. Political Status

    The 14th Dalai Lama is not recognized by the Chinese government as the legitimate ruler of Tibet, which is part of the People's Republic of China (PRC).
    The Central Tibetan Administration (CTA), based in India, is the self-proclaimed government of Tibet.
    The PRC considers Tibet to be an inseparable part of China and has implemented policies to integrate Tibet into the Chinese state.

5. International Recognition

    The 14th Dalai Lama is widely recognized internationally as the spiritual leader of Tibet and as the de facto leader of the Tibetan people.
    Many countries and international organizations support the Tibetan cause, but they are not recognized as a sovereign state.

Summary:

The Dalai Lama is the spiritual and political leader of Tibet, currently Lama Tenzin Gyatso (14th Dalai Lama), who has been in exile in India since 1959. He is a central figure in Tibetan identity and has played a key role in the global Tibetan movement for independence and cultural preservation.

If you'd like to know more about his background, policies, or role in the Tibetan independence movement, feel free to ask!

played around with it a little, works great for - basic conversational tasks - basic reasoning - creating little UI snippets and not good for one shot coding tasks. once it starts hallucinating about something you can't make it rectify that. during a coding task it wrote some madeup library function, despite of pointing it out, giving docs, it kept producing the same code with the madeup function all while acknowledging the problem. btw, thanks for putting it together!

Fyi, I believe `--flash-attn on` doesn't do anything, you should instead use `--flash-attn 1`. I'm getting ~150t/s on a RTX 3080 10GB as well with f16 cache type.

Thanks.. updated my local docs :)

update: Well, spot survived... and since a lot of the folks are still using it, I'm keeping it alive for 2 hours more.

Update 2: Terminated.

Thank you all for trying it out.