Ollama is now available on Windows in preview

For anyone else who missed the announcement a few hours ago, open-webui is the rebranding of the project formerly known as ollama-webui [0].

I can vouch for it as a solid frontend for Ollama. It works really well and has had an astounding pace of development. Every few weeks I pull the latest docker images and am always surprised by how much has improved.

[0] https://github.com/open-webui/open-webui/discussions/764

Out of curiosity, what're you using the fine-tunes for? Do you fine-tune them on your own data or are they just publicly available models you use for different tasks?

Is there something similar but in the terminal instead?

AMD GPU support is definitely an important part of the project roadmap (sorry this isn't better published in a ROADMAP.md or similar for the project – will do that soon).

A few of the maintainers of the project are from the Toronto area, the original home of ATI technologies [1], and so we personally want to see Ollama work well on AMD GPUs :).

One of the test machines we use to work on AMD support for Ollama is running a Radeon RX 7900XT, and it's quite fast. Definitely comparable to a high-end GeForce 40 series GPU.

[1]: https://en.wikipedia.org/wiki/ATI_Technologies

Same. I really want AMD to succeed because as a long time Linux user I have strong distaste for Nvidia and the hell they put me through. I paid a lot for a beastly AMD card in the hopes that it would be shortly behind Nvidia and that has most definitely not been the case, and I blame AMD for not putting the resources behind it.

AMD, you can change, but you need to start NOW.

AMD clearly believes that this newfangled "GPU compute" fad will pass soon, so there's no point to invest in it.

This is one of the worst acts of self-sabotage I have ever seen in the tech business.

llamafile has amd gpu support. on windows, it only depends on the graphics driver, thanks to our tinyBLAS library.

https://github.com/Mozilla-Ocho/llamafile/releases/tag/0.6.2

By default it opens a browser tab with a chat gui. You can run it as a cli chatbot like ollama as follows:

https://justine.lol/oneliners/#chat

As others have mentioned, Ollama uses Llama.CPP under the hood and they recently released Vulkan support which is supposed to work with AMD GPUs. I was able to use llama.cpu compiled with Vulkan support with my app [1] and make it run on an AMD laptop but I was unable to make it work with Ollama as it makes some assumptions about how it goes about searching for available GPUs on a machine.

[1]: https://msty.app

Ollama has a opencl backend. I'm on Linux and clblast works great with AMD cards. As far as I remember opencl on Windows did not have that much issues, but its been a while.

Maybe there’s proper support soon in AI landscape [0].

[0]: https://news.ycombinator.com/item?id=39344815

I've had success using my AMD GPU with the OpenCL backend for llamacpp. The ROCm backend had pretty bad performance though.

Can't edit the parent post, here's the issue to track the Windows ROCM support in ollama: https://github.com/ollama/ollama/issues/2598

AMD is the underdog, and that's what happens when you choose the underdog.

btw, it triggers Program:Win32/Wacapew.C!ml detection

Off topic but what did you use to make your landing page?

Is this similar to LLM Studio?

Any plans for a Linux client?

adding gemini API?

I'm trying the Windows version. What really sticks out is that buttons don't have tooltips. It's impossible to know what they do if you don't click them.

Also in the conversation view you have two buttons "New Chat" and "Add Chat" which do two different things but they both have the same keybind ^T

This is a really interesting question. I think there's definitely a world for both deployment models. Maybe a good analogy is database engines: both SQLite (a library) and Postgres (a long-running service) have widespread use cases with tradeoffs.

In addition to the initial loading time noted by the other posters:

You may want to use the same inference engine or even the same LLM for multiple purposes in multiple applications.

Also, which is a huge factor in my opinion, is getting your machine, environment and OS into a state that can't run the models efficiently. It wasn't trivial to me. Putting all this complexity inside a container (and therefore "server") helps tremendously, a) in setting everything up initially and b) keeping up with the constant improvements and updates that are happening regularly.

It doesn’t make sense to load the weights on the fly- that is gigabits of memory that has to be shuffled around. Instead, you have a long running process that serves up lots of predictions

(edit: someday soon, probably to multiple clients too!)

This is a good thing IMO. I don't have a very powerful laptop or workstation, but do have a multi-GPU headless server. These projects allow me to experiment with LLMs on my server, and expose an API and web UI to my LAN.

In addition to everything that everyone else has said: I run Ollama on a large gaming PC for speed but want to be able to use the models from elsewhere in the house. So I run Open-WebUI at chat.domain.example and Ollama at api.chat.domain.example (both only accessible within my local network).

With this setup I can use my full-speed local models from both my laptop and my phone with the web UI, and my raspberry pi that's running my experimental voice assistant can query Ollama through the API endpoints, all at the full speed enabled by my gaming GPU.

The same logic goes for my Stable Diffusion setup.

Because it adds flexibility. By decoupling the frontend from the backend it's much easier for other devs not directly affiliated with the server repo (e.g. Ollama) to design new frontends that can connect to it.

I also think it allows experts to focus on what they are good at. Some people have a really keen eye for aesthetics and can design amazing front and experiences, and some people are the exact opposite and prefer to work on the backend.

Additionally, since it runs as a server, I can place it on a powerful headless machine that I have and can access that easily from significantly less powerful devices such as my phone and laptop.

> I don’t like running background services locally if I don’t need to. Why do these implementations all seem to operate that way?

Because it's now a simple REST-like query to interact with that server.

Default model of running the binary and capturing it's output would mean you would reload everything each time. Of course, you can write a master process what would actually perform the queries and have a separate executable for querying that master process... wait, you just invented a server.

Because running it locally really means running it on a cloud server that you own and is called by other server that you own. This gives you the ability to make the interfaces lightweight and most importantly to not pay premiums to model servers.

The main reason I see is to use the same AI engine for multiple things like VSCode plugins, UI apps, etc.

That being said I use LM Studio which runs as a UI and allows you to start a local server for coding and editor plugins.

I can run Deepseek Coder in VSCode locally on an M1 Max and it’s actually useful. It’ll just eat the battery quickly if it’s not plugged in since it really slams the GPU. It’s about the only thing I use that will make the M1 make audible fan noise.

I personally find it very useful, because it allows me to run the inference server on a powerful remote server while running the UI locally on a laptop or tablet.

I'll probably uses that because the Rust binding to llamacpp doesn't works on windows (well, cpu only works, so not usable). Python is broken (can't install the deps)

Also mind that loading theses models take dozens of seconds, and you can only load one at a time on your machine, so if you have multiple programs that want to run theses models, it make sense to delegate this job to another program that the user can control.