As you may know, ChatGPT collects a lot of data on the users for the improvement of their AI, but this poses risks in its own way. I was wondering whether there are privacy alternatives to ChatGPT. Perhaps on F-Droid or Aurora/PlayStore, or for Linux.
Are there any alternatives you know of? Or are there other ways to interact with ChatGPT without giving personal information, such as a privacy focussed front-end?
There are private GPT solutions coming, f.e. https://www.fujitsu.com/global/products/data-transformation/data-driven/ai-test-drive/
They are aimed at companies that for compliance reasons want to self host.
I wouldn’t trust an llm solution with sensitive information unless I host it myself.
Do you happen to know how this self-hosting would work? Can I run it at my desktop/phone or even a raspberry pi? How is the quality of generated results compared to ChatGPT?
Typically llm are rather ressource intensive - you need beefy hardware to run those at speed. Especially if you intend to train them with your data to improve their relevance. I don’t think mobile phones or run to the mill laptops are going to be enough for any non-trivial implementations. I might be skewed by experiences on non-personal projects though.
I can run 7B models on my laptop with its embedded GPU. Running on a phone or a Pi is possible with smaller models, but very slow. Expect good speed with a desktop Nvidea GPU. Later this year, there should be new computers with an NPU integrated to the CPU which should speed up computers that don’t have a dedicated GPU. (But a GPU will still outperform them by a lot.)
70B models will run very slowly on even the best consumer hardware due to memory limitations.