As of this podcast there is a literal Artificial Intelligence (AI) landrush to create the tools needed to build and fine-tune language models and what is currently available is only scratching the surface on what can eventually be accomplished. Larger companies such as Meta, Google and Microsoft have been spending considerable resources to create language models that will eventually be used in their own service offerings while also providing the untrained editions for the hungry public to consume and tailor.
Of course, NVIDIA and AMD are busy creating the hardware required to run these AI frameworks and will make a killing in the process. As you know it wasn’t the gold prospectors who got rich during the gold rush, it was the stores selling the shovels. While the sheer size and scalability of this hardware is mindboggling you have to remember that these are intended for large scale AI farms and datacenters which will be accessible by thousands of people at the same time.
As it turns out you can run your own private AI with only a fraction of that hardware. In this episode we explore the Ollama platform and how it can be used to host a private open source LLM (Large Language Model) which is only accessible locally. The system we are testing with is an old AMD Threadripper 3950X with 64GB of DDR4 a single 2TB NVME SSD and a single RTX 3070Ti graphics card. You can run Ollama on just about anything including an old laptop, High-End PC or the gaming rig you built last season. These systems don’t need a GPU but, if you have one, the response time will be considerably faster and require less system resources.
Related Links
Host ALL your AI Locally with NetworkChuck
Episode 161 featured music:
Little People - Start Shootin' (http://www.littlepeoplemusic.com/)
