A very recognizable sentiment on Ben’s blog and bookmark (apologies for the alliteration)(twice)

Running models locally - either directly on researcher laptops, or on researcher-controlled infrastructure - is inevitably going to be a big part of how AI is used in any sensitive context.

At our agency, we also explore the possibilities to have locally run models. Either on our laptop or on a controlled server environment. On my Macbook Pro M3 I can run large models like Llama3 and Gemma2 without it breaking a sweat. I also use local transcribing through Macwhisper for recorded sessions. The development in these fields are very interesting to watch.

However, the speed on those local models needs improvement. When I “talk” to my notes in Obsidian through the (online) Claude API or a local LLM, the online service definitely wins in speed. Which makes sense, because of the gigantic computing power behind the third party services.

The Future is local community-driven AI?

The technology of AI advances at a break-neck speed, both local and online. I hope this performance gap in speed and precision will narrow. When you have your own controlled environment with enough CPU/GPU, for a reasonable fee, this moat might become obsolete.

This opens up new ideas!

  • Custom LLM deployment: We could run our own controlled environment on any (green) server, make sure it’s privacy friendly etc. But we could also do this as a city, as a community, as a family. It also gives me the power to choose a more greener server and have better insight in the carbon footprint of AI use by our group.
  • Collaborative AI: This opens up new ways of communication and sharing information as well. A community of enthousiasts could share prompts and finetune specific models, without any need to send data to third party services.
  • Federated learning: To go one step further, why not make LLM-servers federated as well, so they can talk to each other when communities democratically decide so? Have my family LLM talk to other families, share resources, information and AI output.

The path forward: Decentralized AI

Just like social media networks become less centralized (well, it could speed up a bit) maybe we should start considering how we can move away from centralized one-size-fits-all LLM providers. When we build (or rent) our own AI infrastructure as a community, we can create more tailored, privacy-conscious, and potentially more environmentally friendly AI solutions.

This could lead to a more diverse ecosystem of AI models, adapted to the specific needs and values of a community, while still allowing for collaboration and resource-sharing when desired.

What do you think?