I recently wrote about small language models as the future of enterprise generative AI; here I want to build on this to look specifically at the implications of mass adoption of generative AI models on-device (that is, running locally on your laptop, tablet or phone). With hundreds of millions of laptops and over a billion smartphones sold each year, the potential scale of AI adoption is huge.
This area is gaining huge traction in the media with Apple expected to announce AI integrations across their devices at WWDC next week, Microsoft already defining their future with the Copilot+ PC, and the chip manufacturers (Intel, AMD, Nvidia & Qualcomm) all announcing their own silicon for the AI PC. The commonality with all these announcements is the goal of introducing more AI into your computer, tablet or smartphone (usually by running on a dedicated chip called a Neural Processing Unit, or NPU).
Let’s step back one moment and look at the types of language models which can be broken down by size category:
- Large Language Models (LLMs). Developed and run by the hyperscalers in the cloud. Typically containing 10s or 100s of billions of parameters.
- Small Language Models (SLMs). Custom models that can run locally on-device. Often fine tuned to a variety of use cases. Typically contain 1 to 10 billion parameters.
- Super Tiny Language Models (STLMs). Very small models for running in low power applications such as IoT at the edge. Typically contain up to 500 million parameters.
The selection of a model type may depend upon factors such as market entry, budgetary constraints, usage, access, desired outcomes and the potential return on investment.
However, what is clear is that, apart from LLMs (which are restricted to running in the cloud), there’s going to be a wide and diverse range of SLMs and STLMs running locally on our devices.
Given the wide and diverse range of use cases for locally running AI that will open up, the risks to businesses and consumers associated with running AI locally are significant and need addressing. For example, risks can be associated with privacy concerns, the possibility of toxic & harmful content, subtle and ingrained bias, or straight up security concerns.
At Chatterbox Labs, we have been addressing AI risks for many years within industries such as banking, telecommunications, oil & gas, mining, federal government etc. Our technology can run on a range of devices, from your standard laptop through to massive GPU accelerated cloud infrastructure. Therefore, the opportunity for embedding our AI risk software on devices, to alleviate the risk associated with locally running AI models, is a significant new market opportunity we will explore.
