LLM ASICs on USB sticks?
LLM ASICs on USB sticks?
Never really occurred to me before how huge a 10x savings would be in terms of parameters on consumer hardware.
Like, obviously 10x is a lot, but with the way things are going, it wouldn't surprise me to see that kind of leap in the next year or two tbh.
That would actually be insane. Right now, I still need my GPU and about 8-10 gigs of VRAM to run a 7B model tho, so idk how that's supposed to work on a phone. Still, being able to run a model that's as good as a 70B model but with the speed and memory usage of a 7B model would be huge.
Slowly, is how
I'm even more excited for running 8B models at the speed of 1B! Laughably fast ok-quality generations in JSON format would be crazy useful.
Also yeah, that 7B on mobile was not the best example. Again, probably 1B to 3B is the sweetspot for mobile (I'm running Qwen2.5 0.5B on my phone and it works tel real for simple JSON)
EDIT: And imagine the context lengths we would be ablentonrun on our GPUs at home! What a time to be alive.