-
Notifications
You must be signed in to change notification settings - Fork 975
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature Request: Add support for Raspberry Pi Ai Kit #548
Comments
llamafile already works great on Raspberry Pi 5. It goes very fast with CPU alone. We put a lot of work into making that happen. Their AI accelerator module is cool, but support it isn't on our roadmap. I'm not even sure if it has functionality which would help transformer models. If anyone here understands its capabilities and could explain how it could potentially help make our matrix multiplications go faster, then I'm all ears. But I'm willing to bet we're already offering the best RPI support to you today, that's possible now. |
@jart just little effort from my side and so posting ai reply only. i am not that much technical person into this. The Raspberry Pi 5 AI Kit can potentially help make your matrix multiplications go faster for LLMs in several ways, leveraging its hardware and software capabilities: Hardware Acceleration
Software Optimization
Specific Techniques
|
So what you're telling me is that it's got a 32-bit ARM CPU on it with 2 cores. I doubt there's much advantage offloading to that. Plus having to use a proprietary SDK and drag and drop a special executable and reboot the thing for a program to run. It'd be simpler and platform-agnostic to just plug a second Raspberry Pi into your Raspberry Pi over the ethernet and we'll give you software that lets you cluster llamafile instances. Wouldn't you rather have that instead? |
ok got it, thanks for clarity. |
Ok that sounds awesome! I'm not the OP but I'm VERY interested in that! |
Prerequisites
Feature Description
Add support for raspberry pi ai kit to run llamafile.
Motivation
It will be much faster to run LLM on such a small and cheaper device.
Possible Implementation
Not aware
The text was updated successfully, but these errors were encountered: