r/LocalLLaMA · · 1 min read

Mobile Offline LLMs: What do you use them for?

Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.

I've spent the last year or so playing around with open source MLX and GGUF models on iPhone hardware.

Given the limitations in memory, GPU/CPU/ANE, and in turn the context window I've been trying to figure out the best use cases for them.

I've also done a lot of testing with Apple Foundation model. It is not that capable, but it is good at understanding instructions for tool calling and actually seems to work well for fast summarization and classification prior to feeding data to more capable models.

I've been able to get models ranging from 0.5B to 8B to work well on the higher end devices.

So far I've been testing it out with building tooling such as web search, url scraping, summarization, research, analyzing local photo/video/documents/etc., and basic on-the-go coding help. With continuous compaction I've been able to hold virtually unlimited exchange chats even though context windows are limited to like 8-16k tokens.

Separate from more professional use cases, It also seems like potentially a good option for private, offline chat about your personal thoughts with models tuned with custom system prompts.

I know many here are focused on trying to run as good of a model as possible on their laptop or desktop hardware but it seems like mobile is an area that isn't really being explored extensively yet. What do you use mobile models for?

submitted by /u/James333i
[link] [comments]

Discussion (0)

Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.

Sign in →

No comments yet. Sign in and be the first to say something.

More from r/LocalLLaMA