Literature Review: MELTing point: Mobile Evaluation of Language Transformers | Bnechmarking LLMs on Phones

Finished reading the paper: MELTing point: Mobile Evaluation of Language Transformers

I am starting to benchmark LLMs on edge devices, particularly phones thus been reading a lot on the what has been done and what is currently being done and wanted to share you my journey of reading such papers and my takes on them.

• What is this about?

This is one of the finest papers on mobile-edge performance benchmarking I have read in a while and it is so because of their true-to-user setup, like how a user will actually use the LLMs on their device and their idea of prefill-decode disaggregation!

They have tested - iphone 6 SE - iPhone 14 Pro - S23 - Pixel 6a - NVIDIA JetsonNano - NVIDIA AGX Orin

The models - tinyLlama - Llama-2 - Gemma - Zephyr-3B - Mistral-7B

with backends - MLC-LLM - llama.cpp

Now these are not used barebones but rather with a chat app- MLChat (Android and iOS) and LLMFarm (iOS Metal)

So, they have a sophisticated setup with a RPi-4 at the center of all acting as the controller responsible for

• deployment of tasks to the devices
• collection and monitoring
• interaction with the devices

The two segments - PhoneLab (all Android and iOS) and JetsonLab (NVIDIA devices) connect to this RPi-4.

They used a Thermal camera for measuring temps on phones + SysFS on NVIDIA devices (thus one disadvantage that we dont get per component thermals sadly). They used a relay + a YAKUSH controller to power on the phones all at once and power at once (they did so that the USB only acts as data transfer cables. not charging but it kinda failed for iPhones especially)

They filtered oasst1 dataset and gathered 50 prompts with at least 6-10 prompts and median of ~36 words. They do it 3 times.

Now, for the fun part. They literally made sure that the whole user experience (opening chat app, chatting with LLM, LLM responding etc) is all captured and they do it through ads on Android and their custom HID setup for iOS (using the RPi-4 attached to a keyboard/mouse to send in typed response, scrolling etc).

They also explore the possibility of doing the prefill on NVIDIA devices and sending the rest to the phones to decode through WiFi 6!

Another I liked is their QES score or Quality Exp score which they categorize in three types:

• Responsiveness: The phone still be responsive and not lag or become unresponsive during LLM chatting session
• Stability: There should be stability when doing continuous chatting with the LLM (they tested this with three rounds of continuous hammering all of the 50 conversation in the phones)

• Temperature: Well, the phones did became hot to even touch (skin tmp) 47.1 Celsius to be exact.

• Interesting case was when doing the Stability checks by continuously hammering the phone with 3 x 50 (50 per round) convos, there were spikes in throughput (for NVIDIA AGX too) at two points namely 20 and 32 iteration at both throughput ad prefill. The reason they gave was DVFS which could explain the weird spikes for throughout but for prefill I think its KV cache rebuilding phase which could explain the jump in the prefill cus its compute bound thus data hungry, ... well same could be for throughput ig since its less data now to Read/Write for sometime...

• iPhones loaded the models (all of them) within 5 sec and pixel too but the s23 was more on higher end of 5 sec with exception yes been 14 sec and >30 sec for 7B and 3B respectively and I think for that 3B they tried loading it in fp32?.

Results (not much the usual):

• The GPU is better than CPU obviously here with iPhones taking the lead, especially with LLMFarm as the backend and not MLC as it can effectively use the Metal GPU acceleration than MLC-LLM.

• Androids' a little faster than iPhones in CPU decode

• they used two Q3 and Q4 and Q4 both throughout faster and energy efficient on phone.

• There was this case of phone being unresponsive when loading Zephry 3B on phones (one would have missed it if it were not for them entering and seeing in the command to touch/scroll like a human would)

• Higher latency models tested on phones consume more mAhr

• The ALU not being utilized properly and mostly time spent in Reads and Writes to the memory and forth.

Quirks:

they did grid search for a few conifers for ctx/gen and batch size which makes the test results difficult to compare for every device.

they did do the chat evals which is good but doing the HID and whole "doing the way human will do the chatting" seemed quite too much because such stuff can add in weird reading in the monitoring system, could have juts played an app or do some ml work in the background of the phone...

didnt play with # of threads (they did call to threads usage in LLMFarm for iPhones which could explain their high benchmark numbers like tok/sec )

they didnt give the phone to rest during QES score testing nice!

the DVFS thing could have been explained if the CPU/GPU freq maps were given + fine grained thermal results would have been so

Overall this was a good read for the models and backends they have used and especially including the multi-turn conversations, QES scoring, realistic testing of the the devices. So, if one can set it up, then it becomes a very interesting way to test out edge devices.