MLXHubMLXHub AI Benchmark v1
How the MLXHub benchmark works
The benchmark measures how well an iPhone or iPad runs language models locally: generation speed, prompt processing, model loading, usable memory and sustained performance. The measurements are combined into one global score that can be compared with other devices in the public ranking.
It measures performance only. It does not evaluate the quality of model output.
01 · Tests
Test procedure
A complete run measures usable memory once, then runs a short and a long conversation with each of the four model configurations below: eight scenarios in total. Prompts and generation settings are fixed by the protocol.
| Model | Precision | Purpose |
|---|---|---|
| Gemma 4 E2B Instruct | 4-bit | Small resident model; baseline workload. |
| Gemma 4 E4B Instruct | 4-bit | Larger resident model. |
| Gemma 4 E2B Instruct | 8-bit | Same E2B model at 8-bit: higher precision and memory footprint. |
| LFM2.5 8B A1B | 8-bit | Mixture-of-experts model; measures expert streaming from storage. |
4-bit and 8-bit refer to how the model weights are stored. They are different workloads, not quality levels. LFM streams its experts from storage; the other three configurations stay resident in memory.
Short prompts measure a light input. Long prompts measure the cost of processing accumulated context. Running both exposes different bottlenecks.
Memory test
The test allocates memory gradually and records memory-pressure warnings and peak footprint. A safety margin converts those readings into the recommended usable memory for the run.
Model loading
Each scenario loads its model. Throughput is model size divided by load time, in GB/s, measured separately from generation speed.
Cold prompt processing
The model processes the short or long prompt with an empty prompt cache. Prompt tokens divided by processing time gives the cold prefill rate in tokens per second.
Prompt cache reuse
A second pass with a warm cache checks whether the retained context is reused. Its speed is recorded as a diagnostic only: it does not contribute to the score and never replaces the cold measurement.
Sustained generation
The model generates successive turns of up to 32 tokens for four minutes. The first 10 seconds are warm-up. Only measured generation intervals count toward tokens per second; time spent processing a new prompt is excluded.
Stability and thermal state
The test compares early and late generation speed and records the thermal state throughout. For the mixture-of-experts model it also measures expert read throughput. Each phase waits until the device reaches the required thermal state.
Duration
Eight scenarios of four minutes each give 32 minutes of sustained generation. Downloads, loading, prompt processing, the memory test and cool-down periods add to that. A scenario can extend by up to one minute to collect the minimum evidence: two independent turns and five scored seconds.
Recommended conditions
Use a Release build of the app, turn off Low Power Mode, disconnect external power and let the device cool down before starting. The app checks these conditions and marks phases affected by heat. Keep the app in the foreground: locking the device or leaving the app interrupts the run, and runs cannot be resumed yet.
The memory test and individual models can be run on their own. Partial runs keep their local measurements, but only a complete run produces a global score that can be published.
02 · Score
Score calculation
The global score combines six metrics with fixed weights. Generation speed and usable memory carry the most weight. The weights are a product decision, not the result of a statistical fit.
Generation
30%Tokens per second during scored generation. It has the highest weight because it determines how fast a response appears.
Usable memory
22%Recommended usable memory measured in the run. It determines which models and context sizes fit on the device. Measured once and compared with both references.
Cold prompt processing
15%Input tokens per second with an empty prompt cache. It determines the wait before a response starts, especially with long prompts.
Endurance
15%Combines speed retention and thermal state during sustained generation. It measures whether performance holds beyond a short burst.
Expert reading
10%MiB per second read by LFM’s expert stream. It measures workloads larger than resident memory. Reads may be served from the system cache, so this is not raw flash speed.
Model loading
8%Model size divided by load time, in decimal GB/s. Loading happens less often than generation, so it has the lowest weight.
Calculation steps
- For each scenario, the measurement is divided by the matching reference. A ratio of 1 equals the reference; 2 is twice the reference value. Each ratio has a floor of 0.05 and no upper limit.
- Ratios are combined with the geometric mean within each metric. Generation, prompt processing, loading and endurance combine all eight scenarios with equal weight. Expert reading combines only the two LFM scenarios.
- Memory is a single measured amount compared twice: with the 4-bit memory reference and with the 8-bit reference. In Benchmark v1 both references are the iPad Pro M1 measurement. The two ratios are combined geometrically and the 22% weight is applied once.
D: generation · P: cold prompt · L: loading · M: memory · E: endurance · S: expert reading
The score is not a sum of points or an average of the two category scores. Doubling generation alone multiplies the global score by 2⁰·³⁰, about 1.231: a 23% increase.
Calculation details
Endurance
Speed retention is the last generation window divided by the first, capped at 1. The windows are the first and last 15% of observed generation, taken from different completed turns. Thermal comfort is a time-weighted average: nominal 1, fair 0.85, serious 0.45 and critical 0.15. Endurance is the square root of retention × comfort. Heating during a valid phase lowers comfort; it does not invalidate the run.
Memory safety margin
With warning footprint W and critical footprint K, recommended memory is the smaller of 0.95 × W and 0.85 × K. Without a critical event, it is 0.85 × W. Values are truncated to whole bytes. If no warning occurs, the peak footprint is used as W.
Filtered scores
Model and conversation filters restrict the measurements being combined. An individual metric shows 1000 × its aggregated ratio. For a combined score without expert streaming, the storage factor is omitted and the remaining weights are not rescaled. A 4-bit score and an 8-bit score therefore describe different workloads.
Out-of-memory failures
A documented memory limit keeps the completed phases and assigns a ratio of 0.05 to the interrupted phase and the phases that depend on it. A load failure penalizes all metrics of the scenario; a cold-prompt failure keeps loading; a warm-prompt or generation failure keeps loading and cold prefill. The scenario stays in the average, so running out of memory never improves the score. Cancelled, omitted, heat-affected or invalid tests are not treated this way.
Calculator
Formula calculator
Select a measured example and adjust each metric. Performance controls multiply every scenario’s original ratio before the 0.05 floor is applied. Memory is a single amount compared with both references. The calculator illustrates the formula; it does not predict the score of a real device.
× the selected example
GiB of recommended memory
Calculated global score
iPhone 17 Pro Max
Intermediate values are not rounded; only the displayed score is. Each control changes one metric in isolation; in a real run, metrics affect one another.
03 · References
Reference device
Benchmark v1 uses a single reference device for both the 4-bit and the 8-bit references: the iPad Pro 12.9-inch (5th generation) with M1 and 8 GB of memory. Its run completed all eight scenarios without a pause. M1 is the first chip in Apple’s M series, the starting point of Apple silicon on Mac and iPad, so a score reads as a multiple of that starting point: 1570 is about 1.57 times the M1 reference. The reference is the 8 GB model; an iPad Pro M1 with 16 GB has more usable memory and therefore scores higher.
| Device | 4-bit score | 8-bit score | Global score |
|---|---|---|---|
| iPad Pro M1 (8 GB) | 1000 | 1000 | 1000 |
| iPhone 17 Pro Max | 1352.91 | 1654.93 | 1569.68 |
Because the iPad Pro M1 defines every reference, it scores exactly 1000 in 4-bit, 8-bit and global. The iPhone 17 Pro Max is an example measured against that reference, not a reference itself.
Measured on app Release 2.2.1 (84). iPad (reference): iPadOS 27.0 build 24A435; iPhone (example): iOS 27.0 build 24A437. “Release” refers to the app build, not to the operating system.
Download the Benchmark v1 reference values (JSON)Reference values used for scoring, not ranking data. Scores are only comparable within the same benchmark version.
04 · Ranking
Public ranking
The ranking shows how well a device performs for local inference and how it compares with the ten highest-scoring devices published by MLXHub users.
One row per device
Each row is a device model with a specific memory size, for example iPhone 17 Pro Max with 12 GB. It shows the best global score published for that combination, so one device cannot occupy several positions.
Top 10 and your result
The main view lists the ten highest-scoring devices, for all devices or only iPhone or iPad. If your best result is not in the Top 10, it is shown below the list for direct comparison. The full list is available from the same screen.
Publishing
Publishing is optional and requires an iCloud account. Results appear in the ranking as soon as they are published; review takes place afterwards. Your account keeps one best result per device and benchmark version, and a new run replaces it only if its global score is higher.
Comparing individual tests
Compare tests filters the ranking by quantization, model and conversation length. The score then combines only the selected measurements, as described in Calculation details.
05 · Validation
Result validation
The ranking does not take submitted scores at face value. The app checks every result when it is published and again when the ranking is loaded, and the MLXHub team reviews published results.
Fixed protocol
Scenarios, settings, warm-up, timing and generation windows are fixed, which prevents selecting favorable measurements. A partial run cannot be published as a complete benchmark.
Score recomputation
The app recomputes each score from its measurements using the current calibration and excludes any result whose stored score does not match. Review decisions are bound to the exact submission, so editing a measurement invalidates them.
Plausibility limits
Measurements far above what the reference device achieves are rejected. The whole result is excluded; scores are never capped or adjusted. The limits are deliberately wide so that faster future hardware is not penalized.
Review by the MLXHub team
The MLXHub team periodically verifies published scores, both manually and through automated reviews. Results that do not pass review are removed from the ranking, and accounts that submit manipulated results can be excluded.
Result labels
- Not reviewed
- Published and checked by the app, not yet reviewed. Included in the ranking.
- Reviewed
- Passed the automated consistency review.
- Verified
- Verified by the MLXHub team.
Limitations
- The checks reduce the effect of incorrect or manipulated data, but they cannot prove that a device, account or measurement is genuine.
- Model weights are not yet cryptographically bound to each submission, and model repositories and OS versions can change between runs.
- Ambient temperature and background activity are not controlled. Expert reads may be served from the system cache.
Method versioning
Any change to workloads, timing, weights or references requires a new benchmark version. Validation rules are versioned separately. Published scores keep the meaning of the version they were measured with.
Last updated 11 October 2026 · Benchmark v1