Swiftlet Runs an 80B Qwen Model in 4.3 GB of RAM on Apple Hardware

Swiftlet is a new open-source inference runtime targeting Apple Silicon. The headline numbers: an 80B Qwen model runs inside 4.3 GB of RAM on a Mac, and a 35B model runs on-device on an iPhone.

The memory reduction comes from extreme quantization. Getting an 80B model under 5 GB requires roughly 1-bit or sub-2-bit weight representation — territory that, until recently, caused unacceptable quality degradation. The fact that Qwen-class models survive this compression and remain useful is partly a property of the models themselves, which were trained with quantization-aware techniques, and partly the result of careful layer-wise precision decisions in the runtime.

Swiftlet targets the Apple Metal compute stack directly. This sidesteps Core ML's compilation overhead and lets the runtime manage memory layout for unified memory architectures — the reason the RAM figures are so low. Unified memory means CPU and GPU share the same physical pool, so the model does not need to be duplicated across device boundaries.

For engineers building local inference pipelines on Apple hardware, this changes the constraint model. An 80B parameter model was previously a server-only asset. Swiftlet moves it to a laptop. That affects offline-capable applications, privacy-sensitive workflows, and anything where API latency or cost is a bottleneck.

The iPhone result is the more surprising data point. On-device 35B inference on a phone implies token throughput is slow, but for many use cases — summarization, structured extraction, code completion with local context — throughput matters less than availability and privacy. This opens a class of applications that cannot rely on a network connection.

Swiftlet is available on GitHub under an open-source license. Engineers looking to run large models locally on Apple hardware without a cloud dependency should evaluate it against llama.cpp and MLX to benchmark quality-per-GB on their target models.

Source