Phi-3-MLX: Run Phi-3 Vision and Language Models on Your Mac

TL;DR

TL;DR: Phi-3-MLX is an open-source Python package that runs Microsoft’s Phi-3.5-vision and Phi-3.5-mini models locally on Apple Silicon Macs using the MLX framework. Install with pip and start using vision and language models in under 8GB of RAM.

Source and Accuracy Notes

⚠️ This section is MANDATORY. All links must be verified from actual source, not guessed.

• Project page: github.com/JosefAlbers/Phi-3-Vision-MLX
• Source repository: github.com/JosefAlbers/Phi-3-Vision-MLX — README read in full
• License: MIT (verified from LICENSE file in repo)
• HN launch thread: news.ycombinator.com/item?id=41002393
• PyPI package: phi-3-vision-mlx

What Is Phi-3-MLX?

Phi-3-MLX is a community project that ports Microsoft’s Phi-3.5 family of models to run natively on Apple Silicon via the MLX framework. It supports two model families:

• Phi-3.5-vision-instruct — a multimodal model that handles both text and images, ideal for visual question answering
• Phi-3.5-mini-instruct — a 128K context language model optimized for text-only tasks

The project wraps both models in a unified Python interface and uses MLX for memory-efficient inference on Apple Silicon. Models can be run with quantization to fit within 8GB of RAM on an M-series Mac.

Setup Workflow

Prerequisites

• Apple Silicon Mac (M1, M2, or M3)
• 8GB RAM minimum (with quantization); 16GB recommended
• Python 3.10 or later
• macOS 12 or later

Step 1: Install the Package

The quickest way to install is via pip:

pip install phi-3-vision-mlx

The README notes that the PyPI version may not always be the latest. For the most current version, clone the repo directly:

git clone https://github.com/JosefAlbers/Phi-3-Vision-MLX.git
cd Phi-3-Vision-MLX
pip install -e .

Step 2: Run the CLI

Once installed, the phi3v command launches an interactive session:

phi3v

This starts the visual question answering interface. You can pass an image URL directly:

from phi_3_vision_mlx import generate

generate(
    'What is shown in this image?',
    'https://collectionapi.metmuseum.org/api/collection/v1/iiif/344291/725918/main-image'
)

Step 3: Use Quantization (Optional)

To reduce memory usage, enable model quantization:

generate("Describe the water cycle.", quantize_model=True)

Cache quantization is also supported:

generate("Explain quantum computing.", quantize_cache=True)

Step 4: Batch Text Generation

For batch processing multiple prompts:

prompts = [
    "Write a haiku about spring.",
    "Explain the theory of relativity.",
    "Describe a futuristic city."
]
generate(prompts, max_tokens=100)

To use the language-only model instead of the vision model for batch text:

generate(prompts, max_tokens=100, blind_model=True)

Deeper Analysis

Why this matters for local AI: The MLX framework is Apple’s optimized tensor library for Apple Silicon, offering memory efficiency and speed advantages over CPU-based inference. Phi-3-MLX makes this accessible without needing to manage model files or configuration manually.

Quantization support: The project supports both model quantization and KV cache quantization, which can significantly reduce memory footprint. This is critical for running larger models on base-tier M1/M2 Macs with only 8GB unified memory.

LoRA fine-tuning: The project includes LoRA fine-tuning capabilities, enabling users to adapt the base models to their own use cases without full model retraining.

API integration: The README mentions integration with image generation and text-to-speech APIs, though these appear to be optional extensions rather than core functionality.

Practical Evaluation Checklist

• [ ] Installs cleanly with pip on M1 MacBook Air (8GB)
• [ ] phi3v CLI launches without errors
• [ ] Vision model answers questions about a sample image
• [ ] Quantization reduces RAM usage below 8GB
• [ ] Batch generation processes multiple prompts correctly
• [ ] Language-only model (blind_model=True) works for text-only tasks

Security Notes

• All inference runs locally — no data leaves your machine
• Models are downloaded from Hugging Face on first use; verify you trust the source
• The pip package is community-maintained; audit the code before granting network access in sensitive environments

FAQ

Q: Does this work on Intel Macs? A: No. Phi-3-MLX is built specifically for Apple Silicon using the MLX framework, which only supports M1/M2/M3 chips.

Q: How much RAM do I need? A: The minimum is 8GB with quantization enabled. For comfortable use with larger batches or longer contexts, 16GB is recommended.

Q: Can I fine-tune the models? A: Yes. The project includes LoRA fine-tuning capabilities. The README references this as a built-in feature of the framework.

Q: Is the PyPI version always up to date? A: The README explicitly notes that the PyPI version may lag behind the GitHub repository. For the latest code, install from source with pip install -e . after cloning.

Q: What macOS version is required? A: macOS 12 (Monterey) or later, matching MLX framework requirements.

Conclusion

Phi-3-MLX makes running Microsoft’s Phi-3.5 multimodal models as simple as pip install phi-3-vision-mlx. With quantization support for 8GB Macs, a clean Python API, and both vision and language capabilities, it is a practical entry point for developers who want local AI inference on Apple Silicon without managing model infrastructure. If you have an M-series Mac and want to experiment with Phi-3 class models today, this is the lowest-friction route available.