A locally fine-tuned small model controls 487 Mac apps at 98% accuracy

CJ Zafir@cjzafir

Wao, this post blew up, and I triggered some Saastophers.

But let me tell you more about SLM fine-tuning.

In April-May this year i fine-tuned a 6.5B model: Mac-1 (it controls 487 Mac native apps)

My goal was to build a better Siri (and i achieved that).

Here's how i fine-tuned a model that runs on my Mac:

I used Codex GPT 5.5 model to design the fine-tuning structure.

- what dataset we need?
- which SLM will be perfect?
- How to perform fine-tuning?
- How to run the local model?

And trust me the current models (apart from claude models, are amazing at model fine tuning orchestration)

First we decided the models.
We used 2 models:
- Qwen 3.5 2B (for classification)
- Qwen 3.5 4B (for tool calling and responses)

Then GPT 5.5 suggested that we need a dataset of alteast 20k examples.

So we researched if there's any open source dataset available on macOS native tools. But found nothing.

Then we decided we'll create our own synthetic dataset.

Here's the fun part. For creation of the dataset I used Deepseek models.

Codex GPT 5.5 as planner and reviewer and Deepseek v4 pro (preview) as executor.

In 2 days we had verified, dedupped, evaluated dataset of 30k examples.

Then codex designed the fine tuning configs and decided things like:
- using unsloth studio
- LoRA adapter not QLoRA
- using colab pro (for cheaper 1 H200 WebGPU access)

In 8 hours the fine-tuning job was done, then we ran evals.

And in first try we got 78% accuracy on 400+ native MacOS tools under 45 categories.

I was shocked and excited on what we did.

Next 2 weeks i was obsessed about taking the accuracy to 95% and i achieved 98% on my 5th training.

The day i was about to launch the model, Apple launched new Siri (that use Gemini model at the backend)

So i decided not to go head on with Apple right now cuz you vs Apple is never good.

But I think i might launch the model now or make it open source. I think the model still beats new siri.

So why am i sharing this?

You need to understand that SLMs can beat LLMs in trained environments!

And most companies have these environments.
- customer support
- finance
- insurance
- sales
- inventory
- data entry

And use cases:
- document analysis
- information extraction
- translation
- compliance
- summarization
- classification
and many more.

A fine-tuned SLM can do all this 10x faster, 95% cheaper and on their servers (no data going outside).

And that's why many companies are now building their own AI solutions.

I'd say fine-tuning is is what coding was before 2024 (a proper skill).

That's not it.

I fine-tuned a 9B Ornith 1.5 model that writes like me (fine-tuned on my 10,000 tweets) but that story is for some other day.

I'll share step by step tutorials in coming days cuz i just love this part of AI. Stay tuned.