Fabric notebook development with an AI harness/LLM of your choice

goreavin — Mon, 04 May 2026 18:57:05 GMT

I had a hard transformation to make.
Pandas was too slow.
Polars looked faster.
I wanted the machine to do the work.

The built-in copilot could not carry the load.
Too small. Too careful.
No frontier models. No long leash.

I could use the Pi harness to write the code.
That part was easy.
The hard part came after.

The notebook failed.
I read the errors.
I fixed the code.
I ran it again.

After a while, I became the bottleneck.

So I asked a simple question:

Is there a better way?

There was no clean way to expose Fabric notebooks to an external coding agent.
At least not officially.

But there was VS Code.
There was the Fabric CLI.

There was Livy.
There was the SQL analytics endpoint.

That felt like enough.

So I built around them.

The CLI edits the notebooks.
The CLI runs them too.
Livy works as well, though it fights you on lakehouse writes.

I added a custom Python tool.
Clanker generated it.
It catches cell-level execution and errors.

The harness reads the errors.
Tries again.
Runs again.
Repeats until the notebook executes cleanly.

Then the SQL endpoint checks the transformed data.

After that, I define test cases.
The harness keeps iterating until every test passes.

Closed loop.

The code is here: Link

It works for me.

If you find sharper edges, smoother paths, or stranger failures, change it.
Commit your edge cases.
Let the next person suffer less.

Re: Fabric notebook development with an AI harness/LLM of your choice

v-veshwara-msft — Tue, 05 May 2026 10:42:12 GMT

Hi @goreavin ,

Thanks for sharing this and contributing to the community.

Really appreciate the effort you put into documenting it. This will be helpful for others working on similar scenarios.

If you ever need assistance or have further questions, feel free to reach out or raise a new thread.

Keep engaging with the Fabric Community.

Best Regards,
Vinay.