topic Re: How creating a Python custom library in Data Engineering

How creating a Python custom library

pmscorca — Mon, 25 May 2026 16:07:01 GMT

Hi,

is it possible to create a Python custom library to use in a notebook in order to protect the proprietary code? How?

Thanks

Re: How creating a Python custom library

lbendlin — Mon, 25 May 2026 20:46:24 GMT

please provide some more details. Protect from what?

Re: How creating a Python custom library

tayloramy — Tue, 26 May 2026 13:48:56 GMT

Hi @pmscorca,

As has been stated on multiple of your posts now, if you want to protect custom logic, the best way to do that is to create a custom workload in Fabric.

Anything else, even a custom python library, will expose your code to those who care to look for it.

Re: How creating a Python custom library

apturlov — Tue, 26 May 2026 15:34:50 GMT

Hi @pmscorca, your question is multi-faceted and may require more than a single answer. You state two separate goals: create a custom Python library to use in a Notebook, and protect your propriatary code. The goals may be related but independent. Because code protection is absolutely outside of Fabric concers I will only share my thoughts about custom code in Notebooks.

You create a custom Python library as any other Python development project in your local Python development environment with your regular development process. Before you do this, you need to answer the following quesions:
- decide what execution environment in Fabric you'll be running your code in; you already stated a Notebook.
- make sure that your code is fully compatible with Fabric execution environment; you need to understand dependencies of your code and make sure those dependencies can be resolved in Fabric Notebook execution environment. You may wat to take a look at this guide Use Python experience on Notebook - Microsoft Fabric | Microsoft Learn
- decide how you install your code in your Fabric execution environment. There are two baseline methods in Fabric: install your code using pip command in your Notebook cell, or install your library in a Spark environment. You may want to check out this guide Manage libraries in Fabric environments - Microsoft Fabric | Microsoft Learn.

Once you have you library installed in a Fabric environment, test it by running some code in your Notebook that uses your library.

This is a relatively straightforward process and should let you go ahead with developing your custom Python code and running it in the Fabric Notebooks.

A thought about code protection. The rule of thumb is that Fabric does not provide any features of facilities to protect your code. Any Python code is interpreted and exists in your execution environment as text, so no help in this area. This is a common concern for Python development in general and not Fabric specific, so you need to research protection techniques regardless of where you deploy and run your code.

Hope this answer helps. If so, consider giving it kudos and mark as a solution to help other community members.

Re: How creating a Python custom library

pmscorca — Tue, 26 May 2026 16:39:42 GMT

Hi, I need to find the right solution and so I'm searching the right answers.

The dev toolkit don't seem mature yet. Providing a workloard means that a customer must already have Fabric, but a product could be purchased before Fabric.

So far I seem to have been getting answers from official documentation rather than from experiences.

Thanks

Re: How creating a Python custom library

apturlov — Tue, 26 May 2026 16:51:11 GMT

@pmscorca My answer below is from an experience. But as I stated already, you are mixing different goals in a single question. My advise is to separate your goals into different questions and seek the answer for the Python protection code question outside Fabric community as it is irrelevant. There are literally no obstacles in running any custom Python code in Fabric provided it's compatible with the execution environment. Every way of protecting your Python code from abuse and reverse engineering is bound to affect how your code works and it's a matter of experimentation and finding the right balance. Start from making sure that your code runs in Fabric as intended in your private Fabric environment then move towards protection options. In my experience-based opinion Fabric is not going to help you with protection.

Re: How creating a Python custom library

tayloramy — Wed, 27 May 2026 13:50:40 GMT

Hi @pmscorca,

What is your end goal?
Are you developing a Fabric solution, or are you trying to sell Fabric services to clients that don't have Fabric?

The question asked is how to protect users from viewing your code. In Fabric, the way to do that is to develop a custom workload.

If your goals are different than just protecting the code, then you need to describe what your question is.

Re: How creating a Python custom library

pmscorca — Wed, 27 May 2026 14:16:06 GMT

Hi,

I need to implement and propose a solution to potential customers who have or do not have Fabric; those who don't have Fabric first will buy the solution and then Fabric.
I don't understand how a custom workload could allow to protect the code; it seems that Fabric lays everything out in plain sight.

Re: How creating a Python custom library

apturlov — Wed, 27 May 2026 15:09:53 GMT

@pmscorca I know that @tayloramy knows what he is saying about using Fabric custom workloads as he created them. I have yet to try it out so cannot chime on that. However, Python is Python, and as you correctly said, nothing can be hidden in Fabric once deployed. So, I just repeat, from my first-hand experience Fabric is not your answer for code protection. The fact that your solution is made specifically to run on Fabric platform should not stop your customers from purchasing it. It's similar to if someone wants to use software they need to have a computer too. Most likely your code protection is not a technical concern. Your customers agree with your terms and conditions when they decide to buy and use your solution. That said, it does not mean you should not research various ways of protecting your code, but, don't let it stop you from creating and selling your solution. That's all I have to say on this topic. If you are satisfied, feel free to give it kudos and mark as a solution. If not, keep researching and good luck.

Re: How creating a Python custom library

tayloramy — Wed, 27 May 2026 16:35:06 GMT

Hi @pmscorca,

It might be worth while to join the Fabric Discord that I run (link in my signature) so we can have a more conversational exchange here.

A custom workload will hide your code the same way Micorosft hides their code. FOr example, when you make a pipeline, you don't see the code behind how the pipeline works. When you make a copy job, you don't see the underlaying code for how the copy job operates.

Those are both examples of items inside a workload. WIth the extensibility toolkit, you can create your own custom Fabric Item types that will appear to users who install the workload in their tenant.

Now if your goal is to re-sell Fabric as a solution, maybe a workload isn't what you want.

Do you intend your solution to run on your Fabric tenant permenantly, or do you intend your customers to purchase fabric and then deploy your solution?

If you intend for everything to always run on your tenant/capacity, then you can take the approach that Epic Systems is taking with Cogito Cloud, where they manage the entire thing end to end and make the data available via B2B sharing. In this scenario, because all the notebooks and code is in your tenant, your customers can't see it so there is no need to do anything special to hide it.

If you intend to use your solution as a selling point and expect your customers to pay for a fabric capacity as well as your solution, then a custom workload is what you're going to want.

Re: How creating a Python custom library

pmscorca — Wed, 27 May 2026 17:38:57 GMT

Hi, thanks for your reply.
Let's assume that my solution is composed of a lakehouse, a pipeline for data ingestion, a notebook for data transformation, a semantic model and a set of Power BI reports.

I don't want to offer a service on my tenant; it needs to be on the customer's tenant as much as possible.
In a such case, you say that implementing a custom workload is the right way to follow, ok.

So, what does a customer workload create on the customer tenant? Lakehouse plus pipeline plus notebook plus semantic model plus reports? F.e. if the custom workload creates a notebook, is the related code protected?

Re: How creating a Python custom library

pmscorca — Wed, 27 May 2026 17:44:27 GMT

Hi, thanks for your reply.

Ok, but how protecting code might impact solution architecture.

Re: How creating a Python custom library

tayloramy — Wed, 27 May 2026 19:15:27 GMT

Hi @pmscorca,

A custom workload can create anything that you can code.
There's built in APIs to access system storage, and Fabric has lots of APIs for creating lakehouses, accessing lakehouse data, ect.

I would recommend that you don't use a notebook for data transformation, and instead build a "data transformation" item that does everything you need it to do, and keeps the logic hidden from the end users.

Once you have built this out, then you need to get it published to the Azure marketplace, and from there anyone can purchase it from you.

Note that code is only protected for your custom item types, so if your workload creates a default item type like a notebook using the APIs, then that will behave just like any normal notebook and the customer will be able to see the code.

However if you create a custom "transformation" item type, then your code will be hidden.

Re: How creating a Python custom library

pmscorca — Thu, 28 May 2026 08:29:37 GMT

Hi, thanks for your reply.

So, in a custom workload it is enough to create some custom items in order to protect the code and all these custom items can reside on the customer's tenant, isn'it?

Thanks

Re: How creating a Python custom library

tayloramy — Thu, 28 May 2026 13:50:10 GMT

Hi @pmscorca,

That is correct. Here is an example of a custom workload with the extensibility toolkit: https://community.fabric.microsoft.com/t5/Extensibility-Toolkit-Gallery/Fabric-Job-Registry/m-p/5006346

Re: How creating a Python custom library

v-veshwara-msft — Mon, 01 Jun 2026 06:06:51 GMT

Hi @pmscorca ,
Thanks for reaching out to Microsoft Fabric Community.

Just checking in to see if your query is resolved and if any responses were helpful.
Otherwise, feel free to reach out for further assistance.

Thank you.

Re: How creating a Python custom library

v-veshwara-msft — Thu, 04 Jun 2026 06:11:00 GMT

Hi @pmscorca ,

We wanted to kindly follow up regarding your query. If you need any further assistance, please reach out.
Thank you.