We are excited to announce the preview of Fabric User Data Functions! This feature is made to empower developers to create functions that contain their business logic and connect to Fabric data sources, and/or invoke them from other Fabric items such as Data pipelines, Notebooks and Power BI reports.
Fabric User Data Functions leverage the Python runtime to provide a serverless platform to author runnable applications on Fabric to solve common data engineering use cases such as data cleaning, data validation, data enrichment, integration with third party systems, and much more! Check out What is Fabric User data functions (Preview)? documentation for more details.
What is the need for writing functions in Fabric?
Writing custom solutions is essential in data engineering. This allows for handling complex transformations, optimizing performance, and integrating diverse data sources beyond the capabilities of low-code tools. While existing solutions provide speed and convenience, they often lack flexibility, scalability, and the ability to meet unique business requirements.
During our private preview, we met with several customers who highlighted the benefits for creating reusable libraries of functions. These functions contain standardized functionality that is specific to their business and that they can leverage in other parts of their systems architectures.
What's new in the preview of Fabric User Data Functions?
There are several new features we introduced in the past few months that you can now start using today:
Portal editing experience
In addition to the VS Code extension, you can now write and edit your functions directly in your browser by using the portal editor experience. The portal editor features the following functionality:
- Use the code editor toolbar to access functionality such as Undo, Redo, Copy/Paste, or to Reset your code to the published version.
- Insert a sample to quickly get started with everything you need to connect to a data source or perform data transformations.
- Use Pylance integrated IntelliSense to assist you in writing high-quality code by providing immediate access to documentation and suggestions.
Library management experience
You can now install public libraries from PyPI to enhance the functionality of your functions. Simply access the Library Management button in the portal editor toolbar and use the text field to search for your preferred Python libraries. The search functionality will auto complete the names of the libraries and filter to only those that are compatible with your runtime.
What are the integrations with Fabric items that I connect to with User Data Functions?
Fabric User Data Functions provides the following types of native integrations with your Fabric items:
Fabric data sources
You can create rich data applications by connecting to your Fabric data sources for reading, writing or transforming data. The Manage Connections feature supports creating connections to the following types of Fabric data sources:
- Fabric Warehouse
- Fabric SQL Database
- Fabric Lakehouse
- Mirrored Databases
Invoking from Fabric items
You can invoke User Data Functions from your Fabric items to leverage them in their execution. This allows you to re-use your functions and create sophisticated service architectures with standardized data processing functionality.
The following Fabric items support invoking User Data Functions using native integrations:
- Fabric notebooks
- Fabric Data pipelines
- Power BI reports (Preview)
- Data Warehouse using T-SQL (Preview)
How to get started with Fabric User Data Functions?
To get started with using this feature, simply navigate to the admin portal and turn on User data functions (preview). Make sure that your tenant is in a region that supports Fabric User Data Functions.
After that, you just need to create a new User Data Functions item by clicking on the 'Create' resource button. You can find it under the 'Develop' category:
And that's it! For any questions or thoughts, feel free to reach out to FabricUserDataFunctionsPreview@service.microsoft.com to contact the product and engineering teams. Submit your feedback on Fabric Ideas and join the conversation on the Fabric Community
Resources
- Fabric User Data Functions documentation.
- Check out our YouTube demo video.
- Read the blog post for using User Data Functions with Fabric Data pipelines