Hi, community,
I'm currently working on a complex Microsoft Fabric project that requires going through a CI/CD process to deploy Fabric through dev, test, pre-prod and prod environments.
Because the fabric deployment pipelines are limited: 1. have no parametrisation options; 2. when the workspaces are associated with GitHub where they don't push across the complete item definitions. I have gone the route of using Azure DevOps with GitHub as outlined in this link I found: https://github.com/Azure-Samples/modern-data-warehouse-dataops/tree/main/single_tech_samples/fabric/...
The solution outlined in the Azure Samples repo is centred around using the Fabric API to download, create, and update Fabric items.
For the most part, this is working. However, for some items, there are either heavy limitations or they don't seem to work. I have been using https://learn.microsoft.com/en-us/rest/api/fabric/articles/ as the guideline for what to expect regarding item definitions, as the items don't seem to follow a common format.
There are currently four roadblocks I am facing:
1. For Lakehouse Data Pipeline:
a. Can I create the Data Pipeline using Service Principal? Strangely, other Fabric items support CRUD operations with the Service Principal, but only the Data Pipeline API can be authenticated via user identity (Items - Create Data Pipeline - REST API (DataPipeline) | Microsoft Learn).
b. I can create a new Data pipeline using the API but can't pull down the definitions from the workspace to commit them to GitHub. It only seems to support the name and description. I tried exporting it manually, and it gives the deployment template and a manifest but no .platform file. It's unclear what files I should expect here, as they don't seem to be documented. Am I missing anything here?
2. For KQL Database: Although it's currently supported by Fabric API as stated by Microsoft documentation when exporting KQL Database item definition and config files via the Fabric API, I'm getting an 'unknown error' from the API response. Has anyone experienced the same?
3. Eventhouse and Lakehouse: Child items such as KQL Database and Queryset for Eventhouse, Notebook and Data Pipeline for Lakehouse are created alongside when creating Eventhouse and Lakehouse on the Fabric workspace. However, the hierarchical lineage is not preserved when downloading the item definition files for Eventhouse and Lakehouse. These child items are treated as independent items. Is there any impact when deploying them into the new environment as independent items without the lineage information?
4. Is there any way to programmatically delete Fabrics workspaces? When deleting a workspace via the Fabric API, the recently deleted workspace will remain obsolete depending on the retention period setting and cannot be permanently removed using Fabric API. Manual intervention is required to remove those workspaces. Otherwise, all succeeding automated processes will be stopped.
Can anyone help with any of these questions? That’d be much appreciated
Regards,
Spencer
