data engineering
2268 TopicsFabric Pipeline Error
Today, we noticed that our production pipeline started failing with the following error: Error: BadRequest Error fetching pipeline default identity userToken{ "code": "LSROBOTokenFailure","message": "AADSTS50173: The provided grant has expired due to it being revoked, a fresh auth token is needed. The user might have changed or reset their password. The grant was issued on '2026-08-24T05:50:05.7907970Z' and the TokensValidFrom date (before which tokens are not valid) for this user is '2026-09-18T16:05:07.0000000Z'. Trace ID: 250ed633-08f0-4dcb-bad0-11596983e901 Correlation ID: 430f0833-fbc4-44f8-b19a-883ada2cad74 Timestamp: 2026-09-18 18:29:30Z", "target": "PipelineDefaultIdentity-3e591cb6-2704-49f2-9efd-7196287c9feb","details": null,"error": null }FetchUserTokenForPipelineAsync Has anyone encountered this issue before or have any idea what could be causing it? Any suggestions on how to resolve this would be appreciated.92Views1like5CommentsClaude Code Integrate with Power Bi
Hi everyone, I’m working on a POC for Power BI + Claude Integration, specifically the Report Authoring skill.We need to use Claude Code to create the required report pages and visuals. I’m currently trying to estimate the Claude Code credit/token consumption and approximate cost for completing this POC. Has anyone worked on a similar POC using Claude Code? If so, could you please share: Approximate credits/tokens consumed Estimated cost Any recommendations for estimating the usage before starting Any guidance or experience would be really helpful. Thanks!136Views2likes7CommentsCopy Job - Amazon S3 connector fails with region signing error on generic endpoint
I'm trying to connect Fabric's Amazon S3 connector (Copy activity, pipelines) to a bucket in ap-southeast-2, using Access Key authentication. When I create a connection with the generic regional endpoint: https://s3.ap-southeast-2.amazonaws.com connecting fails with: Expression.Error: Invalid Url. The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'ap-southeast-2' I've added s3:ListAllMyBuckets, s3:ListBucket, and s3:GetBucketLocation to the IAM user, and the error persists identically.Solved88Views0likes8CommentsFabric Copy Activity fails in Query mode but succeeds in Table mode through an on-premises gateway
Hi Fabric Community, I am using a Microsoft Fabric pipeline Copy Activity to copy data from a Fabric Lakehouse to an on-premises SQL Server database through a dedicated on-premises data gateway. I tested the Copy Activity using both of the following source connection types: Lakehouse connection Lakehouse SQL analytics endpoint connection With both connection types, the behavior is the same: Lookup Activity: Successful Script Activity: Successful Copy Activity using Table mode: Successful Copy Activity using Query mode: Fails When the Copy Activity source is configured with Use query = Query, it fails with the following error: ErrorCode=SqlFailedToConnect,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Cannot connect to SQL Database. Please contact SQL server team for further support. Server: 'xxxxxxxx.datawarehouse.fabric.microsoft.com', Database: 'LH_XYZ', User: ''. Check the connection configuration is correct, and make sure the SQL Database firewall allows the Data Factory runtime to access.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Data.SqlClient.SqlException,Message=A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server),Source=.Net SqlClient Data Provider,SqlErrorNumber=53,Class=20,ErrorCode=-2146232060,State=0,Errors=[{Class=20,Number=53,State=0,Message=A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server),},],''Type=System.ComponentModel.Win32Exception,Message=The network path was not found,Source=,' However, when I change the source configuration from Query to Table and select the source table directly, the Copy Activity completes successfully. Has anyone encountered this behavior? Does Query mode use a different runtime, connector path, or metadata-discovery process when the Copy Activity runs through an on-premises gateway? Are there any known limitations or additional gateway requirements for using a custom query as the source? Any guidance on how to diagnose or resolve this would be appreciated.105Views0likes10CommentsSharePoint Excel ingestion, Dataflow Gen2 vs shortcut + notebook CU efficiency
I need to ingest and transform multiple Excel files stored in SharePoint folders. I am comparing: Pure Dataflow Gen2 using SharePoint.Files and Power Query transformations. Lakehouse Files shortcut to the SharePoint folder, followed by all transformations in a Fabric notebook using PySpark. For the same files, transformations, output, schedule, and capacity: Is the shortcut + notebook approach expected to consume fewer Fabric Capacity Units than pure Dataflow Gen2? What is the recommended approach for SharePoint Excel-folder ingestion?130Views1like6CommentsPower Automate Export to PDF Returns Blank Report for Report Connected to Shared Semantic Model
Hi Team, I am facing an issue with Power Automate's "Export To File for Power BI Reports" action. Scenario I have a Power BI Semantic Model (Dataset A). Report A is built directly on Dataset A. Report B is another report built using the same semantic model(shared dataset/thin report approach). Expected Behavior When Power Automate exports Report B to PDF, the report should contain the same data that is visible in Power BI Service. Actual Behavior Exporting Report A through Power Automate generates a PDF with data correctly displayed. Exporting Report B through Power Automate generates a PDF, but the visuals are blank and no data is shown. There are no export errors. Additional Findings Manual export from Power BI Service (File > Export > PDF) works correctly for both reports and the generated PDF contains data. The issue only occurs when exporting through Power Automate. Both reports use the same semantic model. The semantic model is accessible and contains data. If I export pages from Report A, data is visible in the PDF. If I export pages from Report B (connected report/thin report), the PDF is blank. Questions Does the Export To File for Power BI Reports action have any limitations with thin reports or reports connected to a shared semantic model? Are there any permission requirements (Build permission, RLS, semantic model access, etc.) that differ between manual export and Power Automate export? Has anyone experienced blank PDF exports when using a report connected to an existing semantic model while manual exports continue to work? Any guidance would be appreciated. Thank you.57Views1like3CommentsSalesforce data in Fabric with bronze/silver/gold: what would you change?
Hi community, I've recently built a Salesforce analytics setup on Microsoft Fabric with my team at datatobiz and I'd like to hear how you would do differently. Here is the setup: Ingestion: Dataflow Gen2 connects Salesforce to Fabric, with scheduled, incremental loads for objects such as Accounts, Opportunities, Orders, Products, Campaigns and Cases Bronze: raw Salesforce tables land in OneLake with the schema preserved, for auditability and lineage Silver: Fabric and Databricks notebooks handle duplicates, data types, business rules and object relationships Gold: subject-specific marts for Sales Performance, Customer 360 and Operations Pipelines: scheduled refreshes with monitoring and alerts Reporting: Power BI semantic models on star schemas, with DAX measures and role-based views Governance: Azure AD role-based access, Microsoft Purview lineage and Azure Key Vault I'd like your views on: Is Dataflow Gen2 a good fit for incremental Salesforce loads, or would you use Copy activity or notebooks instead? Has anyone used Fabric and Databricks notebooks together in a silver layer? How did it work for lineage and monitoring? What data quality checks do you automate between bronze, silver and gold? Would you change anything in this layering for Salesforce data? Thanks in advance!23Views0likes2CommentsFabric Link Setup
Hello Community, I'm trying to Configure Fabric Link with FnO to sync My FnO data to Fabric. While creating Fabric Link for FnO I'm getting 2 Options as per below SS. One is F&O Entities (Caption 1) Second is F&O tables (Caption 2). I want to Understand How "FnO entities" and "FnO tables" different from each other, In which case I need to choose from second Option(F&O tables Caption 2) & in which case I need to choose from (F&O Entities Caption 1). Thank you56Views0likes3CommentsHow to design a metadata-driven framework to run 600+ Fabric notebooks with parallel execution?
Hi Fabric Community, We are currently planning to migrate 600+ notebooks from Azure Databricks to Microsoft Fabric. Our current requirement is to build a metadata-driven orchestration framework using: Fabric Data Pipelines for orchestration SQL Database for metadata/control tables Fabric Notebooks for data processing OneLake/Lakehouse as the target storage Instead of creating separate pipelines for each notebook, we would like to have a generic metadata-driven pipeline that dynamically reads notebook information from SQL Database and executes the required notebooks. For example, our metadata table could contain: Notebook ID Notebook name/path Priority Dependency Parameters Data volume Expected runtime Compute/pool requirement Retry count Active/inactive flag Our main challenge We may need to execute approximately 40 notebooks in parallel. We would like to understand the recommended Fabric architecture for this scenario. Specifically: How should we design the metadata-driven pipeline to dynamically execute 600+ Fabric notebooks? How should we control parallelism when around 40 notebooks need to run simultaneously? Should we use a single Spark environment/pool, multiple environments/pools, or some other approach? How should we decide which Spark compute configuration should be used for each notebook based on: Data volume Memory requirement Processing time Shuffle-intensive workloads Small vs. large workloads? If 40 notebooks run simultaneously, how does Fabric manage the underlying Spark compute and capacity? Should we be concerned about resource contention or throttling? Is it recommended to classify notebooks into workload groups such as:and then control concurrency separately for each group? Small / Medium / Large / XLarge What is the recommended way to implement dependency management? For example:where Notebook B should only start after A succeeds. Notebook A → Notebook B → Notebook C What is the recommended approach for retry, failure handling, logging, and restartability for 600+ notebooks? Is there a recommended metadata-driven orchestration pattern/reference architecture in Microsoft Fabric for this scale? Are there any Fabric-specific limitations or best practices we should consider when running hundreds of Spark notebooks with high concurrency? Our main objective is to build a scalable framework where we can manage 600+ notebooks from metadata instead of maintaining hundreds of individual pipelines, while still controlling Spark compute and parallel execution efficiently. Any architectural recommendations, reference implementations, or real-world experience with similar workloads would be highly appreciated. Thanks!146Views1like12Comments