DataFlow gen 2 not updating the data in my Warehouse
Hey Community, I have this automated pipeline: The DataFlow performs some transformations and dumps data into the Warehouse. And from the warehouse, I create my report. The pipeline ran successfully, however, the new columns I added in my dataflow are not reflected in my warehouse. Why is this happenning? And how can I make sure the data reflects? Cause I'm getting a lot of data and want it to see the warehouse and dashboard updated. Thanks !
Issue with PowerBIDatasetRefreshOperator in Airflow: Authentication Error
I’m currently using Airflow to orchestrate some notebooks in Microsoft Fabric. My setup leverages the FabricUser connection and FabricRunItemOperator for earlier steps, and everything works smoothly up to this point. However, when I attempt to use the PowerBIDatasetRefreshOperator in the final step with the same FabricUser connection, I encounter the following error: Authentication failed: AADSTS7000215: Invalid client secret provided. Ensure the secret being sent in the request is the client secret value, not the client secret ID, for a secret added to app 'b39a6123-149f-4fad-8659-55f249cb5e9a'. Trace ID: 9516efd5-2ead-4059-8cae-957fdd72f600 Correlation ID: 8ec4dfbf-8e41-497e-aaec-1061d6204919 Timestamp: 2024-11-27 16:17:48Z What I’ve Tried Verified the connection details for the FabricUser. Checked the app registration in Azure Active Directory to ensure the client secret is correctly configured. Confirmed that the secret being used is the client secret value, not the client secret ID. Despite these checks, the issue persists. My Questions Is there a specific configuration or prerequisite I might be missing when using PowerBIDatasetRefreshOperator in conjunction with the FabricUser connection? Are there any known limitations or best practices for using this operator in Airflow with Microsoft Fabric? How can I ensure that the authentication process works consistently for this operator? Thank you in advance for your help.
Df gen2 Save Changes deficiency
miguel A major diffrence between gen1 and gen2 is while exiting the gen1 df it gives the following options Gen 1 at exit at bottom vs Gen2 at exit at bottom The implication of the above is, if I open the same gen1 df in two different browser tabs and edit and save one (tabl1 -v1 to v2) and then I can simply exit out of the other dataflow in other browser tab (tab2 - v1) by choosing any of the options (at exit, at bottom). The current datatflow is v2 (desired v2). but for gen2 df if I open the same gen2 df in two different browser tabs and edit and save one (tabl1 - v1 to v2) and then if I exit out of the other dataflow in other browser tab (tab2 - v1) because it neither gives Save/Dsicard/Cancel option at exit nor the Cancel option at bottom, v2 gets overwritten by v1. The current datatflow is v1 (desired v2). It means one would be heavily penalized if unaware of this behavior and can loose hours of work. Can you please test this on your side and let me know if a permanent fix is expected? This has undesirable effects.
Is query folding in data flow related to user credentials access limitations?
I am trying to replicate a Data flow which is created by another person to test out Increemental refresh. I observed that their query steps are folding and my steps are not folding. In general if query is not folding it means that we are unable apply transformations steps directly on the source. Iam referencing following articles. https://learn.microsoft.com/en-us/power-query/query-folding-basics Understanding query evaluation and query folding in Power Query - Power Query | Microsoft Learn Query folding indicators in Power Query - Power Query | Microsoft Learn Please let me know if the following queries makes sense as credentials might be different: Is query folding in data flow related to user credentials access limitations? Ref image: Fold/unfold FAQ on the Fabric User Groups
Dataflow Gen2: Data source connection error (API)
mslearn-fabric (microsoftlearning.github.io) I have Data Source connection issues while trying to create Dataflow(Gen2) via Synapse Data Engineering using the guidelines in the link above. The import tabs (Excel, SQL, Text/CSV, dataflows) give the error message below. "Something went wrong while loading required resources. Please try refreshing the page or come back later to try again. (APIs: connections In Scope)." Much more, both import from Excel and Text/CSV tab generated "Unexpected error" page restricting further access to the import pages. Please see error message below. ---------- Message ---------- Unhandled HTTP Request Exception. Status Code: 400. Reason: undefined. Response Text: "ExternalServiceError". ---------- Stack ---------- ApiResponseError: Unhandled HTTP Request Exception. Status Code: 400. Reason: undefined. Response Text: "ExternalServiceError". at p.handleLoadEvent (https://uk.prod.powerquery.microsoft.com/MashupEditor/scripts/p.h.a.c2dedd539a7896950b5f.bundle.js:2:34257) at XMLHttpRequest.<anonymous> (https://uk.prod.powerquery.microsoft.com/MashupEditor/scripts/p.h.a.c2dedd539a7896950b5f.bundle.js:2:35587) ---------- Session ID ---------- e58577fb-a25f-4462-9228-a9742e974b16 ---------- Request ID ---------- f52b4c18-1052-464d-93eb-7b381f53fe01 Please advise how to resolve this issue.
SQL Analytics Endpoint - Table constraints disappear after running Dataflow Gen2
Hi, I created a Lakehouse with 2 tables: Fact table: RandomNumbers Dimensions table: Dim_Date I created a relationship between the tables in the SQL Analytics Endpoint When I query the [INFORMATION_SCHEMA].[TABLE_CONSTRAINTS] view, I can see that these relationships have been registered as constraints on the tables in the SQL Analytics Endpoint. When I connect to the RandomNumbers table of the SQL Analytics Endpoint from Power Query in Power BI Desktop, I can see that there is an expandable column so I can easily expand columns from Dim_Date into the RandomNumbers query. I have a Dataflow Gen2 which, when it runs, does a Replace with Dynamic schema of the Dim_Date table. After running the Dataflow Gen2, these are my observations: The relationship still appears in the model pane of the SQL Analytics Endpoint ✔️ The relationship still works in the Default Direct Lake semantic model ✔️ The constraints are not found anymore when I query the [INFORMATION_SCHEMA].[TABLE_CONSTRAINTS] view ❌ The expandable column is not found anymore in Power Query in Power BI Desktop. An import mode semantic model I built which uses this expandable column, get failed refreshes because the column can't be found anymore. ❌ My questions: - Is this the expected behavior? - Shouldn't the table constraints and the model relationships in the SQL Analytics Endpoint be automatically synchronized? Thank you! 😀QUESTION::PIPELINE::SET VARIABLE::RACE CONDITION
Let's say you have a situation like this one: and there is only one variable, meaning the variable being referenced by both Set variable activities is the same. And after the Set variable, to the right (not shown here), the next activity, say a IF condition, depends on the value of that one variable. Can the following scenario happen? --> after the top Set variable completes can the bottom Set variable also complete BEFORE the value of the variable reaches the next activity in the top path (stuff on the right that is not shown here)? Let's say the activity that comes after the Set variable in the top path is a condition IF statement that expects a value of 1 and let's say that the activity that comes after the Set variable in the bottom path is a condition IF statement that expects a value of 2. If top completes first, then the variable value = 1. Then the bottom path completes, and the variable value = 2. Then the execution of the top path reaches a condition testing for IF variable = 1. But since the bottom path changed the value of the variable to 2, the top condition in the top path with evalute to false instead of true (remember that the top path Set variable was 1). Is this scenario possible? And if yes, is there a mechanism by which we can specify how the pipeline variables should be accessed , ie no shared memory, or some lock mechanism, etc..., instead of having to define 2 different variable names?
