Dataflow refresh transaction failed with status 22
Hi, I am getting this error very often, we are running a pipeline where we have multiple dataflows, and usually there is atleast one which fails with this error, which messes up the whole ETL. We have even reconfigured it, so it runs just one by one and not in parallel and we are still getting this.
Fabrick - write to lakehouse - get error 24596 when accesssing table using sql afterwards
Sql error number: 24596. Error Message: Failed to complete the command because the underlying location does not exist. Underlying data description: table 'dbo.Source_D365_LH_CustInvoiceTrans', file 'https://onelake.dfs.fabric.microsoft.com/9de3f770-392b-41e5-b6e4-d23d6f17560d/77ec4415-7fc0-43e3-8418-b547c4e514b9/Tables/Source_D365_LH_CustInvoiceTrans/07471e8c-1637-412d-aacb-3a6a3753162f.parquet'. Observation is: I load the data to the parquet file using a copy command. The second afterwards I am trying to read data using the sql endpoint. And get the above error message. If I wait a couple of minutes some metadata will be updated automatically and the select query will perform fine. Any chance I can update the metadata? br David
Unexpected Pipeline Slowness / Increase in Data Flow vCore Consumption
Hi, We have been running the same pipelines on a weekly basis and have seen consistent results when it comes to run time / vCore consumption / cost. A week ago however, we noticed that our pipelines have been running extremely slowly, and as a result have consumed a huge amount of V-core processing. All our pipelines are impacted, but is a single example: Pipeline duration: 07/14/2024 - 30s 07/20/2024 - 10m 6s Data flow vCore-hour: 07/14/2024 - 0.0722 07/20/2024 - 2.614 Any ideas on what is causing the impact to our pipeline performance? We have not changed any run time variables, and the number of rows / records being processed has remained the same. One theory we have is that the Azure resource / VM associated with the run time was low on space, would this have an impact? Thanks, Angusstart.datetime is NULL for CalendarView object from Microsoft Data Connect
I am trying to extract all calendar data(BasicDataSet_v0.CalendarView_v0) using our Microsoft 365 connector and Grap Data Connect app using Data Pipeline on Microsft Fabric. I am able to get data in some fields but the start.datetime is always NULL. Can you please help me why this is happening. Followed the instructions from here: https://learn.microsoft.com/en-us/graph/app-registration https://learn.microsoft.com/en-us/graph/data-connect-quickstart?tabs=AzureSynapsePipeline&tutorial-step=3 https://learn.microsoft.com/en-us/entra/identity/enterprise-apps/configure-admin-consent-workflow I have also added the following consents to the app.
need to run java before datapipline copy from rest api
hi, i need to copy data from a rest api over to data lake. before accessing the rest api, i need to execute smal java jar file to extract some info from the result of the jar execution. simple like that: step1) result = jar(param1, param2) string res = result[0] step2) call the api with the value of 'res'. question: how can i make this api call incorporating the variable, and mostly: where do i make that java call? Thank you in advance!Copying data gets failed
Hi, I was copying data from postgres to lakehouse in fabric. Some tables get easily copied but for some tables the task gets failed det to the following error: Failure happened on 'Source' side. 'Type=Npgsql.NpgsqlException,Message=Exception while reading from stream,Source=Npgsql,''Type=System.IO.IOException,Message=Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.,Source=System,''Type=System.Net.Sockets.SocketException,Message=A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond,Source=System,'Two developers not able to work on their feature branches independently - Data Factory
Hi all, If I start working on feature_a branch and in the meantime my colleague starts working on his feature_b branch (or it creates it) I will automatically switch to feature_b branch. Why is that? Does anyone know? Regards
