Forum Discussion
Something went wrong, Dataflow publish failed
- 2 years ago
I managed to solve this, I was changing the base URL of the Web.Contents which meant the flow didn't know what connector to use everytime it changed.
Fixed by using one Base URL and then using RelativePath and Query properly.
So I changed this
let FetchPage = (url) => let Source = Json.Document(Web.Contents(url)), Data = Source[data], NextPage = Source[meta][pagination][next], More = Source[meta][pagination][more], Result = if More then Data & @FetchPage(NextPage) else Data in Result, StartURL = "https://api.wonde.com/v1.0/schools?page=1", AllData = FetchPage(StartURL), ...and so on with some more data changesinto this:
FetchPage = (url) => let baseURL = "https://api.wonde.com/v1.0/", relativePath = "schools", queryList = Uri.Parts(url)[Query], response = Web.Contents(baseURL, [ RelativePath = relativePath, Query = queryList ]), Source = Json.Document(response), Data = Source[data], NextPage = Source[meta][pagination][next], More = Source[meta][pagination][more], Result = if More then Data & @FetchPage(NextPage) else Data in Result, StartURL = "https://api.wonde.com/v1.0/schools?page=1", AllData = FetchPage(StartURL),which means it always uses the same connector and is using the relative path correctly.
I encountered similar issue recently with Dataflow Gen2 in Microsoft Fabric. I mean nothing really fancy here, only selecting a .csv file uploaded in Lakehouse folder which I had previously created, then define a destination through Add Data Destination to very same Lakehouse with new table name though in different folder. "Dataflow Failed to Publish!" [sigh...]
Okay never mind... solved it! It seems Dataflow Gen2 didn't seem to accept any blank space after a comma... in csv. So I simply removed all blank spaces in original csv file, deleted the wrong formatted csv file version in Lakehouse, then re-uploaded new corrected csv version in same Lakehouse and it worked this time...just as stupid as that... [sigh...]
- frithjof_v2 years agoCommunity Champion
Interesting.
How did you find out that it was the reason?
Was there some error values in some of the data columns?
- MawashiKid22 years agoAdvocate I
To resume...
It turns out a Column name from a .csv sample file I had downloaded to use for simple SCD2 test- and had previously uploaded in a Lakehouse under Files folder - was not well formatted.
So I then created DataFlow Gen2 (WhateverDF...). Once the csv file as loaded in the DataFlow,
I then selected Use first row as header, defined a new Table name in same Lakehouse as destination using Add data destination, then simply clicked on Publish.
A popup notification displayed Dataflow WhateverDF failed to pulished! error.
I then located the WhateverDF Dataflow instance in main Workspace Name column and noticed a small red border triangle next to it.I simply clicked on it to get more details about the error.
So that was it. The Name column name was formatted as ', Name' when it should have been formatted as ',Name' that is with NO SPACE between the comma and first letter of column name. So I simply fixed the error and was then able to Publish and run DataFlow Gen2 as expected. N.B. The DataFlow Gen2 doesn't imply much transformation here - beside selecting first row as header - and is used mainly in combination with a Notebook in a data pipeline. All SCD2 logic is defined in Notebook. That's when the rubber hits the road!😉- frithjof_v2 years agoCommunity Champion
Thank you for sharing!
Nice to learn that an explanatory error message for failed publishes can be found in the workspace interface (triangle icon) 😀Since the error popup you received immediately when the publish failed, didn't seem to provide much guidance.