topic Something went wrong, Dataflow publish failed in Dataflow

Something went wrong, Dataflow publish failed

danielmccluskey — Thu, 28 Sep 2023 22:14:18 GMT

Hi, Im getting a strange and vague error when I publish my data flow gen 2.

All I get is "Something went wrong" and can't seem to find out anything else about where it is going wrong.

I can publish without error but then I get:

(Clicking See details just shows me the IDs of the flow and what time the error occurred.)

let FetchPage = (url) => let Source = Json.Document(Web.Contents(url)), Data = Source[data], NextPage = Source[meta][pagination][next], More = Source[meta][pagination][more], Result = if More then Data & @FetchPage(NextPage) else Data in Result, StartURL = "https://api.wonde.com/v1.0/schools?page=1", AllData = FetchPage(StartURL), ...and so on with some more data changes

Re: Something went wrong, Dataflow publish failed

miguel — Thu, 28 Sep 2023 22:35:54 GMT

Hey!

I'd recommend that you raise a support ticket so an engineer can take a closer look at it.

Re: Something went wrong, Dataflow publish failed

danielmccluskey — Fri, 29 Sep 2023 07:06:25 GMT

I managed to solve this, I was changing the base URL of the Web.Contents which meant the flow didn't know what connector to use everytime it changed.

Fixed by using one Base URL and then using RelativePath and Query properly.

So I changed this

into this:

FetchPage = (url) => let baseURL = "https://api.wonde.com/v1.0/", relativePath = "schools", queryList = Uri.Parts(url)[Query], response = Web.Contents(baseURL, [ RelativePath = relativePath, Query = queryList ]), Source = Json.Document(response), Data = Source[data], NextPage = Source[meta][pagination][next], More = Source[meta][pagination][more], Result = if More then Data & @FetchPage(NextPage) else Data in Result, StartURL = "https://api.wonde.com/v1.0/schools?page=1", AllData = FetchPage(StartURL),

which means it always uses the same connector and is using the relative path correctly.

Re: Something went wrong, Dataflow publish failed

Anonymous — Thu, 05 Oct 2023 08:43:41 GMT

Hi danielmccluskey ,
Glad that your issue got resolved. Please continue using Fabric Community for help regarding your queries.

Re: Something went wrong, Dataflow publish failed

MawashiKid2 — Fri, 19 Jan 2024 22:03:05 GMT

I encountered similar issue recently with Dataflow Gen2 in Microsoft Fabric. I mean nothing really fancy here, only selecting a .csv file uploaded in Lakehouse folder which I had previously created, then define a destination through Add Data Destination to very same Lakehouse with new table name though in different folder. "Dataflow Failed to Publish!" [sigh...]

Re: Something went wrong, Dataflow publish failed

MawashiKid2 — Fri, 19 Jan 2024 22:28:55 GMT

Okay never mind... solved it! It seems Dataflow Gen2 didn't seem to accept any blank space after a comma... in csv. So I simply removed all blank spaces in original csv file, deleted the wrong formatted csv file version in Lakehouse, then re-uploaded new corrected csv version in same Lakehouse and it worked this time...just as stupid as that... [sigh...]

Re: Something went wrong, Dataflow publish failed

frithjof_v — Fri, 19 Jan 2024 22:38:33 GMT

Interesting.

How did you find out that it was the reason?

Was there some error values in some of the data columns?

Re: Something went wrong, Dataflow publish failed

MawashiKid2 — Sat, 20 Jan 2024 20:46:01 GMT

To resume...
It turns out a Column name from a .csv sample file I had downloaded to use for simple SCD2 test- and had previously uploaded in a Lakehouse under Files folder - was not well formatted.
So I then created DataFlow Gen2 (WhateverDF...). Once the csv file as loaded in the DataFlow,
I then selected Use first row as header, defined a new Table name in same Lakehouse as destination using Add data destination, then simply clicked on Publish.
A popup notification displayed Dataflow WhateverDF failed to pulished! error.
I then located the WhateverDF Dataflow instance in main Workspace Name column and noticed a small red border triangle next to it.

I simply clicked on it to get more details about the error.

So that was it. The Name column name was formatted as ', Name' when it should have been formatted as ',Name' that is with NO SPACE between the comma and first letter of column name. So I simply fixed the error and was then able to Publish and run DataFlow Gen2 as expected. N.B. The DataFlow Gen2 doesn't imply much transformation here - beside selecting first row as header - and is used mainly in combination with a Notebook in a data pipeline. All SCD2 logic is defined in Notebook. That's when the rubber hits the road!😉

Re: Something went wrong, Dataflow publish failed

frithjof_v — Sat, 20 Jan 2024 21:19:25 GMT

Thank you for sharing!

Nice to learn that an explanatory error message for failed publishes can be found in the workspace interface (triangle icon) 😀

Since the error popup you received immediately when the publish failed, didn't seem to provide much guidance.