Something went wrong, Dataflow publish failed

[danielmccluskey]

Hi, Im getting a strange and vague error when I publish my data flow gen 2.

All I get is "Something went wrong" and can't seem to find out anything else about where it is going wrong.

I can publish without error but then I get:

(Clicking See details just shows me the IDs of the flow and what time the error occurred.)

let
    FetchPage = (url) =>
        let
            Source = Json.Document(Web.Contents(url)),
            Data = Source[data],
            NextPage = Source[meta][pagination][next],
            More = Source[meta][pagination][more],
            Result = if More then Data & @FetchPage(NextPage) else Data
        in
            Result,
    StartURL = "https://api.wonde.com/v1.0/schools?page=1",
    AllData = FetchPage(StartURL),

...and so on with some more data changes

[MawashiKid2]

I encountered similar issue recently with Dataflow Gen2 in Microsoft Fabric. I mean nothing really fancy here, only selecting a .csv file uploaded in Lakehouse folder which I had previously created, then define a destination through Add Data Destination to very same Lakehouse with new table name though in different folder. "Dataflow Failed to Publish!"  [sigh...]

[MawashiKid2]

Okay never mind... solved it! It seems Dataflow Gen2 didn't seem to accept any blank space after a comma... in csv. So I simply removed all blank spaces in original csv file, deleted the wrong formatted csv file version in Lakehouse, then re-uploaded new corrected csv version in same Lakehouse and it worked this time...just as stupid as that... [sigh...]

[frithjof_v]

Interesting. 

How did you find out that it was the reason?

Was there some error values in some of the data columns?

[MawashiKid2]

To resume...
It turns out a Column name from a .csv sample file I had downloaded to use for simple SCD2 test- and had previously uploaded in a Lakehouse under Files folder - was not well formatted.
So I then created DataFlow Gen2 (WhateverDF...). Once the csv file as loaded in the DataFlow,
I then selected Use first row as header, defined a new Table name in same Lakehouse as destination using Add data destination, then simply clicked on Publish.
A popup notification displayed Dataflow WhateverDF failed to pulished! error.
I then located the WhateverDF Dataflow instance in main Workspace Name column and noticed a small red border triangle next to it.

I simply clicked on it to get more details about the error.

So that was it. The Name column name was formatted as ', Name' when it should have been formatted as ',Name' that is with NO SPACE between the comma and first letter of column name. So I simply fixed the error and was then able to Publish and run DataFlow Gen2 as expected. N.B. The DataFlow Gen2 doesn't imply much transformation here - beside selecting first row as header - and is used mainly in combination with a Notebook in a data pipeline.  All SCD2 logic is defined in Notebook. That's when the rubber hits the road!😉 

[frithjof_v]

Thank you for sharing!

Nice to learn that an explanatory error message for failed publishes can be found in the workspace interface (triangle icon) 😀

Since the error popup you received immediately when the publish failed, didn't seem to provide much guidance.

[danielmccluskey · Accepted solution]

I managed to solve this, I was changing the base URL of the Web.Contents which meant the flow didn't know what connector to use everytime it changed.

Fixed by using one Base URL and then using RelativePath and Query properly.

So I changed this

let
    FetchPage = (url) =>
        let
            Source = Json.Document(Web.Contents(url)),
            Data = Source[data],
            NextPage = Source[meta][pagination][next],
            More = Source[meta][pagination][more],
            Result = if More then Data & @FetchPage(NextPage) else Data
        in
            Result,
    StartURL = "https://api.wonde.com/v1.0/schools?page=1",
    AllData = FetchPage(StartURL),

...and so on with some more data changes

into this:

FetchPage = (url) =>
        let
            baseURL = "https://api.wonde.com/v1.0/",
            relativePath = "schools",
            queryList = Uri.Parts(url)[Query],
            response = Web.Contents(baseURL,
            [
                RelativePath = relativePath,
                Query = queryList
            ]),
            Source = Json.Document(response),
            Data = Source[data],
            NextPage = Source[meta][pagination][next],
            More = Source[meta][pagination][more],
            Result = if More then Data & @FetchPage(NextPage) else Data
        in
            Result,
    StartURL = "https://api.wonde.com/v1.0/schools?page=1",
    AllData = FetchPage(StartURL),

which means it always uses the same connector and is using the relative path correctly.

[Anonymous]

Hi @danielmccluskey ,
Glad that your issue got resolved. Please continue using Fabric Community for help regarding your queries.

[miguel]

Hey!

I'd recommend that you raise a support ticket so an engineer can take a closer look at it.