Issue: dataflow gen2 created in subfolder or with git integration preview causing issues

[nioj2S]

Dunno where this should be posted, so trying here under "Issues" forum.

I recently saw that I am able to create a dataflow gen2 directly in a subfolder of my workspace, which was not previously possible. Previously I had to create a dataflow gen2 in my startpage of workspace, then when the dataflow gen2 was published and done refreshing, I could move it into a subfolder of my choice. I also clicked the git integration preview option. When creating the dataflow gen2 in the subfolder, I noticed the "Publish" button in lower right corner was missing and new button "Save & run" in top left corner was added. Naturally I assumed this new "Save & run" button was replacing the "Publish" button and therefor the publish function.

It does not.

When saving and running the dataflow, it fails without a clear error message. It says:
Error Details: We encountered an error during evaluation. Details: A localization error has been logged, it is safe to continue. Message: "Unknown evaluation error code: {errorCode}"

This is something I saw when checking the recent runs for the dataflow gen2, it did not update the regular information columns on the page so it was very unclear it failed.

I then tried creating a dataflow gen2 in the startpage of my workspace, without git integration preview, and now the "Save & run" button is missing but the "Publish" button is back. I created my dataflow, hit "Publish" and it works as expected.

TL;DR: Creating a dataflow gen2 in a subfolder and/or with git integration preview introduces errors that does not exist when creating a dataflow gen2 in workspace.

[v-zhangtin-msft]

Hi, @nioj2S 

I tried following your steps, saving and running datalfow gen2 in a subfolder and no errors were reported. Tried refreshing and no errors were reported either. Can you try again to see if there is a problem?

Best Regards,
Community Support Team- Charlotte Zhang

[nioj2S]

@v-zhangtin-msftHi! Tried again now, same issue, the "Publish" button is missing. Here are the steps I did:

1. Went into subfolder (workspace -> folder1 -> folder2) and clicked "New item" and "Dataflow gen" which gave me the "New Dataflow Gen2" popup in which I could enter a name and the "Enable Git integration, deployment pipelines and Public API scenarios (preview)" option is both already checked and also greyed out so I can not uncheck.

2. Click "Create"

3. "Get data" from my lakehouse.

4. Set "Data destination" to my lakehouse.

5. Still no "Publish" button in bottom right corner. Can not publish, only "Save & run" at top left corner.

I tried creating dataflows in my main workspace. The regular kind worked as expected, but when I checked the "Enable Git integration, deployment pipelines and Public API scenarios (preview)" which was not prechecked or greyed out in main workspace, then I got the same issue with missing "Publish" button.

[v-zhangtin-msft]

Hi, @nioj2S 

I tested it and it completely restored your problem. Unfortunately, this is by design. There is no publish button when Git integration is enabled.

Dataflow Gen2 with CI/CD and Git integration - Microsoft Fabric | Microsoft Learn

Best Regards,
Community Support Team- Charlotte Zhang

[nioj2S]

@v-zhangtin-msft Ah, this creates the issue that I can not refer to that dataflow in a pipeline, because it seems like a dataflow has to be published in order for it to be seen by a pipeline. But, at this point in time, it is more important that I can use a dataflow in a pipeline, so I will just avoid using the Git Integration preview option. Thanks!

[mandania]

@v-zhangtin-msft this problem is pointing more towards gen2 bug. Scenario 1: I created a Dataflow Gen2 1 without checking the " Enable GIT Integration..." checkbox. configured the source and destination and published it. Created a pipeline dataflow1 and linked the Dataflow Gen2 1. As expected. Scenario 2: I created a Dataflow Gen2 this time enabling the " Enable GIT Integration..." checkbox. configured the source and destination and clicked on "Save & Run". NO PUBLISH option available. Created a pipeline dataflow2 BUT could not see the Dataflow Gen2 in the dropdown. Now if i cannot publish my Gen2 pipeline, i wont be able be able to run them from my pipeline dataflows since they are not published , hence wont be visible in dropdown., so whats the logic behind this implementation?