Hi!

Our recommendation is to adopt either one of the approaches from the article below:

CI/CD and ALM solution architectures for Dataflow Gen2 - Microsoft Fabric | Microsoft Learn

The closest would be to use Fabric variable libraries which Dataflow Gen2 fully supports. 

The benefits of this approach over the delployment rules is that from a Git perspective you are not affecting the actual mashup document. This means that the deployment is really just copying the same Dataflow from one environemnt to the next and the variables are the one passing the correct values for execution, but they simply remain as variables.

The bad effect of using deployment rules was that each deployment would trigger 2 changes:

1. Updating the content of the mashup document of the Dataflow with the one from the earlier environment

2. Doing a sort of "find and replace" over the outcome of #1 

This behavior would actually go against some principles of how ALM should be handled that the exact code that you tested in your earlier environment is just being pushed to a different environment because #2 fundamentally changes that code.

Hope this helps!

Hi! 

Could you tell us a bit more about the scenario for #1 and #3?

For #2, and specifically for your case with Lakehouse destinations, changing the name of your table is completely supported for Dataflow public parameters. You can follow the tutorial below for more information on how to make that happen:

Parameterized Dataflow Gen2 - Microsoft Fabric | Microsoft Learn

Hi @VictorLojero  can you please elaborate your ask?

If you are a tenant admin, you have a robust report in OneLake Catalog - Govern tab

click 'View more' and there you can explore the data in your tenant.

for example, filtering on 'reports' in the 'feature usage' tab you can see what activities were taken and in which report

additonally, per report, you have 'usage metrics report' (assuming its enabled in your tenant through tenant settings). there you can see how many users accessed the report and track some metrics around that.

Thanks for the input.

Trying to understand the scenario better - what would be the reason for doing this?
Eventhouse data and OneLake Availability is considered as one logical copy with same access level and retention. You can query the data in OneLake without using the Eventhouse compute for historical analysis, while querying the Eventhouse direcely for timeseries/real-time analytics.

If you would like to manage the data in lake seperately, then copying it outside Eventhouse might be a better way think about it. 

Thanks for the detailed suggestion. Behind the scene, Eventstream custom endpoint is an Event Hub. Customers can use Entra Id authentication. Eventhub namespace and Eventhub name are also exposed. Can you please give more details on why this approach didn't meet your goal? 

Thanks

Hi! Are you looking to ingest into Eventstreams or Eventhouse?

Would you mind letting us know more a bit what kind of background logging you'd like to see?

We do have some telemetry that we use to help troubleshoot scenarios, but there are scenarios in which we simply do not store detailed information as it doesn't run under our platform. The primary example of such cases are evaluations that happen through the Gateway and this is the reason why advanced logging is a feature that you can enable at the Gateway level to capture them and go deeper into the investigation, but the logging only happens at the gateway and such information is not sent or captured by the Fabric platform.

In order to compute each step, we require to send the request to your source. Turning it off would mean that we wouldn't be able to communicate with your data source and validate that the request is valid. Imagine a scenario in which you manually type the name of a value which is incorrect and we aren't able to filter your table to say that there are no values with that string in your column. This wouldn't be possible to compute solely on the data preview as the preview typically only holds around 200 rows unless you start scrolling further down.

If perhaps the idea is more geared towards creating a local cache of your data and only working against that cached and sampled dataset, definitely feel free to edit your idea to fit such description.

We recommend using either one of the two features below to reduce the number of requests as well as the load to your data source:

Using Schema view (Preview) - Power Query | Microsoft Learn - this makes sure that you only work with your table schema and are not loading anything other than just the schema

Preview only step in Dataflow Gen2 (Preview) - Microsoft Fabric | Microsoft Learn - you can set a filter against your data to, fundamentally, work on a sample of your while authoring your dataflow without ever impacting the end-result which would go against the unfiltered dataset.

Hi!

Could you please clarify your scenario and what is the expectation for the OVERWRITE?

At the moment, Dataflow Gen2 with default destination does support a Replace update method for Lakehouse.

Here's the link to the official documentation: Dataflow Gen2 default destination - Microsoft Fabric | Microsoft Learn

Hi @Shahid12523 , have you tried using the OneLake Catalog in Fabric service?

You can actually accomplish this today in Dataflow Gen2 (CI/CD). Dataflow doesn't automatically save your changes, so you need to click the "Save" button in order to commit / publish those changes.

Let us know if that works for you today or if you have something else in mind that could improve your experience.

Hi @Gavin_Pedersen ,can you use Tags - Unapply Tags - REST API (Core) for you usecase?

Given the latest response in the comments, I'm removing the Dataflow label and adding the Pipeline one so the Pipelines team can investigate a bit further as to what the true ask is here.

If this is a support case, I highly recommend reaching out to our support team by creating a support ticket:

http://support.fabric.microsoft.com/

Hi @aleroux , could you please provide more clarity on the timeout issue/request you are encountering? Am not sure if we have any settings specifically for timeout. We have a system stored procedure sp-change-feed-configure-parameters that allows fine tuning of operational performance. for Fabric SQL database mirrored databases these values are dynamically determined and auto set.

The GIT approach is our suggested approach when version control is required. GIT brings a number of capabilities that do fit perfectly with the task of handling version control.

Would you mind sharing more about what sort of scenarios or challenges you're facing today that cannot be addressed with git? Or perhaps what scenarios could only be addressed by having a JSON export of the Dataflow definition?

Please do bare in mind that you can also leverage the REST API today to get the full Dataflow definition without ever setting up GIT for your workspace:

https://learn.microsoft.com/en-us/rest/api/fabric/dataflow/items/get-dataflow-definition

Could you share some repro steps of the issue? I've just given it a try and it appears to work as designed.

For example, for the query below:

let
  Source = OData.Feed("https://services.odata.org/V4/Northwind/Northwind.svc/", null, [Implementation = "2.0"]),
  #"Navigation 1" = Source{[Name = "Customers", Signature = "table"]}[Data]
in
  #"Navigation 1"

I am able to see that it has many columns and the data preview doesn't allow me to see all of them.

I can then use the "Go To column", select a column that I'm unable to see like "Fax":

And it takes me to the Fax column:

Hi @lincoln_flores  - can you please clarify how this relates to SQL database in Fabric?  This sounds like an Azure functionality that you are asking for.

Thanks @ITmerrr  for submitting this idea

as we are working on enhancements to Admin Portal, I'd be happy to better understand:

- Tenant settings - would you expect non admins to be able to access that page? those non admins will be able to change settings? will be happy to learn more about this use case

- usage metrics - that's a deperecated tab, we removed it, the change is rolling to production

- premium per user - is setup - would you expect non admins to be able to change it?

- Audit logs and Users redirect to external system (M365 ) - same quetsion , what would you expect the non admins to be able to see here?