Forum Discussion
Dataflows Gen2 cannot access Lakehouse Files — missing OneLake internal provider
Hello pbiadminjenhill,
What you’re seeing is expected behavior based on the current design of Dataflows Gen2 in Microsoft Fabric.
Dataflows Gen2 connects to a Lakehouse through the SQL analytics endpoint, which exposes Delta tables only. It does not surface the Lakehouse Files folder structure. As a result:
The Lakehouse connector shows tables, not /Files/...
File paths are treated as external sources and may prompt for a gateway
Combining Parquet files directly from Lakehouse Files isn’t supported
Incremental refresh applies to table sources, not raw file paths
This is a product capability boundary, not a tenant, region, trial, or GoDaddy limitation.
If you need to transform Parquet files stored in Files, first materialize them as Delta tables (via Notebook or Pipeline). Once the data exists as a Lakehouse table, Dataflows Gen2 can access it and support transformations and incremental refresh.
Hi pbiadminjenhill,
Thank you for reaching out to the Microsoft Fabric Community Forum and sharing the screenshot and the details. Also thanks to Olufemi7, deborshi_nag, for those inputs on this thread.
Based on your scenario, this behaviour is expected and not related to your tenant, region, trial status, or GoDaddy licensing. Your Dataflows Gen2 is already connected correctly to One Lake. Currently, when accessing the Lakehouse Files section through the One Lake Catalog connector, Dataflows Gen2 may show files in a flattened view and does not always allow deeper folder-level navigation or direct transformation of Parquet files from nested folders.
To work with these files in Dataflows Gen2, the recommended approach is to first load the Parquet files into Lakehouse Delta tables. You can do this by going to your Lakehouse, navigating to Files, and selecting Load to Table, or by using a Fabric Notebook or Pipeline. Once the data is available under the Tables section, you can connect to it from Dataflows Gen2 without gateway prompts and use features like transformations and incremental refresh.
Refer these links:
1. https://learn.microsoft.com/en-us/fabric/data-factory/dataflows-gen2-overview
2. https://learn.microsoft.com/en-us/fabric/onelake/onelake-overview
3. https://learn.microsoft.com/en-us/fabric/data-factory/connector-lakehouse
4. https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-and-delta-tables
Thank you for using the Microsoft Fabric Community Forum.
Thank you for this information. What I am doing is bringing in data into a Datalake for multiple subsidiaries of a master company and so putting them in a landing zone in their own folder and then within their folder we have the files like accounts, customers etc as additional files. I am using a 3rd party connector to get the data out and only incremental data gets added to the raw files in the data lake. These files are already coming in parquet format. I was hoping that I could then use dataflows gen 2 on top of these raw files, make some transformations and move them to the tables area. Today I am doing this via pipeline however I am having to overwrite the data everyday rather than an incremental refresh. It is my understanding that incremental refresh is much easier to do in Dataflows Gen 2. Will I have to figure out a way to do it only in pipelines and once the data has moved into a table then only can I use Dataflows Gen 2 for additional transformations?
Hi deborshi_nag,
Thank you for explaining your ingestion process, your approach of landing incremental Parquet files into the Lakehouse Files section is correct and follows a proper landing zone pattern. However, incremental refresh in Dataflows Gen2 is supported when working with structured Lakehouse Tables (Delta tables), not directly from raw Parquet files in the Files section. Because of this, Dataflows Gen2 cannot manage incremental logic directly on those landing files. This is why you are currently needing to overwrite data when using pipelines, and it is expected behaviour when the incremental logic is not yet applied at the table level.
To resolve this, the recommended approach is to continue landing incremental Parquet files in the Files section but configure your Pipeline or Notebook to append or merge only new data into a Lakehouse Delta table instead of overwriting it. Once the data is maintained incrementally in the Tables section, you can connect Dataflows Gen2 to that table and perform further transformations efficiently without full reloads. In summary, Pipelines or Notebooks should handle incremental ingestion into Tables, and Dataflows Gen2 can then be used on top of those Tables for transformation and downstream processing.
Thank you again for using the Microsoft Fabric Community Forum.
Hello pbiadminjenhill
You can use Dataflows Gen2 directly on top of Lakehouse Files and implement incremental refresh there, without overwriting the table every day. In fact, Dataflows Gen2 is the recommended place to handle incremental transformations in your scenario.Dataflows Gen2 can:
- Read parquet files from the Files section
- Apply Power Query transformations
- Write to Lakehouse Tables
- Use Incremental Refresh to only process new data.
