Forum Discussion

pbiadminjenhill's avatar
pbiadminjenhill
Regular Visitor
6 months ago
Solved

Dataflows Gen2 cannot access Lakehouse Files — missing OneLake internal provider

We are evaluating Microsoft Fabric and have encountered what appears to be a tenant‑level capability issue. Our Dataflows Gen2 environment cannot access Lakehouse Files. Symptoms: - Lakehouse (Files...
capacities
one lake
workspaces
  • Olufemi7's avatar
    Olufemi7
    6 months ago

    Hello pbiadminjenhill, 

    What you’re seeing is expected behavior based on the current design of Dataflows Gen2 in Microsoft Fabric.

    Dataflows Gen2 connects to a Lakehouse through the SQL analytics endpoint, which exposes Delta tables only. It does not surface the Lakehouse Files folder structure. As a result:

    • The Lakehouse connector shows tables, not /Files/...

    • File paths are treated as external sources and may prompt for a gateway

    • Combining Parquet files directly from Lakehouse Files isn’t supported

    • Incremental refresh applies to table sources, not raw file paths

    This is a product capability boundary, not a tenant, region, trial, or GoDaddy limitation.

    If you need to transform Parquet files stored in Files, first materialize them as Delta tables (via Notebook or Pipeline). Once the data exists as a Lakehouse table, Dataflows Gen2 can access it and support transformations and incremental refresh.

pbiadminjenhill's avatar
pbiadminjenhill
Regular Visitor
6 months ago

Thank you for this information. What I am doing is bringing in data into a Datalake for multiple subsidiaries of a master company and so putting them in a landing zone in their own folder and then within their folder we have the files like accounts, customers etc as additional files. I am using a 3rd party connector to get the data out and only incremental data gets added to the raw files in the data lake. These files are already coming in parquet format. I was hoping that I could then use dataflows gen 2 on top of these raw files, make some  transformations and move them to the tables area. Today I am doing this via pipeline however I am having to overwrite the data everyday rather than an incremental refresh. It is my understanding that incremental refresh is much easier to do in Dataflows Gen 2. Will I have to figure out a way to do it only in pipelines and once the data has moved into a table then only can I use Dataflows Gen 2 for additional transformations?

deborshi_nag's avatar
deborshi_nag
Super User
6 months ago

Hello pbiadminjenhill 

 

You can use Dataflows Gen2 directly on top of Lakehouse Files and implement incremental refresh there, without overwriting the table every day. In fact, Dataflows Gen2 is the recommended place to handle incremental transformations in your scenario.
 

Dataflows Gen2 can:

  • Read parquet files from the Files section
  • Apply Power Query transformations
  • Write to Lakehouse Tables
  • Use Incremental Refresh to only process new data.