Forum Discussion
Dataflows Gen2 cannot access Lakehouse Files — missing OneLake internal provider
Hello pbiadminjenhill,
What you’re seeing is expected behavior based on the current design of Dataflows Gen2 in Microsoft Fabric.
Dataflows Gen2 connects to a Lakehouse through the SQL analytics endpoint, which exposes Delta tables only. It does not surface the Lakehouse Files folder structure. As a result:
The Lakehouse connector shows tables, not /Files/...
File paths are treated as external sources and may prompt for a gateway
Combining Parquet files directly from Lakehouse Files isn’t supported
Incremental refresh applies to table sources, not raw file paths
This is a product capability boundary, not a tenant, region, trial, or GoDaddy limitation.
If you need to transform Parquet files stored in Files, first materialize them as Delta tables (via Notebook or Pipeline). Once the data exists as a Lakehouse table, Dataflows Gen2 can access it and support transformations and incremental refresh.
Hello pbiadminjenhill Using Dataflow Gen2 can you perform a Get Data > OneLake Catalog > Select your Lakehouse using your Organisational Account? This usually creates a connection to your OneLake and shows all your managed tables as well as all files underneath a folder named Files.
That is what I thought as well. Here is a screenshot and you can see when I go to Files I have a folder for each company and underneath that are a bunch of files. For some reason when I go to the company it does not let me go one level lower- it just shows all of them together, at least online researching suggested maybe an issue due to the license being bought by Go Daddy vs directly with Microsoft? Here is a screenshot. I really appreciate your help. Thank you!
Hello pbiadminjenhill
Thank you for the screenshot and confirming that the files are in Parquet. I have replicated this on my Fabric workspace and I can tell you how you can load the content of the files using Dataflow Gen2. But bear in mind, incremental load won't work, as the file format is in parquet.
The first change you'd have to do is to use Azure Data Lake Storage Gen2 connector instead of OneLake connector when you do Get Data step. In the Azure Data Lake Storage connection specify the https URI of the sub folder name inside your Raw Data folder. You will get this from your OneLake if you look at the Properties menu of the folder in Files/ section. Press Create to load them into Power Query. You will see one record for each parquet file in the folder. At this stage you can remove _SUCCESS record if present. Press Combine files from the ribbon menu and it will load all the data inside those parquet files.
Thank you for the suggestion. I will definitely try this solution. I do have the option of bringing the file in as csv, avro, or parquet. Should I be thinking of bringing it in one of those formats so that incremental refresh could work or are none of those options going to work for incremental refresh? If not incremental refresh how about a full rewrite. The amount of data is not too crazy - overall across all tables maybe 500k rows? Thank you so much for your advice!
Hello pbiadminjenhill
Unfortunately Delta tables (i.e., files stored in Delta format, is what Dataflow Gen2 expects to use for Incremental Refresh - csv/avro or parquet won't work!
This is because Incremental Refresh with Dataflow Gen2 works on Query Folding, and those file types don't allow query folding.
Hello pbiadminjenhill what is the format of files in those company folders? Did you try clicking the [Table] link under the Content column?