Incremental load of onedrive files into fabric

[holgergubbels]

Hi Community,

I work with many customers who use OneDrive (csv-, xlsx-files) as data source for Power BI reports. These files exist for valid reasons— they’re sent by partners, suppliers, customers, and so on. I want to avoid transforming them locally into a database or similar structure using something like a C# tool.

Power BI can read data from SharePoint folders without issues, but it’s often time-consuming. A better solution would be to use a service, like a dataflow, to import the data, and then pull all the data from the dataflow into Power BI.

With the introduction of Fabric, I’d love to migrate all file-based data into a warehouse or a lakehouse—the new SQL server-like option in Power BI. I envision a workflow where

• files are stored in a specific OneDrive folder (either manually or via Power Apps).
• Some tool (Pipelines? Dataflow Gen2?) would be triggered by new files (or run periodically) to read them, import the data, and then move the files to an "imported" folder.

In essence: an incremental refresh for file-based data sources with post-import file management.

I’m struggling to navigate the many options within Fabric. How can I address this use-case effectively?

Thanks in advance!

Holger

[v-prasare · Accepted solution]

@holgergubbels, Thanks for reaching MS Fabric community support.

You can use Fabric Pipelines + Dataflows Gen2 + Lakehouse to setup your flow

• Triggering Mechanism: Use Fabric Pipelines  to automate the process . Pipelines can be scheduled or triggered by changes in a folder
• Data Ingestion with Dataflows Gen2: Create a Dataflow Gen2 that reads the files from OneDrive (or SharePoint) and ingests them into a Fabric Lakehouse. Use incremental refresh to ensure only new or updated data is processed.
• Data Consumption in Power BI: once data ingested Connect Power BI to the Fabric Lakehouse or Warehouse for reporting.

additionally, Dataflow gen2 are more powerful than traditional data pipelines. you can have advanced options like joins ..etc

let me know if you need any further assistance here.

Thanks,

Prashanth Are

Please mark this as solution if it helps. Appreciate Kudos.

[lbendlin]

One more thing!  Fabric is based on Parquet files.  So while you can certainly store your CSVs in the data lake it would be better for performance to convert them into Delta Lake format (with Vorder optimization).  Then you can access them in Power BI directly.