Forum Discussion
Design question for importing data from ADLS Gen2 Data Lake
Hi All,
I need to design importing data from ADLS Gen2 Data Lake into Power BI and need your recommendations. Here are my requirements:
- Data to be in Parquet (preferably) or CSV format
- There will be new data coming to ADLS Gen2 from different applications every hour
- We should be able to import new fact table data refresh hourly. Dimensions can refresh daily
Questions
- Do we really need a dataflow or it is better to import directly to PBI dataset? My understanding is that the refresh time will increase if we need to refresh dataflow and then dataset
- How do we incrementally import flat files in ADLS Gen2?
- Is it better to implement datetime based folders in ADLS Gen 2 (partitioning) to efficiently query and import last-hour changes into PBI?
Thanks
2 Replies
Hi amalekshahi ,
According to your description, I think you can refer to the following blog link, which basically can answer your confusion.
Parquet, ADLS Gen2, ETL, and Incremental Refresh i... - Microsoft Power BI Community
Please point out if there is still confusion. Looking forward to your feedback.
Best Regards,
Henry
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.Thanks Henry.
There is no confusion on my part. I just have some design questions. Specifically the 2 questions still remain:
- Do we really need a dataflow or it is better to import directly to PBI dataset? My understanding is that the refresh time will increase if we need to refresh dataflow and then dataset
- Is it better to implement datetime based folders in ADLS Gen 2 (partitioning) to efficiently query and import last-hour changes into PBI?
