Best way to manage data - looking for advice…

[Matty_77]

Hi All,

I am looking for some advice as to how to most efficiently make data available on the Power BI Service to our citizen developers, avoiding data duplication wherever possible.

We have data held in an Azure data lake as well as in an on prem SQL Server database, which utilises a gateway.

I have looked at setting up dataflows (incremental refresh where appropriate) and then pushing the data over to datasets held in a workspace of their own. Devs wanting to build reports would then consume the data from these datasets, importing or direct querying data as required.

The Power BI ecosystem is changes so quickly, it's quite difficult to know what to plumb for, so I'm looking for any advice anyone can offer.

Thanks,

Matty

[GilbertQ · Accepted solution]

Hi @Matty_77 

What you are doing is certainly working. What you could look at moving forward is looking to see if using fabric with lake houses would solve your challenge. The reason for using a layout is that you could have all the data, whether it is from your azure data lake or a Prem sql server database, land in a single lakehouse that the users could then connect to to build their reports off. What is a lakehouse? - Microsoft Fabric | Microsoft Learn

[Matty_77]

Thanks for your responses, Gilbert and Wenbin!  They've given me some good food for thought!

Cheers,

Matty

[v-zhouwen-msft · Accepted solution]

Hi GilbertQ ,thanks for the quick reply, I'll add more.

Hi @Matty_77 ,

Regarding your question, my idea is to create the Frabic item “LakeHouse” in the workspace, (WareHouse can only store structured data). Then create a Data flow gen2 and set the output to the previously created “LakeHouse”. Finally with the data pipeline you can set the schedule refresh time. the 'DataFactory' in Frabic should be able to meet your needs.

What is Data Factory - Microsoft Fabric | Microsoft Learn

Best Regards,
Wenbin Zhou

[GilbertQ · Accepted solution]

Hi @Matty_77 

What you are doing is certainly working. What you could look at moving forward is looking to see if using fabric with lake houses would solve your challenge. The reason for using a layout is that you could have all the data, whether it is from your azure data lake or a Prem sql server database, land in a single lakehouse that the users could then connect to to build their reports off. What is a lakehouse? - Microsoft Fabric | Microsoft Learn