Forum Discussion

Don-Bot's avatar
Don-Bot
Icon for Helper V rankHelper V
2 years ago

Power BI Embedded using Databricks sql warehouse or data lake?

We are currently looking into moving from using a SQL Server datawarehouse to something new.  We have large amount of data, currently in the millions of rows used by hundreds of clients.  The current...
Need Help
lbendlin's avatar
lbendlin
Icon for Super User rankSuper User
2 years ago

millions of rows should not take several hours. Are you doing a lot of transforms in Power Query? Are you attempting merges?

 

Consider using dataflows, or flat out Parquet files.  Let the data model do the work for you.

  • Don-Bot's avatar
    Don-Bot
    Icon for Helper V rankHelper V
    2 years ago

    Currently the data model is connecting to a view in SQL Server to pull the information.  Hence the long load times and huge memory hogging.  

    I am looking into enhancing it to utilize Databricks Warehouse or Direct Lake parquet files.  Either one is probably a huge improvement over what we have today.  

    • lbendlin's avatar
      lbendlin
      Icon for Super User rankSuper User
      2 years ago
      Currently the data model is connecting to a view in SQL Server to pull the information.  Hence the long load times and huge memory hogging.

      Optimize the query behind the view.  Add indexes as needed and make sure the statistics are up to date.