Loading star schema from Parquet via data bricks

Hi All,

What solutions have you done that had Power BI consuming data that was stored in Parquet?

I am designing a star schema model and using Data Bricks as the compute platform for the ETL and so far the plan was to store the tables in parquet on blob storage.

But we need a way for Power BI to consume it for use and we have thought of the below possible solutions;

1. Keep the data bricks cluster on all the time and refresh spark data frames from the parquet when the data needs to be updated (Once a day) Power BI will connect to the cluster via direct query.

2. Load the Data into txt or csv's on blob storage and have Power BI connect to those

3. Push the data into physical tables on SQL server Database or SQL Data Warehouse

We are not sure at this stage if option 1 is a waste of compute resources and if having the cluster on all the time will be feasible but considering we need to use direct query as the data is too large to load into memory this is still a valid option.

One of the requirements I have is that the end users can still query this star schema via SQL but this could be via spark SQL.

Interested to herar your thoughts.