Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
7 years ago

Architecture for refreshes

Hello!

 

We are an ISV and will be utilizing a premium capacity to embed data for our external customers, and I wanted to see if someone could provide some guidance on how best to set up dataset refreshes.   

 

We have a multi-tenant setup (many customers in a single database and their data is loaded at different times throughout the day)  We have RLS in place to filter data based on the user who logs in. 

 

Here are the approaches I came up with, and the "Cons" I'm seeing for each:

 

1) Have 1 dataset with all of our customer's data in it and just use incremental refresh to update it throughout the day. 

Pros:

  • Easier to maintain 1 dataset
  • Can be more easily shared amongst other reports

Cons:

  • Since customers send data it at different times throughout the day, they could potentially have to wait a long time between when they send and the next refresh
  • The dataset would be pretty large overall, and might run into limitations, both size and performance
  • We do have instances where customers send us bad data or for other reasons, we need to do a reload, and I don't know how a situation for 1 customer might impact the rest (can you remove import data for just 1 customer in a dataset?)

 

2) Create a dataset for each tenant and add a step to the end of ETL to refresh the data.  

Pros:

  • Have flexibility to refresh a given customers data shortly after their load completes
  • If there is an issue with a refresh or dataset, only that 1 customer is impacted (isolation)
  • Smaller datasets, and thus hopefully shorter refresh times

 

Cons:

  • Could be a pain to maintain as continue to add customers 

 

Are there other approaches I'm missing?  Any recommendations to the 2 I've outlined above?   (side note:  we initially started down the road of Direct Query, but the performance has been so-so, and we're afraid as we have a lot of customers using the site, it might be even worse)

 

Thanks!

3 Replies