Forum Discussion
Import or Directy query with about a billion data
Hi changhai ,
For your first question, we think using Import Mode might reach limitation of dataset (For example 1,999,999,997 limit on the number of distinct values that can be stored in a column and 1-GB limit for datasets stored in Shared capacities). And If your first table will update frequently, it might take much time in scheduled refresh because of the large dataset. The most important thing is, although DirectQuery Mode has some limitation in tranforming data and DAX, but we can easily change dataset from DirectQuery to Import. Consider your large dataset, we suggest you to try the DirectQuery Mode first, if it cannot meet your requirement, you can change it to Import Mode.
For the second question, When using Import Mode, the data will be refresh and stored in the model, we can execute the on-demand refresh or set the scheduled refresh to make it up-to-date in service and do not need to re-publish again.
For the last question, Use of DirectQuery does have potentially negative implications, as detailed in this section. Some of those limitations are slightly different depending upon the exact source that is being used. Please refer to this document about the Implications of using DirectQuery. The implication of Import mode mostly is the limitation on size of dataset and the refresh time if your dataset is large, also it will take much more menory while refresh.
Best regards,
v-lid-msft Thanks for your reply.
My data will not exceed 1 billion so I think for import mode it will not reach limitation.
According to my test, I imported almost 100 million data, I used advanced filter for the big table and found it too slow, most of the time it will time out.
I want to know how to set the timeout to a longer time. And on the other hand how to speed up the query?
Thanks!