Import large set of data and refreshing with new data without importing already imported data.

[TcT85]

I have alot of data measurements and result from out testequipment and process controls.

The issue is that I have to much data a range from 1 million to 1 billiion of rows of data., all of this data is located in a SQL server.

New set of data is being written every minutes or seconds in some cases.

For some reports I have created I'm using Direct Query because it takes a few hours to refresh new data with Transform data.

But I prefer to use Transform data since Direct Query seems to have less compatible DAX functions than Transfrom data.

Is there a way to use Transform data and only refresh with new set of data based on date?

Lets say I have Imported data from 2021-01-01 to 2023-03-01 and I want to add additional data from 2023-03-01 to TODAY.

Instead of loading data from 2021-01-01 to TODAY , it will add the additonal data from 2023-03-01 to Today into the report that already has data from 2021-01-01 to 2023-03-01. This is to reduce the time of getting new refreshed data.

I understand the downside to do this way, that if some measurements were changed between 2021-01-01 to 2023-03-01 it will not be added into this report.

But that is fine for us, since we don't change the recoreded data.

Is this possible?

[TcT85]

@Idrissshatila 

Thanks for the help this was exactly what I needed.

[Idrissshatila · Accepted solution]

Hello @TcT85 ,

what you explained could be accomplished using incremental refresh.

check this link https://youtu.be/Kui_1G6kQIQ

If I answered your question, please mark my post as solution so it would appeare to others, Appreciate your Kudos 👍

Follow me on Linkedin