Deployment Pipelines - Datamarts with incremental refresh

[mike9999]

Hi-

I'm trying to learn what the best practices are around datamarts and deployment pipelines.

I notice when promoting the datamart from DEV -> TEST -> PROD the datamart is empty and requires a full refresh. Settings such as incremental refresh are lost when promoting.

Typically, are three full sets of data kept with incremental refresh going? Or do people just keep one datamart and promote the datasets/reports?

Our full datamart refresh (pulls from web api) - is quite slow - so it's not realistic to quickly populate an empty datamart, if it needs a full refresh every time it is promoted.

Thanks!

[Anonymous]

Hi @mike9999 ,

Datamart deployment pipelines can be complex and require careful planning to ensure that data is properly managed and secured as it moves through the various stages. Typically, it is recommended to keep separate datamarts for each environment (DEV, TEST, PROD) to ensure that data is not inadvertently mixed or corrupted during the promotion process. Regarding incremental refresh, it is generally not recommended to use this feature during the promotion process as it can lead to data inconsistencies and errors. Instead, it is best to perform a full refresh of the datamart each time it is promoted to ensure that the data is accurate and up-to-date. In terms of managing the time it takes to refresh the datamart, there are a few strategies that can be employed. One option is to use a data warehouse or other data storage solution that allows for faster data retrieval and processing. Another option is to schedule the datamart refresh to occur during off-peak hours when system usage is lower. Finally, it may be possible to optimize the data retrieval process itself to reduce the time it takes to refresh the datamart.

Best Regards,

Neeko Tang

If this post  helps, then please consider Accept it as the solution  to help the other members find it more quickly.