DataMart with 3 M rows of data very slow tiring to use Transform

[nrowey]

I am using Premium Per User licence, Added a new DataMart in a Premium Workspace with a Postgres connector.

The data has aprox 3 million rows , Caching is turned on but itis very slow to load the Transform function to build joins to a second datamart. Is there any way to make this faster.  Taking aprox 15 to 20 mins to load every time I select transform.

[otravers]

The patterns I'd go in most cases, depending on whether I want to use Import, are:

- Dataflow(s) > Datamart > SQL endpoint > Import (or composite) dataset > reports

- Dataflow(s) > Datamart > auto-generated DQ dataset > reports

This is based on my current understanding of datamarts, I reserve the right to change my mind!

[otravers]

In their present state it is completely impractical to do any ETL within the datamart, you want to do all your transformations in upstream dataflows.

[nrowey]

It sounds like i am using this Datamart incorectly, it automaticly created a Dataset with the same name.

Are you saying I should create a Dataflow and connect to the autocreated dataset, do all the ETL there and point my report to the Dataflow?

I guess I dont get the advantage?

[GilbertQ]

Hi @nrowey 

Yeah this is expected to happen because it has to first load all the data before it can do the join. So each time this happens it has to load the data to then find the joins.

The only way to do this faster is to do the joins in the PostGres database query?