(Help) Modelling Using 2 Dataflows

[felipejeunon]

Hello everybody.

I read some documentations about modeling using two dataflows and one dataset. The first dataflow would bring the data from the source without any transformation being applied (raw data). The second dataflow would be the layer where we would apply the necessary transformations. When we use a dataflow as a source to create a second dataflow, it comes in the model as an entity, and when we try to do some transformation we get the error message bellow. I am also leaving the image with the modeling I want to implement.

My question is: how do I apply transformations to the second Dataflow since I can't because the first one exists as an entity? What is the correct way to do this modeling?

Thank you very much.

https://dataandai.files.wordpress.com/2020/06/organizedataflows_linkedentitiesnamingconvention.png 

https://i.imgur.com/acaycqg.png 

[felipejeunon]

Hello everybody.

I read some documentations about modeling using two dataflows and one dataset. The first dataflow would bring the data from the source without any transformation being applied (raw data). The second dataflow would be the layer where we would apply the necessary transformations. When we use a dataflow as a source to create a second dataflow, it comes in the model as an entity, and when we try to do some transformation we get the error message bellow. I am also leaving the image with the modeling I want to implement.

My question is: how do I apply transformations to the second Dataflow since I can't because the first one exists as an entity? What is the correct way to do this modeling?

Thank you very much.

https://dataandai.files.wordpress.com/2020/06/organizedataflows_linkedentitiesnamingconvention.png 

https://i.imgur.com/acaycqg.png 

[jeffshieldsdev]

Right click on it and create a Reference query to the Linked Table, then make the transformations you need. If the Workspace is not in Premium capacity, you'll have to uncheck "Enable Load" on the Linked Table.

[felipejeunon]

Hi Jeff.

Thank you for the answer.

When I create a reference, I'm doubling the base, in terms of data, right?
Given this, is it more interesting that I have a single dataflow bringing the data from the base already with the transformation or modeling as I initially proposed?

Thanks.

[jeffshieldsdev · Accepted solution]

A reference is just a pointer, nothing is duplicated. When a dataflow refreshes, data is read from the source and stored in data lake storage behind the scenes. A second dataflow will read from that store.

If you plan on doing large transformations (aggregations, merges, sorts) or have a large volume of data, seperate ingest and transform dataflows can be helpful, or if you want to leverage the ingest dataflow in another way.

[felipejeunon]

Understood. Perfect Jeff.

Thank you so much. Have a good Day!