Linked entity data gets refreshed/duplicated for dataflow that uses it to create computed entity

[Anonymous]

Why does a linked table/entity get duplicated when refreshing a dataflow that uses the linked table/entity to create a computed table/entity?

I understand linked entities as shortcuts to Azure Data Lake entities when I read the MS documentation:

I'm trying to build a data pipeline that extracts data from a database view and performs some transformations without doing unnecessary work. However it appears that the transformation dataflow that ingests the extracted data as linked table also refreshes/duplicates the data. Detailed scenario of what I'm doing below.

Can anyone shed some light on why this could be happening? Am I doing something wrong? Would using the PowerBI.Dataflows() connector to link to the table instead of the PowerPlatform.Dataflows() connector be a better choice here?

Scenario

1. Database view

Oracle database contains view V_DIM_SIEBEL_CONTACT

2. Landing zone dataflow

LZ_Dim Siebel Contact dataflow contains 1 query LZ_Dim Siebel Contact that selects all data from the V_DIM_SIEBEL_CONTACT view

Power Query code

let

  ConnectToDB = Oracle.Database("TST_DWH", [HierarchicalNavigation = true]),

  SchemaDWPBI = ConnectToDB{[Schema = "DWPBI"]}[Data],

  SelectTable = SchemaDWPBI{[Name = "V_DIM_SIEBEL_CONTACT"]}[Data]

in

  SelectTable

Query plan

Refresh behavior

Requested on

Dataflow name

Dataflow refresh status

Table name

Partition name

Refresh status

Start time

End time

Duration

Rows processed

Bytes processed (KB)

Max commit (KB)

Processor Time

Wait time

Compute engine

Error

2022-02-23 15:43

LZ_Dim Siebel Contact

Completed

LZ_Dim Siebel Contact

FullRefreshPolicyPartition

Completed

2022-02-23 15:43

2022-02-23 17:07

01:23:28

NA

5804536

54864

NA

00:00.1

Not used

NA

3. Transformation dataflow

Dim Siebel Contact dataflow contains 2 queries:

• LZ_Dim Siebel Contact as linked table
• Dim Siebel Contact as computed table

Power Query code

LZ_Dim Siebel Contact (linked table)

Sensitive information scrubbed from code

let

  Source = PowerPlatform.Dataflows([]),

  Navigation = Source{[Id = "Workspaces"]}[Data],

  #"Navigation 1" = Navigation{[workspaceId = "<WORKSPACEID>"]}[Data],

  #"Navigation 2" = #"Navigation 1"{[dataflowId = "<DATAFLOWID>"]}[Data],

  #"Navigation 3" = #"Navigation 2"{[entity = "LZ_Dim Siebel Contact", version = ""]}[Data]

in

  #"Navigation 3"

Dim Siebel Contact (computed table)

Sensitive information scrubbed from code

let

  Source = #"LZ_Dim Siebel Contact",

  #"Changed column type" = Table.TransformColumnTypes(Source, {<COLUMN DATA TYPES>}),

  #"Marked key columns" = Table.AddKey(#"Changed column type", {"Tech Dim Siebel Contact Key"}, true)

in

  #"Marked key columns"

Query plan

LZ_Dim Siebel Contact

Query plan not available

Dim Siebel Contact

Refresh behavior

Requested on

Dataflow name

Dataflow refresh status

Table name

Partition name

Refresh status

Start time

End time

Duration

Rows processed

Bytes processed (KB)

Max commit (KB)

Processor Time

Wait time

Compute engine

Error

2022-02-24 09:18

Dim Siebel Contact

Completed

LZ_Dim Siebel Contact

FullRefreshPolicyPartition

Completed

2022-02-24 09:18

2022-02-24 09:58

39:32.5

9661869

5804536

56080

19:12.7

00:00.1

Cached

NA

2022-02-24 09:58

Dim Siebel Contact

Completed

Dim Siebel Contact

FullRefreshPolicyPartition

Completed

2022-02-24 09:58

2022-02-24 10:47

48:45.4

9661869

5804536

56640

33:41.6

00:00.1

Cached + folded

NA

[Anonymous]

Hi @Anonymous ,

Here are several relationships between Power Platform dataflows and Power BI dataflows.

https://docs.microsoft.com/en-us/power-platform-release-plan/2019wave2/cdm-data-integration/power-pl...

If you want to know more about Power Platform dataflows, there is a document for your reference.

https://docs.microsoft.com/en-us/data-integration/dataflows/dataflows-integration-overview

In summary, Power Platform dataflow will be easier and faster.

Best Regards,

Jay