topic Re: check accuracy of data load in Pipelines

check accuracy of data load

arkiboys2 — Sat, 13 Jul 2024 17:44:17 GMT

how is it possible to check/compare the source to the destination?

for example, the data factory pipelines load data from sql server into warehouse tables and now I would like to see if the loaded data is the same as source and report if there are any differences.

thank you

Re: check accuracy of data load

NandanHegde — Mon, 15 Jul 2024 03:19:07 GMT

Below links can help :

https://stackoverflow.com/questions/60669045/azure-data-factory-how-to-fetch-source-count-and-sink-count-for-data-validation

https://github.com/MicrosoftDocs/azure-docs/issues/64333

Re: check accuracy of data load

Anonymous — Mon, 15 Jul 2024 03:55:50 GMT

Hi arkiboys2 ,

I think you can join TABLE1 and TABLE2 first, then use filter to get the rows which's type isn't same. Next create a DerivedColumn to generate Type Change column. Finally, use Select transformation to remain the columns you need.

1. Create Join transformation to TABLE1 and TABLE2.

2. Filter the rows which's type isn't same. Expression: source2@Type != source1@Type

3. Add Type Change column. Expression:concat('from ',source1@Type,' to ', source2@Type)

4. Use Select transformation to remain the columns you need.

Then you will see what you want.

Best Regards

Yilong Zhou

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Re: check accuracy of data load

arkiboys2 — Tue, 16 Jul 2024 10:07:57 GMT

hello, thank you for the details. Is it possible you provide the same but for fabric data factory/dataflow gen2 thank you

Re: check accuracy of data load

arkiboys2 — Tue, 16 Jul 2024 10:13:59 GMT

Hi, thank you for the reply, however I am looking for the fabric version.