Ways can we validation Fabric and redshift tables after migration

[Saadather]

Hi all, 

we are currently in process of our migration from redshift to fabric warehouse and i would like to know the ways we can do our validation/testing of same tables between source(redshift) and sink(fabric warehouse). And would like to hear from the experts in industry which are the possible ways we can do that?

Thank you!

[v-sdhruv]

Hi @Saadather ,

Just wanted to check if you got a chance to review the suggestions provided and whether that helped you resolve your query?

Thank You

[Saadather · Accepted solution]

Hi @v-sdhruv 

The solution only works when have to write two seperate expectations and create a script that could compare the results from these two, what i wanted was something like reconsiliation checks that could give me the results all together.

In GX fourm they also mentioned that currently this cross datasource feature is not supported yet..
https://discourse.greatexpectations.io/t/how-to-compare-two-tables-across-different-databases/588 

My further RnD for this use case i found these can be achieved by using another tool called SODA which supports such reconsicliation requiremnts.

[v-sdhruv]

Hi @Saadather ,

You might want to check -
Data Validations with Great Expectations in MS Fabric - ISE Developer Blog
Great Expectations (GX) can be used to validate your data migration from Amazon Redshift to Microsoft Fabric Warehouse.

You can begin by:

• Connect GX to your Redshift and Fabric Warehouse environments.
• Define expectations for key tables and columns.
• Run validations in parallel

Hope this helps!
Thank You

[BalajiL]

@Saadather 

If you are looking testing perspective, plan your test strategy to look for 

• Schema validation
• Row Count check 
• Hash check / Checksum
• Do parallel run testing 
• Create reconciliation reports 

If you are looking for migration aspect look at the below link:

https://learn.microsoft.com/en-us/fabric/data-warehouse/migration-assistant

Also check "Spectral Core tool" is offering redshift migration and validation. 

Thanks !

[Saadather]

hi @BalajiL 

Thankyou for clarifying, what i am intrested is by which ways i can do the above checks you mentioned i know via python i do that by connecting to both datasources and writing the queries and comparing the result for both.
What i really want to know is there any tool/way which azure/fabric provide to do so or any thrid party which supports that via automated way instead of me writing the sqls comparison for all the tables?