Issues with D365 Data Ingestion into Fabric Lakehouse: Mismatch in Row and Column Counts

[Pradeepsingh]

Hello, Fabric Community!

I am working on ingesting data from Microsoft Dynamics 365 into a Fabric Lakehouse. To achieve this, I created a shortcut to link the source data and ingested it into the Lakehouse tables.

However, I noticed discrepancies between the source and destination tables after ingestion:

1. Row Count Mismatch: The destination table contains additional rows with NULL values.
2. Column Count Mismatch: Some columns seem to be missing or have extra fields compared to the source.

I have already verified the schema and checked for issues during the ingestion process but haven't been able to pinpoint the problem.

Has anyone encountered similar challenges? Any suggestions, insights, or best practices to ensure schema and data consistency would be greatly appreciated!

Thank you for your help

[saud968]

Here are a few things that I can think off. 

Row Count Mismatch
Check for Duplicates: Ensure that there are no duplicate rows being ingested. Sometimes, data duplication can occur during the ingestion process.
Null Value Handling: Investigate why rows with NULL values are being added. This could be due to incomplete data in the source or issues during the transformation process.
Data Validation: Implement data validation checks before and after ingestion to ensure consistency.

Column Count Mismatch
Schema Mapping: Double-check the schema mapping between the source and destination tables. Ensure that all columns are correctly mapped and that there are no extra or missing fields.
Transformation Logic: Review any transformation logic applied during ingestion. Sometimes, transformations can inadvertently add or remove columns.
Dataflow Settings: If you're using Dataflow Gen2, enable verbose diagnostics to capture detailed logs and identify where the discrepancies are occurring

Best Regards
Saud Ansari
If this post helps, please Accept it as a Solution to help other members find it. I appreciate your Kudos!

[Pradeepsingh]

Thank you for your response.

In MS Fabric Shortcut ingestion there is no transformation , no schema mapping. Its pretty straight forward a link service something similar to Azure External Tables.