Forum Discussion
Duplicate data appearing with no merges or relationships
I found the answer! As it turned out it was minor differences in upper/lower case that weren't immediately visible from the way in which I was inspecting the data at both ends. In Power BI visuals the differences in capitalisation weren't showing (like in Excel when you have a filter "today" and "Today" will be treated the same) meaning that the count for the value was duplicated, even though it would have been a count of one against each variation. Then of course at the Databricks end, the querying was case sensitive so I was only returning the specific version of the data with that capitalisation when I tried to find the relevant rows.
Eventually found it all once I moved to pulling the row counts for everything from both Databricks and PBI and comparing in Excel.
What a ride!
Thanks for your offers of possible things to test!
Super UserHow many columns are you highlighting with the remove duplciates step?
If more than one it will be looking at the unique combinations.
Assuming you are selecting only one is there a chance of whitespace being the difference between the two rows?
Try filtering in the QE to two of the rows that are duplicating. See if there are any differences in the QE rows compared to the table view versions.
I found the answer! As it turned out it was minor differences in upper/lower case that weren't immediately visible from the way in which I was inspecting the data at both ends. In Power BI visuals the differences in capitalisation weren't showing (like in Excel when you have a filter "today" and "Today" will be treated the same) meaning that the count for the value was duplicated, even though it would have been a count of one against each variation. Then of course at the Databricks end, the querying was case sensitive so I was only returning the specific version of the data with that capitalisation when I tried to find the relevant rows.
Eventually found it all once I moved to pulling the row counts for everything from both Databricks and PBI and comparing in Excel.
What a ride!
Thanks for your offers of possible things to test!