topic Re: Fabric storage issue in Data Engineering

Fabric storage issue

Kr97 — Thu, 29 Aug 2024 14:14:52 GMT

Hello we have implemented the Medallion architecture in the Fabric workspace and while Transforming the data a wrong Dataflow got triggered and Generated Huge stupendous amount of Errorneous data. We tried to find out where the data is stored but got no luck and we are being charged for that storage .
So now we want to get rid of that Data without affecting any of the original data. Can any one throw some light on how to Trace and troubleshoot that data and get rid off it.

Thanks in advance .

Re: Fabric storage issue

AndyDDC — Thu, 29 Aug 2024 15:16:26 GMT

Hi Kr97 what was the output destination of the Dataflow? e.g a lakehouse etc

Re: Fabric storage issue

Kr97 — Thu, 29 Aug 2024 15:18:33 GMT

Hello AndyDDC yes the destination is lakehouse

Re: Fabric storage issue

frithjof_v — Thu, 29 Aug 2024 19:30:27 GMT

Which table did the erroneous Dataflow write the data to?

Did you afterwards overwrite the erroneous data with good data? So now you cannot see the erroneous data anymore, you just see the good data?

Re: Fabric storage issue

AndyDDC — Thu, 29 Aug 2024 20:30:24 GMT

If you want to see the underlying data, install Azure Storage Explorer and connect to the Fabric workspace the lakehouse resides in. You can then see how much storage is being taken up.

https://learn.microsoft.com/en-us/fabric/onelake/onelake-azure-storage-explorer

If this dataflow wrote to a lakehouse table, you should just be able to delete the table

Re: Fabric storage issue

Anonymous — Fri, 30 Aug 2024 03:17:30 GMT

Hi Kr97

If you want to find the detailed destination information of each query, you can open that dataflow, switch to Diagram view. Then in the top right corner of each query pane, hover over the destination icon to find its details and update method. You can click the icon to edit the destination.

Best Regards,
Jing