Databricks loading dataflows from Azure gen2 storage doesn't load table

[jacinto]

I have multiple text files and used custom function to merge and process in dataflows. The dataflows is connected to azure gen2 storage. I was trying to load the final table from storage to databricks notebook, however, instead of the finale table, I see the first unprocessed file (basically the sample file). In the dataflows, I don't see any issue, the files are combined without any issues. Anyone faced similar issue? Thank you

[StrategicSavvy]

hi @jacinto 

I think you cannot connect directly to dataflow because it is just a query and your data is not saved anywhere. I think you can consider 2 options: 

1. create dataflow and saved output in power bi datamart - then you will be able to see this table once you connect with databricks

2. connect your databictks notebook to azure data storage gen2 and make transformations in databricks

I hope this post helps to solve your issue and if it does consider accepting it as a solution and giving the post a thumbs up!

My LinkedIn: www.linkedin.com/in/lukasz-kozdron