Dataflows would be really useful if you could connect to them via DirectQuery. This would mean that we could query enormous dataflows that are not practical to load to the Data model and can make agg...
tmckenzie3
6 years agoNew Member
The other aspect I am finding the more I tinker with Dataflows is increasing the importance of being able to DQ these for a few reasons. 1) Getting the timing schedules for refreshes has just got a lot more complex - To get a Dataset refreshed you need to have the Dataflow refresh complete to get the latest data, then only after that is finished start refreshing the Dataset to get the latest data. The sequence of timings is going to be complex, plus also time debilitating. My first dataflows are taking around 1 hour to process, and then my datasets after that are taking a bit of time. Waiting a few hours to get the latest data is feeling like a step backwards compared to the refresh times I was getting with my existing models. 2) We can use Dataflows as a way to generate a 'Single Model of Truth' with the data preparation. However, since anyone can modify a DF in their Datasets this also breaks the certainty that the data validity is maintained. By allowing a DQ connection you can be assured that the Single Model of Truth can be maintained.
Recent ideas
OneLake Catalog full-text and content search
Why is it not possible to search OneLake by item description or even content? So I was thinking about how a user can find a table if they don't know in which lakehouse it is located? Why can't you se...TePe8 hours agoHelper IVPlanned1.1KViews21likes5Comments