Forum Discussion
Dataflow data
Using sempy one can query the data from a dataset directly into notebook.
Is there any way to similarly query/read the data directly from dataflow (gen1/gen2 df) in a notebook?
I am not sure if sempy can query the data from a df.
Is there any method available whatsoever (sempy, API, other workaround)?
Thank you in advance.
Hi smpa01,
Thank you for reaching out to Microsoft Fabric Community Forum.
SemPy is designed primarily for querying Power BI datasets. However, it is important to note that it does not have built-in capabilities for querying dataflows. For querying dataflows, you may need to consider alternative tools or methods, as suggested by the super users below.
Regards,
Vinay Pabbu
10 Replies
Dataflows Gen1 are stored as Azure Blobs. It is technically possible, but very, very convoluted, to access them directly. Not worth the effort in my opinion.
Not really sure about Dataflows Gen2 - they give you the option to specify alternative destinations, so they may not even store anyting themselves.
lbendlin Thanks. But the method described here is not straightforward.
I ideally want to be able to read the data by making an API call or so in a notebook.
The fact that,
a. there is a way to read the data in notebook directly from some sources (databricks,ssas,smartsheet etc as API access is secured)
and
b. there would always be data residing in sources (sharepoint, sql, others - can only be read through dataflow as direct API access is restricted by the data owners)
if MS can provide access to dataflow data (for #b) using API (last time I checked it is currently not possible, PBI access token does not have Dataflow as audience), it would be so much more easier to utilize notebook as the single source to read data from different sources and build downstream applications.
The alternative is to scrape dataflow altogether and build ssas db (to utilize #a in notebook) but the caveat is dataflow usually has more memory attributed than dataset.
I don't to how to solve this problem. The ecosystem is following
previously power query (limited opportunites, not optimized),notebook ( query/ transform/ wrangle/ ML from different sources) -> downstream storage (Lakehouse,KQL, DW) ->downstream applications (BI, others)Occam's Razor - challenge the need for the dataflow if you can do all this with Data Factory, Pipelines and notebooks.
Even in legacy Power BI dataflows are very often a solution looking for a problem.
If you're using Dataflow Gen2, why not write the outputs to a Lakehouse and the Notebook can query the Lakehouse
Concur. That 1433 issue is an unvoluntary showstopper for us too. Out Cyber Security team does not take kindly to Microsoft's suggestion to "just open this port for these couple endpoints".
Hi smpa01,
Thank you for reaching out to Microsoft Fabric Community Forum.
SemPy is designed primarily for querying Power BI datasets. However, it is important to note that it does not have built-in capabilities for querying dataflows. For querying dataflows, you may need to consider alternative tools or methods, as suggested by the super users below.
Regards,
Vinay Pabbu
Hi smpa01,
As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for the issue worked? or Let us know if you need any further assistance?
If our response addressed, please mark it as Accept as solution and click Yes if you found it helpful.Regards,
Vinay Pabbu
Hi smpa01,
we wanted to kindly follow up to check if the solution provided for the issue worked? or Let us know if you need any further assistance?
If our response addressed, please mark it as Accept as solution and click Yes if you found it helpful.Regards,
Vinay Pabbu
Hi smpa01,
May I ask if you have gotten this issue resolved?
If it is solved, please mark the helpful reply or share your solution and accept it as solution, it will be helpful for other members of the community who have similar problems as yours to solve it faster.
Regards,
Vinay Pabbu
