The great thing about Fabric is that we can share data between Lakehouse and Warehouse in one workspace. However, the utility is massively unlocked when we can share workload/processing/pipeline across the workspace too. This means the ability to run a bit of Spark notebook, then a stored proc, then another notebook, then stored proc again etc.
Usually a mixed pipeline would need a secondary tool like Airflow or Prefect. However, if we can execute T-SQL stored procs from the Notebook, then we can orchestrate a pipeline across lakehouse/warehouse from within Fabric itself. This will be massive.
Please add the ability to easily execute Warehouse T-SQL from Spark notebook. Presumably we can try to do this by setting up the Apache Spark connector but it would be quite complicated.
2 Comments
- fbcideas_migusrNew Member
Up-voting! Also adding the ability to run TSQL Notebooks from PySpark Notebooks (Run-all Notebooks) to orchestrate workflows across Lakehouse & Warehouse! :)
- fbcideas_migusrNew MemberStatus added:Needs Votes
Recent ideas
Justification Prompts When Downgrading or Removing Sensitivity Labels in Power BI Desktop
Description Currently, organizations can configure Microsoft Purview sensitivity label policies to require users to provide a justification when removing a label or lowering its classification. T...vvargasv12 hours agoMicrosoft EmployeeNew11Views0likes0CommentsVisual Type Defaults in Visualization Pane
This would change my life with fussy clients and client/public facing apps. In addition, it would allow me to add more value to the clients on retainers who do not have the 'hours' to allocate to...IolaWhiteley13 hours agoAdvocate IINew48Views7likes0CommentsMatrix: column totals as a bar chart docked to the edge
What's missing Data bars already work on the Total row and column through conditional formatting, so row totals can be read as length. Column totals cannot: they render as a horizontal bar inside a ...Sayurivalente14 hours agoRegular VisitorNew79Views6likes1CommentAllow Bookmarks to Affect Multiple Report Pages
Description: I would like to request the ability for a single Power BI bookmark to affect and maintain a state across multiple report pages, rather than being limited to a single page. For example,...Josegonzalez2114 hours agoNew MemberNew20Views4likes1CommentFeature Proposal: Expandable/Collapsible Page Sections for Reports on Same Semantic Model
Subject: Feature Proposal: Expandable/Collapsible Page Sections for Reports Using the Same Semantic Model Dear Microsoft Power BI Team, I would like to propose an enhancement to the Power BI report...Dr7awb14 hours agoNew MemberNew32Views1like1Comment