The great thing about Fabric is that we can share data between Lakehouse and Warehouse in one workspace. However, the utility is massively unlocked when we can share workload/processing/pipeline across the workspace too. This means the ability to run a bit of Spark notebook, then a stored proc, then another notebook, then stored proc again etc.
Usually a mixed pipeline would need a secondary tool like Airflow or Prefect. However, if we can execute T-SQL stored procs from the Notebook, then we can orchestrate a pipeline across lakehouse/warehouse from within Fabric itself. This will be massive.
Please add the ability to easily execute Warehouse T-SQL from Spark notebook. Presumably we can try to do this by setting up the Apache Spark connector but it would be quite complicated.
2 Comments
- fbcideas_migusrNew Member
Up-voting! Also adding the ability to run TSQL Notebooks from PySpark Notebooks (Run-all Notebooks) to orchestrate workflows across Lakehouse & Warehouse! :)
- fbcideas_migusrNew MemberStatus added:Needs Votes
Recent ideas
Allow NotebookUtils getSecret() to authenticate with Workspace Identity
Current behavior In Microsoft Fabric, notebookutils.credentials.getSecret() authenticates against Azure Key Vault using the identity of the user who executes the notebook. This behavior appli...tmihara1 hour agoNew MemberNew3Views0likes0CommentsSupport Synonyms in Fabric Warehouse
Microsoft SQL Server has a very powerful feature by the way of "synonyms." It allows users and DBAs to do all sorts of powerful magic such as rewiring objects under the hood (e.g. run the code agains...matthias-bi5 hours agoRegular VisitorNew1.3KViews18likes2CommentsExpose Refresh Warnings and Informational Messages via Notifications and API
When a Power BI semantic model refresh completes successfully, the status shows Completed, even when the refresh details contain warnings or informational messages that require attention. Please pro...Jashwanth_K7 hours agoMicrosoft EmployeeNew22Views6likes0CommentsInvoke Pipeline Task - Workspace Identity Authentication
Currently, the Fabric Data Factory Invoke Pipeline task uses the user's credentials who saved the pipeline to then authenticate to the Azure Data Factory to execute the ADF pipeline. When that user'...dzebrowitz11 hours agoAdvocate IPlanned1.8KViews61likes5CommentsFabric Pipeline should run as workspace identity
Currently, Microsoft Fabric pipelines run under the identity of the last user who modified them, which can cause disruptions when tenant administrators make changes to security policies, such as enab...pellitteris12 hours agoAdvocate IINew1.4KViews27likes3Comments