The great thing about Fabric is that we can share data between Lakehouse and Warehouse in one workspace. However, the utility is massively unlocked when we can share workload/processing/pipeline across the workspace too. This means the ability to run a bit of Spark notebook, then a stored proc, then another notebook, then stored proc again etc.
Usually a mixed pipeline would need a secondary tool like Airflow or Prefect. However, if we can execute T-SQL stored procs from the Notebook, then we can orchestrate a pipeline across lakehouse/warehouse from within Fabric itself. This will be massive.
Please add the ability to easily execute Warehouse T-SQL from Spark notebook. Presumably we can try to do this by setting up the Apache Spark connector but it would be quite complicated.
2 Comments
- fbcideas_migusrNew Member
Up-voting! Also adding the ability to run TSQL Notebooks from PySpark Notebooks (Run-all Notebooks) to orchestrate workflows across Lakehouse & Warehouse! :)
- fbcideas_migusrNew MemberStatus added:Needs Votes
Recent ideas
Data Pipelines - Run only selected activities
For debugging and testing pipeline activities during development, allow us to select one or multiple activities and run only the selected pipeline activities. For example, I'm working on editing ...frithjof_v1 hour agoCommunity ChampionNew606Views11likes2CommentsSemantic model connection bindings should be in source control (Git)
Semantic model data source connection bindings should be source controlled. A semantic model can contain multiple data source references, each of which can be mapped to a separate Fabric data connec...frithjof_v8 hours agoCommunity ChampionNew12Views1like0CommentsBulk changing column names in Visualizations Pane
We often use raw/api column names or measures with a set nomenclature to be consistent and to keep track of them but we do not want to display these names in the visuals. Currently we have to change ...vishal14019712 hours agoFrequent VisitorNew6Views0likes0Comments