The great thing about Fabric is that we can share data between Lakehouse and Warehouse in one workspace. However, the utility is massively unlocked when we can share workload/processing/pipeline across the workspace too. This means the ability to run a bit of Spark notebook, then a stored proc, then another notebook, then stored proc again etc.
Usually a mixed pipeline would need a secondary tool like Airflow or Prefect. However, if we can execute T-SQL stored procs from the Notebook, then we can orchestrate a pipeline across lakehouse/warehouse from within Fabric itself. This will be massive.
Please add the ability to easily execute Warehouse T-SQL from Spark notebook. Presumably we can try to do this by setting up the Apache Spark connector but it would be quite complicated.
2 Comments
- fbcideas_migusrNew Member
Up-voting! Also adding the ability to run TSQL Notebooks from PySpark Notebooks (Run-all Notebooks) to orchestrate workflows across Lakehouse & Warehouse! :)
- fbcideas_migusrNew MemberStatus added:Needs Votes
Recent ideas
About "Download SQL database project" and fix the downloaded project
1. The sqlproj must be fixed, it is downloaded without the xml markup line <?xml version="1.0" encoding="utf-8"> 2. The sdk must be updated: Actual: <Sdk Name="Microsoft.Build.Sql" Version="0.1...Jarrinf7 hours agoFrequent VisitorNew48Views3likes2CommentsEventstream MQTT broker source: V3.1.1 support
The Eventstream MQTT broker source supports connections with public MQTT brokers that offer MQTT V5 support. Please offer support for V3.1.1 too, because this is still a valid and widely used ver...svelde8 hours agoSuper UserCompleted1.4KViews6likes4CommentsGet data from Apache Kafka topics via private networking
Current prerequisites limit to public Kafka instances which is not feasible for most enterprise customers. When will this limitation be lifted so that it can be used in private network setups, for ...Jugi8 hours agoMicrosoft EmployeePlanned3.5KViews2likes3CommentsSet default x axis scrolling position
Enable selection of the default scroll position for the x-axis on a chart. For example, on a line chart with dates as the x-axis, when not all of the data can display in the view at once, the scroll...Lisa_Krieg17 hours agoNew MemberNeeds Votes7.8KViews1154likes104Comments