The great thing about Fabric is that we can share data between Lakehouse and Warehouse in one workspace. However, the utility is massively unlocked when we can share workload/processing/pipeline across the workspace too. This means the ability to run a bit of Spark notebook, then a stored proc, then another notebook, then stored proc again etc.
Usually a mixed pipeline would need a secondary tool like Airflow or Prefect. However, if we can execute T-SQL stored procs from the Notebook, then we can orchestrate a pipeline across lakehouse/warehouse from within Fabric itself. This will be massive.
Please add the ability to easily execute Warehouse T-SQL from Spark notebook. Presumably we can try to do this by setting up the Apache Spark connector but it would be quite complicated.
2 Comments
- fbcideas_migusrNew Member
Up-voting! Also adding the ability to run TSQL Notebooks from PySpark Notebooks (Run-all Notebooks) to orchestrate workflows across Lakehouse & Warehouse! :)
- fbcideas_migusrNew MemberStatus added:Needs Votes
Recent ideas
Set default x axis scrolling position
Enable selection of the default scroll position for the x-axis on a chart. For example, on a line chart with dates as the x-axis, when not all of the data can display in the view at once, the scroll...Lisa_Krieg5 hours agoNew MemberNeeds Votes7.8KViews1154likes104CommentsTable Visual — Rounded Corners and Pill-Style Cell Backgrounds
Add the option to display cell background colors as rounded rectangles or pill-shaped labels within the native Table visual. Currently, applying a background color fills the entire cell with a solid...urawizardharry6 hours agoRegular VisitorNew10Views0likes0CommentsAdd Global Alignment Controls for Power BI Matrix Values and Headers
Please add a global alignment setting for Matrix visuals that allows users to apply the same alignment, left, center, or right, to all measure columns at once. This should apply to column headers, d...AaronReasoner7 hours agoNew MemberNew4Views0likes0CommentsCase sensitive-insensitive option in Power BI
There should be an option in Power BI for the data in the columns to be either case sensitive or insensitive. When one deals with data involving client names or entity names, case sensitivity-sinsens...sohail_syed_moh7 hours agoNew MemberNeeds Votes2.3KViews124likes11CommentsGoverned & Regulation for Publish to Web feature
Notifying certain group of individuals if there is any enhancements performed on the semantic model which already hold an active Publish to Web code in place. This ensure in having more security and...Jaswanth_UNHCR7 hours agoNew MemberNew5Views0likes0Comments