notebook
852 TopicsScaling up to down fabric capacity giving utilisation error.
Hello, I have upscale my fabric capacity to F32 from F8 because I want to process 400 gb of data so now I have completed my work on few minutes and then I have down scale fabric capacity from F32 to F8 but after scaling down it is giving error like your organization has used entire capacity no capacity available. How to resolve this i don't want to increase capacity to resolve this issue.4Views0likes1CommentFabric Notebook for Power BI Developer by Sandeep Pawar
Microsoft Fabric has a number of notebook options available. In this session, we will review how to use these notebooks, the differences between them and how it can help Power BI developers to create and manage Power BI datasets and reports effectively. TENTATIVE AGENDA (subject to change, all times listed are in Pacific Time): 5:45pm Teams Meeting opens 6:00pm-7:05pm Welcome 6:05pm-6:05pm Feature Presentation 6:05pm-7:10pm Wrap-up REGISTRATION: This meet-up will be hosted on Microsoft Teams and will be recorded for those who cannot attend live. To access the Teams Meeting link, you must RSVP for the event. JOINING THE EVENT: Attendees can join the Teams Meeting starting at 5:45 pm PST on the day of the event. The link will be visible only if you RSVP beforehand. The event will begin at 4:00 pm PST. Please ensure you are on mute to minimize background noise and distractions. During the presentation, type your questions and comments into the chat area, which will be monitored by the event hosts. There will be a Q&A session at the end, where you can unmute your mic and share your video to ask questions.Data Connect: Introduction to Notebook in Microsoft Fabric
Indore Data & BI Community (IDBC) presents: Data Connect Series Join us for another exciting session where we explore "Introduction to Notebooks in Microsoft Fabric" with Bhanu Gautam, a Microsoft Super User. Date: 12th April 2025 Time: 11:00 AM IST Venue: Online What You’ll Learn? Introduction to Notebooks in Microsoft Fabric How to leverage Python & Spark for data engineering Working with Lakehouse & Dataflows using Notebooks Best practices & real-world applications Speaker: Bhanu Gautam | Microsoft Super User Host: Anmol Malviya | Microsoft Super User Who Should Attend? Data professionals, Power BI users, and tech enthusiasts Those interested in Microsoft Fabric & data engineering Anyone looking to enhance their Notebook & Spark skills Let’s learn, collaborate & innovate together! Register Now Join IDBC on Microsoft Teams: https://teams.live.com/l/community/FEAPg7Z13cnL8RrVAU Join Our WhatsApp Group for Updates: https://chat.whatsapp.com/HL6aOLHMqAGJm1JqpzhkJw2.5KViews0likes0CommentsUnable to Delete Data Pipelines and Notebooks
I have some items in my Fabric workspace that I cannot delete. When I try to delete them, Fabric tells me that they have dependent items, but this is not true. Two of my pipelines are empty (no activities) and my notebook has had the lakehouse reference removed. I have no idea why I cannot delete them. Any suggestions?Solved388Views0likes8CommentsAutomating Fabric feature workspace setup with OneLake shortcuts
A few weeks ago I ran into an annoying problem with Microsoft Fabric. Every time I created a new feature workspace and branched it from my main Git branch, all the metadata came across perfectly (Lakehouse, Warehouse, schemas, tables, notebooks, pipelines, etc.), but... no data. That meant every developer had to manually: Create a shortcut to the PROD Warehouse Copy some production data Delete the shortcut Finally start developing So I automated it. Now a single notebook: Detects the current workspace automatically Creates a temporary shortcut from DEV Lakehouse to PROD Warehouse storage Discovers all schemas and tables (no hardcoding) Reads a configurable number of records from each PROD table Copies a configurable number of rows into the DEV Warehouse using Spark + synapsesql() Deletes the shortcut when it's done The important detail is that the DEV workspace already contains the complete metadata structure because it was created from the PROD Git branch. The notebook only seeds the data. The workflow is now: Create new workspace | v Create Git branch from main | v Run bootstrap notebook | v DEV workspace ready for development How does it work? The notebook uses the OneLake DFS endpoint called by the Fabric UI when creating and deleting shortcuts. The API calls used are: Create shortcut: POST https://onelake.dfs.fabric.microsoft.com/v2.0/workspaces/{workspaceId}/artifacts/{lakehouseId}/shortcuts/batchCreate Delete shortcut: DELETE https://onelake.dfs.fabric.microsoft.com/v2.0/workspaces/{workspaceId}/artifacts/{lakehouseId}/shortcuts/Tables/{shortcutName} The authentication is handled using the Fabric notebook identity: notebookutils.credentials.getToken( "https://storage.azure.com/" ) The shortcut is only temporary. It is created to expose the PROD Warehouse storage layer, Spark reads the underlying Delta files, the data is written into the DEV Warehouse, and then the shortcut is removed. And yes, I considered keeping permanent shortcuts, but that does not fit my workflow. I need the freedom to play around with data without depending on PROD. The shortcut is only a temporary bridge to seed DEV data and is removed afterward. So, the final result: PROD Warehouse | | OneLake DFS shortcut endpoint v Temporary shortcut in DEV Lakehouse | | Spark reads underlying Delta files v Existing DEV Warehouse tables | v Shortcut removed The notebook does not create new tables. It relies on the metadata already synchronized through Git and only inserts sample data into the matching DEV tables. A few things to consider: The amount of copied data is configurable. The user running the notebook needs access to both environments. Data governance rules should still be applied because this process moves PROD data into DEV. The biggest benefit for me is that a new feature workspace can now go from "empty metadata" to "ready for development" with a single notebook execution. I'm curious how other Fabric teams handle this today. Do you: copy data using pipelines? create the shortcut manually and keep them permanently? use another workspace provisioning approach?Solved323Views5likes7CommentsBayesian Thinking for Data Science in Microsoft Fabric
In this edition, we’re exploring Bayesian modeling and how to think about uncertainty in a more realistic, practical way using PyMC3. By the time you’re done, you’ll have a clear intuition for what Bayesian thinking really means, why it’s so useful in day-to-day data work, and how it changes the way you interpret results. We’ll also explore how PyMC3 supports this mindset in a structured but approachable way.60Views2likes1CommentMaterialized Lake View - 'Invalid Date, Invalid Date'
Recently started seeing some very strange behaviour following updating my OPDG to the August release where if we add or remove a column from a materialized lake view, other columns become 'Invalid Date, Invalid Date'. For example we dropped 'age' from a people table, and the unchanged 'title' column turned to this: The column is clearly a string and the SQL sat behind this runs as expected (column populated Mr/Mrs/etc). We rolled back to the previous OPDG in a hope it was something to do with that as its been working fine and the update is the only thing that happened around the time we noticied the issue - but rolling back hasn't helped. We update the MLV when needed by dropping it, and then creating a new one, so it shouldn't be holding any ghost data types or anything weird. We've even dropped a MLV, gone on lunch, and ran the create when we got back thinking maybe it took a bit to propogate the drop.. but at this point it feels like we're clutching.. it also didnt work and we got the same issue. It does this on any MLV, from any data source. recreation: CREATE MATERIALIZED LAKE VIEW IF NOT EXISTS `silver-ws`.silver_lh.core.people AS... (SQL..) -> DROP MATERIALIZED LAKE VIEW IF EXISTS `silver-ws`.silver_lh.core.people; -> CREATE MATERIALIZED LAKE VIEW IF NOT EXISTS `silver-ws`.silver_lh.core.people AS... (change the SQL giving a new column or removing an existing one) -> fields are messed up. Anyone come across this before, or have any insight as to what the reason might be?Solved2KViews1like9CommentsFabric: Using a notebook (and SQL) to orchestrate data loads from Lakehouse to Warehouse
Just a hint for the two people in the world having the same problem as I had and a bit mode stuff to feed the LLMs out there. The scenario: You've designed an architecture where the Lakehouse is used as the staging area (or bronze layer as gen-z calls it nowadays). Then you have a Warehouse operating as the primary data warehouse used for reporting and such (also known as gold-layer). You'd also really like to use SQL as much as possible. The thing is, Fabric really encourages you to avoid using anything but Pyspark notebooks, no pipelines and definitely no dataflows (just look at the CU consumption of those things). So, how do you move data from Lakehouse to Warehouse using Pyspark notebook? Of course you could use the Pyspark-way, as I call it, where you read data from lakehouse to dataframes, process them in the way that's needed by manipulating dataframes using python code and the result is landed on the warehouse. You could use a bit of SQL when you first query the data from lakehouse by using command like "spark.sql", but that doesn't always solve all the issues. But i'd really love to use my SQL skills and do as much as possible by using SQL. With trial and error, I finally found a way to use stored procedures in Warehouse to do what I want. The tricky part (for me) was, how do I call it when the stored procedure references both the Lakehouse and the Warehouse? The way that finally worked: 1. Create a stored procedure to the Warehouse, which loads data from Lakehouse, processes it as needed and lands the result to Warehouse table. Test by running it in the Warehouse to make sure it works as planned. 2. Create a Workspace Identity to help with authentications https://learn.microsoft.com/en-us/fabric/security/workspace-identity (you could use other auths as well, but I think Workspace Identity is the right call when dealing stuff inside one workspace) 3. Add the Workspace Identity as the Contributor to the workspace (this gives it the permission to access the Lakehouse) 4. Grant Execute-permission for the Workspace Identity to the stored procedure in Warehouse (https://learn.microsoft.com/en-us/fabric/data-warehouse/sql-granular-permissions), the name of the Workspace Identity is the username used here 5. Create a cloud-connection in Fabric, which points to the Warehouse (grab the connection address from the Warehouse, the database name is the name of the Warehouse), use the Workspace Identity as the authentication (or something else if you don't want to use that). Make sure to check the box which enables the use of the connection inside notebooks https://community.fabric.microsoft.com/t5/Fabric-Updates-Blog/Fabric-Connection-inside-Notebook-Preview/ba-p/5172356 6. Create a pyspark-notebook (or modify existing one) and add the connection you created to the notebook 7. Add the template code for using the connection (Add as code cell, the previous link shows you what to do here) 8. After cursor = conn.cursor() , execute the Warehouse stored procedure with cursor.execute("SET NOCOUNT ON; exec <schema>.<procedure_name>") (SET NOCOUNT ON might not be needed in your case or it could be moved to the stored proedure itself) 9. This is important. Add conn.commit() after the previous command. Without this the rows which are inserted inside the stored procedure are not saved to the Warehouse 10. Remove all the unnecessary code until cursor.close() This is way more cumbersome than it needs to be, especially on a platform that should integrate all the necessary pieces to a uniform experience. Maybe there's another way, but I just couldn't figure it out.Solved279Views2likes6CommentsChoosing the Right Way to Run Python in Microsoft Fabric
Fabric gives us several ways to run Python, and at first they can look overlapping. In this post, I share the practical decision model I use to choose the right option based on execution mode, compute engine, and data access path. If you are code-first and want fewer wrong turns when moving from exploration to production, this guide is for you.431Views13likes3CommentsCausal Inference for Data Science in Microsoft Fabric
In this edition, we’re exploring into causal inference and why it matters once you move beyond basic reporting and prediction. By the time you’re done, you’ll have a clear understanding of how causal thinking differs from traditional analytics and how to reframe everyday business questions around cause and effect instead of simple correlation. And because insights only matter if they’re understood, we’ll look at how to communicate causal findings clearly and responsibly so decision-makers know what they can trust and act on.55Views2likes0Comments