Vibe Coding Fabric Notebooks in VS Code: A Production Checklist
1. Key Insights Level: 300 - practical implementation. You need to know Spark and Fabric notebooks. You do not need to know how the extension works internally. The situation. Fabric notebooks are an excellent environment for developing and operating Spark workloads. But once a notebook passes a few hundred lines, the engineering activities - refactoring, documentation, architecture review, large-scale changes - become hard to do in a browser. The alternative. The Fabric Data Engineering extension for VS Code opens a different workflow: Download the notebook from Fabric Work on it locally with a modern coding agent - Claude Code, the Claude extension, GitHub Copilot agent mode, OpenAI Codex, or a local agent Publish it back You keep Fabric's managed Spark platform, and you add what VS Code already has: advanced refactoring, Source Control, review workflows, and project-wide AI context. What surprised me. The value was not where I expected it, and the line is not where most people draw it. Generating a notebook from scratch was the weakest use. Ask for the whole thing at once and you get something plausible that you then have to verify end to end - which costs more than writing it yourself. Generating code block by block worked well. "Add one more cell that does X" is bounded. I can read it, run it, and reject it in seconds. Improving notebooks that already exist was the strongest of all. The existing code is the specification, so the agent has something to be correct against. So the useful rule is not generation bad, refactoring good. It is about scope: the agent is reliable when the change is bounded and verifiable in one reading, and unreliable when it is unbounded. What it costs. Every capability here has its limitation, and they belong in the same paragraph: Notebook synchronization is not a real-time collaborative experience. Treat it as checkout, not as co-editing. AI-generated changes require human review - and the notebook diff you review is not a code diff. That is Section 3, and it is the part I did not expect. Production workloads still need governance, validation and security controls. The agent does not provide them. So, the conclusion up front. Successful Spark vibe coding is less about replacing engineering discipline and more about augmenting it. If you skip the review step, you have not saved time - you have only moved the risk later. What is in this article. A practical workflow: prerequisites, the development patterns that worked, three review strategies with their trade-offs, production safeguards, the prompt set I reuse, and the lessons from real experimentation. The prompts are at the end. 2. When Spark Notebooks Vibe Coding is Helpful Experimenting, I found a few cases where having Spark in VSCode is more beneficial than having to solve its complexity and logistic: Converting messy 30-cells notebook into a well-structured sanctuary Solving complex technical problems (API ingestion, complex transformation, complex dependencies), to save time for copy-paste, and doing that coding step-by-step, with powerful AI assistance in place Refactoring becomes just easier using VSCode. It may be done using AI, by prompting. Or, if we want to do it manually, F2 (Windows) or Fn-F2 when the desired variable is selected will help. It is a very clean way to refactor code at scale, with minimal risk. 3. When is Vibe Coding not worth the cost? Simple, straightforward notebooks - easier to modify using the web or copy-paste AI-generated code. Highly critical pipelines, where the mistake costs a lot or the privacy model is incompatible with AI provider terms - these are subject to reconsider using AI in autonomous or partly autonomous mode. Now, let's observe some workflows and prerequisites to be able to vibe-code in Spark notebooks. 4. Prerequisites What is needed to start vibe-coding in Fabric/Spark? VSCode - indeed; Fabric Data Engineering VS Code - to connect to Fabric; Coding agent (I use the Claude extension, installed in VSCode) Turned on AutoSave in VSCode will help avoiding overwriting and keep the changes actual. 5. Workflow Extension -> Workspace -> Notebook -> Download -> Open Then - modifying the code, using vibe-coding. Then - publishing back. If the notebook was updated on the web, it is possible but harder to merge all the changes. I would omit that scenario entirely to avoid any issues. Full journey may look like that: During testing, I found that notebook synchronization between Fabric and VS Code should not be treated as real-time collaboration. If substantial changes were made through the Fabric web interface, re-downloading the notebook may be safer than attempting to merge changes manually. Disclaimer: If somebody changed the code in the web interface, getting these changes may be a challenge. Simple Update icon doesn't help, even though it says the update was successful. So, additional awareness is required for that matter. It will be an alert during publishing the code with the option to accept local or remote change: 6. Production-ready best practices 6.1. Security & Privacy If we work in production, it is important to follow the organization's rules, and use only appropriate AI tools. 6.2. Responsibility AI may change code very unpredictably. It may break production silently. Two options for handling that. First is asking AI to show all the changes. The second one is to manually check points at the beginning of vibe-coding and at the end. 6.2.1. Strategy one - risky: Asking AI to show differences Before vibe-coding, give an instruction similar to this: Before making any file modifications, show me the proposed diff and wait for my approval. As a result, the agent may showcase the changes it made. The key thing here is it may miss something or hallucinate. So, for highly critical pipelines, it may not work. This strategy is good for prototype, PoC, and MVP development. But, actually, I would say, it is overall the best time to use vibe-coding. Fully productionized critical system - it is debatable whether the outcome outweighs risks using AI assistants to write code in agentic mode. It is much safer to use manual coding, by copy-paste-debug without writing to the final tables. 6.2.2. Strategy two - conservative: Backup notebook and related data Backing up related data and code - it is a highly conservative strategy. It make sence doint this only if vibe-coding will bring much more benefits than the other ways of changing the logic 6.2.3. Strategy three: Checking the changes before and after Open notebook in VS Code. Initialize Source Control (Git). Let Claude modify the notebook. Open Source Control (Cmd+Shift+G). Click the modified notebook, to see the differences, before publication. 7. Additional block - Prompts for success 7.1. Prompt 1 - Notebook Structure Overview the notebook, and add its title, description, and after that - all the Headlines and sub-headlines, with prefixes like 1., 2., 3., 1.1., 2.2., etc 7.2. Prompt 2 - Find weak points Check code and find what may be improved. Don't do any changes. Only give the comments. 7.3. Prompt 3 - Refactor Notebook Refactor this notebook without changing functionality. Extract duplicated logic into reusable functions, improve variable naming, add comments where necessary, and reorganize code into logical sections. 7.4. Prompt 4 - Documentation This is probably the highest ROI prompt. Generate markdown documentation for this notebook. Explain business purpose, inputs, outputs, dependencies, assumptions, and execution flow. Many Fabric notebooks have zero documentation. 7.5. Prompt 5 - Performance Review This is very Spark-specific. Review this PySpark code for performance issues. Identify unnecessary shuffles, collect operations, driver-side processing, inefficient joins, and repeated scans. 7.6. Prompt Set for Feature Building 7.6.1. API Ingestion Prompt: Review this notebook and create a reusable API ingestion framework with retry logic, pagination support, logging, and configuration-driven endpoints. 7.6.2. Data Quality Framework Prompt: Convert all validation checks into reusable functions and generate a summary report at the end of execution. 7.6.3. Monitoring Notebook Prompt: Extract all monitoring logic into reusable functions and create a centralized alerting section. 7.6.4. Legacy Notebook Cleanup Prompt: Refactor this notebook into logical sections and eliminate duplicated code. 8. Before & After examples Before: After: 9. What agents may we use here? The agents selection is not limited. When we work with the notebooks that way, they are stored locally. It means, Claude Code, OpenAI Code, Loca agent from VSCode, Github Copilot Agend mode and others are fully available for us. 10. Lessons Learned Documentation generation produced the highest ROI. Structural refactoring was more reliable than business logic generation. Source Control review was essential. Small notebooks rarely justified the overhead. Large notebooks benefited the most. This makes the article feel based on real experimentation rather than theory. 11. Conclusion Spark notebooks are a surprisingly good candidate for vibe coding. The extension also supports Python notebooks and Spark SQL, from what I tested. Large notebooks often contain repetitive patterns, technical debt, and insufficient documentation-areas where modern coding agents can provide significant value. However, AI-generated changes should always be reviewed before publication, especially for production workloads. Combined with VS Code Source Control and the Fabric Data Engineering extension, vibe coding can become a practical addition to a data engineer's toolkit rather than a replacement for engineering judgment.
Table is not created in lakehouse after save & run in dataflow
i was exploring dataflows gen 2 in Fabric. I created a dataflow and gave data destination as a salesdata lakehouse which i already created but inside the lakehouse, that table is not created. i clicked save & run. In dataflow, data destination is correctly set as the salesdata lakehouse. it will be helpful if anyone can clarify me on this.
Azure Artifact Feed - Connection string not supported
Hi everyone, I'm trying to create a new Azure Artifact Feed (Preview) connection in Microsoft Fabric, but I'm consistently getting the following error: Unable to create connection for the following reason: Connection string is not supported. Details: Query contains unsupported function. Function name: AzureArtifactFeed.Contents Environment Microsoft Fabric Connection Type: Azure Artifact Feed (Preview) PyPi Feed URL format: https://pkgs.dev.azure.com/<organization>/_packaging/<feed>/... Creating the connection from Manage Connections and Gateways > New Connection > Cloud Questions Has anyone successfully configured an Azure Artifact Feed connection in Fabric recently? Could this be related to the note displayed in the UI indicating that some cloud connections are not supported by Dataflows Gen1 and Datamarts? Any insights or confirmation that this is a known issue would be greatly appreciated. Thanks!
Best practices for ingesting Marketo data into Microsoft Fabric using REST API / Bulk API
Hi Fabric Community, We are building a data platform in Microsoft Fabric to ingest and store Marketo data in a Fabric Lakehouse. For Salesforce, we can use the standard connector available in Microsoft Fabric. However, as far as we understand, there is currently no equivalent standard connector for Marketo. Therefore, we are considering using the Marketo REST API / Bulk API to ingest Marketo data into Fabric. We would appreciate any guidance on recommended implementation patterns or best practices for the following two scenarios. 1. Ingesting Marketo data using the REST API We are considering ingesting Marketo data, such as Person data, into Microsoft Fabric. If possible, we would prefer a low-code / no-code implementation. We are currently considering the following options: Calling the Marketo REST API from Dataflow Gen2 Calling the Marketo REST API from a Fabric Data Pipeline Calling the Marketo REST API from a Fabric Notebook For periodically retrieving Marketo data and storing it in a Fabric Lakehouse, which of these approaches would generally be recommended or considered a best practice? We would prefer to use Dataflow Gen2 or another low-code approach if possible. However, considering requirements such as Marketo API authentication, pagination, and API rate limits, are there cases where a Notebook-based implementation would be more appropriate or necessary? We also plan to use Marketo Activity Log data together with Salesforce Lead / Contact data. In this scenario, are there any recommended approaches regarding which keys or fields should be retained from Marketo and how Marketo Person / Activity data should be linked with Salesforce Lead / Contact data in Fabric? 2. Ingesting large volumes of Activity Log data using the Marketo Bulk API For high-volume data such as the Marketo Activity Log, we are considering using the Marketo Bulk Extract API. As we understand it, the Bulk API exports data as files such as CSV. When ingesting this data into Fabric, we are considering two approaches: Option A Store the CSV files as-is in the Files area of the Lakehouse. Option B Transform the retrieved CSV data and store it directly as Delta tables. Which approach would generally be recommended? In particular, if we adopt a Medallion Architecture (Bronze / Silver / Gold), would it be a common or recommended pattern to: rather than converting the data into tables immediately during ingestion? Also, for continuously ingesting Marketo data via the Bulk API, are there any recommended Fabric implementation patterns using Data Pipeline, Notebook, or Dataflow Gen2, especially when considering incremental ingestion and Marketo API limits? Any examples, architecture recommendations, or lessons learned from similar Marketo–Fabric integrations would be greatly appreciated. Thank you! #Microsoft Fabric #Data Factory #Dataflow Gen2 #Data Pipeline #REST API #Lakehouse #Data Integration #Marketo
Sharepoint shortcut folder reads empty after overwriting a file; only a shortcut rename refreshes it
Product experience: OneLake (SharePoint/OneDrive shortcuts, GA June 2026) Summary: Since approximately 28 July 2026, when I overwrite an existing file (same filename) in a SharePoint document library that is surfaced into a Fabric Lakehouse via an ODSP shortcut, the shortcut folder in the Lakehouse reads as EMPTY. The updated content does not appear. The only reliable fix is to RENAME the shortcut, which forces it to re-enumerate and the files reappear correctly. This worked fine before late July with no changes on our side. Impact: breaks scheduled pipelines that rely on overwriting files in SharePoint-shortcut folders. The rename workaround is not automatable at scale. Asking: can someone confirm whether same-name overwrites are expected to invalidate the ODSP shortcut metadata cache, and whether this is a known regression from late July 2026?
SharePoint shortcut folder reads empty after overwriting a file; only a shortcut rename refreshes it
Product experience: OneLake (SharePoint/OneDrive shortcuts, GA June 2026) Summary: Since approximately 28 July 2026, when I overwrite an existing file (same filename) in a SharePoint document library that is surfaced into a Fabric Lakehouse via an ODSP shortcut, the shortcut folder in the Lakehouse reads as EMPTY. The updated content does not appear. The only reliable fix is to RENAME the shortcut, which forces it to re-enumerate and the files reappear correctly. This worked fine before late July with no changes on our side. Impact: breaks scheduled pipelines that rely on overwriting files in SharePoint-shortcut folders. The rename workaround is not automatable at scale. Asking: can someone confirm whether same-name overwrites are expected to invalidate the ODSP shortcut metadata cache, and whether this is a known regression from late July 2026?
Why leaving Fabric to find data in Fabric?
The Explore tab in OneLake catalog, in my opinion, does not live up to its name. If an organisation has all its data in Fabric, well-governed with descriptions and ownership, why can't users find metadata (at least name and description) of such data? Well, yes: Purview. But why isolating metadata asynchronously? Why can't users see what's out there? It's like hiding countries from world maps - how would you know where to travel and apply for visa? Is there any plan to solve this problem? Someone else facing this issue?Datagateway stuck at "Failed to sign in. Please Verify your credentials and try again."
Im droping this here as i didnt see anywhere where this had been addressed. When trying to signin to the data gateway the app freezes and says "Failed to sign in. Please Verify your credentials and try again." The credentials are good and everything was fine prior to update. I got this after updateing the datagateway. The app freezes up and you have to force close. The work around is that you click signin options prior to entering email select these options and it will redirect you to your browser to login and wont lock up hope this someoneDirect Lake on SQL Endpoint + OneLake Security: Delegated vs. User Identity Mode (A Field Guide)
Direct Lake on SQL Endpoint + OneLake Security: the field guide. Model owned by an SPN, report published, refresh green — but every visual fails with QueryUserError? The reason is which identity actually reads the data, and it's not the one you think. Delegated vs. User identity mode, shortcut + native tables, SPN refresh traps, and the new Fabric default — decoded with diagrams and permission matrices. Save yourself the debugging session.
