Immense SQL endpoint lag with external data share
Hello everyone, our setup is: Tenant A: Mirroring a Business Central database into Fabric, external data share into Tenant B Tenant B: external data share shortcut landing in lakehouse (arrival in lakehouse works fine within a minute), DBT jobs access data via SLQ (that's why we're quite insistent on using SQL, DBT jobs are great) Our problem is: We're no strangers to SQL endpoint lag, but it is reaching unsustainable levels. Refreshing via the "Sync Metadata" button in the SQL endpoint GUI takes multiple hours. Without refresh the data becomes stale, up to several days old. Tables appear in the SQL endpoint GUI table list but are unreachable ("not existing") via SQL queries. Pipeline refresh activity and refresh via notebook take less but still too much time. Our capacity load is well within the limits, that's no probable cause. Our questions would be: A simple fix or workaround is always welcome, but also Is this a specific/known problem with external data shares (in combination with SQL endpoint)? Shortcuts should be a simple, latency free thing, right?
Fabric Lakehouse – Is there a limitation on concurrent Livy/Spark JDBC sessions?
Hi, I am working on an external Java integration with Microsoft Fabric Lakehouse where I need to perform CRUD operations on Lakehouse Delta tables. I created a standalone Java application using the Microsoft Fabric Spark JDBC driver (ms-sparksql-jdbc), which internally uses the Fabric Livy/Spark session. The basic CRUD operations are working successfully. For example, I can execute an INSERT like: Class.forName("com.microsoft.spark.livy.jdbc.LivyDriver"); String connectionUrl = String.format( "jdbc:fabricspark://api.fabric.microsoft.com;" + "FabricWorkspaceID=%s;" + "FabricLakehouseID=%s;" + "AuthFlow=5;" + "AuthAccessToken=%s;" + "LogLevel=INFO;" + "ReuseSession=true", WORKSPACE_ID, LAKEHOUSE_ID, accessToken ); Connection conn = DriverManager.getConnection(connectionUrl); String sql = "INSERT INTO snaplogic_test_customers " + "VALUES (?, ?, ?, ?)"; try (PreparedStatement pstmt = conn.prepareStatement(sql)) { pstmt.setInt(1, 1); pstmt.setString(2, "Alice Johnson"); pstmt.setString(3, "[email protected]"); pstmt.setDouble(4, 1250.50); pstmt.executeUpdate(); } Issue I am seeing When I run the standalone application for the first time, it works successfully. In the Fabric Monitoring Hub, I can see a Fabric -JDBC- session being created, and the CRUD operation completes successfully. However, if I run the same Java application again immediately after the previous execution completes, the second execution gets stuck during: Connection conn = DriverManager.getConnection(connectionUrl); The application remains stuck at the connection creation stage. Interestingly, if I wait approximately 5–10 minutes and then run the application again, it works successfully. My questions Is there any limitation on the number of active/concurrent Livy/Spark sessions for a Fabric Lakehouse? Is there a limitation where only one regular Livy/Spark session can actively run against a particular Lakehouse/workspace/capacity at a time? If multiple external applications need to access the same Lakehouse concurrently, are multiple Livy/Spark sessions expected to work in parallel? If multiple sessions are supported, what could cause the second call to:DriverManager.getConnection(connectionUrl) to wait until the previous JDBC/Spark session is completed or released? I need multiple external applications or requests to perform CRUD operations concurrently on Fabric Lakehouse tables, what is the recommended approach? My use case is to build an external application where multiple requests may need to perform SELECT, INSERT, UPDATE, and DELETE operations on Lakehouse tables concurrently. I am currently using the Fabric Livy API/Spark SQL approach. I want to understand whether there is a better or recommended Fabric-supported approach for implementing concurrent CRUD operations against Lakehouse tables. Any guidance or recommendations on the best approach would be greatly appreciated. Thanks in advance! Regards, BalramHands-On Data Engineering with Microsoft Fabric by Estera Kot
Hey Everyone! We're excited to announce our very first kickoff session, featuring none other than Estera Kot, Senior Product Manager at Microsoft and an influential speaker in the field of big data and spark. Some of you might already be familiar with her work from the Azure Synapse Analytics channel on YouTube. Best part? This is a virtual event. So whether you're sipping coffee at office or enjoying the sun outside, you can jump right in. All you need is some good old internet! Join our community group Meetup:-https://www.meetup.com/fabrics-data-engineering-explorers-user-group-norway Microsoft Community:https://tinyurl.com/v4j7wkau LinkedIn:-https://www.linkedin.com/groups/9381253/ The session is titled "Hands-On Data Engineering with Microsoft Fabric". Estera will guide us through the process of building an end-to-end lakehouse architecture. From data ingestion best practices to data transformation and even installing your own library for reporting. She'll show us how to leverage the tailored features of Microsoft Fabric to ace our data engineering tasks. It's going to be packed with insights! We'd also like to extend our heartfelt thanks to Stijn Wynants for helping us facilitating this session, and for his continued support of our community. So, mark your calendars for June 28th at 0900 PDT / 1800 CET and make sure you join our community group on LinkedIn/Meetup/Microsoft Community. We'll be posting all the details there. Can't wait to see you all at the session!Zürich - 49th Power BI User Group Meetup [ONLINE]
Dear Data Wizards We are looking forward to inviting all of you to our next meetup. This time in online Mode! Topics: What's New - Kristian Microsoft Fabric, Lakehouses and Power BI: A guide for BI developers - Kasper Microsoft Fabric Part 3: Deep Dive - Kristian The session will be recorded. Microsoft Fabric, Lakehouses and Power BI: A guide for BI developers Microsoft Fabric is here, and it fundamentally changes our data landscape. It introduces a Power BI like SaaS model for the data platform. You can now with a few clicks ingest data into the cloud by running pipelines, create a Lakehouse to make that data accessible and run Python, Spark, SQL and DAX on top of it. What does this all mean for Power BI? There are a lot of architectural changes, and you might wonder if you now need to start refactoring your Power BI solutions. In this session we get you ready to decide whether you want to start using the new Fabric workloads for your Power BI solutions. First, we do an introduction of Microsoft Fabric is, then we will get an overview of lakehouse and how the data is stored in parquet files under the covers. Finally, we will see how this data is being exposed to Power BI through Directlake and how this differs from DirectQuery or Import. Then we will close off with demos where we solve some common Power BI patterns like incremental refresh and transactions on top of data getting loaded into the Fabric lakehouse. Good to know We want this group to be a safe environment that encourages open discussion, exchange of ideas and problems you may face. Therefore, we kindly ask that no members will leverage the information for unsolicited acquisitions of new customers or projects. This group builds on trust, and without it we cannot learn from each other and excel on this topic. Want to be a presenter? We are always looking for new speaker. If you are interested and would like to show something to the Power BI Meetup Group please feel free to contact us!Zürich - 50th Power BI User Group Meetup [HYBRID]
Dear Data Wizards We are looking forward to inviting all of you to our next meetup. This time in hybrid Mode! If you wish to attend in-person, please send me an email to register - this is required! Topics: What's New - Kristian A new approach to generating Power BI Themes - Diana & Oscar Microsoft Fabric Part 4: Deep Dive - Kristian The session will be recorded. A new approach to generating Power BI Themes In this session, Diana and Oscar will discuss their approach to generating themes for Power BI and show us how to use the tool they developed for the community. Good to know We want this group to be a safe environment that encourages open discussion, exchange of ideas and problems you may face. Therefore, we kindly ask that no members will leverage the information for unsolicited acquisitions of new customers or projects. This group builds on trust, and without it we cannot learn from each other and excel on this topic. Want to be a presenter? We are always looking for new speaker. If you are interested and would like to show something to the Power BI Meetup Group please feel free to contact us!Meetup #1 - Fabric Overview - Online
Welcome to our first session of the Sydney Fabric user group! As we are just getting started our first couple of sessions will be online, then we will move to in person sessions in Sydney CBD. In this initial session James & Dean will give an overview of the Fabric platform, do a demo of building a Lakehouse form scratch, and run through some deployment points for consideration. Thankyou for joining us!Zürich - 51st Fabric User Group [ONLINE]
Dear Data Wizards We are looking forward to inviting all of you to our next meetup. This time in online Mode! Topics: What's New - Kristian Professional Developer Experiences in Power BI Microsoft Fabric Part 5: Deep Dive into Data Activator - Denis The session will be recorded. Professional Developer Experiences in Power BI Developer mode seamlessly integrates developer-centric features directly into Microsoft Power BI Desktop. This advancement empowers users to effortlessly implement source control for their work, thereby boosting efficiency and enhancing the maintenance of Power BI projects. When combined with Azure DevOps and Fabric Git Integration, Developer Mode opens up unprecedented collaboration possibilities, facilitating the establishment of robust Continuous Integration/Continuous Deployment (CI/CD) pipelines. These pipelines not only improve the reliability of Power BI projects but also enable scalability, ultimately elevating the development and collaboration experience to new heights. Join me to explore how these powerful tools can transform your Power BI development experience. Good to know We want this group to be a safe environment that encourages open discussion, exchange of ideas and problems you may face. Therefore, we kindly ask that no members will leverage the information for unsolicited acquisitions of new customers or projects. This group builds on trust, and without it we cannot learn from each other and excel on this topic. Want to be a presenter? We are always looking for new speaker. If you are interested and would like to show something to the Power BI Meetup Group please feel free to contact us!Zürich - 52nd Fabric User Group [ONLINE]
Dear Data Wizards We are looking forward to inviting all of you to our next meetup. This time in online Mode! Topics: What's New - Kristian Highlights 2023 - Kristian & Denis The session will be recorded. Good to know We want this group to be a safe environment that encourages open discussion, exchange of ideas and problems you may face. Therefore, we kindly ask that no members will leverage the information for unsolicited acquisitions of new customers or projects. This group builds on trust, and without it we cannot learn from each other and excel on this topic. Want to be a presenter? We are always looking for new speaker. If you are interested and would like to show something to the Power BI Meetup Group please feel free to contact us!Data Lakehouse Architecture with Microsoft Fabric
What are Data Lakehouses? When and why should you use this design pattern? How are they different from Data Warehouses? This session will answer these questions and expand on how new architecture design patterns can help you solve some of your business problems and prepare for the future. You may have heard about Data Lakes and data warehouses, but as technology evolves, there are new design patterns like Data Lakehouses that you should take into account when building data solutions. 𝙋𝙚𝙧𝙨𝙞𝙖𝙣 𝙋𝘽𝙄𝙐𝙂- 𝙈𝙞𝙘𝙧𝙤𝙨𝙤𝙛𝙩 𝘾𝙤𝙢𝙢𝙪𝙣𝙞𝙩𝙮: https://bit.ly/3IAg7xT 𝙇𝙞𝙣𝙠𝙚𝙙𝙄𝙣: https://bit.ly/32tGkif 𝙏𝙚𝙡𝙚𝙜𝙧𝙖𝙢 𝘾𝙝𝙖𝙣𝙣𝙚𝙡: https://t.me/PersianPBIUG 𝙔𝙤𝙪𝙏𝙪𝙗𝙚: https://bit.ly/3hk20RL Language: Persian - English
