How do I unzip a .gz file?
I used a data pipeline to make a web call (http) and get a file. The file has been downloaded to the Files area in the lakehouse. How can I uncompress this file? I am using a PySpark notebook to unzip this file. The file itself is good, since I was able to download the file and uncompress it on my windows machine. I tried this code, but that fails. df = spark.read.format("json").option("multiLine", "false").load("Files/bronze/github-events-2025-01-15-12.json.gz") # Display a preview display(df.limit(5)) # Save as Delta table df.write.mode("overwrite").format("delta").saveAsTable("github_events_bronze") print(f"Successfully loaded {df.count()} records")
FTL4 Trial Spark failing - InvalidRequestClusterFromFabricDenyList - Cluster Cancelled before Ready
FTL4 Trial, Canada Central. Spark sessions cannot start in multiple workspaces. Both Starter Pool and custom Small 1-node pool fail. Error is InvalidRequestClusterFromFabricDenyList and cluster is cancelled before reaching Ready. Please advise whether the Fabric Trial capacity/Spark backend needs to be reprovisioned or removed from the deny list.
Is a table ordinary, or special (= from a shortcut, a system table or materialized lake view table)?
Hi, In Python/PySpark/Spark SQL, is there a way to determine what kind of lakehouse table a table is—that is whether the table is a) an ordinary table, b) a materialized lake view table, c) a table from a shortcut, or d) a system table (e.g. like sys_dq_metrics)? In PySpark, I can use `spark.catalog.listTables()` to get details about the tables in a lakehouse. In Spark SQL, I can use `SHOW TABLES` to get the table list then `DESCRIBE DETAIL some_table_name` on each to get the particular table's details. However, in both cases, these details returned don't seem to give me an easy way to differentiate between ordinary tables and special tables (materialized views, shortcust, system tables). I'd really like to be able to differentiate between the two.... Any ideas? Thanks!
Best Practice for Ingesting External APIs and CSV Data into Microsoft Fabric
Hi everyone, I'm looking for guidance on the recommended approach for ingesting external data into Microsoft Fabric. From sources, including WooCommerce, Zoho, and third-party service providers such as Bobgo that expose data through REST APIs. We also receive data in CSV files from various external sources. What is considered best practice in Fabric for bringing this type of data into the platform? For example: Use a Notebook (Python/PySpark) to call APIs and land data in a Lakehouse? Use a Dataflow Gen2 to consume API and CSV data and load it into a Lakehouse? Use a Data Pipeline for orchestration and scheduling? Is Lakehouse the recommended landing destination? Is there a preferred medallion architecture (Bronze → Silver → Gold) for these types of sources? I'm particularly interested in understanding: What ingestion method is preferred for API-based sources. What ingestion method is preferred for CSV files that arrive on a schedule or are uploaded manually. When to choose Notebooks versus Dataflow Gen2. Whether Lakehouse is the recommended destination for raw ingestion. How others are handling authentication, pagination, incremental loads, retries, and error handling for APIs. How others are managing schema drift and changing file structures for CSV-based sources. Any real-world architecture patterns, lessons learned, or recommendations for production workloads. We're looking to establish a standard approach for onboarding new external data sources into Fabric, so any guidance or examples would be greatly appreciated. Thanks in advance!Direct OneLake / ADLS Gen2 Access to Lakehouse & Open Mirroring Tables
Hi Fabric Community, I'm currently exploring whether it's possible for external users to connect directly to Fabric data at the OneLake storage layer using the Azure Data Lake Storage Gen2 connector in Power BI, rather than accessing the data through the SQL Endpoint. The reason for this investigation is that the recent SQL endpoint metering changes have significantly increased costs in some of our environments. We're looking at alternatives where clients could build and refresh their own semantic models directly against the underlying OneLake data, thereby avoiding SQL endpoint usage where possible. What I've Tried I granted a guest user access to the workspace and attempted to connect to both: Lakehouse tables Open Mirroring tables using the Azure Data Lake Storage Gen2 connector in Power BI Desktop. I copied the ABFS(https path rather since the abfs did not work) path from the table properties and connected successfully. However, instead of seeing the Delta table itself, Power BI only exposes the underlying files and folders (Parquet and JSON files), as shown below: While I can browse the files, I have not been able to get Power BI to recognise the table as a Delta table automatically. Goal The objective is to allow clients to: Connect directly to OneLake storage using their guest account. Build Import semantic models in their own tenant. Refresh those semantic models without querying our Fabric SQL Endpoint. Reduce or eliminate SQL Endpoint compute costs resulting from semantic model refreshes and user queries. Questions Has anyone successfully connected to Lakehouse tables through the ADLS Gen2 connector and had Power BI recognise the Delta table structure automatically? Is this supported for Open Mirroring tables as well? Is there a recommended approach for reading Delta tables directly from OneLake in Power BI Desktop? Are there any limitations or permissions that would prevent external B2B users from accessing Delta tables in this manner? Has anyone implemented a similar architecture to avoid SQL Endpoint consumption after the recent metering changes? I also came across the Fabric connector that exposes Lakehouse tables directly through OneLake Data Hub and wanted to understand whether this could be a viable option as well. However, it appears that this method creates the semantic model in our tenant, alongside the Lakehouse where the data is stored. If so, I assume refreshes and queries would still consume resources within our Fabric environment. Has anyone tested this approach and compared the cost implications against direct access to the underlying Delta tables through ADLS Gen2? Here are a few concepts I am currently exploring: Copying the materialized/gold tables into the client's tenant, allowing them to build and manage their own semantic models and reports entirely within their environment. Creating and maintaining the semantic model in our tenant, with the client connecting their reports directly to that shared semantic model. Creating shortcuts from our gold tables into the client's tenant, enabling them to build semantic models and reports against the shortcut data in their own Fabric environment. I'm not entirely sure whether this approach is supported across tenants, but it is something I'm currently investigating. At this stage, I'm evaluating the pros and cons of each option, particularly with regard to performance, governance, data ownership, and the impact of the recent SQL endpoint metering changes. It is important to note, however, that these clients do not currently have Microsoft Fabric licenses or capacities in their own tenants. They only have Power BI Pro licenses available. Any potential solution would therefore need to operate within those licensing constraints and ideally avoid requiring the client to purchase or maintain Fabric capacity in their environment. Any guidance, best practices, or pointers would be greatly appreciated. Thanks in advance! Kind regards,DataFlow Gen2 Issue
Hi, Could someone please help me? I'm trying to load data from my on-premises SQL Server database into a Lakehouse in Microsoft Fabric. Step 1: Load data into Power Query (Dataflow Gen2) from SQL Server on-premises. Result: Success. Step 2: Load the data into my Lakehouse. This is where I'm encountering the problem. When I try to connect to the Lakehouse destination, I receive the following error: Additional information: The SQL Server connection is working correctly. The On-premises Data Gateway status is Online. I can access and write to the Lakehouse successfully from Fabric Notebooks. The issue occurs only when trying to use the Lakehouse as the destination in Dataflow Gen2. Has anyone experienced a similar issue or found a solution? Thank you.
