The ultimate Microsoft Fabric, Power BI, Azure AI, and SQL learning event! Join us in Stockholm, Sweden from September 24-27, 2024.
2-for-1 sale on June 20 only!
Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started
Hi there,
Hoping someone can help. I want to write a parquet file to Lakehouse, but can't see how to include storage otions (access token, use_fabric_endpoint). Polars does this as part of its "write_delta" process, eg.
, but "polardataframe.write_parquet" doesn't have the 'storage_options' field.
Note my motivation for using parquet rather than deltatables is so I can implement partitions for faster querying. If some kind person could tell me how to write deltatables with partitions this would aso solve my problem. I did try adding "partition_by" as an delta_write option, but Fabric would only accept a blank list
Some forums suggest using "df.to_pandas().to_parquet(filepath, storage_options={...}) ", but I don't know what to put in the filepath or storage options.
Others suggest "... stop using the storage_options parameter and just use the aabfs.open handler for both reading and writing", which sounds like a solution but I've never (consciously) used aabfs before and wouldn't know where to begin
Solved! Go to Solution.
Actually I managed to apply partitions using write deltatables. The issue was I had been trying to apply partitions to an existing rather than a new table. D'oh! 😛
Actually I managed to apply partitions using write deltatables. The issue was I had been trying to apply partitions to an existing rather than a new table. D'oh! 😛
Hi @mhaupt ,
Glad to know that your query got resolved. Please continue using Fabric Community on your further queries.
Hi @mhaupt ,
Thanks for using Fabric Community.
As I understand you are trying to write a partitioned parquet file to Microsoft Fabric Lakehouse.
Can you please check this doc - Microsoft Fabric: using Notebooks and Table Partitioning to Convert Files to Tables
Hope this might give you some idea over your query. Do let me know incase of further queries.
ta!
Join the community in Stockholm for expert Microsoft Fabric learning including a very exciting keynote from Arun Ulag, Corporate Vice President, Azure Data.
Ask questions in Data Engineering, Data Science, Data Warehouse and General Discussion.
Ask questions in Eventhouse and KQL, Eventstream, and Reflex.
User | Count |
---|---|
2 | |
1 | |
1 | |
1 | |
1 |
User | Count |
---|---|
7 | |
6 | |
4 | |
3 | |
3 |