Forum Discussion
Dataflow Gen2 File Destination - JSON and Parquet Format Support Needed for Nested OData Data
Hi pwr20 ,
Currently, Microsoft Fabric Dataflow Gen2 supports only CSV/delimited formats for file-based destinations such as Lakehouse Files and ADLS Gen2. These formats do not support nested data structures, which makes them unsuitable for landing raw nested OData data without flattening.
When working with nested OData navigation properties, Dataflow Gen2 treats nested records as unsupported for tabular or file destinations. As a result, nested columns are either dropped or replaced with null values when writing to Lakehouse tables, staging destinations, or files. Fully expanding nested data within Dataflow can work for small volumes but does not scale reliably, especially when using an on‑premises data gateway, due to sequential API calls and timeout limitations.
At this time, JSON or Parquet file destinations are not available in Dataflow Gen2, and there is no built‑in, scalable solution to preserve nested OData structures end‑to‑end. This is a current product limitation, not a configuration issue.
I’d encourage you to submit your detailed feedback and ideas via Microsoft's official feedback channels, such as the Microsoft Fabric Ideas.
Feedback submitted here is often reviewed by the product teams and can lead to meaningful improvement.
Thanks,
Prashanth
Hi pwr20 ,
Currently, Microsoft Fabric Dataflow Gen2 supports only CSV/delimited formats for file-based destinations such as Lakehouse Files and ADLS Gen2. These formats do not support nested data structures, which makes them unsuitable for landing raw nested OData data without flattening.
When working with nested OData navigation properties, Dataflow Gen2 treats nested records as unsupported for tabular or file destinations. As a result, nested columns are either dropped or replaced with null values when writing to Lakehouse tables, staging destinations, or files. Fully expanding nested data within Dataflow can work for small volumes but does not scale reliably, especially when using an on‑premises data gateway, due to sequential API calls and timeout limitations.
At this time, JSON or Parquet file destinations are not available in Dataflow Gen2, and there is no built‑in, scalable solution to preserve nested OData structures end‑to‑end. This is a current product limitation, not a configuration issue.
I’d encourage you to submit your detailed feedback and ideas via Microsoft's official feedback channels, such as the Microsoft Fabric Ideas.
Feedback submitted here is often reviewed by the product teams and can lead to meaningful improvement.
Thanks,
Prashanth
Thank you for the detailed explanation. This really clears things up and confirms what I have been experiencing. It is reassuring to know this is a product limitation rather than a configuration issue on our end.
Based on your feedback and our own investigation, we are now looking at two alternative approaches for the nested OData entities:
1. Using the Fabric Pipeline Copy Data Activity with the on-premises data gateway to land raw JSON or Parquet files into Lakehouse Files, and then using a Fabric Notebook to flatten the nested structures from there. we are currently investigating whether Fabric Pipeline Copy Data Activity supports our OData on-premises scenario and will update the thread with our findings
2. Azure Data Factory with a self-hosted integration runtime as a fallback option if the Copy Data Activity approach does not work out. For the flat/scalar entities, we will continue using Dataflow Gen2 as it works well for those.
I will also submit feedback via Microsoft Fabric Ideas as you suggested, as JSON and Parquet file destinations in Dataflow Gen2 would be a very valuable addition for scenarios like this.
Thanks again for taking the time to respond.