Forum Discussion
Limitations of bulk insert and openrowset
Hi Team,
I understand that Microsoft has released BULK INSERT and OPENROWSET in preview. However, I would like to gain more insight into their limitations.
My requirement involves loading Parquet files from the Lakehouse into Delta tables. Can the SQL endpoint be used for this process via OPENROWSET or BULK INSERT?
I appreciate your insights.
Thanks,
Jaganath
Hello Jaganath
the SQL endpoint cannot be used to load Parquet files into Delta tables via `OPENROWSET` or `BULK INSERT`. While Fabric’s Data Warehouse preview allows previewing Parquet files with `OPENROWSET`, direct ingestion into Delta tables requires Spark-based processing to convert Parquet to Delta format first
• `OPENROWSET` can preview Parquet/CSV files stored in Azure Blob or ADLS Gen2. It can also load filtered/transformed Parquet data into Fabric Warehouse tables via `INSERT INTO`.
• `BULK INSERT` (preview) supports loading CSV data into Warehouse tables but lacks direct Parquet ingestion supportFabric’s SQL analytics endpoint auto-discovers Delta Lake tables (stored as Parquet files with transaction logs) but cannot directly ingest raw Parquet files via T-SQL
Use Spark notebooks to convert Parquet to Delta format (e.g., `spark.read.parquet(...).write.format("delta").saveAsTable(...)`).
• Only Delta tables in the Lakehouse’s `/tables` folder are auto-discovered by the SQL endpointIf this is helpful please accept the answer
3 Replies
- nilendraFabric
Super User
Hello Jaganath
the SQL endpoint cannot be used to load Parquet files into Delta tables via `OPENROWSET` or `BULK INSERT`. While Fabric’s Data Warehouse preview allows previewing Parquet files with `OPENROWSET`, direct ingestion into Delta tables requires Spark-based processing to convert Parquet to Delta format first
• `OPENROWSET` can preview Parquet/CSV files stored in Azure Blob or ADLS Gen2. It can also load filtered/transformed Parquet data into Fabric Warehouse tables via `INSERT INTO`.
• `BULK INSERT` (preview) supports loading CSV data into Warehouse tables but lacks direct Parquet ingestion supportFabric’s SQL analytics endpoint auto-discovers Delta Lake tables (stored as Parquet files with transaction logs) but cannot directly ingest raw Parquet files via T-SQL
Use Spark notebooks to convert Parquet to Delta format (e.g., `spark.read.parquet(...).write.format("delta").saveAsTable(...)`).
• Only Delta tables in the Lakehouse’s `/tables` folder are auto-discovered by the SQL endpointIf this is helpful please accept the answer
Hi Jaganath ,
Thank you nilendraFabric for the helpful response.
I wanted to check in on your situation regarding the issue. Have you resolved it? If you have, please consider marking the reply that helped you or sharing your solution. It would be greatly appreciated by others in the community who may have the same question.
Thank you for your cooperation. Have a great day.Thanks nilendraFabric for your assistance and guidance.
