Forum Discussion
Using Delta Sharing IN Fabric Notebook
Hi ericjo ,
let me clarify both points.
1. 10-minute timeout :The 10-minute limit does not apply between tables. It applies to the validation and publishing process of the dataflow: each query must complete that process within 10 minutes. (Microsoft Learn) If it exceeds that, the dataflow fails at publish time, not during the data refresh itself. If you have three tables and one of them is slow or complex, that specific query can cause the publish to fail. The documented workaround is to simplify the queries or split them across separate dataflows.
2. Schema sync : Yes, errors can occur. If the provider adds, removes, or renames columns and you do not update the dataflow manually, the load will either fail or produce incorrect data. In a Notebook with Spark you can mitigate this with mergeSchema, which automatically reconciles schema differences:
df.write.format("delta").option("mergeSchema", "true").mode("append").save(path)
If my commenta helped solve your question, it would be great if you could like all comment and mark it as the accepted solution. It helps others with the same issue and also motivates me to keep contributing.
Thanks a lot, I really appreciate it.
Hi ericjo ,
You can try this:
- Step 1— Upload the `.share` file to your Lakehouse under Files/
- Step 2 — First cell in the notebook:
```python
%pip install delta-sharing
```
- Step 3— Second cell:
```python
import delta_sharing
profile_path = "/lakehouse/default/Files/config.share"
client = delta_sharing.SharingClient(profile_path)
print(client.list_all_tables())
```
If the tables appear in the output, the connection is working. Share the exact error if something fails in any of these steps.
I think that perhaps, you can create environment for this.
Dataflow Gen2 — My personal opinion: it works, but for this case I would go with a Notebook. It is more scalable, you have more control over the ingestion logic, it integrates better into pipelines, and it is easier to maintain as the project grows.
Dataflow Gen2 has some limitations worth knowing:
- 10-minute timeout per query
- Schema is not automatically synced if the provider changes columns
- Issues with OAuth2 tokens on long-running refreshes
If my comment helped solve your question, it would be great if you could like the comment and mark it as the accepted solution. It helps others with the same issue and also motivates me to keep contributing.
Thanks a lot, I really appreciate it.
Thank you for your reply, and I have a few questions.
1. Does "10-minute timeout per query" mean that if there are three queries to retrieve tables within a Data Flow, the second table will be retrieved 10 minutes after the first table is retrieved?
2. The schema is not automatically synchronized even if the provider changes columns. -> If automatic synchronization does not occur, could errors occur during data migration?
Hi ericjo ,
let me clarify both points.
1. 10-minute timeout :The 10-minute limit does not apply between tables. It applies to the validation and publishing process of the dataflow: each query must complete that process within 10 minutes. (Microsoft Learn) If it exceeds that, the dataflow fails at publish time, not during the data refresh itself. If you have three tables and one of them is slow or complex, that specific query can cause the publish to fail. The documented workaround is to simplify the queries or split them across separate dataflows.
2. Schema sync : Yes, errors can occur. If the provider adds, removes, or renames columns and you do not update the dataflow manually, the load will either fail or produce incorrect data. In a Notebook with Spark you can mitigate this with mergeSchema, which automatically reconciles schema differences:
df.write.format("delta").option("mergeSchema", "true").mode("append").save(path)
If my commenta helped solve your question, it would be great if you could like all comment and mark it as the accepted solution. It helps others with the same issue and also motivates me to keep contributing.
Thanks a lot, I really appreciate it.