Forum Discussion
Difference between PySpark .saveAsTable() and .save()
Hello mmcanelly That's an interesting observation! I have not experienced that myself, but it is probably because of the order of things in which data is written to using those methods. By the way, both methods end up producing a managed Delta table, however internally it interprets as "path-write" (.save) and "table-write" (.saveAsTable).
When you're using table-write Fabric registers with catalog and sql_endpoint so that the table is immediately visible, whereas using path-write it simply writes to the /Tables area and let Fabric auto-register to the catalog, that will eventually surface the table on the sql_endpoint. That explains your experience with latency using the saveAsTable method.
If your use case doesn't demand that the table is immediately available on sql_endpoint, you could follow the appraoch of using save() rather than saveAsTable().
I would like to point out that I could not find any documentation on Fabric that explains the internal working. In documentation it always uses .save to write to /Files and .saveAsTable to write to /Tables.
Hello mmcanelly That's an interesting observation! I have not experienced that myself, but it is probably because of the order of things in which data is written to using those methods. By the way, both methods end up producing a managed Delta table, however internally it interprets as "path-write" (.save) and "table-write" (.saveAsTable).
When you're using table-write Fabric registers with catalog and sql_endpoint so that the table is immediately visible, whereas using path-write it simply writes to the /Tables area and let Fabric auto-register to the catalog, that will eventually surface the table on the sql_endpoint. That explains your experience with latency using the saveAsTable method.
If your use case doesn't demand that the table is immediately available on sql_endpoint, you could follow the appraoch of using save() rather than saveAsTable().
I would like to point out that I could not find any documentation on Fabric that explains the internal working. In documentation it always uses .save to write to /Files and .saveAsTable to write to /Tables.
Thank you deborshi_nag! So it sounds like as long as I refresh the SQL endpoint metadata after making my changes (per something like this: Programmatically Refresh & Sync SQL Analytics Endp... - Microsoft Fabric Community) so that the SQL endpoint sees the latest data right away, then .save() should be OK for what I'm doing. Now I'll have to see if it's still faster once I include that extra step. I feel like it should be because I'm only doing it once at the end as opposed to updating the catalog with every table write.
Thanks for your insight!
