Forum Discussion
Pandas or Spark
PySpark is more native as format in Fabric than Pandas. Especially for read and write to Tables in a Lakehouse. There are possibilities to directly write a table a Pandas DataFrame to a delta table (https://delta.io/blog/2023-04-01-create-append-delta-lake-table-pandas/), but that still feels more like a workaround then the native way.
However, I you read data from other sources and/or write data to other destinations than the Tables section of the Lakehouse, Pandas will have scenario's where it will be preferred over PySpark. Your tables are indeed not very large and Pandas will perform well in these cases. Looking at your description, an F8 will be performing well with both Pandas and PySpark dataframes.
Yeah, the main reason we fell back onto Pandas for this particular CSV is because the format sucked and wasn't playing nicely with the native pySpark ingestion routines.
All of our other CSV sources don't suffer from these problems.
There are also reasons to use Pandas for file export too - it's the easiest way of getting a single file output, rather than 1 file per partition with system generated names.