Forum Discussion
Solved
Pandas or Spark
Hi all, I am beginning the research and setup of Fabric in a Trial Environment and had some questions about Spark vs. Pandas. I understand that Spark is more tuned for "Big Data", due to it's...
spencer_sa
Impactful Individual
Impactful IndividualWe've got a 1.5M row, 145 column (poorly formed*) CSV that we are ingesting monthly using Pandas to then fix and write to a Delta table.
Pandas quite happily handles 1.5M rows on an F8. (We currently run at F32 becuase our semantic model is >5GB)
* CRLF in fields, 2 columns with same name, columns with bad characters in the header.