Forum Discussion
How do you remove top N rows from a CSV when loading it into a notebook?
Greetings, community. I have a scenario where I need to skip the first few rows of a CSV file and then save that back into a lakehouse. I need the lakehouse to be dynamic since I'll be deploying the ...
Some methods are mentioned in this thread: https://community.databricks.com/t5/data-engineering/skip-number-of-rows-when-reading-csv-files/td-p/28059
Some thoughts / suggestions to try:
Does the order of the options matter in PySpark? I don't know.
Does it make a difference if you rearrange the expression like this?
df = spark.read.format("csv").option("header","true").option("skipRows",25).load(ABFSPath)
or remove the header option like this
df = spark.read.format("csv").option("skipRows",25).load(ABFSPath)