Forum Discussion
How to Validate Blank Lines in CSV Files Using Microsoft Fabric Data Pipelines
Hi Mamatha77 ,
This is a really good quality-check requirement, but with the constraints you listed it’s important to be clear about what Fabric can and can’t do today.
Fabric Data Pipelines by themselves (Copy, Lookup, Get Metadata, etc.) can see files and basic properties (name, size, modified time), but they cannot read file content line-by-line. That means they can’t look inside a CSV and say “this line is blank”, “this is the header”, or “this row is empty after the header”.
Those kinds of checks need something that can actually parse the text — in Fabric that’s:
a Notebook (Spark / Python), or
Dataflow Gen2 (Power Query),
and you’ve already ruled out Dataflow Gen2.
So, staying fully inside Fabric and keeping the files in place, the realistic pattern is:
use a Data Pipeline to orchestrate,
call a Notebook activity that does the actual validation,
have the notebook write a small log table (e.g. csv_validation_log) in the Lakehouse with columns like file_path, is_valid, reason.
The notebook would just read each CSV as text, find the first non-blank line as the header, then check for blank lines before or after it and record the result for each file. The CSVs themselves stay where they are; only the log is written.
If you are not allowed to use notebooks either, and the rule is “pipelines only, no Dataflow Gen2, no ADF, no external services”, then the honest answer is:
With the current features, Fabric Data Pipelines alone cannot implement this exact validation, because there is no built-in activity that can inspect CSV content at the line level.
In that case, the design would need to be relaxed slightly (for example, allow a small notebook just for validation) to be achievable purely in Fabric.
So just to summarise in plain terms:
Yes, you can do this inside Fabric with a pipeline + notebook and a validation log.
No, you cannot do it with pipeline activities only today, because they can’t look inside the file content.
– Gopi Krishna
Hi Mamatha77 ,
This is a really good quality-check requirement, but with the constraints you listed it’s important to be clear about what Fabric can and can’t do today.
Fabric Data Pipelines by themselves (Copy, Lookup, Get Metadata, etc.) can see files and basic properties (name, size, modified time), but they cannot read file content line-by-line. That means they can’t look inside a CSV and say “this line is blank”, “this is the header”, or “this row is empty after the header”.
Those kinds of checks need something that can actually parse the text — in Fabric that’s:
a Notebook (Spark / Python), or
Dataflow Gen2 (Power Query),
and you’ve already ruled out Dataflow Gen2.
So, staying fully inside Fabric and keeping the files in place, the realistic pattern is:
use a Data Pipeline to orchestrate,
call a Notebook activity that does the actual validation,
have the notebook write a small log table (e.g. csv_validation_log) in the Lakehouse with columns like file_path, is_valid, reason.
The notebook would just read each CSV as text, find the first non-blank line as the header, then check for blank lines before or after it and record the result for each file. The CSVs themselves stay where they are; only the log is written.
If you are not allowed to use notebooks either, and the rule is “pipelines only, no Dataflow Gen2, no ADF, no external services”, then the honest answer is:
With the current features, Fabric Data Pipelines alone cannot implement this exact validation, because there is no built-in activity that can inspect CSV content at the line level.
In that case, the design would need to be relaxed slightly (for example, allow a small notebook just for validation) to be achievable purely in Fabric.
So just to summarise in plain terms:
Yes, you can do this inside Fabric with a pipeline + notebook and a validation log.
No, you cannot do it with pipeline activities only today, because they can’t look inside the file content.
– Gopi Krishna