Forum Discussion

SamyAbdul's avatar
SamyAbdul
Frequent Visitor
1 day ago

The Source has too many files

Hi experts, we ingesting the data through API , for some data sources the business has shared nearly 150 files for one source, the schema is different sparing one or two similar columns , by going th...
data engineering
data pipeline
lakehouse
v-aatheeque's avatar
v-aatheeque
Icon for Community Support rankCommunity Support
1 day ago

Hi SamyAbdul​ 

Thanks for sharing the issue.

The error “The source has too many files” may be related to the number of files being processed by the source operation. To narrow this down, could you please confirm which Fabric item or activity is failing (for example, a Data Pipeline Copy activity, Dataflow Gen2, or a notebook) and share the complete error message?

If you are using a Data Pipeline Copy activity to load many small files into a Fabric Warehouse, one approach worth testing is to merge the small files into larger files before loading them. Fabric Community discussions have recommended the Copy activity's Merge files option for scenarios where numerous small files affect Warehouse loading performance.

<Link url="https://community.fabric.microsoft.com/discussions/df_pipelines/speed-of-writing-to-fabric-data-warehouse-from-fabric-pipeline-vs-adf-pipeline/3871823/replies/3899516

The appropriate solution depends on the failing activity and source type. Please share the activity name, source type, approximate number of files, and complete error details so we can identify the most suitable next step.

Thanks!