Forum Discussion
Need help to speed up folder source
Use Folder Path Parameter: Set up a parameter to dynamically select the folder path for the year 2024, reducing unnecessary data loading.
Combine Queries: Merge multiple query steps to minimize operations during data loading, like filtering .csv files and folders for 2024 together.
Reduce Data Cleaning: Only apply essential data cleaning steps to minimize processing time. Focus on transformations necessary for your analysis.
Query Folding: Utilize query folding to push transformation operations to the data source, leveraging its processing capabilities for faster data retrieval.
Incremental Loading: Implement incremental loading to only load new or updated data since the last refresh, reducing the amount of data loaded each time.
Data Compression: Optimize data compression settings to balance file size and performance, experimenting with different options.
Partitioning: If applicable, partition your data based on criteria like date to improve query performance by accessing relevant partitions only.
Data Model Simplification: Review your data model to remove unnecessary relationships or columns, simplifying it for better query performance and reduced memory usage.
Did I answer your question? Mark my post as a solution! Appreciate your Kudos !!
Use Folder Path Parameter: Set up a parameter to dynamically select the folder path for the year 2024, reducing unnecessary data loading.
Combine Queries: Merge multiple query steps to minimize operations during data loading, like filtering .csv files and folders for 2024 together.
Reduce Data Cleaning: Only apply essential data cleaning steps to minimize processing time. Focus on transformations necessary for your analysis.
Query Folding: Utilize query folding to push transformation operations to the data source, leveraging its processing capabilities for faster data retrieval.
Incremental Loading: Implement incremental loading to only load new or updated data since the last refresh, reducing the amount of data loaded each time.
Data Compression: Optimize data compression settings to balance file size and performance, experimenting with different options.
Partitioning: If applicable, partition your data based on criteria like date to improve query performance by accessing relevant partitions only.
Data Model Simplification: Review your data model to remove unnecessary relationships or columns, simplifying it for better query performance and reduced memory usage.
Did I answer your question? Mark my post as a solution! Appreciate your Kudos !!