Forum Discussion
Small datasets, many queries/steps or large data sets, fewer queries/steps?
- 4 years ago
In my experience working with one Excel file with multiple tables of same structure, I generally get best performance by immediately expanding (and automatically combining) tables off the initial Excel.Workbook access function. Pre-reqs/steps:
- Make sure inside your Excel workbook that your data tables are formatted as tables. This let's you skip any need to do any transforms before combining tables (namely, Table.PromoteHeaders). They show up as Kind = Table in the access output
- From the initial access step (e.g. Excel.Workbook( Web.Contents( [workbook url] ) ) ), filter the table of content to just your tables (not sheets) of same-structure data.
- Now, you can simply expand the Data column to get your combined table. You'll have the added benefit of keeping a "Source Table" column with the Name or Item column (others can be removed before expanding)
I've found that this minimizes repeated queries to the data source, and makes maintenance easier (e.g. schema drift like when column name changes; can just fix in the one expand step rather than multiple places)
- Make sure inside your Excel workbook that your data tables are formatted as tables. This let's you skip any need to do any transforms before combining tables (namely, Table.PromoteHeaders). They show up as Kind = Table in the access output
In my experience working with one Excel file with multiple tables of same structure, I generally get best performance by immediately expanding (and automatically combining) tables off the initial Excel.Workbook access function. Pre-reqs/steps:
- Make sure inside your Excel workbook that your data tables are formatted as tables. This let's you skip any need to do any transforms before combining tables (namely, Table.PromoteHeaders). They show up as Kind = Table in the access output
- From the initial access step (e.g. Excel.Workbook( Web.Contents( [workbook url] ) ) ), filter the table of content to just your tables (not sheets) of same-structure data.
- Now, you can simply expand the Data column to get your combined table. You'll have the added benefit of keeping a "Source Table" column with the Name or Item column (others can be removed before expanding)
I've found that this minimizes repeated queries to the data source, and makes maintenance easier (e.g. schema drift like when column name changes; can just fix in the one expand step rather than multiple places)
MarkLaf I don't think I could get over to you how much this post has transformed my work! Thank you!!