Forum Discussion
Dataflow Optimization: Breaking down a 100-col flat file into different queries?
Our current data structure is coming from 100+ col flat (Excel) files stores on Sharepoint which we run through a dataflow. There's basically 3 dataflows: A) data from the legacy system (XLSX), B) data from the current system (XLSX), and then C) combining both data sources into one "universal" dataflow. We made it so because the legacy system's data is no longer updated, and the only one that sees updates is dataflow B/C if operations asks to add new columns to dataflow B.
Currently, our queries in dataflow B are structured as follows:
1) Initial Staging of the flat file which is just pulling the data from Sharepoint, renaming columns and applying initial data types ; then,
2) Adding calculated columns, data corrections, finalizing data types, cleanup
I am thinking of breaking up the 2nd query into separate queries that processes the string columns and the date columns for ease of maintenance, still using the same initial query as their source and then combining the string and date datasets into one. My question is: on the optimization POV, will this tank our resources? Aside from what I'm thinking, is there a better way to optimize this dataflow?
Our enterprise system does not support a "entity" system, wherein each form is an entity of itself that cannot relate to anything because everything is entered manually (ie: no Client/Client ID, Staff/StaffID, etc.).
2 Replies
- olimiloPost Prodigy
danextian wrote:Hi olimilo
I would break B further into what's stale and what will still be updated. With this B.1 will be refreshed only once or when the are changes to its files. B.2 will be refreshed regularly. Another question is do you really all those 100+ columns?
If the answer is there's no way to tell which ones get updated and when, what would be the proper course of action?
Also yes, we do need the 100+ columns. I understand it's not the best setup but our BU unfortunately does not want to invest in a dedicated database solution for it. As I'm writing this, I'm getting riddled by a bunch of 0x0000DEAD from this specific dataflow so it really needs to be optimized somehow.