Forum Discussion

olimilo's avatar
olimilo
Icon for Post Prodigy rankPost Prodigy
1 year ago

Dataflow Optimization: Breaking down a 100-col flat file into different queries?

Our current data structure is coming from 100+ col flat (Excel) files stores on Sharepoint which we run through a dataflow. There's basically 3 dataflows: A) data from the legacy system (XLSX), B) ...
Need Help
danextian's avatar
danextian
Icon for Super User rankSuper User
1 year ago

Hi olimilo 

 

I would break B further into what's stale and what will still be updated. With this B.1 will be refreshed only once or when the are changes to its files. B.2 will be refreshed regularly. Another question is do you really all those 100+ columns?

 

olimilo's avatar
olimilo
Icon for Post Prodigy rankPost Prodigy
1 year ago
danextian wrote:

Hi olimilo 

 

I would break B further into what's stale and what will still be updated. With this B.1 will be refreshed only once or when the are changes to its files. B.2 will be refreshed regularly. Another question is do you really all those 100+ columns?

 

If the answer is there's no way to tell which ones get updated and when, what would be the proper course of action?

 

Also yes, we do need the 100+ columns. I understand it's not the best setup but our BU unfortunately does not want to invest in a dedicated database solution for it. As I'm writing this, I'm getting riddled by a bunch of 0x0000DEAD from this specific dataflow so it really needs to be optimized somehow.