Forum Discussion
Data Inconsistency excel files different structure - build a model
HI Anonymous ,
For large amount of randomly structure files, I'd like to suggest you do these transform/conversation by external tools instead of M query code. (notice: these M query codes of transform are complex, low performance and hard to maintain, you need to manually fix these steps codes every time major data structure changes)
In fact, current I haven't found any simple solutions to automatically handle fully randomly data structure formatting, maybe you can try to machine learning?
Guide to File Formats for Machine Learning: Columnar, Training, Inferencing, and the Feature Store
Regards,
Xiaoxin Sheng
Hi Anonymous
thanks for your message
I'll try to use the fuzzy Match,
for you solution, I can do manually for one or two excel files but do you think would be good if I have hundreds of excel files? I don't know excel how many but I know this app where I'm getting the excel files provides all in different patterns, for example It doesn't provide exctely the same strucure of spaces or columns names and etc,
so I need somehow to automate this ETL,
Fabricio,
HI Anonymous ,
For large amount of randomly structure files, I'd like to suggest you do these transform/conversation by external tools instead of M query code. (notice: these M query codes of transform are complex, low performance and hard to maintain, you need to manually fix these steps codes every time major data structure changes)
In fact, current I haven't found any simple solutions to automatically handle fully randomly data structure formatting, maybe you can try to machine learning?
Guide to File Formats for Machine Learning: Columnar, Training, Inferencing, and the Feature Store
Regards,
Xiaoxin Sheng