Forum Discussion

AussieCoops's avatar
AussieCoops
Regular Visitor
6 years ago

Datacleanse before or after appending tables or within same table?

I have five databases for my companies.  Each deals with a separate country/company.

Currently I Import the db from sqlserver one instance of SYSPRO with 5 db.

I am trying to speed things up a little.

So what is best?

Import each and do cleansing in each ===> then Append    OR

Import each without any cleanse and do all the cleaning up in the Appended table?

 

Currently I can View Native Query in the base imported tables right to the end of the query.

 

A third alternative is to write a longer single query bringing each db in, then Appending those within the same table using the technique mentioned by v-sihou-msft  here:  https://community.powerbi.com/t5/user/viewprofilepage/user-id/6969 .   I have trialled this on a much smaller table and it seems fine.  Great tip btw.

Before I embark on a major writing exercise, any tips on speed issues:

1.  Import each table, cleanse then Append all in another table;

2.  Import basic tables, Append all in another table, then perform cleansing in the new table;

3.  Import using v-sihou-msft method.

 

 

1 Reply

  • az38's avatar
    az38
    Community Champion

    Hi AussieCoops 

    i have no investigated with digit but i think that 1 cleansing is better then 5, it should work a little bit faster as there will be less initializing procedures

    and the second, and most important point - if you will discover some mistake or incompletness in your data after cleansing you will have to make change to the only datasource in the second option and not to the 5 different data sources with first option