Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
6 years ago

Using Parameter to define if a table is loaded or not

Hi,   I have a large data set that I have chunked up into separate Dataflows  of Current, Archive #1, Archive #2 and when developing and testing I only want to load the Current Dataflow   The cod...
Need Help
lbendlin's avatar
lbendlin
Icon for Super User rankSuper User
6 years ago

You're not doing anything wrong per se, that's more of a question if that is a bug in the lazy execution or not.  

 

Try to disable the parallel loading of sources - but it shouldn't really make a difference.

 

What you actually want to do is to control that at the source. For example if Current, Archive1, Archive2 etc are CSV files on a file share then you can easily control the Table.Combine by moving the archive files in and out of the source folder as needed.

Anonymous's avatar
Anonymous
Not applicable
6 years ago

Hi, I tried disabling the parellel loading but no improvement.

 

The data sets are hundreds of csv files so I can't move the files for testing.

 

So it a bug in the evaluation process then?

 

Cheers

 

Andrew 

  • lbendlin's avatar
    lbendlin
    Icon for Super User rankSuper User
    6 years ago

    "The data sets are hundreds of csv files so I can't move the files for testing."

     

    Why not?

    • Anonymous's avatar
      Anonymous
      Not applicable
      6 years ago

      Two reasons really,

       

      1. The size they about a GB each and its never a good idea to mess with the source data
      2. We have other models and users that mine this data so in effect taking it off line so we can test and dev models would be complicated and disruptive. 
      • lbendlin's avatar
        lbendlin
        Icon for Super User rankSuper User
        6 years ago

        1. file size is irrelevant if the move is within the same folder structure. All you do is change pointers.

        2. Would it be an option for you to have your own copy of the source data?

  • Anonymous's avatar
    Anonymous
    Not applicable
    6 years ago

    Pinging this again.

     

    Does anyone know how to use Parameter to define if a Dataflow is loaded into the model or not.

     

    Seems simple - But I have tried it a number of ways and it still ingests all the data when you refresh the Desktop model instead of ignoring the ones that the logic would say not to look at.

     

    Any ideas?