Forum Discussion

madzgalj_darko1's avatar
madzgalj_darko1
Regular Visitor
3 years ago

Help, about setting up data.

Hello everyone, I'm pretty new at PBI, so basically what I need help for is setting up my data so i can do scheduled refresh without problem. I have a lot of rows 4-5 mil. I'm pulling some of them ...
Need Help
Show and Tell
madzgalj_darko's avatar
madzgalj_darko
Frequent Visitor
3 years ago

Yes, that is exactly what i'm doing. 
I have 5-6 csv files in same sharepoint, one as sample file and the rest of them load based on sample file. 
So now, if I understood correctly i have to set up parameters for date filters. Which is going to be what? If you can help me?
My data starts from 01-Jan-2023, and in csv files are transactions until 01-Jun-2023. From 1st of June it goes true postgre sql import. 
So it's 2 querys one from sharepoint and one from postgre. And I appended that in one query. 
Basically I need last 10-15 days to be refreshed when i schedule refresh in the night and the rest remains the same.
I need to set up incremental only on Appened table is that correct? 
Thank you 

  • lbendlin's avatar
    lbendlin
    Icon for Super User rankSuper User
    3 years ago
    one as sample file and the rest of them load based on sample file.

    Don't do that, the generic process is inefficient. Create your own process via Table.AddColumn and with a fixed list of column Headers.

     

    Partitions in Power BI are controlled by datetime fields RangeStart and RangeEnd.  You need to find a data point that is usable for this, for example the file names. You can't use the File Modified date as that can change. 

     

    So let's say your file is named "Data 01-Jan-2023.csv"  (which is a horrible name, by the way - much better to use ISO8601 format).  In your Power Query you need to have steps that convert that to 2023-01-01T00:00:00.00 for RangeStart and 2023-02-01T00:00:00.00 for RangeEnd  (for example).