Forum Discussion

Noxy's avatar
Noxy
Frequent Visitor
3 years ago

Unstructured data taking values from one record and combining those values with other records.

Hi all, Given records like these below AAA,time,00:00:11 AAA,date,07-JUL-2023 CPU_ALL,T0001,56.1,23.5,0.4,20.1,79.6,32 CPU_ALL,T0002,57.1,22.2,0.3,20.4,79.3,32 AAA,time,00:10:11 AAA,date,07-JU...
Need Help
Noxy's avatar
Noxy
Frequent Visitor
2 years ago

Hi whitch and thank you very much for sending me in the right direction.

I do have a further issue now in that given I am reading in any number of unstructured data files from a folder, my query steps even with 2 or 3 files of 10MB seem to take for ever and I see the bytes read going into the hundreds of MB.  I guess the data is being read multiple times as the query goes through the steps. Is there any way to make this more efficient? I am farily new to PowerBI. I was hoping that as I filter the data for the records I want to concentrate on that the query would get faster as it went through but that doesn't appear to happen.

whitch's avatar
whitch
Resolver I
2 years ago

Do you have the option to put steps 1 & 2 into a Dataflow?  This would force that stage to be evaluated only once.

  • Noxy's avatar
    Noxy
    Frequent Visitor
    2 years ago

    I'm not sure I have access to create a connection to a local folder from PowerBI online. The next button is greyed out. If I create a connection using a blank query and cut paste in from my original PowerBI desktop query, it comes up saying credentials are required to connect to the folder source. When I click to modify those, the connect button is greyed out. 

     

    Going back to my original query, I was hoping that as I filter the data that the query would run faster as it completes through the query steps. However what I seem to see is multiple reads back to the original input text files. I have 3 10MB files but the read count bottom right is reaching hundreds of MB. I am new to PowerBI (probably mentioned that before). It would be good to read the data and filter it once. Does that form the data in the data model? Not sure of the correct terminology. Then the rest of the steps work on that cached data? 

    • SRHayward's avatar
      SRHayward
      New Member
      2 years ago

      Hi Noxy,

       

      Did you manage to get this to work?  We have seen this before and developed a solution to import nmon data into PowerBI.  Please let me know if you would like to know more.

       

      Stuart

      • Noxy's avatar
        Noxy
        Frequent Visitor
        2 years ago

        Hi Stuart, thanks for replying. Yes I'm all good now thank you. As I have progressed, I've actually ended up going back to the raw data and transforming / extracting what I need at that end which vastly speeds up the process. It was good to have a go at this to continue learning about PowerBI. Sadly I am addicted at the moment 🙂 Regards, Noxy