Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
6 years ago
Solved

CSV Web connection doesn't update my report properly

Hi Folks!

 

Yesterday I did a report and I connected it to this website

 

https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_19-covid-Confirmed.csv

 

As you can see the last day with information si 02/29, but yesterday it was 02/28. It's being updated day after day.

 

The thing is I woke up this morning and refreshed my report expecting to see my data updated as it's shown in the website, till 02/29, but that's not what it happened. In my report, the last date that I can see is 02/28.

 

When I go to Advance Editor, I can read this:

 

Origen = Csv.Document(Web.Contents("https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_19-covid-Confirmed.csv"),[Delimiter=",", Columns=42, Encoding=65001, QuoteStyle=QuoteStyle.None]),

 

If you try to connect to the source the report today, you will see "Columns=43", because it's updated till 02/29. If you do it tomorrow, you will see "Columns=44" because it'll be updated in the website till 03/01, and so on.

 

How can I update it in my report automatically, just refreshing it? Why it doesn't update the date?

 

Thank you, I really appreciate your help

5 Replies

  • Greg_Deckler's avatar
    Greg_Deckler
    Community Champion

    Try removing your Columns=42 completely. I will see if I can test this.

  • Greg_Deckler's avatar
    Greg_Deckler
    Community Champion

    Try replacing your Origin line with:

     

    Origin = Csv.Document(Web.Contents("https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_19-covid-Confirmed.csv")),

     

    This gets rid of the definition for how many columns there are. That being said, you will have to be careful with the rest of your query. A dynamic set of columns is not the most pleasant thing to deal with. It can play havoc with the rest of your query. I would promote the first row and then immediately use unpivot other columns to unpivot the day columns. Also get rid of any automatic Change Type step. Use something like the following for the first part of your query:

     

    let
        Source = Csv.Document(Web.Contents("https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_19-covid-Confirmed.csv")),
        #"Promoted Headers" = Table.PromoteHeaders(Source, [PromoteAllScalars=true]),
        #"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"Promoted Headers", {"Province/State", "Country/Region", "Lat", "Long"}, "Attribute", "Value"),
        #"Renamed Columns" = Table.RenameColumns(#"Unpivoted Other Columns",{{"Attribute", "Day"}}),
        #"Changed Type" = Table.TransformColumnTypes(#"Renamed Columns",{{"Value", Int64.Type}, {"Day", type date}})
    in
        #"Changed Type"
    • Anonymous's avatar
      Anonymous
      Not applicable

      Super! Thank you Greg_Deckler !


      Deleting the part of the definition has been enough.

       

      Thank you for your other advice. In any case, the next step in my query was unpivot the other columns. 🙂

    • ghada_shawish's avatar
      ghada_shawish
      New Member

      I had the same issue and yes its solved and the latest column is showing now but may I ask you if you want to display the latest date column in a graph .. would that be possible? i mean if you want to keep using the latest column (for the last date inserted) added to the data for the graph(after an update) would that be possible?

       

      Greg_Deckler Anonymous