Forum Discussion

sebbyp's avatar
sebbyp
Helper III
8 years ago
Solved

Get data using API link to OED data

Can somebody help with API access.  I would like to access the economic data tables provided by the OECD.  Here is the link

 

http://www.oecd-ilibrary.org/economics/data/main-economic-indicators/composite-leading-indicators_data-00042-en?isPartOf=/content/datacollection/mei-data-en

 

How do you tell Power BI to get this infomation?

  • This function should work for all OECD-datasets via the JSON-API in Flat Format:

     

    (URL) =>
    let
    
        fnConvertRecordList = (Recordlist) => Table.ExpandRecordColumn(Table.FromList(Recordlist, Splitter.SplitByNothing(), null, null, ExtraValues.Error), "Column1", Record.FieldNames(Record.Combine(Table.FromList(Recordlist, Splitter.SplitByNothing(), null, null, ExtraValues.Error)[Column1]))),
        fnAddKeyToList = (list) => Table.AddIndexColumn(Table.FromColumns({list}), "Key",0,1),
        Source = Json.Document(Web.Contents(URL)),
        ToTable = Table.FromRecords({Source}),
        Expand = Table.ExpandRecordColumn(ToTable, "structure", {"links", "name", "description", "dimensions", "attributes", "annotations"}, {"links", "name", "description", "dimensions", "attributes", "annotations"}),
        Attr = fnConvertRecordList(Expand[attributes]{0}[observation]),
        Dim = fnConvertRecordList(Expand[dimensions]{0}[observation]),
        Dims = Table.Combine({Attr,Dim}),
        #"Added Index1" = Table.AddIndexColumn(Dims, "Index", 1, 1),
        AddKeyToValues = Table.AddColumn(#"Added Index1", "Custom", each fnAddKeyToList([values])),
        ExpandValues = Table.ExpandTableColumn(AddKeyToValues, "Custom", {"Column1", "Key"}, {"Column1", "Key"}),
        FilterNotNull = Table.SelectRows(ExpandValues, each ([Key] <> null)),
        LookupTbl = Table.ExpandRecordColumn(FilterNotNull, "Column1", {"id", "name"}, {"id.1", "name.1"}),
        Datasets = Expand[dataSets]{0}{0}[observations],
        ConvertToTable = Record.ToTable(Datasets),
        AddKeys = Table.AddColumn(ConvertToTable, "Custom", each fnAddKeyToList(List.Combine({[Value],Text.Split([Name], ":")}))),
        Cleanup = Table.RemoveColumns(AddKeys,{"Value"}),
        ExpandValues2 = Table.ExpandTableColumn(Cleanup, "Custom", {"Column1", "Key"}, {"Value", "Key"}),
        ChgType = Table.TransformColumnTypes(ExpandValues2,{{"Value", type number}}),
        Amount = Table.AddColumn(ChgType, "Amount", each if [Key]=0 then [Value] else null),
        FillDownAmount = Table.FillDown(Amount,{"Amount"}),
        #"Filtered Rows" = Table.SelectRows(FillDownAmount, each ([Value] <> null) and ([Key] <> 0)),
        MergeLookup = Table.NestedJoin(#"Filtered Rows",{"Key", "Value"},LookupTbl,{"Index", "Key"},"Expanded Custom",JoinKind.LeftOuter),
        ExpandLookup = Table.ExpandTableColumn(MergeLookup, "Expanded Custom", {"id", "id.1"}, {"id", "id.1"}),
        Cleanup3 = Table.RemoveColumns(ExpandLookup,{"Key", "Value"}),
        Pivot = Table.Pivot(Cleanup3, List.Distinct(Cleanup3[id]), "id", "id.1"),
        MergeLookup2 = Table.NestedJoin(Pivot,{"Name"},MergeLookup,{"Name"},"Pivoted Column",JoinKind.LeftOuter),
        Expand2 = Table.ExpandTableColumn(MergeLookup2, "Pivoted Column", {"Expanded Custom"}, {"Expanded Custom"}),
        Expand3 = Table.ExpandTableColumn(Expand2, "Expanded Custom", {"name", "name.1"}, {"name.2", "name.1"}),
        AddSpace = Table.TransformColumns(Expand3,{{"name.2", each _&" "}}),
        Pivot2 = Table.Pivot(AddSpace, List.Distinct(AddSpace[name.2]), "name.2", "name.1"),
        Cleanup4 = Table.RemoveColumns(Pivot2,{"Name"}),
        ChgType2 = Table.TransformColumnTypes(Cleanup4,{{"Amount", type number}})
    in
        ChgType2

    Please give a shout if it doesn't!

     

18 Replies

  • ImkeF's avatar
    ImkeF
    Community Champion

    Thy have an open API, so you don't need an API key:

     

    let
        Source = Json.Document(Web.Contents("http://stats.oecd.org/SDMX-JSON/data/MEI_CLI/LOLITOAA+LOLITONO+LOLITOTR_STSA+LOLITOTR_GYSA+BSCICP03+CSCICP03+LORSGPRT+LORSGPNO+LORSGPTD+LORSGPOR_IXOBSA.AUS+AUT+BEL+CAN+CHL+CZE+DNK+EST+FIN+FRA+DEU+GRC+HUN+IRL+ISR+ITA+JPN+KOR+LUX+MEX+NLD+NZL+NOR+POL+PRT+SVK+SVN+ESP+SWE+CHE+TUR+GBR+USA+EA19+G4E+G-7+NAFTA+OECDE+OECD+ONM+A5M+BRA+CHN+IND+IDN+RUS+ZAF.M/all?startTime=2015-08&endTime=2017-07&dimensionAtObservation=allDimensions"))
    in
        Source

     

    Taken from here:

    • sebbyp's avatar
      sebbyp
      Helper III

      ImkeFthanks for your help.  The issue i am having is extracting the tables from the API link.  I cannot get the tables in a user friendly format.  Is there a known set of steps to extract the data tables which i can then work with?

      • ImkeF's avatar
        ImkeF
        Community Champion

        This function should work for all OECD-datasets via the JSON-API in Flat Format:

         

        (URL) =>
        let
        
            fnConvertRecordList = (Recordlist) => Table.ExpandRecordColumn(Table.FromList(Recordlist, Splitter.SplitByNothing(), null, null, ExtraValues.Error), "Column1", Record.FieldNames(Record.Combine(Table.FromList(Recordlist, Splitter.SplitByNothing(), null, null, ExtraValues.Error)[Column1]))),
            fnAddKeyToList = (list) => Table.AddIndexColumn(Table.FromColumns({list}), "Key",0,1),
            Source = Json.Document(Web.Contents(URL)),
            ToTable = Table.FromRecords({Source}),
            Expand = Table.ExpandRecordColumn(ToTable, "structure", {"links", "name", "description", "dimensions", "attributes", "annotations"}, {"links", "name", "description", "dimensions", "attributes", "annotations"}),
            Attr = fnConvertRecordList(Expand[attributes]{0}[observation]),
            Dim = fnConvertRecordList(Expand[dimensions]{0}[observation]),
            Dims = Table.Combine({Attr,Dim}),
            #"Added Index1" = Table.AddIndexColumn(Dims, "Index", 1, 1),
            AddKeyToValues = Table.AddColumn(#"Added Index1", "Custom", each fnAddKeyToList([values])),
            ExpandValues = Table.ExpandTableColumn(AddKeyToValues, "Custom", {"Column1", "Key"}, {"Column1", "Key"}),
            FilterNotNull = Table.SelectRows(ExpandValues, each ([Key] <> null)),
            LookupTbl = Table.ExpandRecordColumn(FilterNotNull, "Column1", {"id", "name"}, {"id.1", "name.1"}),
            Datasets = Expand[dataSets]{0}{0}[observations],
            ConvertToTable = Record.ToTable(Datasets),
            AddKeys = Table.AddColumn(ConvertToTable, "Custom", each fnAddKeyToList(List.Combine({[Value],Text.Split([Name], ":")}))),
            Cleanup = Table.RemoveColumns(AddKeys,{"Value"}),
            ExpandValues2 = Table.ExpandTableColumn(Cleanup, "Custom", {"Column1", "Key"}, {"Value", "Key"}),
            ChgType = Table.TransformColumnTypes(ExpandValues2,{{"Value", type number}}),
            Amount = Table.AddColumn(ChgType, "Amount", each if [Key]=0 then [Value] else null),
            FillDownAmount = Table.FillDown(Amount,{"Amount"}),
            #"Filtered Rows" = Table.SelectRows(FillDownAmount, each ([Value] <> null) and ([Key] <> 0)),
            MergeLookup = Table.NestedJoin(#"Filtered Rows",{"Key", "Value"},LookupTbl,{"Index", "Key"},"Expanded Custom",JoinKind.LeftOuter),
            ExpandLookup = Table.ExpandTableColumn(MergeLookup, "Expanded Custom", {"id", "id.1"}, {"id", "id.1"}),
            Cleanup3 = Table.RemoveColumns(ExpandLookup,{"Key", "Value"}),
            Pivot = Table.Pivot(Cleanup3, List.Distinct(Cleanup3[id]), "id", "id.1"),
            MergeLookup2 = Table.NestedJoin(Pivot,{"Name"},MergeLookup,{"Name"},"Pivoted Column",JoinKind.LeftOuter),
            Expand2 = Table.ExpandTableColumn(MergeLookup2, "Pivoted Column", {"Expanded Custom"}, {"Expanded Custom"}),
            Expand3 = Table.ExpandTableColumn(Expand2, "Expanded Custom", {"name", "name.1"}, {"name.2", "name.1"}),
            AddSpace = Table.TransformColumns(Expand3,{{"name.2", each _&" "}}),
            Pivot2 = Table.Pivot(AddSpace, List.Distinct(AddSpace[name.2]), "name.2", "name.1"),
            Cleanup4 = Table.RemoveColumns(Pivot2,{"Name"}),
            ChgType2 = Table.TransformColumnTypes(Cleanup4,{{"Amount", type number}})
        in
            ChgType2

        Please give a shout if it doesn't!

         

    • ImkeF's avatar
      ImkeF
      Community Champion

      Hey folks,

      I think they moved to a general API-page: API Documentation (oecd.org)
      So to query the house prices from 2020 and 2021 your query would look like so:

      Json.Document(Web.Contents("http://stats.oecd.org/sdmx-json/data/HOUSE_PRICES?startTime=2020&endTime=2021"))

       
      XML-Specs can be found here: sdmx-ml-documentation (oecd.org)
      Query would be so:

      Xml.Tables(Web.Contents("https://stats.oecd.org/restsdmx/sdmx.ashx/GetData/HOUSE_PRICES?startTime=2020&endTime=2021"))

       Looks like you can retrieve all data with this method.
      Please let me know how it worked out.

       

      • lakshayreddy's avatar
        lakshayreddy
        Frequent Visitor

        Heyy ImkeF ,

         

        I tried using this function after changing the Json.Document(Web.Contents(URL)) to Xml.Tables(Web.Contents(URL)). But, I seem to get an error. Can you please tell me where I'm going wrong?

         

        I basically want to import SDMX-XML typye OECD datasets into PowerBI. The function for Developer API aka JSON.Document is working fine.

         

        let

        fnConvertRecordList = (Recordlist) => Table.ExpandRecordColumn(Table.FromList(Recordlist, Splitter.SplitByNothing(), null, null, ExtraValues.Error), "Column1", Record.FieldNames(Record.Combine(Table.FromList(Recordlist, Splitter.SplitByNothing(), null, null, ExtraValues.Error)[Column1]))),
        fnAddKeyToList = (list) => Table.AddIndexColumn(Table.FromColumns({list}), "Key",0,1),
        Source = Xml.Tables(Web.Contents("https://stats.oecd.org/restsdmx/sdmx.ashx/GetData/HOUSE_PRICES?startTime=2020&endTime=2021")),
        ToTable = Table.FromRecords({Source}),
        Expand = Table.ExpandRecordColumn(ToTable, "structure", {"links", "name", "description", "dimensions", "attributes", "annotations"}, {"links", "name", "description", "dimensions", "attributes", "annotations"}),
        Attr = fnConvertRecordList(Expand[attributes]{0}[observation]),
        Dim = fnConvertRecordList(Expand[dimensions]{0}[observation]),
        Dims = Table.Combine({Attr,Dim}),
        #"Added Index1" = Table.AddIndexColumn(Dims, "Index", 1, 1),
        AddKeyToValues = Table.AddColumn(#"Added Index1", "Custom", each fnAddKeyToList([values])),
        ExpandValues = Table.ExpandTableColumn(AddKeyToValues, "Custom", {"Column1", "Key"}, {"Column1", "Key"}),
        FilterNotNull = Table.SelectRows(ExpandValues, each ([Key] <> null)),
        LookupTbl = Table.ExpandRecordColumn(FilterNotNull, "Column1", {"id", "name"}, {"id.1", "name.1"}),
        Datasets = Expand[dataSets]{0}{0}[observations],
        ConvertToTable = Record.ToTable(Datasets),
        AddKeys = Table.AddColumn(ConvertToTable, "Custom", each fnAddKeyToList(List.Combine({[Value],Text.Split([Name], ":")}))),
        Cleanup = Table.RemoveColumns(AddKeys,{"Value"}),
        ExpandValues2 = Table.ExpandTableColumn(Cleanup, "Custom", {"Column1", "Key"}, {"Value", "Key"}),
        ChgType = Table.TransformColumnTypes(ExpandValues2,{{"Value", type number}}),
        Amount = Table.AddColumn(ChgType, "Amount", each if [Key]=0 then [Value] else null),
        FillDownAmount = Table.FillDown(Amount,{"Amount"}),
        #"Filtered Rows" = Table.SelectRows(FillDownAmount, each ([Value] <> null) and ([Key] <> 0)),
        MergeLookup = Table.NestedJoin(#"Filtered Rows",{"Key", "Value"},LookupTbl,{"Index", "Key"},"Expanded Custom",JoinKind.LeftOuter),
        ExpandLookup = Table.ExpandTableColumn(MergeLookup, "Expanded Custom", {"id", "id.1"}, {"id", "id.1"}),
        Cleanup3 = Table.RemoveColumns(ExpandLookup,{"Key", "Value"}),
        Pivot = Table.Pivot(Cleanup3, List.Distinct(Cleanup3[id]), "id", "id.1"),
        MergeLookup2 = Table.NestedJoin(Pivot,{"Name"},MergeLookup,{"Name"},"Pivoted Column",JoinKind.LeftOuter),
        Expand2 = Table.ExpandTableColumn(MergeLookup2, "Pivoted Column", {"Expanded Custom"}, {"Expanded Custom"}),
        Expand3 = Table.ExpandTableColumn(Expand2, "Expanded Custom", {"name", "name.1"}, {"name.2", "name.1"}),
        AddSpace = Table.TransformColumns(Expand3,{{"name.2", each _&" "}}),
        Pivot2 = Table.Pivot(AddSpace, List.Distinct(AddSpace[name.2]), "name.2", "name.1"),
        Cleanup4 = Table.RemoveColumns(Pivot2,{"Name"}),
        ChgType2 = Table.TransformColumnTypes(Cleanup4,{{"Amount", type number}})
        in
        ChgType2 

         

         

        I wish there's a way. Thanks again for the help!😀 

  • Hello ImkeF ,

     

    I using the API I'm getting below error, I'm trying to download the Economic outlook data. Please help me how can I resolve the below error.