Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
10 years ago
Solved

Change JSON Blob data during import

Hi,   I'm trying to import data from an Azure Blob Storage. The files that are getting imported are JSON files, with one entry on each line.   They are lacking 2 things :  An opening "[" and a...
  • Anonymous's avatar
    Anonymous
    10 years ago

    Hi again,

     

     

    I've managed to display what I wanted ! The importation of data is really slow (it takes about 10 min to process the data for a week, which represents about 250k lines of data), but it works.

     

    Here's how I did it, in case someone tries to do the same thing as I did :

     

    let
    Source = AzureStorage.Blobs("myazureblobsource"),
    #"datastorage" = Source{[Name="datastorage"]}[Data],
    #"Filtered Rows" = Table.SelectRows(#"atastorage", each [Date modified] > #datetime(2016, 4, 18, 12, 3, 22)),
    #"Combined Binaries" = Binary.Combine(#"Filtered Rows"[Content]),
    #"Combined Text" = Lines.FromBinary(#"Combined Binaries",1),
    #"Json joined" = Replacer.ReplaceText(Lines.ToText(#"Combined Text"),"}{","}#(cr)#(lf){"),
    #"Json Text" = Replacer.ReplaceText(#"Json joined","#(cr)#(lf){",",#(lf){"),
    #"Json Text Complete" = Text.Combine({"[", #"Json Text", "]"}, ""),
    #"Json list" = Json.Document(#"Json Text Complete"),
    #"Data table" = Table.FromList(#"Json list", Splitter.SplitByNothing(), null, null, ExtraValues.Error),

     

    Basically I had to transform the combined binaries into a list of Lines, then join the Lines into a Text (with Lines.ToText), then I replaced what I needed to be replaced in the text. Finally, I passed this text as an argument for Json.Document, split the result for it to be interpreted correctly by Power BI.

    I did not include the last part of the process, which is the expansion of the JSON into a table, because the code was generated by Power BI when I clicked on the double arrows and it's really long.

    Cheers :)