Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
4 years ago
Solved

In a DataFlow are partitioned Azure DataLake Gen2 Parquet files supported

I generated partitioned Azure DataLake Gen2 Parquet files using a spark notebook and can query them with no issues in Synapse Analytics using serverless SQL.       When I attempt to create ...
  • Anonymous's avatar
    Anonymous
    4 years ago

    I am going to reply to my own question and hopefully it will close this issue and help someone else.

     

    • What I needed was a Proof of Concept (POC) of putting data into a ADLS Gen2 storage account container where that container is partitioned by DataType/Year=2022/Month=01/Day=01.
    • Then I wanted to create a DataFlow referencing that container to see if Power BI Desktop could use the entities in that DataFlow. 

     

    Anyway the issue was that I could not get it to work in a dataflow and even if I did I doubt that Query Folding would work.

    So the alternative was to use serverless sql in Azure Synapse.

    • I could create multiple databases with each database targeting a container within ADLS Gen2.
    • Rather than use AAD passthru I setup access to the CETAS table using a SAS token.
    • Then I used TSQL to create a specific username/password for access to each database.
    • Then I gave that username specific permissions.

    This all worked fine until I discovered that you cannot refresh CETAS tables.

     

    But it you do everything else the same and rather than create a CETAS table you create a view using OpenRowset then

    "Bobs You Uncle".

    Appending additional data to the underlying ADLS Gen 2 container as Year=xxxx/Month=xx/Day=xx works as expected.  I can see the refreshed data if I am attached using SSMS or if I  query in Azure Synapse.  I still have to refresh the DataFlow and for now until I can test using a Computed Engine I have to also refresh the DataSet but it does work in a DataFlow as expected.

     

    Now I need to test using Lake Database tables to determine if they see appended data automatically.