Hi all,
I am trying to load parquet data from my ADLS Gen 2 storage. The data is being generated by a Data Flow in ADF/Synapse which stores it in the ADLS Gen 2 Sink with a key partition on 2 columns. As a result, the folder structure looks a bit like this.
MyDataStore/CategoryName=AI/SubCategoryName=Swarm Intelligence/2023-02-27.parquet
However, the data in the parquet file doesn't have the 2 columns used in the key. Now when I try loading this into Power BI using the ADLS Gen 2 connector, I get all the data except the 2 columns that I have partitioned with. I am trying to load the parquet files using the following URL
https://pbixdemo.dfs.core.windows.net/pbixdemo2gen2storage/MyDataStore/ with a SAS token.
I have tried using something like https://pbixdemo.dfs.core.windows.net/pbixdemo2gen2storage/MyDataStore/CategoryName=*/SubCategoryNam... to see if we can infer the CategoryName and SubCategoryName columns, but this threw a 404 error.
Also, after I transform and load the data, I have no way to recompute the partition keys since I only have the Source column as an additional column in the transformed data that houses the name of the parquet file, which won't help in backtracking to the keys.
