Forum Discussion
Helper ILayered approach for LH
Hi experts, current we are ingesting the data in bronze layer as of the source format in Excel /CSV format. Now we have to transition to Silver layer and here lies the challenge, we are told by business the parquet format is preferred, and tabular structure is expected, I understand it is stored in par1 ,part 2... and transaction log. This is not warehouse to be in tabular format , how to navigate. Please help
Hi SamyAbdul
Fabric is built upon delta parquet file formats. Any tables in a Lakehouse or Warehouse are Delta Parquet files under the hood.
Now that you have your bronze data, use a notebook to move the data from CSV/Excel files into bronze tables, and bam you have it delta parquet files.
3 Replies
SamyAbdulThank you tayloramy but we do need to aggregate those excel file into one broader table as per semantics and commonality , as Silver layer suppose to have.
- tayloramy
Super User
Hi SamyAbdul
Yes, that is expected. Once you read those excel files in a notebook and write them back to tables in whatever way you need to, the tables written will be stored as delta parquet files.
You would still need to create a notebook with pyspark code to be able to read the excel files, and transform them into whatever your desired structure is.
- tayloramy
Hi SamyAbdul
Fabric is built upon delta parquet file formats. Any tables in a Lakehouse or Warehouse are Delta Parquet files under the hood.
Now that you have your bronze data, use a notebook to move the data from CSV/Excel files into bronze tables, and bam you have it delta parquet files.