Forum Discussion
Helper ILayered approach for LH
Hi experts, current we are ingesting the data in bronze layer as of the source format in Excel /CSV format. Now we have to transition to Silver layer and here lies the challenge, we are told by business the parquet format is preferred, and tabular structure is expected, I understand it is stored in par1 ,part 2... and transaction log. This is not warehouse to be in tabular format , how to navigate. Please help
Hi SamyAbdul
Fabric is built upon delta parquet file formats. Any tables in a Lakehouse or Warehouse are Delta Parquet files under the hood.
Now that you have your bronze data, use a notebook to move the data from CSV/Excel files into bronze tables, and bam you have it delta parquet files.
3 Replies
- tayloramy
Super User
Hi SamyAbdul
Fabric is built upon delta parquet file formats. Any tables in a Lakehouse or Warehouse are Delta Parquet files under the hood.
Now that you have your bronze data, use a notebook to move the data from CSV/Excel files into bronze tables, and bam you have it delta parquet files.
SamyAbdulThank you tayloramy but we do need to aggregate those excel file into one broader table as per semantics and commonality , as Silver layer suppose to have.
- tayloramy
Hi SamyAbdul
Yes, that is expected. Once you read those excel files in a notebook and write them back to tables in whatever way you need to, the tables written will be stored as delta parquet files.
You would still need to create a notebook with pyspark code to be able to read the excel files, and transform them into whatever your desired structure is.