topic Re: What is the best way to load a fixed-size CSV file to the Warehouse (Microsoft Fabric)? in Pipelines

What is the best way to load a fixed-size CSV file to the Warehouse (Microsoft Fabric)?

marcosvin — Fri, 09 Jun 2023 03:07:53 GMT

I did a test of using Dataflow Gen2 and was able to read the CSV file by position, but I can only upload to Lakehouse (not Warehouse). In the test using the data pipelines I did not find the option to read by position. Can you tell if there is such an option? Or thinking about the architecture the option of loading first to Lakehouse is correct?

Re: What is the best way to load a fixed-size CSV file to the Warehouse (Microsoft Fabric)?

MartinEstZurita — Sat, 03 Jun 2023 18:05:24 GMT

Hi marcosvin, as far as I know it is not required to upload first to lakehouse and then to warehouse.

I dont understand what you mean with "read by position", could you clarify that?

Re: What is the best way to load a fixed-size CSV file to the Warehouse (Microsoft Fabric)?

makromer — Sat, 03 Jun 2023 21:18:56 GMT

In Fabric Data Pipelines, you can use the Copy task to bring data directly into the Warehouse. Create a new data pipeline, click on "Copy Data" and select your CSV file. Currently in pipelines, your file must be in Azure, but we are adding access to on-prem data as well. Then you can select "Data Warehouse" as your target. It is in the Workspace category for data destinations.

Re: What is the best way to load a fixed-size CSV file to the Warehouse (Microsoft Fabric)?

marcosvin — Mon, 05 Jun 2023 13:21:21 GMT

Hi! The CSV file is delimited by position, that is, the first column starts at character 1 and goes up to 10, the second from 11 to 20, and so on... I didn't find the functionality to read the file this way.

Re: What is the best way to load a fixed-size CSV file to the Warehouse (Microsoft Fabric)?

marcosvin — Mon, 05 Jun 2023 13:59:41 GMT

I found the solution to use Dataflow Gen2 and warehouse in this topic:

Solved: Dataflow (Gen2): Fabric Warehouse Destination...th... - Microsoft Fabric Community