Building Your First Pipeline in Microsoft Fabric
Data rarely lives where you need it. More often than not, the first real challenge of any analytics project is simply getting information from one place to another, reliably and on a schedule. This is exactly where Microsoft Fabric pipelines shine. If you have worked with Azure Data Factory before, much of this will feel familiar, but Fabric brings everything together inside a single, unified workspace and adds a few welcome conveniences along the way. In this article, we will walk through creating your very first Fabric pipeline from the ground up. We will start by setting up the pipeline and exploring the activities available to us, then move on to scheduling and monitoring. Finally, we will build a complete, end-to-end data movement that copies a file from an Azure Data Lake Storage (ADLS) account into a lakehouse. By the end, you will have a working pipeline and a solid mental model of how the pieces fit together. Creating the Pipeline Begin by opening your Newsletter_Pipelines workspace. To create a new pipeline, click on New Item. Give your pipeline a name — in this case, we will call it Ingestion Data. Once you click OK, you are taken to the pipeline authoring canvas shown below. From here, we can start building the pipeline from scratch. Click on the activity you want to add — we will choose Copy Data. Next, click on Activities. Exploring Activities At the top of the canvas you will find the activities bar. Clicking the Activities tab reveals the most commonly used activities right away. If you need something beyond those, the three dots open up a much longer list of options. From here you can, for instance, connect to Databricks or trigger a notification. One of the most useful additions in Fabric is the Outlook activity, which lets you send an email directly from the pipeline. You can even post a notification to Microsoft Teams. These options were not available in Azure Data Factory, so for data engineers they are a genuinely handy way to keep stakeholders informed. Scheduling and Running the Pipeline Adding activities is only half the story — at some point you will want the pipeline to run automatically. To handle that, click the Run button at the top. From there you can run the pipeline immediately, or set up a schedule. Clicking the Schedule button opens the scheduling options, where you can define how and when the pipeline runs. If you would rather use a different kind of trigger — a storage-based trigger, for example — you can configure that here as well. And whenever you want to check on past executions, View Run History gives you a complete record of every run. On the right-hand side, a slider lets you zoom in and out of the canvas as needed. A quick orientation tip for anyone new to Data Factory and pipelines: clicking anywhere outside the activity box surfaces the properties for the overall pipeline configuration in the bottom panel, while clicking inside the box shows the settings specific to that individual activity. Your First End-to-End Data Movement Now it is time for a complete Fabric data movement. Our goal is to build a pipeline that moves data from an Azure Data Lake Storage account into a lakehouse. To see what we are working with, head back to the ADLS account and open the landing container. Inside, you will find the source files. Of the three files here, suppose we only want to move orders.csv from ADLS to the lakehouse. To do that, we will use the Data Factory pipeline we just started building. We already have a pipeline with a single Copy activity in place. The first thing worth doing is renaming that activity to something meaningful — for example, Transfer data from ADLS to a Lakehouse. Naming the activity this way is not mandatory, but it is good practice. A descriptive name makes the pipeline far easier for other developers to understand at a glance; they can tell what the activity does without having to open it and inspect the source and destination. Configuring the Source To move the data, click on the Source tab and set up a connection to the source. Selecting Browse all reveals the wide range of data sources Fabric supports. We want to connect to an Azure Data Lake Storage account, but if you choose View more you will see that Fabric can also connect to SharePoint, Salesforce, Oracle, FTP, SFTP, and many others. Since we only need Data Lake Storage, type Data Lake at the top, find Azure Data Lake Storage, and click it to open the connector. With the connector open, we need to supply the URL, and Fabric will then ask for authentication. The credentials can be an organizational account, a SAS token, or an account key. In this example we will use a SAS token to connect to the Azure Data Lake Storage account. To generate one, go back to the Data Lake Storage account and search for SAS, then select Shared access signature. On the SAS configuration screen, allow all resource types, grant all permissions, and set the expiry far enough out so that you can still access it later. Then click Generate SAS and connection string. You will receive both a connection string and a SAS token. For now, copy the connection string and return to Fabric. When Fabric asks whether you are creating a new connection, choose yes, and paste in the URL. Paste the URL carefully. The example shows a format like https://<storage-account>.dfs.core.windows.net. When you copy and paste, however, the value often comes through as a blob type rather than dfs. Simply change blob to dfs manually and it will work correctly. We want the path to point as far as the landing container, so type landing here. Finally, give the connection a name — we will call it ADLS connection. Since this is neither a private nor an on-premises network, no data gateway is required, so leave that blank. Under authentication type, choose Shared access signature. Now return to the Azure portal, copy the SAS token, come back to Fabric, and paste it in. To recap: we supplied the URL, changed blob to dfs, named the connection, and provided the SAS token. With that done, click Connect. The connection takes a moment to establish and will eventually succeed. To confirm everything is working, click Test connection — you should see that the connection is successful. Next, choose which file or folder to bring in using the file path. The connection is made, but now we browse to a specific file — in this case, orders.csv. Click Browse, open the landing container, select orders.csv, and click OK. Because the file is a CSV, set the file format to DelimitedText. If you need finer control, the Settings tab lets you adjust details such as the column delimiter (useful when the file is not comma-separated) and whether the first line is a header. For now, click OK and move on to the destination. Before moving on, you can confirm the data looks right using the Preview tab. Clicking Preview Data shows the incoming records, and in this case everything reads perfectly. Configuring the Destination Now switch to the Destination tab. As before, there is no existing connection to our lakehouse, so click Browse All. We want to connect to the lakehouse, so type lakehouse. Rather than selecting the New Fabric item option, we will use the OneLake Catalog. Click View More, and under it you will find the e-commerce catalog — exactly where we want our data to land. Select it to make the connection to Orders_lakehouse. With the connection in place, we are almost ready to run. Fabric asks whether to copy the data into the Tables folder or the Files folder; we will save to Files for now. When prompted for a file path, you can browse the folders available in the lakehouse. We will create a new path called copied via pipeline and copy the data there. You can also set the output file format by clicking through the options — we will keep the data in Parquet format. Select Parquet, and we are done. All that remains is to execute the pipeline so it pulls the data from ADLS and pushes it into the lakehouse. Running and Verifying the Pipeline Click the Run button, then choose Save The pipeline now starts running, and full execution takes a little while. The Output tab shows live details. Initially the run sits in a queued state, waiting for resources. Once resources are assigned, it moves into the in-progress state and then completes. You can watch it progress from queued, to in progress, to succeeded. Our pipeline has executed successfully. Clicking into the run details, we can see it copied 3,000 records from the source file into the lakehouse, which now also holds 3,000 records. You will notice that the data read size and data written size differ. That is because Parquet is a compressed format, so the data footprint shrinks once written. To confirm the data actually landed, navigate to the lakehouse, where you will see the new copied via pipeline folder. Open it, and the Parquet files are right there. Wrapping Up And that is how you copy data from an ADLS account into a lakehouse using Fabric pipelines. In just a handful of steps, we created a pipeline, explored the activities Fabric offers, set up scheduling and monitoring, and built a complete data movement from source to destination — verifying along the way that every record arrived safely. The real takeaway is how approachable this process has become. What once required stitching together separate tools now happens inside a single, cohesive workspace, complete with conveniences like Outlook and Teams notifications that simply were not available before. With this foundation in place, you are well positioned to build richer pipelines: chaining multiple activities, adding transformations, and orchestrating sophisticated, scheduled workflows. Your first pipeline is rarely your last, but it is the one that makes everything that follows feel possible.
