Join us for an expert-led overview of the tools and concepts you'll need to pass exam PL-300. The first session starts on June 11th. See you there!
Get registeredJoin us at FabCon Vienna from September 15-18, 2025, for the ultimate Fabric, Power BI, SQL, and AI community-led learning event. Save €200 with code FABCOMM. Get registered
I am struggling to understand how to add a new column from my sql database into my pipeline.
i have a workspace setup with
pipeline - get data from sql database, sql server as source.
lakehouse - destinatione of pipeline
in one of my tables i have a new column. that column is now showing up in my lakehouse.
I can not find how i can update my schema and re-map my columns for one table so i can add the new column to the pipeline.
Still havent been able to figure this out.
Can you share the JSON part of your pipeline?
It seems you have a meta data driven framework, so any changes made in the source automatically gets integrated and gets reflected in the sink; so there shouldnt be any need to do anything manually in case of any changes in the source schema .
Once you share the JSON, that can be confirmed
are you able to see how i should go about to add the new column now that i shared my JSON?
This is my JSON for my pipeline:
would it be possible to just edit my json and update with a new column like this:
{
"source": {
"name": "HandoverType",
"type": "String",
"physicalType": "nvarchar",
"length": "20"
},
"sink": {
"name": "HandoverType",
"physicalType": "string"
}
},
I could not paste it here since it was to large.
Hi @Oskar ,
Here are some of my personal thoughts:
1. Ensure that the SQL Server source schema is updated to include the new columns. This may require refreshing the schema in the Data Pipeline tool.
2. Modify the pipeline by editing the source component (SQL Server) to refresh the schema. This will pull in the new columns. Map the new column to the target (Lakehouse). This may require updating the mapping settings in the Pipeline tool. Make sure to update the Lakehouse schema to include the new columns. If it does not update automatically, you may need to add the new column manually.
3. Run the pipeline to ensure that the new column is correctly transferred from SQL Server to Lakehouse.
Best Regards
Yilong Zhou
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
1. my sql database schema is updated
2. this is the part i dont get. i have updated as much as i can and re-run the pipeline. but nothing new is imported.
if i setup a brand new pipeline, then i get the new column and everything works.
but i want to ad to my existing pipeline.
Adding some info.
My pipeline handles multiple tables in a for each.
This was setup by the wizard when configuring the pipeline
So i reccon it would be easier if i only had one copy job, but now it is in a for each and i cant find out how to edit one table.
This is your chance to engage directly with the engineering team behind Fabric and Power BI. Share your experiences and shape the future.