Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Join us at FabCon Vienna from September 15-18, 2025, for the ultimate Fabric, Power BI, SQL, and AI community-led learning event. Save €200 with code FABCOMM. Get registered

Reply
Oskar
Helper I
Helper I

Pipeline - new column in source how to add it to my pipeline for fetching data from sql db to lake

I am struggling to understand how to add a new column from my sql database into my pipeline.

 

i have a workspace setup with

pipeline - get data from sql database, sql server as source.

lakehouse - destinatione of pipeline

 

in one of my tables i have a new column. that column is now showing up in my lakehouse.

 

Oskar_0-1733342969080.png



Oskar_1-1733343000877.png



I can not find how i can update my schema and re-map my columns for one table so i can add the new column to the pipeline.

 

Oskar_0-1733343097923.png

 

 

7 REPLIES 7
Oskar
Helper I
Helper I

Still havent been able to figure this out.

NandanHegde
Super User
Super User

Can you share the JSON part of your pipeline?
It seems you have a meta data driven framework, so any changes made in the source automatically gets integrated and gets reflected in the sink; so there shouldnt be any need to do anything manually in case of any changes in the source schema .

 

Once you share the JSON, that can be confirmed




----------------------------------------------------------------------------------------------
Nandan Hegde (MSFT Data MVP)
LinkedIn Profile : www.linkedin.com/in/nandan-hegde-4a195a66
GitHUB Profile : https://github.com/NandanHegde15
Twitter Profile : @nandan_hegde15
MSFT MVP Profile : https://mvp.microsoft.com/en-US/MVP/profile/8977819f-95fb-ed11-8f6d-000d3a560942
Topmate : https://topmate.io/nandan_hegde
Blog :https://datasharkx.wordpress.com

are you able to see how i should go about to add the new column now that i shared my JSON?

 

This is my JSON for my pipeline:

json.json

 

would it be possible to just edit my json and update with a new column like this:

                                    {
                                        "source": {
                                            "name": "HandoverType",
                                            "type": "String",
                                            "physicalType": "nvarchar",
                                            "length": "20"
                                        },
                                        "sink": {
                                            "name": "HandoverType",
                                            "physicalType": "string"
                                        }
                                    },

I could not paste it here since it was to large.

 

 

Anonymous
Not applicable

Hi @Oskar ,

Here are some of my personal thoughts:

 

1. Ensure that the SQL Server source schema is updated to include the new columns. This may require refreshing the schema in the Data Pipeline tool.

 

2. Modify the pipeline by editing the source component (SQL Server) to refresh the schema. This will pull in the new columns. Map the new column to the target (Lakehouse). This may require updating the mapping settings in the Pipeline tool. Make sure to update the Lakehouse schema to include the new columns. If it does not update automatically, you may need to add the new column manually.

 

3. Run the pipeline to ensure that the new column is correctly transferred from SQL Server to Lakehouse.

 

 

Best Regards

Yilong Zhou

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

1. my sql database schema is updated

2. this is the part i dont get. i have updated as much as i can and re-run the pipeline. but nothing new is imported.

 

 

if i setup a brand new pipeline, then i get the new column and everything works.

but i want to ad to my existing pipeline.

 

Oskar
Helper I
Helper I

Adding some info.

My pipeline handles multiple tables in a for each.

This was setup by the wizard when configuring the pipeline

 

Oskar_0-1733343742237.png

 

So i reccon it would be easier if i only had one copy job, but now it is in a for each and i cant find out how to edit one table.

Helpful resources

Announcements
Join our Fabric User Panel

Join our Fabric User Panel

This is your chance to engage directly with the engineering team behind Fabric and Power BI. Share your experiences and shape the future.

May FBC25 Carousel

Fabric Monthly Update - May 2025

Check out the May 2025 Fabric update to learn about new features.

June 2025 community update carousel

Fabric Community Update - June 2025

Find out what's new and trending in the Fabric community.