Pipeline - new column in source how to add it to my pipeline for fetching data from sql db to lake

[Oskar]

I am struggling to understand how to add a new column from my sql database into my pipeline.

i have a workspace setup with

pipeline - get data from sql database, sql server as source.

lakehouse - destinatione of pipeline

in one of my tables i have a new column. that column is now showing up in my lakehouse.

I can not find how i can update my schema and re-map my columns for one table so i can add the new column to the pipeline.

[Oskar]

Still havent been able to figure this out.

[NandanHegde]

Can you share the JSON part of your pipeline?
It seems you have a meta data driven framework, so any changes made in the source automatically gets integrated and gets reflected in the sink; so there shouldnt be any need to do anything manually in case of any changes in the source schema .

Once you share the JSON, that can be confirmed

[Oskar]

are you able to see how i should go about to add the new column now that i shared my JSON?

[Oskar]

This is my JSON for my pipeline:

json.json

would it be possible to just edit my json and update with a new column like this:

                                    {
                                        "source": {
                                            "name": "HandoverType",
                                            "type": "String",
                                            "physicalType": "nvarchar",
                                            "length": "20"
                                        },
                                        "sink": {
                                            "name": "HandoverType",
                                            "physicalType": "string"
                                        }
                                    },

I could not paste it here since it was to large.

[Anonymous]

Hi @Oskar ,

Here are some of my personal thoughts:

1. Ensure that the SQL Server source schema is updated to include the new columns. This may require refreshing the schema in the Data Pipeline tool.

2. Modify the pipeline by editing the source component (SQL Server) to refresh the schema. This will pull in the new columns. Map the new column to the target (Lakehouse). This may require updating the mapping settings in the Pipeline tool. Make sure to update the Lakehouse schema to include the new columns. If it does not update automatically, you may need to add the new column manually.

3. Run the pipeline to ensure that the new column is correctly transferred from SQL Server to Lakehouse.

Best Regards

Yilong Zhou

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[Oskar]

1. my sql database schema is updated

2. this is the part i dont get. i have updated as much as i can and re-run the pipeline. but nothing new is imported.

if i setup a brand new pipeline, then i get the new column and everything works.

but i want to ad to my existing pipeline.

[Oskar]

Adding some info.

My pipeline handles multiple tables in a for each.

This was setup by the wizard when configuring the pipeline

So i reccon it would be easier if i only had one copy job, but now it is in a for each and i cant find out how to edit one table.