Forum Discussion
Helper VBC2ADLS to fabrics
hi, i am using BC2ADLS extension to get large amount of data from BC to fabric lakehouse using the provided code on github https://github.com/Bertverbeek4PS/bc2adls/blob/2a2d5693a9f8fa1b3b9bd5f6591fbf06ded028f7/fabric/CopyBusinessCentral.ipynb .
i was testing and found that if publish a table with specific columns and use the code to get the tables, then i add more columns on BC and publish the table again, i dont get the new added columns. it is only working with editing,deleting or inserting rows, not new columns. does anyone have information on how we should work with adding new columns? should we maybe delete the table from fabrics and then run the code again? or what else?
HI Hussein_charif,
To correctly refresh the schema and allow the new columns to be ingested:
Drop the table using Spark SQL -- This removes the logical table and all metadata from the Spark metastore.
DROP TABLE IF EXISTS YourTableName
Delete physical files -- If you also want to clear out the actual data or Delta files, delete the corresponding folder from the Lakehouse Files directory. It is useful when doing a full refresh.After these steps, re-running the ingestion code will recreate the table and correctly pick up the new schema, including any added columns.
Happy to help! If this addressed your concern, marking it as "Accepted Solution" and giving us "kudos" would be valuable for others in the community.
Thank you.
4 Replies
v-sgandrathiCommunity Support
Hi Hussein_charif,
When using the BC2ADLS extension to transfer data from Business Central (BC) to Microsoft Fabric Lakehouse, changes such as inserting, updating, or deleting rows are successfully reflected.However, if you add new columns to a BC table and republish it, those new columns do not automatically appear in Fabric. This occurs because the existing table schema in the Fabric Lakehouse is not automatically updated to reflect structural changes in the source.
BC2ADLS focuses on row-level data changes and does not perform schema evolution by default. To ensure the new columns are recognized, you should manually delete or drop the existing table in Fabric and re-run the sync process. This forces Fabric to read the table schema again and include the new columns during data ingestion. Without this step, Fabric will continue using the previous schema definition, which excludes any newly added fields.
If this solution worked for you, kindly mark it as Accept as Solution and feel free to give a Kudos, it would be much appreciated!
Thank you.
Hussein_charifHi v-sgandrathi , thank you for clarifying!
so we tried yesterday deleting the table directly from the lakehouse and run the code but it would bug out, as in being deleted but when running the code it is as if it still exists. should deleting the table be dropped with code and not deleted from the tables file as shown in the screenshot below?
here is where we deleted the table from, we clicked on the 3 dots and then clicked delete
- v-sgandrathi
HI Hussein_charif,
To correctly refresh the schema and allow the new columns to be ingested:
Drop the table using Spark SQL -- This removes the logical table and all metadata from the Spark metastore.
DROP TABLE IF EXISTS YourTableName
Delete physical files -- If you also want to clear out the actual data or Delta files, delete the corresponding folder from the Lakehouse Files directory. It is useful when doing a full refresh.After these steps, re-running the ingestion code will recreate the table and correctly pick up the new schema, including any added columns.
Happy to help! If this addressed your concern, marking it as "Accepted Solution" and giving us "kudos" would be valuable for others in the community.
Thank you.