nested Array Issue in Mapping the key Values Part
I have a nested list of the array within the API Response in the Azure Data Factory. When I am mapping it with the column its is showing me null value instead of giving me the actual value which exist. Here rthe Product List Part part of the reposnse that I wanted to map "products": [ { "id": 1, "catalogItem": { "id": 1, "identifier": "1", "_info": { "catalog_href": "https://example.com/1" } }, "description": "1", "sequenceNumber": 1, "quantity": 1, "price": 1, "cost": 1, "agreementAmount": 1, "billableOption": "1", "agreement": { "id": 1, "name": "1", "_info": { "agreement_href": "https://example.com/1" } }, "locationId": 1, "businessUnitId": 1, "taxableFlag": false, "dropshipFlag": false, "specialOrderFlag": false, "phaseProductFlag": false, "cancelledFlag": false, "quantityCancelled": 1, "customerDescription": "1", "productSuppliedFlag": false, "invoice": { "id": 1, "identifier": "1", "_info": { "invoice_href": "https://example.com/1" } }, "warehouseId": 1, "warehouseBinId": 1, "calculatedPriceFlag": false, "calculatedCostFlag": false, "warehouse": "1", "warehouseBin": "1", "company": { "id": 1, "identifier": "1", "name": "1", "_info": { "company_href": "https://example.com/1" } }, "productClass": "1", "needToPurchaseFlag": false, "minimumStockFlag": false, "calculatedPrice": 1, "calculatedCost": 1, "_info": { "lastUpdated": "2026-01-01T00:00:00Z", "updatedBy": "1" } } ] I am trying the catlogitem.id and the the Same Format you can see here but still cant get an data in the rows of the SQL Table How to refrenece it or how to flat it?
Copy Job - Lakehouse Files to Lakehouse Table
Hi there, I am exactly trying to achieve what Patrick is doing in the video: Nutzen Sie den Kopierauftrag, um die neueste Datei in die Data Factory zu kopieren. - YouTube Shortcut loading Blob Files into Lakehouses Files and then using a Copy Job to move in the same Lakehouse with incremental load into the Table section. However this seems to be not working anymore? I am receiving a error message stating that: "ErrorCode=ApplyChangesNotSupported,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ApplyChanges is not supported for the copy pair from Lakehouse to LakehouseTable.,Source=Microsoft.DataTransfer.ClientLibrary,'" However I am curious if there is anything that I am doing wrong? Are there better ways to move within a Lakehouse from Files into Table incrementally with a low-code experience? Thanks for the help!
Incremental Copy Job error from NetSuite
I am trying to configure a copy job to incrementally load data from NetSuite SuiteAnalytics Connect. The source is accessed using the NetSuite ODBC driver installed on an on-prem data gateway. I receive this error when trying to run the job, regardless of table selected: Failure happened on 'Source' side. ErrorCode=UserErrorOdbcOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ERROR [HY000] [NetSuite][ODBC 64bit driver][OpenAccess SDK SQL Engine]Error getting the literal value of right operand.[10169],Source=Microsoft.DataTransfer.Runtime.GenericOdbcConnectors,''Type=System.Data.Odbc.OdbcException,Message=ERROR [HY000] [NetSuite][ODBC 64bit driver][OpenAccess SDK SQL Engine]Error getting the literal value of right operand.[10169],Source=NQoa27.dll,' The field used as a watermark is recognised by Fabric as a datetime column. Before I raise a support request with MS, has anyone else encountered this issue and found a solution?Copy Job deceivingly says "Overwrite" when some tables are "Append"
Hi, My data analysts noticed some inflated numbers on tables and found sets of duplicated data in their Lakehouse. I use a Copy Job daily to fully copy all data from my GoldLakehouse into their AnalystLakehouse in a separate workspace. The Copy Job is set to Overwrite in the Advanced Settings tab in the ribbon at the top. Yet, I dive into the JSON for the Copy Job, and it reveals that some of the tables are actually set to Append individually. There is no setting in the UI that lets you set Overwrite/Append on individual tables, so why does the field exist for individual tables in the JSON? Moreso, how did some of them get set to Append when the only available setting in the Copy Job is set to Overwrite? I believe the ones that got set to Append were the new ones I selected recently to copy. They were inserted as "Append" by default, even though the Copy Job is set to Overwrite. This has to be fixed. Microsoft, this has caused our reports for C-Suite to be off by potentially millions of dollars for a reason out of our control, this is unacceptable. Please look into this and see who else this is effecting without their notice. Update Copy Job to be without confusing and misleading settings, and please do not push updates to the backend logic before providing UI to manage it. Please provide a stable product. To the community, has anyone else run into this?CopyJob → Snowflake: Database name is hard‑coded in activity JSON and cannot be parameterised
When using CopyJob in Microsoft Fabric to load data into Snowflake, the selected Database becomes permanently embedded inside the CopyJob’s internal JSON definition. Unlike the connection (server, user, warehouse), the Database is not part of the connection object and is not exposed to dynamic content or environment variables. As a result, when deploying pipelines from Dev → SIT → UAT using Deployment Pipelines, the CopyJob in SIT/UAT continues to write to the Dev Snowflake database, even though the connection has been correctly parameterised and switched to SIT/UAT. The CopyJob JSON shows the database name hard‑coded, and this JSON is not editable in the UI or via dynamic content. This prevents environment‑agnostic pipelines and breaks standard Dev→Test→Prod deployment patterns. Detailed Problem Description 1. Fabric Snowflake connections do not include a Database field The Snowflake connector only supports: Server (account URL) Username Password Warehouse There is no field for Database or Schema. 2. CopyJob stores the Database inside the activity JSON, not the connection When configuring a CopyJob in Dev: The user selects a Snowflake database (e.g., DATA_LAKE_DEV) Fabric stores this value inside the CopyJob JSON This value is not parameterisable This value is not editable after deployment Dynamic content is not supported for this field 3. The hard‑coded JSON looks like this Inside the CopyJob definition, the destination section contains: "destination": { "type": "SnowflakeTable", "connectionSettings": { "type": "Snowflake", "typeProperties": { "database": "DATA_LAKE_DEV" } } } This JSON is not exposed in the UI, not editable, and not overridable via variable libraries. 4. Deployment Pipelines cannot adjust the Snowflake database After deploying to SIT: The connection correctly switches to SIT But the CopyJob still contains "database": "DATA_LAKE_DEV" The UI does not allow changing the database Dynamic content is not supported Incremental CopyJob mode does not support Query mode (so no workaround) 5. Result: SIT and UAT pipelines write into the Dev database This breaks: Environment isolation Data governance CI/CD deployment patterns Automated promotion workflows Impact CopyJobs cannot be made environment‑aware. Pipelines deployed to SIT/UAT continue writing into Dev Snowflake databases. Incremental loads cannot be parameterised. Deployment Pipelines lose their value for Snowflake workloads. Teams must manually recreate CopyJobs in each environment, which is error‑prone and not scalable. Requested Enhancements 1. Add Database and Schema fields to the Snowflake connection object This would align Fabric with ADF and allow environment‑specific connections. 2. Allow Database and Schema to be parameterised in CopyJob Expose these fields to dynamic content so variable libraries can control them. 3. Allow editing the Database/Schema in CopyJob after deployment This would allow SIT/UAT to rebind the target database without recreating the activity.OData Error When Connecting to SharePoint Online
Hi Fabric Community, I'm encountering an interesting error when trying to use a copy job to pull data from a sharepoint list. After I set up the connection, when I get to the choose data screen, I get the following error: Failed to get metadata of odata service, please check if service url and credential is correct and your application has permission to the resource. Expected status code: 200, actual status code: Conflict, response is : <?xml version="1.0" encoding="utf-8"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <title>Request Error</title> <style>BODY { color: #000000; background-color: white; font-family: Verdana; margin-left: 0px; margin-top: 0px; } #content { margin-left: 30px; font-size: .70em; padding-bottom: 2em; } A:link { color: #336699; font-weight: bold; text-decoration: underline; } A:visited { color: #6699cc; font-weight: bold; text-decoration: underline; } A:active { color: #336699; font-weight: bold; text-decoration: underline; } .heading1 { background-color: #003366; border-bottom: #336699 6px solid; color: #ffffff; font-family: Tahoma; font-size: 26px; font-weight: normal;margin: 0em 0em 10px -20px; padding-bottom: 8px; padding-left: 30px;padding-top: 16px;} pre { font-size:small; background-color: #e5e5cc; padding: 5px; font-family: Courier New; margin-top: 0px; border: 1px #f0f0e0 solid; white-space: pre-wrap; white-space: -pre-wrap; word-wrap: break-word; } table { border-collapse: collapse; border-spacing: 0px; font-family: Verdana;} table th { border-right: 2px white solid; border-bottom: 2px white solid; font-weight: bold; background-color: #cecf9c;} table td { border-right: 2px white solid; border-bottom: 2px white solid; background-color: #e5e5cc;}</style> </head> <body> <div id="content"> <p class="heading1">Request Error</p> <p>The server encountered an error processing the request. See server logs for more details.</p> </div> </body> </html>. I am logged in with an account that has site owner permissions, so there shouldn't be any permissions issues. I think this is an issue with this specific sharepoint site, as other sites work fine, but I'm not sure what could be misconfigured on the site for this to happen. Any ideas?
