topic Re: Could not determine JSON object type for type Microsoft.DataTransfer.DataTypes.Date. in Pipelines

Could not determine JSON object type for type Microsoft.DataTransfer.DataTypes.Date.

Hendrik1980 — Fri, 10 May 2024 10:08:12 GMT

I'm trying to use a Copy Data node to send a table to an external WebHook. The data source is a Lakehouse table that I've created using a Notebook and PySpark.

I'm getting the following error message:

ErrorCode=InvalidParameter,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The value of the property '' is invalid: 'Could not determine JSON object type for type Microsoft.DataTransfer.DataTypes.Date.'.,Source=,''Type=System.ArgumentException,Message=Could not determine JSON object type for type Microsoft.DataTransfer.DataTypes.Date.,Source=Newtonsoft.Json,'

Please check why this is happening. It's entirely blocking me right now.

Re: Could not determine JSON object type for type Microsoft.DataTransfer.DataTypes.Date.

Anonymous — Fri, 10 May 2024 11:39:53 GMT

Hi Hendrik1980 ,

Thanks for using Fabric Community.
Can you please share the screenshot of the pipeline along with activity configration?
Inorder to help you out, I need these details - what exactly are you doing here?

Re: Could not determine JSON object type for type Microsoft.DataTransfer.DataTypes.Date.

Hendrik1980 — Fri, 10 May 2024 11:58:11 GMT

Here's a screenshot of the pipeline.

And the Copy Data activity is configured like this:

{ "name": "Copy data1", "type": "Copy", "dependsOn": [ { "activity": "Clockify JSON to ADV CSV", "dependencyConditions": [ "Succeeded" ] } ], "policy": { "timeout": "0.12:00:00", "retry": 0, "retryIntervalInSeconds": 30, "secureOutput": false, "secureInput": false }, "typeProperties": { "source": { "type": "LakehouseTableSource", "datasetSettings": { "annotations": [], "linkedService": { "name": "Clockify", "properties": { "annotations": [], "type": "Lakehouse", "typeProperties": { "workspaceId": "f81cf571-7587-428c-a68d-25b3177c337b", "artifactId": "ee28671f-e7db-4649-a545-ea868e2af123", "rootFolder": "Tables" } } }, "type": "LakehouseTable", "schema": [], "typeProperties": { "table": "adv_output" } } }, "sink": { "type": "RestSink", "httpRequestTimeout": "00:05:00", "requestInterval": 10, "requestMethod": "PUT", "writeBatchSize": 10000, "httpCompressionType": "none", "datasetSettings": { "annotations": [], "type": "RestResource", "typeProperties": { "relativeUrl": "webhook-test/fabric-clockify-adv" }, "schema": [], "externalReferences": { "connection": "01fb0275-f785-4223-93fa-02f3eb148cff" } } }, "enableStaging": false, "translator": { "type": "TabularTranslator", "typeConversion": true, "typeConversionSettings": { "allowDataTruncation": true, "treatBooleanAsNumber": false } } } }

Re: Could not determine JSON object type for type Microsoft.DataTransfer.DataTypes.Date.

Anonymous — Fri, 10 May 2024 13:58:20 GMT

Hi Hendrik1980 ,

The error message you're encountering, ErrorCode=InvalidParameter, indicates that there's an issue with the way a date value is being formatted in your JSON data. It seems Microsoft Data Factory is expecting a specific format for dates, but the data you're providing doesn't match that format.

Please check with date values, you should be able to get some resolution.

Re: Could not determine JSON object type for type Microsoft.DataTransfer.DataTypes.Date.

Hendrik1980 — Fri, 10 May 2024 14:07:13 GMT

The values in the date column are just that: dates. Timestamps. Some kind of representation of a date. Whatever format is required, under the hood it should be nothing other than a timestamp. If it was a string column, I could understand the format argument. But the date column has no "format" other than what's displayed to the user. It has some date time information, timezone naive or timezone aware. How that date is made readable, should be irrelevant for the JSON conversion. I would expect the JSON conversion, which essentially converts the date time object into a string (yes), to use some universal or configurable format instead of just failing. I see no option to choose a format for the date to JSON conversion anywhere. It just fails. As a bare minimum/default/fallback format I would expect the datetime to be redered into an ISO string - instead of failing.

Re: Could not determine JSON object type for type Microsoft.DataTransfer.DataTypes.Date.

Anonymous — Mon, 13 May 2024 07:41:15 GMT

Hi Hendrik1980 ,

Just checking back to see if you have a resolution yet .
In case if you have any resolution please do share that same with the community as it can be helpful to others .
Otherwise, will respond back with the more details and we will try to help .

Re: Could not determine JSON object type for type Microsoft.DataTransfer.DataTypes.Date.

Anonymous — Tue, 14 May 2024 13:17:54 GMT

Hello Hendrik1980 ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
In case if you have any resolution please do share that same with the community as it can be helpful to others .
If you have any question relating to the current thread, please do let us know and we will try out best to help you.
In case if you have any other question on a different issue, we request you to open a new thread .
Thanks

Re: Could not determine JSON object type for type Microsoft.DataTransfer.DataTypes.Date.

Hendrik1980 — Tue, 14 May 2024 13:38:18 GMT

No. I have a workaround: converting the column to string in the Data Flow.

However, I do not want that workaround. I expect the involved components (Lakehouse table, Copy Data node) to work together regardless of the data type. I expect the Copy Data node to be able to convert whatever Fabric data source is used to whatever Copy Data target is used. It's both integrated and core functionality of Fabric. Converting a date field from a Lakehouse table into a JSON field (string, essentially) should be no rocket science (if in doubt about the target format, either use ISO date per default or offer a way to configure a desired format). I expect the Copy Data node to be able to do that as it converts any other data type.

I see the error as a bug and would like to report it as that.

Re: Could not determine JSON object type for type Microsoft.DataTransfer.DataTypes.Date.

Anonymous — Tue, 14 May 2024 17:23:48 GMT

Hi Hendrik1980 ,

The best course of action is to open a support ticket and have our support team take a closer look at it.

Please reach out to our support team so they can do a more thorough investigation on why this it is happening: Link

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

Hope this helps. Please let us know if you have any other queries.

Re: Could not determine JSON object type for type Microsoft.DataTransfer.DataTypes.Date.

Hendrik1980 — Tue, 14 May 2024 17:54:50 GMT

Will do. Thank you for the link.

Re: Could not determine JSON object type for type Microsoft.DataTransfer.DataTypes.Date.

Anonymous — Tue, 14 May 2024 17:56:48 GMT

I hope you got some insights over your query. Please continue using Fabric Community on your further queries.