Fabric pipeline error

[Hussein_charif]

i am trying to get data from dataverse virtual table using a pipeline. i am getting this error everytime :
Failure happened on 'Source' side. ErrorCode=UserErrorWriteFailedFileOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The file operation is failed, upload file failed at path: '0f1e85e7-0992-4618-a232-d50b497751a3/6e2ceb53-4803-4936-8e57-9acfd7d0c58e/Tables/mserp_generaljournalaccountentrybientity/c48f78c1-65fc-4a90-bf6b-a4666652c5e7.parquet'.,Source=Microsoft.DataTransfer.Common,''Type=System.ServiceModel.FaultException`1[[Microsoft.Xrm.Sdk.OrganizationServiceFault, Microsoft.Xrm.Sdk, Version=9.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]],Message=System.OverflowException: Value was either too large or too small for an Int64.
at System.Decimal.ToInt64(Decimal d) +0x5c
at System.Decimal.System.IConvertible.ToInt64(IFormatProvider provider) +0x0
at Microsoft.Crm.Formatting.CrmFormatter.CrmFormatNumber[TIntermediateType](Object value, Int32 precision, Boolean isCurrency, String currencySymbol, ICrmFormatterContext context) +0x1b
at Microsoft.Crm.Sdk.OnReadPropertyFilter.FormatNumberProperty(AttributeType attributeType, Object value, Entity entity, AttributeMetadata attributeMetadata, Boolean isCurrency, ExecutionContext context, String& formattedValue) +0x122
at Microsoft.Crm.Sdk.OnReadPropertyFilter.FormatProperty(KeyValuePair`2 property, Entity entity, AttributeMetadata attributeMetadata, ExecutionContext context) +0x0
at Microsoft.Crm.Sdk.PropertyFilter.Filter(Entity entity, String[] excluded, ExecutionContext context) +0x24a
at Microsoft.Crm.Extensibility.MessageProcessor.PostProcessEntity(Entity entity, ExecutionContext context) +0x0
at Microsoft.Crm.Extensibility.MessageProcessor.PostProcessEntityCollection(EntityCollection entities, ExecutionContext context) +0x1a
at Microsoft.Crm.Extensibility.MessageProcessor.FilterEntityForOutboundImpl(PipelineExecutionContext context) +0x23
at Microsoft.Xrm.Telemetry.XrmTelemetryExtensions.Execute[TArg](ILogger logger, XrmTelemetryActivityType activityType, Action`1 action, TArg arg) +0x24
at Microsoft.Crm.Extensibility.MessageProcessor.FilterEntityForOutbound(PipelineExecutionContext context) +0x2f
at Microsoft.Crm.Extensibility.MessageProcessor.Execute(PipelineExecutionContext context) +0x6c9
at Microsoft.Crm.Extensibility.InternalMessageDispatcher.<>c__DisplayClass5_0.<Execute>b__0() +0x262
at Microsoft.PowerApps.CoreFramework.ActivityLoggerExtensions.Execute(ILogger logger, EventId eventId, ActivityType activityType, Action action, IEnumerable`1 additionalCustomProperties) +0x90
at Microsoft.Xrm.Telemetry.XrmTelemetryExtensions.Execute(ILogger logger, XrmTelemetryActivityType activityType, Action action) +0x13
at Microsoft.Crm.Extensibility.ExternalMessageDispatcher.ExecuteInternal(IInProcessOrganizationServiceFactory serviceFactory, IPlatformMessageDispatcherFactory dispatcherFactory, Guid callerRegardingObjectId, Int32 invocationSource, Version endpointVersion, OrganizationRequest request, OrganizationInternalRequest organizationInternalRequest) +0x7ea
at Microsoft.Crm.Extensibility.OrganizationSdkServiceInternal.DispatchRequest(OrganizationRequest request, OrganizationInternalRequest organizationInternalRequest, SdkResponseDescription responseDescription, SdkPerformanceCounterContext sdkPerformanceCounterContext, IHttpContext httpContext) +0x5b
at Microsoft.Crm.Extensibility.OrganizationSdkServiceInternal.ExecuteRequestRequestWithInstrumentation(OrganizationRequest request, OrganizationInternalRequest organizationInternalRequest, IHttpContext httpContext) +0x1a5
at Microsoft.Crm.Extensibility.OrganizationSdkServiceInternal.ExecuteRequestRequestWithInstrumentationOperation(OrganizationRequest request, OrganizationInternalRequest organizationInternalRequest, IHttpContext httpContext) +0x8f
at Microsoft.Crm.Extensibility.OrganizationSdkServiceInternal.<>c__DisplayClass27_0.<ExecuteRequest>b__0() +0x2d2
at Microsoft.PowerApps.CoreFramework.ActivityLoggerExtensions.Execute[TResult](ILogger logger, EventId eventId, ActivityType activityType, Func`1 func, IEnumerable`1 additionalCustomProperties) +0x91
at Microsoft.Xrm.Telemetry.XrmTelemetryExtensions.Execute[TResult](ILogger logger, XrmTelemetryActivityType activityType, Func`1 func) +0x0
at Microsoft.Crm.Extensibility.OrganizationSdkServiceInternal.ExecuteRequest(OrganizationRequest request, InvocationContext invocationContext, CallerOriginToken callerOriginToken, WebServiceType serviceType, UserAuth userAuth, Guid targetUserId, Guid targetCallerRegardingObjectId, UserType targetUserType, OrganizationContext context, Boolean returnResponse, Boolean checkAdminMode, IHttpContext httpContext, Dictionary`2 optionalParameters) +0x231
at Microsoft.Crm.Extensibility.OrganizationSdkServiceInternal.ExecuteRequestInternal(OrganizationRequest request, InvocationContext invocationContext, CallerOriginToken callerOriginToken, WebServiceType serviceType, Boolean checkAdminMode, ExecutionContext executionContext, Dictionary`2 optionalParameters) +0x7a
at Microsoft.Crm.Extensibility.OrganizationSdkServiceInternal.Execute(OrganizationRequest request, InvocationContext invocationContext, CallerOriginToken callerOriginToken, WebServiceType serviceType, Boolean checkAdminMode, ExecutionContext executionContext, Dictionary`2 optionalParameters) +0x63: Microsoft Dynamics CRM has experienced an error. Reference number for administrators or support: #F27DEC89,Source=mscorlib,'

i have tried removing all columns with int64 or int32 data types to try and tackle the column with the issue, but i still couldnt extract the data. 

is the problem from my end and should be fixed or does it need fixing on dataverse?

[Vinodh247 · Accepted solution]

Hi,

Removing Int32/Int64 columns in your copy activity does not help because dataverse formats all attributes of the virtual entity before sending them out. This is a virtual table plugin/provider issue. The backend data source is returning a decimal/numeric value that is too large or too small for Int64 and dataverse chokes. This is not something Fabric can fix, fabric simply receives whatever dataverse sends.

[v-sgandrathi]

Hi @Hussein_charif,

Just wanted to follow up and confirm that everything has been going well on this. Please let me know if there’s anything from our end.
Please feel free to reach out Microsoft fabric community forum.

[v-sgandrathi]

Hi @Hussein_charif,

Just looping back once to check if everything's good on your end. Let me know if you need any final support happy to assist if anything’s still open.

Thank you.

[v-sgandrathi · Accepted solution]

Hi @Hussein_charif,

Thank you @Vinodh247 and @tayloramy for the responses to the query.

In addition to the previous suggestions, you can try to pinpoint where the overflow is happening on the Dataverse side. Since the error occurs before Fabric accesses the data, the next step is to check the values returned by the virtual table provider. Try limiting the records such as by date or ID range to find which rows cause the issue. Once you’ve identified the problematic rows, review the relevant numeric or calculated fields in the external source, as one of them might be producing a value outside the allowed range.

If you find the field causing the problem, you can modify the mapping in the virtual table provider, perhaps by reducing precision, lowering the scale, or converting the value to text if it’s only for display. If you don’t manage the provider, the Dataverse or external system team may need to create a safer view with properly constrained numeric types and update the virtual table to use that view. Adjusting the mapping at the source is the best way to ensure Fabric pipelines can read the table without errors.

Thank you.

[Vinodh247 · Accepted solution]

Hi,

Removing Int32/Int64 columns in your copy activity does not help because dataverse formats all attributes of the virtual entity before sending them out. This is a virtual table plugin/provider issue. The backend data source is returning a decimal/numeric value that is too large or too small for Int64 and dataverse chokes. This is not something Fabric can fix, fabric simply receives whatever dataverse sends.

[tayloramy]

Hi @Hussein_charif, 

The error System.OverflowException: Value was either too large or too small for an Int64 means that your values are too large for the column type. 

You said you tried removing the int columns but are still not having any luck. Are you able to send us the structure of the dataverse table? 

This is a failure onthe source side, so the copy activity is failing when reading the data, not when writing it. 

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.