On Prem SQL Server to Fabric Lakehouse - copy data activity stuck In progress and never completes

[GLWY]

Hi,

I created a pipeline with one copy data activity, copying from Onprem server to lakehouse using copy data assistant. Previewing the data works fine, I have updated the power bi gateway to the latest.

When it runs, it stays In progress. I even let it run overnight and it was still running for 16 hours for just a table with a few records. In the destination lakehouse, it created the delta files with the header but no data.

What am I missing here?

Thanks,

Load in running status:

Preview data:

Copy data activity input :

{
    "source": {
        "type": "SqlServerSource",
        "partitionOption": "None",
        "datasetSettings": {
            "annotations": [],
            "type": "SqlServerTable",
            "schema": [],
            "typeProperties": {
                "schema": "dbo",
                "table": "__RefactorLog",
                "database": "DataWarehouse"
            },
            "externalReferences": {
                "connection": "c87c9225-17e4-41b3-86ec-ab4e602871e1"
            }
        }
    },
    "sink": {
        "type": "LakehouseTableSink",
        "tableActionOption": "OverwriteSchema",
        "partitionOption": "None",
        "datasetSettings": {
            "annotations": [],
            "linkedService": {
                "name": "test",
                "properties": {
                    "annotations": [],
                    "type": "Lakehouse",
                    "typeProperties": {
                        "workspaceId": "b18b9c0f-4256-4e80-9cef-906c265b8983",
                        "artifactId": "47202686-375f-4e4c-bdc0-68a665f1db55",
                        "rootFolder": "Tables"
                    }
                }
            },
            "type": "LakehouseTable",
            "schema": [],
            "typeProperties": {
                "table": "test"
            }
        }
    },
    "enableStaging": false,
    "translator": {
        "type": "TabularTranslator",
        "mappings": [
            {
                "source": {
                    "name": "OperationKey",
                    "type": "Guid",
                    "physicalType": "uniqueidentifier"
                },
                "sink": {
                    "name": "OperationKey",
                    "physicalType": "string"
                }
            }
        ],
        "typeConversion": true,
        "typeConversionSettings": {
            "allowDataTruncation": true,
            "treatBooleanAsNumber": false
        }
    }
}

Result in Destination lakehouse:

[TomT131]

Hi @v-cboorla-msft,

I am experiencing the same issue

SQL Onprem (Gateway 3000.218.9) copy data activity hangs on "In Progress".
Dataflow works fine through same gateway.

Have tried using Edge browser and no difference.
Would be great to get this resolved?
Thanks 

[v-cboorla-msft]

Hi @GLWY 

The internal team has replied has follows:

Solution: This will be fixed in the official release of OPDG by starting week of July.

You can follow this workaround untill it get fixed:
1. Stop the on-prem gateway service
2. replace ParquetSharpNative.dll under this folder "C:\Program Files\On-premises data gateway\FabricIntegrationRuntime\5.0\Gateway" with another version that can be downloaded from:https://adfxingwewu3.z11.blob.storage.azure.net/bits/ParquetSharpNative.dll?sp=r&st=2024-05-16T03:49....
3. restart the service in #1

I hope this information helps.

Thank you.

[GLWY]

Hi @v-cboorla-msft ,

This is good news, thanks for quick turnaround.

However I cannot access the file. I am getting error: 

Can you please advise.

Thanks,

[v-cboorla-msft · Accepted solution]

Hi @GLWY 

Could you please try installing the latest on-prem data gateway version (in case if you are using older version) and let me know if the issue still persists. You can download the latest version here:

Latest version : Download On-premises data gateway from Official Microsoft Download Center

I hope this information helps.

Thank you.

[v-cboorla-msft]

Hi @GLWY 

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others .
If you have any question relating to the current thread, please do let us know and we will try out best to help you.
In case if you have any other question on a different issue, we request you to open a new thread.

Thank you.

[GLWY]

Hi @v-cboorla-msft ,

We have installed the latest version of the Power BI gateway (JUNE 2024 release) and it fixed the issue. 

Thank you for your help. Appreciate it.

[GLWY]

Hi,

Thanks for the response but I cannot see how cache or cookies in the web browser affects how the pipeline run in Power BI Service.

The monitor hub clearly shows that the pipeline is running but not moving any data across

[v-cboorla-msft]

Hi @GLWY 

Thanks for using Microsoft Fabric Community.

There might be a chance of an intermittent issue could you please follow the below steps that might help you.

Temporary Glitch: Clearing cookies and caches can sometimes resolve temporary glitches within the application that might be causing the issue.

Corrupted Data: In rare cases, corrupted data stored in the browser's cache related to Microsoft Fabric might be causing the issue. Clearing the cache removes this potentially problematic data.

Hard Refresh: A hard refresh bypasses the cached version of the webpage and forces the browser to download the latest version from the server. Press Ctrl+Shift+R (Windows) or Cmd+Shift+R (Mac).

If the issue still persist, please do let us know. Glad to help.

I hope this information helps.

Thank you.