topic Re: Write data from Web connector to lakehouse table in Data Engineering

Write data from Web connector to lakehouse table

amaaiia — Tue, 04 Feb 2025 16:17:00 GMT

Hi,

I have one Web Connector connected to a web service.

When I cerate a Web activity in a Data Pipeline and I use this connection, it retrieves me data as follows (json objects list):

Where each object should be a row in the table.

If I try to use the same conneciton in COPY activity, I get an error in the preview of the data:

Rest call failed with client error, status code 404 NotFound, please check your activity settings. Response: <html> <head><title>404 Not Found</title></head> <body> <center><h1>404 Not Found</h1></center> <hr><center>nginx</center> </body> </html>

Any ideas of how to ingest data from web connector into lakehouse table?

Re: Write data from Web connector to lakehouse table

nilendraFabric — Tue, 04 Feb 2025 18:42:38 GMT

Hello amaaiia

Save the output of the Web Activity as a variable:
• Use a Set Variable Activity after the Web Activity.
• Assign the output of the Web Activity to a variable using dynamic content, such as:

@activity('WebActivityName').output

In the Copy Activity settings:
• Source Tab:
• Select “Dynamic Content” as your source type. @variables('webData')

• Map it to the variable containing your JSON data.

Re: Write data from Web connector to lakehouse table

amaaiia — Wed, 05 Feb 2025 07:28:45 GMT

When I set connection as variable content, it asks me in Source to select a lakehouse table. The source shouldn't be a lakehouse table but the content of the variable itself.

Re: Write data from Web connector to lakehouse table

nilendraFabric — Wed, 05 Feb 2025 08:45:18 GMT

Use dynamic content optipn from the dropdown and set @variables('webData')

Re: Write data from Web connector to lakehouse table

nilendraFabric — Wed, 05 Feb 2025 08:45:26 GMT

Use dynamic content optipn from the dropdown and set @variables('webData')

Re: Write data from Web connector to lakehouse table

amaaiia — Wed, 05 Feb 2025 11:39:47 GMT

If you see the screenshot, this is what I've done

Re: Write data from Web connector to lakehouse table

amaaiia — Wed, 05 Feb 2025 11:40:50 GMT

Solved.

RelativePath for Web activity starts with slash, but relative path in COPY activity doesn't start with slash. That's why it said NOT FOUND

Re: Write data from Web connector to lakehouse table

nilendraFabric — Wed, 05 Feb 2025 11:53:26 GMT

No screenshots visible