Could not read message metadata

[girishtharwani2]

Hi,

I am using stored procedure to load data from Stg to DWH in Snowflake.

The stored procedure worked in snowflake UI however same stored procedure is not working in fabric pipeline.

Interesting thing is even though pipeline failed, the data in DWH is loaded.

Can someone assist.

Thanks,

Girish

[girishtharwani2]

Thanks @v-karpurapud for the quick assistance.

It is expected that SP will return counts of rows updated and inserted and it has been working for other tables.

I use lookup activity to execute the SP.

Your help will be appreciated.

Thanks,

Girish

[v-karpurapud]

Hi @girishtharwani2 

The Lookup activity in Microsoft Fabric pipelines expects the output to be a single-row result set with supported scalar datatypes. Although it works for other tables, it fails with this stored procedure because the output format doesn't match the required structure, even though the DWH updates are successful.

Modify the output to return a single SELECT statement with two columns: rows_inserted and rows_updated. This will enable the Lookup activity to process the response correctly.

For Example:

CREATE OR REPLACE PROCEDURE DWH.LOAD_FROM_STG()

RETURNS TABLE (rows_inserted NUMBER, rows_updated NUMBER)

LANGUAGE SQL

AS

$$

DECLARE

v_inserted NUMBER;

v_updated  NUMBER;

BEGIN

 -- logic --

 INSERT INTO ...;

LET v_inserted = SQLROWCOUNT;

UPDATE ...;

LET v_updated = SQLROWCOUNT;

RETURN TABLE (SELECT v_inserted, v_updated);

END;

$$;

Should you have any additional questions, please feel free to reach out, and we will be glad to assist you further.

Regards,

Microsoft Fabric Community Support Team.

[v-karpurapud]

Hi @girishtharwani2 

Thank you for your question on the Microsoft Fabric Community Forum.

The error occurs because Microsoft Fabric pipelines are unable to process result sets returned by Snowflake stored procedures. While the procedure executes successfully in Snowflake and loads the data as expected, the pipeline fails when attempting to parse the output. The message "arrow/ipc: could not read message metadata: unexpected EOF" appears when the stored procedure returns a result set or status message that the connector cannot interpret.

To resolve this, ensure that the stored procedure does not return any result sets. Remove or suppress any SELECT statements used for status messages, and consider returning a simple scalar value (such as 'OK') or no value at all. If the existing procedure must remain unchanged for other use cases, you may create a wrapper procedure that calls the original and only returns a basic status. Alternatively, use a pipeline activity that executes the procedure without attempting to read its results.
 
By preventing the procedure from returning a result set, the Fabric pipeline will execute successfully and load your data into the warehouse without encountering parsing errors.

I hope this information is helpful. . If you have any further questions, please let us know. we can assist you further.

Regards,

Microsoft Fabric Community Support Team.

[girishtharwani2 · Accepted solution]

Thanks for the suggestions.

The issue was, snowflake was returning column which has length more than 100000 and lookup can not return lengths till 99000.

Trimmed the message to 99000 in snowflake and it worked.

Thanks,

Girish