Forum Discussion
Could not read message metadata
Hi,
I am using stored procedure to load data from Stg to DWH in Snowflake.
The stored procedure worked in snowflake UI however same stored procedure is not working in fabric pipeline.
Interesting thing is even though pipeline failed, the data in DWH is loaded.
Can someone assist.
Thanks,
Girish
Thanks for the suggestions.
The issue was, snowflake was returning column which has length more than 100000 and lookup can not return lengths till 99000.
Trimmed the message to 99000 in snowflake and it worked.
Thanks,
Girish
6 Replies
Thanks v-karpurapud for the quick assistance.
It is expected that SP will return counts of rows updated and inserted and it has been working for other tables.
I use lookup activity to execute the SP.
Your help will be appreciated.
Thanks,
Girish
The Lookup activity in Microsoft Fabric pipelines expects the output to be a single-row result set with supported scalar datatypes. Although it works for other tables, it fails with this stored procedure because the output format doesn't match the required structure, even though the DWH updates are successful.
Modify the output to return a single SELECT statement with two columns: rows_inserted and rows_updated. This will enable the Lookup activity to process the response correctly.
For Example:
CREATE OR REPLACE PROCEDURE DWH.LOAD_FROM_STG()
RETURNS TABLE (rows_inserted NUMBER, rows_updated NUMBER)
LANGUAGE SQL
AS
$$
DECLARE
v_inserted NUMBER;
v_updated NUMBER;
BEGIN
-- logic --
INSERT INTO ...;
LET v_inserted = SQLROWCOUNT;
UPDATE ...;
LET v_updated = SQLROWCOUNT;
RETURN TABLE (SELECT v_inserted, v_updated);
END;
$$;
Should you have any additional questions, please feel free to reach out, and we will be glad to assist you further.
Regards,
Microsoft Fabric Community Support Team.
Thank you for your question on the Microsoft Fabric Community Forum.
The error occurs because Microsoft Fabric pipelines are unable to process result sets returned by Snowflake stored procedures. While the procedure executes successfully in Snowflake and loads the data as expected, the pipeline fails when attempting to parse the output. The message "arrow/ipc: could not read message metadata: unexpected EOF" appears when the stored procedure returns a result set or status message that the connector cannot interpret.
To resolve this, ensure that the stored procedure does not return any result sets. Remove or suppress any SELECT statements used for status messages, and consider returning a simple scalar value (such as 'OK') or no value at all. If the existing procedure must remain unchanged for other use cases, you may create a wrapper procedure that calls the original and only returns a basic status. Alternatively, use a pipeline activity that executes the procedure without attempting to read its results.
By preventing the procedure from returning a result set, the Fabric pipeline will execute successfully and load your data into the warehouse without encountering parsing errors.
I hope this information is helpful. . If you have any further questions, please let us know. we can assist you further.Regards,
Microsoft Fabric Community Support Team.
Thanks for the suggestions.
The issue was, snowflake was returning column which has length more than 100000 and lookup can not return lengths till 99000.
Trimmed the message to 99000 in snowflake and it worked.
Thanks,
Girish
Hi,
We are using the Power BI - Fabric Capacity Metrics
And in one of the workspace it says we have consumed more GB.
How to check the details of it and where exactly it is consumed.
Hi girishtharwani2
Sorry for delay response. The Fabric Capacity Metrics app shows storage at the workspace level, so the value is aggregated and does not point to the exact source of the consumption right away.
To see which artifact is driving the usage, open the Storage page in the Capacity Metrics app and drill down from the workspace to the individual items. That can help you determine whether a Lakehouse, Warehouse, Semantic Model, or KQL Database is contributing to the storage footprint.
If you need a deeper view, use the drillthrough options in the Capacity Metrics app or export the underlying data for a more detailed breakdown.Please feel free to contact us if you have any further questions.
Regards,
Microsoft Fabric Community Support Team.
