Lakehouse error: "An internal error has occurred while applying table changes to SQL"

[RishikeshPashte]

I have two EventStreams setup that fetches the data from App Insights and evetually dumps the selected set of columns into the respective tables created in OneLake but, for some reason it recently started throwing the error "An internal error has occurred while applying table changes to SQL" for both the tables

The latest data is successfully fetched from the App Insights to the EventStream:

But, the tables are returning the below error:

NOTE: Ealrier, the flow was working flawlessly and the table contains data

[v-achippa]

Hi @RishikeshPashte,

Thank you for reaching out to Microsoft Fabric Community.

The error here indicates an issue when the SQL Analytics endpoint tries to update the tables in OneLake. Since the flow was working previously and the EventStream is successfully fetching data, the issue is likely related to a schema mismatch or a temporary issue with the SQL endpoint.

• Compare the schema of the data in the EventStream with the schema of the target table in OneLake. If there is a mismatch for example like new columns or data type changes, either modify the EventStream to match with the table schema or update the table schema to match the incoming data.
• Create a new table in OneLake with the same schema and redirect the EventStream output to this new table. If it works the issue might be with the original table, so migrate the existing data to the new table and update your pipeline to use the new table.
• In the Fabric workspace, restart the SQL Analytics endpoint or refresh the Lakehouse to clear any temporary issues.
• And also ensure that your account has the necessary permissions to write the table and you have not exceeded any fabric capacity limits, check this in the fabric workspace settings under Capacity.

If this post helps, then please consider Accepting as solution to help the other members find it more quickly, don't forget to give a "Kudos" – I’d truly appreciate it! 

Thanks and regards,

Anjan Kumar Chippa

[v-achippa]

Hi @RishikeshPashte,

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution I have provided for the issue worked? or let us know if you need any further assistance.
If my response addressed, please mark it as "Accept as solution" and click "Yes" if you found it helpful.

Thanks and regards,

Anjan Kumar Chippa

[v-achippa]

Hi @RishikeshPashte,

We wanted to kindly follow up to check if the solution I have provided for the issue worked.
If my response addressed, please mark it as "Accept as solution" and click "Yes" if you found it helpful.

Thanks and regards,

Anjan Kumar Chippa

[RishikeshPashte]

Hey @v-achippa ,

I created a new table, dumped the previous data in the new table and again set up an EventStream for the new table but, it's again giving me the same error as mentioned above

Also, when I query the table via the SQL endpoint it doesn't show me the latest data whereas, when I query the table using a Fabric notebook it fetches the lata data from the table. Please find the screenshots for the same below:

Queried using SQL Endpoint:

Queried using Fabric notebook:

[v-achippa · Accepted solution]

Hi @RishikeshPashte,

The issue here is that the EventStream writes are succeeding but SQL endpoint queries not reflecting the latest data, this looks like a metadata sync issue between the Spark engine and the SQL analytics endpoint in microsoft Fabric.

Create a new Lakehouse or new table and use a Fabric notebook to manually copy the data using Spark. This ensures that the metadata is clean and synchronized correctly with the SQL endpoint. Once done, refresh the SQL endpoint and re run the query, this should now reflect the latest data.

If this post helps, then please consider Accepting as solution to help the other members find it more quickly, don't forget to give a "Kudos" – I’d truly appreciate it! 

Thanks and regards,

Anjan Kumar Chippa

[v-achippa]

Hi @RishikeshPashte,

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution I have provided for the issue worked.
If my response addressed, please mark it as "Accept as solution" and click "Yes" if you found it helpful.

Thanks and regards,

Anjan Kumar Chippa