Forum Discussion
Microsoft EmployeeCopy Data into a new Kusto table
I ended up replacing the original KQL activity with the following sequence:
- A Lookup activity to retrieve the schema of the incoming table
- A KQL activity to drop the table if it already exists
- A KQL activity to create an empty table
- A Copy Data activity to populate the table with content
While this approach isn't fully compliant—since not all activities are workspace identity (WI) based—it’s still an improvement over the previous setup. At least the activities that interact with external databases, namely Lookup and Copy Data, are WI-based.
Community SupportHi Jing1018
Thank you for reaching out to the Microsoft Fabric Community Forum.
The “Table doesn’t exist” error appears because the Copy Data activity in Fabric pipelines uses Kusto ingestion commands such as .ingest into or .ingest-from-storage, which require the target table to be present beforehand. These commands do not create tables automatically and are mainly intended for prototyping, not production. Additionally, the KQL activity does not currently support Managed Identity (MI), so control commands like .create table or .set-or-replace cannot be executed within it. To address this, the table should be created in advance. A recommended MI-compatible solution is to use a Web (REST) activity in the pipeline, authenticated via MI, to call the Kusto management endpoint and execute table creation commands. Once the table exists, named ingestion mappings can provide schema flexibility during ingestion. The most straightforward MI-based approach is to run a Web activity for table creation, followed by the Copy Data activity for ingestion.
I hope this information is helpful. . If you have any further questions, please let us know. we can assist you further.
Regards,
Microsoft Fabric Community Support Team.
Microsoft EmployeeHi v-karpurapud ,
Thank you for getting back to me. I have two follow-up questions:
1. According to the official documentation: Authenticate with Microsoft Fabric workspace identity - Microsoft Fabric | Microsoft Learn, MI authentication is currently supported only in three activities: Copy, Lookup, and GetMetadata. Are you suggesting that Web activity also supports MI authentication, even though it's not explicitly mentioned in the documentation?
2. I assume that the Kusto table created via the Web activity won’t reflect the actual data schema, since we haven’t connected to the real data source yet. You mentioned that named ingestion mappings can help provide schema flexibility. I’m not very familiar with this approach—could you please share more insights on the best way to construct such mappings in this scenario? For example, if the Kusto table has a dummy column of type string, but the incoming data could contain any number of columns with varying types, how should we handle that?
Thanks in advance!