Forum Discussion
Source option missing
Hi smpa01,
Thank you for reaching out to the Microsoft Fabric Community Forum. The solutions provided by the datacoffee, are correct according to the thread.
And, I have identified few alternative workarounds that may help resolve the issue. Please follow these steps:
You are correct. When you create a KQL database directly from the Event house or Fabric UI, the "Source" step does not appear in the "New Table" wizard. This behaviour is intentional.
Navigate to the workspace settings by clicking the workspace name and selecting "Settings." Under "Workspace type," you should see “Real-Time Intelligence” listed. If this is not visible, you are not in an RTI workspace. Only RTI workspaces support streaming scenarios with the full table wizard, including the Source step.
Tables created manually from One Lake files will not automatically refresh when the files are updated. This information is crucial for setting expectations and directing them towards the appropriate methods, such as Event Streams or Notebooks.
Option A: Use Event Streams- Create a table from Event Streams > Add Destination > New Table.
- This enables the full wizard with Source, supports auto-refreshing data when new events arrive.
Option B: Use a Notebook to Manage Refresh
- Use a notebook to drop and recreate/update the table regularly.
- This is more manual but gives you control if Event Streams are not part of your current setup.
Kindly refer to the below documentation links:
Get data from Event stream - Microsoft Fabric | Microsoft Learn
Create an event stream in Microsoft Fabric - Microsoft Fabric | Microsoft Learn
Real-Time Intelligence in Microsoft Fabric documentation - Microsoft Fabric | Microsoft Learn
If this post helps, then please give us ‘Kudos’ and consider Accept it as a solution to help the other members find it more quickly.
Thank you for using Microsoft Community Forum.
Yes - the KQL engine will "listen" to the underlying parquet files and update accordingly 😊
Thanks for the response. That's great and sorry to be a pain.
I need to clarify few more aspects beofre I can out this in PROD.
I am not being ble to create a shortcut in KQL databse for a file residing in lakehouse that contains a output.json like this
[
{
"dev": "dev_1",
"id": 1,
"comment": "some_comment"
},
{
"dev": "dev_2",
"id": 2,
"comment": "some_other_comment"
}
]I am getting
Failed to create external table "st_2" with error: Action: command cannot be executed due to an invalid argument: argument: 'compressed'; reason: "Server is not configured for compression. Compression or compression properties aren't allowed for this schema table. Either drop the compression properties or use one of the following formats: csv, tsv, json."Oh - I had the idea that it was a parquet file.
then you can't use a shortcut.
let me try to understand the situation.
you have a json file from which you want to have the data in a KQL database.
how is that file generated? Can it be send to an eventhub from the source instead?
if so, then you have a perfect option to use the Eventstream to get the data into the KQL database and from there, do what you need with the data.
let me know - we can surely figure out a good solution for you