topic Re: Autoloader Feature in Data Engineering

Autoloader Feature

poojitha_yeluru — Tue, 07 May 2024 06:35:38 GMT

As we know in Databricks there is Autoloader Feature , Is there a way in fabric where it will behave same as autoloader

Re: Autoloader Feature

Anonymous — Tue, 07 May 2024 06:45:17 GMT

Hi @poojitha_yeluru ,

Thanks for using Fabric Community.
At this time, we are reaching out to the internal team to get some help on this .
We will update you once we hear back from them.

Re: Autoloader Feature

Anonymous — Fri, 10 May 2024 07:15:43 GMT

Hi @poojitha_yeluru ,

Unfortunately we don't have any similar feature in Fabric that behave same like Databricks - autoloader.

Appreciate if you could share the feedback on our feedback channel. Which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.

Hope this helps. Please let me know if you have any further queries.

Re: Autoloader Feature

Anonymous — Fri, 10 May 2024 06:50:26 GMT

Hi @poojitha_yeluru ,

We haven’t heard from you on the last response and was just checking back to see if your query was answered.
Otherwise, will respond back with the more details and we will try to help .

Thanks

Re: Autoloader Feature

poojitha_yeluru — Fri, 10 May 2024 07:13:26 GMT

Hi @Anonymous , u had mentioned Autodiscovery may I know what its about.

Re: Autoloader Feature

Anonymous — Fri, 10 May 2024 07:18:00 GMT

Hi @poojitha_yeluru ,

Apologies for the confusion it's a typo error. I have editied my previous comment.
Its by mistake I mentioned auto discovery, inplace of autoloader.

Re: Autoloader Feature

poojitha_yeluru — Fri, 10 May 2024 07:30:37 GMT

Hi @Anonymous , In Fabric how can I approach this scenario, I have ADLS Gen2 account there I have a folder name Employee it contains 10 parquet files , with the help of notebook I have read the folder and I created the table, The next day the new parquet file is added when I run the notebook It should only read the new file not all older file how can I acheive this in Ms fabric.

Re: Autoloader Feature

Anonymous — Mon, 13 May 2024 09:59:24 GMT

Hi @poojitha_yeluru ,

I am adding some usefull docs that might give you some idea on how to deal with this scenario.

Docs to refer -
(27) Fabric Lakehouse merge operation, schema evolution | LinkedIn
Solved: Incremental write - Databricks - 14562

Hope this is helpful.

Re: Autoloader Feature

Anonymous — Tue, 14 May 2024 12:38:39 GMT

Hi @poojitha_yeluru ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
In case if you have any resolution please do share that same with the community as it can be helpful to others .
Otherwise, will respond back with the more details and we will try to help .
Thanks