Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
2 years ago

Fabric Data Factory Copy Data Pipeline - recursive excel file processing

Have a directory of 100s of excel files, all with one worksheet, and all having the same column structure but the worksheet name is different in each file (its is the same as the actual file name) Re...
data engineering
Data Integration
one lake
v-cboorla-msft's avatar
v-cboorla-msft
Icon for Microsoft Employee rankMicrosoft Employee
2 years ago

Hi Anonymous 

 

Thanks for using Microsoft Fabric Community.

Apologies for the inconvenience that you are facing here.

As I understand that you are trying to load the contents of multiple Excel files with varying worksheet names into a single table using Microsoft Fabric pipeline.

I tried to repro the same scenario and was successful in executing the copy data activity in pipeline without any difficulty. Please find the screenshots for reference.

For doing repro I took 10 sample excel files which are in a directory named as SampleData. Just for reference I took only 3 excel data files and all the excel files have the same data.

Source data:

Copy activity configuration:

Data preview:

Execution details:

Destination:

Here are some common reasons for this error and potential solutions that might help.

  • Verify that the column name you are referencing in your query matches the actual column name in the Excel worksheet.
  • If your Excel column names contain special characters (such as spaces, hyphens, or other non-alphanumeric characters), it can lead to issues.
  • Sometimes refreshing the query or unmerging previously merged columns can resolve issues.

I hope this information helps. Please do let us know if the issue still persists. Glad to help.

 

Thanks.

  • v-cboorla-msft's avatar
    v-cboorla-msft
    Icon for Microsoft Employee rankMicrosoft Employee
    2 years ago

    Hi Anonymous 

     

    We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
    In case if you have any resolution please do share that same with the community as it can be helpful to others.
    Otherwise, will respond back with the more details and we will try to help.


    Thanks.

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Thanks for taking time to respond, good to know that it works as I had expected.  Will check your recommendations and revert back

    • v-cboorla-msft's avatar
      v-cboorla-msft
      Icon for Microsoft Employee rankMicrosoft Employee
      2 years ago

      Hi Anonymous 


      We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
      Otherwise, will respond back with the more details and we will try to help.


      Thanks.

      • v-cboorla-msft's avatar
        v-cboorla-msft
        Icon for Microsoft Employee rankMicrosoft Employee
        2 years ago

        Hi Anonymous 

         

        We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
        If you have any question relating to the current thread, please do let us know and we will try out best to help you.
        In case if you have any other question on a different issue, we request you to open a new thread.


        Thanks.

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    During first load, I can able to  load the data. However, for second run I added one extra colum to one of the sheet, it is not detecting that column after the copy data activity completion.

     

    Do I need to do anything with the settings?