tutorial requests

14 Topics

Gen2 Dataflow Refresh Failure - Staging Lakehouse Not Found
I am unable to successfully refresh a Gen2 Dataflow configured to fetch data from SQL Server and store it in a Lakehouse. The dataflow creation process appears to complete, but when attempting to refresh the dataflow, it consistently fails with a staging lakehouse error Error Details: Error Message: "There was a problem refreshing the dataflow: 'Staging lakehouse for dataflow with id ... was not found.'" Error Code: StagingLakehouseMissingError Request ID: 00000000-0000-0000-0000-000000000000 Steps Taken: Created a new Gen2 Dataflow Configured data source connection to SQL Server Set destination to Lakehouse Attempted to refresh the dataflow I would appreciate hearing if others have experienced this problem and successfully resolved it
Solved
bunny96
6 months ago Place Dataflow
7.6KViews
0likes
4Comments
Get email attachment with receivedtime
Hello, I'm learning Dataflow and Queries and I'm trying to load into a Lake an email with an excel attachment with the email received date. I cannot find the way to add the email received date as a column of the excel. Thanks for your help.
Solved
Comprateur
9 months ago Place Dataflow
3.5KViews
0likes
9Comments
DataFlow(Gen2) managed by multiple users
When DataFlow(Gen2) is used for work, it is necessary for multiple people to manage it together. However, at present, only one owner is able to check the contents, let alone touch them. Does this not meet the business needs at all? Or is there a proper way to do this that I just don't know about? If anyone knows, I would be very grateful if you could enlighten me.
Solved
murat-to
10 months ago Place Dataflow
5.5KViews
0likes
5Comments
Would I use Dataflow for this use case? Or is it easier to use notebook for transformations
Hello, I am very new to Azure and MS services so I hope this is the right place to ask. I have a list of csv files for 5 different tables that are generated daily (5 new csvs daily) with new data and placed in our Azure Blob storage account. I want to create a pipeline that automates the ETL process. Where the new csv files are processed each day then appended onto the tables that I have created in my lake house. If I create a dataflow, do I need to select Azure blobs or Azure data lake storage? If I connect to the storage account via either method, I am given just a single query with a table that lists all the contents of the container (all csvs, all subfolders each csv is located in etc). If I open up one of the csv files and do transformations on it and then set the destination to append onto the table I have already created. How does Dataflow know to get all the files from that folder to process it using the same transformations that I had applied already. Also how does it know which datasets it would need to apply the transfomations to based on which set of data I am transforming? The csv files always have the same column headers etc, just different rows of data. There are no Youtube videos I can find that show situations where you want to automate an ETL process using dataflows and pipelines where you would want to be processing a different csv file each day for multiple tables. Is this possible with dataflows or do I just need to use a notebook instead? Thank You, Paul S
Solved
SamsonYoung20
1 year ago Place Dataflow
2.6KViews
0likes
5Comments
JSON data structure reconstruction
Hi Everyone, I am working on an Azure Cosmos DB migration where I need to transform the structure of documents in the "Notes" container. While I have identified most of the migration steps, I am currently facing a challenge with structural conversion and parsing of a JSON array field inside Azure Data Factory (ADF) Data Flow. Problem Context: I need to convert the "NoteDetails" array field from the original document format to a new normalized structure. The transformation involves flattening the array and restructuring it into a key-value format within a new field called "NormalizedNoteDetails". Original Document (Current Cosmos DB Structure): { "id": "123", "ownerId" : "1234", "noteCategory" : "somecategory", "NoteDetails": [ { "Key": "PropertyNameKey1", "Value": ["prop1of1", "prop2of1"] }, { "Key": "PropertyNameKey2", "Value": ["prop1of2", "prop2of2"] }] } Expected Transformed Structure (Target Format in Cosmos DB): { "id": "123", "ownerId" : "1234", "noteCategory" : "somecategory", "normalizedNoteDetails": { "PropertyNameKey1": ["prop1of1", "prop2of1"], "PropertyNameKey2": ["prop1of2", "prop2of2"] } } Specific Question: How can I efficiently parse and transform the "Notes" JSON array field inside an ADF Data Flow activity to achieve the new structure? I have tried flatten and pivot with no success but I am newbie to the ADF. A concrete example would help me tremendously. Thanks.
Solved
cyberman
1 year ago Place Dataflow
1.5KViews
0likes
1Comment
Facts and Dims
Hi everyone, How does everyone approach modeling in Dataflows Gen2? For our client it seems easier to do it with a dataflow because they are not on a sufficient SQL or notebook level. Would you create a dataflow for every fact/dim? It makes sense for me to keep it seperate. But i'm also thinking - if multiple facts/dims use the same source tables that we might combine them into one dataflow Thanks
Solved
dpjatochh
1 year ago Place Dataflow
6.1KViews
1like
1Comment
Installing a Data Gateway
I'm trying to connect to a SAP database using Dataflows Gen 2, however it requests an On-premises Data Gateway. I've read a lot about these data gateways. I need to install it on a computer that is ON all the time, which is something that I do not have. I'm assuming that the ideal solution would be to install the gateway on the same machine that is hosting the SAP database? What if I don't have any way of doing this? Is there any solution where Microsoft Fabric can 'host' these data gateways so we don't have to worry about installing in some computers just for the gateway? Thanks in advance
SergioSurfer
1 year ago Place Dataflow
3.5KViews
0likes
5Comments
Ms Fabric SAP Hana Connector Issue
Hi all, Background: I have been successful in connecting to SAP BW (hosted in cloud) with Ms Fabric using Dataflow Gen 2 connector. However, I am now working on connecting a SAP HANA Database (hosted by SAP in cloud) with Microsoft Fabric using the Dataflow Gen 2 connector, and have issues configuring the ODBC connector and the Online Power Query connector as follows: 1. ODBC Connector Issue: I have downloaded the ODBC client form SAP development tools and have configured it as follows: However the Test Connection is failing with the following error: 2. Connect to SAP Hana using Dataflow Gen2 Issue: Going forward when I try to connect to SAP Hana using Dataflow Gen2 in MS Fabric after creating a gateway, I get the following error. Error Message: ‘An exception occurred: ODBC: ERROR [08S01] [SAP AG][LIBODBCHDB DLL][HDBODBC] Communication link failure;-10709 Connection failed (RTE:[89006] System call 'connect' failed, rc=10061:No connection could be established because the target computer refused the connection {**.***.***.**:30015} (**.***.***.**:30015)) (Session ID: 16d99661-7b85-41e9-9a4e-2e345c90ce50, Region: germany)’ I currently understand that the Online Power Query error is occurring because the ODBC connection could not be established successfully. Therefore, any help with in resolving the ODBC error and the Dataflow Gen2 error this issue would be highly appreciated. Warm Regards, Kartik
Solved
Kartik_K
1 year ago Place Dataflow
4.1KViews
0likes
1Comment
We reached the end of the buffer - Dataflow
I have the following dataflow containing : The Entity and the GL/Mapping (I am sharing only screenshots for Entity since the GL/Mapping has the same) are failing to retrieve data when I check them, the sources seems to be pointing to inexisting dataflows : When I check the workspace it is existing but the Dataflow I get this message : The workspace pointing to the dataflow exists but the dataflow is missing but the error message is different : I get this error message : We reached the end of the buffer section Section1; shared #"AFAS Openingsbalans" = let /*#"2018" = let Source = Sql.Database("XXXXXXXXXXX.database.windows.net", "XXXXXXXXXXX"), afas_vw_FinancieleMutaties_Current = Source{[Schema="afas",Item="vw_getAllEmps"]}[Data], #"Filtered Rows" = Table.SelectRows(afas_vw_FinancieleMutaties_Current, each ([Jaar] < 2018) and ([Administratie_code] <> 1) and ([Administratie_code] <> 2) and [Dagboek_code] = "99"), #"Removed Other Columns" = Table.SelectColumns(#"Filtered Rows",{"Administratie_code", "Rekeningnummer", "Rekening_omschrijving", "Bedrag_saldo", "Verzamelen_op_rekeningnummer"}), #"Grouped rows" = Table.Group(#"Removed Other Columns", {"Administratie_code", "Rekeningnummer", "Rekening_omschrijving", "Verzamelen_op_rekeningnummer"}, {{"Bedrag_Saldo", each List.Sum([Bedrag_saldo]), type number}}), #"Added Year" = Table.AddColumn(#"Grouped rows", "Year", each 2018), #"Added Period" = Table.AddColumn(#"Added Year", "Period", each 0), #"Renamed columns" = Table.RenameColumns(#"Added Period", {{"Bedrag_Saldo", "Amount"}}) in #"Renamed columns", #"2019" = let Source = Sql.Database("XXXXXXXXXXX.database.windows.net", "XXXXXXXXXXX"), afas_vw_FinancieleMutaties_Current = Source{[Schema="afas",Item="vw_getAllEmps"]}[Data], #"Filtered Rows" = Table.SelectRows(afas_vw_FinancieleMutaties_Current, each ([Jaar] < 2019) and ([Administratie_code] <> 1) and ([Administratie_code] <> 2)), #"Removed Other Columns" = Table.SelectColumns(#"Filtered Rows",{"Administratie_code", "Rekeningnummer", "Rekening_omschrijving", "Bedrag_saldo", "Verzamelen_op_rekeningnummer"}), #"Grouped rows" = Table.Group(#"Removed Other Columns", {"Administratie_code", "Rekeningnummer", "Rekening_omschrijving", "Verzamelen_op_rekeningnummer"}, {{"Bedrag_Saldo", each List.Sum([Bedrag_saldo]), type number}}), #"Added Year" = Table.AddColumn(#"Grouped rows", "Year", each 2019), #"Added Period" = Table.AddColumn(#"Added Year", "Period", each 0), #"Renamed columns" = Table.RenameColumns(#"Added Period", {{"Bedrag_Saldo", "Amount"}}) in #"Renamed columns", #"2020" = let Source = Sql.Database("XXXXXXXXXXX.database.windows.net", "XXXXXXXXXXX"), afas_vw_FinancieleMutaties_Current = Source{[Schema="afas",Item="vw_getAllEmps"]}[Data], #"Filtered Rows" = Table.SelectRows(afas_vw_FinancieleMutaties_Current, each ([Jaar] < 2020) and ([Administratie_code] <> 1) and ([Administratie_code] <> 2)), #"Removed Other Columns" = Table.SelectColumns(#"Filtered Rows",{"Administratie_code", "Rekeningnummer", "Rekening_omschrijving", "Bedrag_saldo", "Verzamelen_op_rekeningnummer"}), #"Grouped rows" = Table.Group(#"Removed Other Columns", {"Administratie_code", "Rekeningnummer", "Rekening_omschrijving", "Verzamelen_op_rekeningnummer"}, {{"Bedrag_Saldo", each List.Sum([Bedrag_saldo]), type number}}), #"Added Year" = Table.AddColumn(#"Grouped rows", "Year", each 2020), #"Added Period" = Table.AddColumn(#"Added Year", "Period", each 0), #"Renamed columns" = Table.RenameColumns(#"Added Period", {{"Bedrag_Saldo", "Amount"}}) in #"Renamed columns", #"2021" = let Source = Sql.Database("XXXXXXXXXXX.database.windows.net", "XXXXXXXXXXX"), afas_vw_FinancieleMutaties_Current = Source{[Schema="afas",Item="vw_getAllEmps"]}[Data], #"Filtered Rows" = Table.SelectRows(afas_vw_FinancieleMutaties_Current, each ([Jaar] < 2021) and ([Administratie_code] <> 1) and ([Administratie_code] <> 2)), #"Removed Other Columns" = Table.SelectColumns(#"Filtered Rows",{"Administratie_code", "Rekeningnummer", "Rekening_omschrijving", "Bedrag_saldo", "Verzamelen_op_rekeningnummer"}), #"Grouped rows" = Table.Group(#"Removed Other Columns", {"Administratie_code", "Rekeningnummer", "Rekening_omschrijving", "Verzamelen_op_rekeningnummer"}, {{"Bedrag_Saldo", each List.Sum([Bedrag_saldo]), type number}}), #"Added Year" = Table.AddColumn(#"Grouped rows", "Year", each 2021), #"Added Period" = Table.AddColumn(#"Added Year", "Period", each 0), #"Renamed columns" = Table.RenameColumns(#"Added Period", {{"Bedrag_Saldo", "Amount"}}) in #"Renamed columns", */
Solved
AmiraBedh
2 years ago Place Dataflow
1.4KViews
0likes
2Comments
Connecting fabric with On-Premises SSAS Cubes
I've been struggling to find a straightforward solution for connecting Fabric or Azure data factory to an on-premises SSAS cube. I've searched extensively, but all I come across are guides on connecting to SQL Server or discussions stating that connecting to SSAS from Fabric isn't currently available. I'm unsure if I'm pursuing the right approach. Any guidance would be greatly appreciated. Would it be sufficient to install a linked server on the machine and utilize SQL Server as the connector?
Solved
Chelly
2 years ago Place Dataflow
4.9KViews
0likes
6Comments