tutorial requests
74 TopicsRefresh Semantic Model - Table Dynamic Content
I've set up dynamic content inside of a Refresh Semantic Model activity inside of a data pipeline. So far I've dynamically set the Workspace Id and the Semantic Model but I've previously left the table line blank. The pipeline would refresh properly until now and the only way it seems to run even with static Workspace Id and Semantic Model values is if I specify the tables. For the static values I can wait for the drop down menu to load and select all but when I use dynamic content in Workspace and Semantic Model ID, the table names won't load and I can only input dynamic content. How do I write in the tables through dynamic content. I've tried the following based off the JSON file: {"table": "TableName"} TableName table:TableName [{"table": "TableName"}] TableName1, TableName2 Editing the JSON file directly by copying and pasting how a non-dynamic content inputted Workspace and Semantic model ID has the table names set up How do I dynamically write in multiple tables for the dynamic content? Is there a way to specify select all when I write in dynamic content?Solved5.4KViews1like7CommentsFilter CSV dataset based on a column
Hi, I am a newbie to ADF. i have CSV dataset referring from SFTP holding 14 million records. Currently, it has copyactivity which copies to CSV dataset(unzipped) from source zipped format dataset! Then dataflow , which refers this CSV and loads to AzureSQL (upsert operation) . The pipeline fails at times. Its an ask to filter read data only from 8 days backdated based on a column "DateColumn" present in the CSV file. I added this condition(filter) in dataflow (Source ->filter ->sink) where Filter takes expression as DateColumn>= toString(addDays(currentDate(), -8)). The pipeline doesnt haveconsistent success! any other method would help since the data would be growing for years .Solved801Views0likes4CommentsIs there an ETA on event based trigger for pipeline (specially integration with sharepoint)?
Hi, I was wondering if there is a way to trigger a pipeline when a file at a sharepoint point location is modifed/created without using REST API and power shell? I have been using power automate to do this but would like to remove this dependency. If this trigger is not yet present in fabric pipeline....is there an ETA when it will be ? ThankyouSolved1.5KViews1like4CommentsGen2 Dataflow Refresh Failure - Staging Lakehouse Not Found
I am unable to successfully refresh a Gen2 Dataflow configured to fetch data from SQL Server and store it in a Lakehouse. The dataflow creation process appears to complete, but when attempting to refresh the dataflow, it consistently fails with a staging lakehouse error Error Details: Error Message: "There was a problem refreshing the dataflow: 'Staging lakehouse for dataflow with id ... was not found.'" Error Code: StagingLakehouseMissingError Request ID: 00000000-0000-0000-0000-000000000000 Steps Taken: Created a new Gen2 Dataflow Configured data source connection to SQL Server Set destination to Lakehouse Attempted to refresh the dataflow I would appreciate hearing if others have experienced this problem and successfully resolved itSolved7.6KViews0likes4CommentsHow to retrieve the output of a look-up activity in a notebook
Hi, I have a pipeline in which I perform a lookup on a table and retrieve this: { "count": 7, "value": [ { "WorkspaceID": "WKS13", "WorkspaceName": "Test Workspace 5", "UserEmail": "email" }, { "WorkspaceID": "WKS9", "WorkspaceName": "Test Workspace 9", "UserEmail": "email" }, { "WorkspaceID": "WKS9", "WorkspaceName": "Test Workspace 9", "UserEmail": "email" }, { "WorkspaceID": "WKS12", "WorkspaceName": "Test Workspace 12", "UserEmail": "email" }, { "WorkspaceID": "WKS11", "WorkspaceName": "Test Workspace 11", "UserEmail": "email" }, { "WorkspaceID": "WKS15", "WorkspaceName": "Test Workspace 11", "UserEmail": "email" }, { "WorkspaceID": "WKS16", "WorkspaceName": "Test Workspace 11", "UserEmail": "email" } ] } And I would like to retrieve this in a notebook. Do you have a solution?Solved2.6KViews0likes7CommentsGet email attachment with receivedtime
Hello, I'm learning Dataflow and Queries and I'm trying to load into a Lake an email with an excel attachment with the email received date. I cannot find the way to add the email received date as a column of the excel. Thanks for your help.Solved3.5KViews0likes9CommentsDataFlow(Gen2) managed by multiple users
When DataFlow(Gen2) is used for work, it is necessary for multiple people to manage it together. However, at present, only one owner is able to check the contents, let alone touch them. Does this not meet the business needs at all? Or is there a proper way to do this that I just don't know about? If anyone knows, I would be very grateful if you could enlighten me.Solved5.5KViews0likes5CommentsNew in Airflow Fabric
Hello, dear all. I'm starting with Airflow in Fabric. So, I want to know how I can migrate my project that is working in my local machine if Fabric only allows add .py files. Usually, in a Airflow project we have to work with folders containing other scripts, for example. My project get data from a API and store json files 5 or 6 times per day. So, how can I connect the json results to a fabric workspace lakehouse? Finally, once stored I have to transform them using notebooks in pipelines to be shown in dashboards. I would appreciate your help. Thanks in advance!Solved4.5KViews0likes4Commentsstuck with MS Learn Tutorial for Apache Airflow Jobs for dbt-fabric Orchestration
Hello, does anyone managed to implement the sample/tutorial Transform data using dbt - Microsoft Fabric | Microsoft Learn ? For us, with a Starter Pool running on Apache Airflow version 2.10.5, only the version of astronomer-cosmos==1.5.1 and dbt-fabric==1.5.0 are valid. ....but when running the DAG 'dags/my_cosmos_dag.py we systematically get a Failed to start error. ThanksSolved7.6KViews1like9CommentsStore Activity Data into a Table
I am wanting to store the data collected from the Get Metadata activity into a lakehouse table, along with the Copy Data activity output for each table loaded in the ForEach activity. I have tried using variables of type array, and have now been testing out system variables. I haven't had any luck with figuring it out so far. Does anyone have any suggestions?Solved3.1KViews1like8Comments