Data Wrangler Aggregation does not work properly for Spark Data Frames
Hi Team, When we are creating a Spark Data Frame and open Data Wrangler and if we try to add "Group by and aggregate" operation under "aggregation" section when we select a particular column it does not show all the aggregation options for the selected column. It just shows 5 aggregations i.e. count, first value, last value, number of distinct values and mode. I think this is a bug. I hope you are able to understand the issue.
Unable see the Tables in Fabric that have status as active in Power Platform.
Hi Community Using Link to Fabric in Power apps i have synced my D365 FnO to Fabric lakehouse there are some tables that have status as active in power apps as show in screenshot below but are found in Unidentified folder in Fabric lakehouse what might be the reason behind it and how to solve this problem. The above two screenshots one is from power apps and the other is from Fabric lakehouse Can any one in the commuinty provide the solution for this problem any small information provided will be so helpfull.
Developing Dashboards from FnO Data
Hi, I am developing Power BI Dashboard in Fabric. My flow of Data is from, FnO to Dataverse to Microsoft Fabric (As Shortcut Tables, here data is not present in OneLake, It is present in Dataverse Environment). My Requirement is to Develop a dashboard on top of it. I have came to know about two methods: 1. Using SQL End-Point - Creating a visual query, choosing specified columns, replacing values, etc.. and then saving as a view (The required view alone saves in OneLake). If needed we can also create a measure for the same in SQL EndPoint. 2. Using Notebooks on Lakehouse - Using a Notebook, loading tables as df, making transformations and then saving the same (Here the transformed gets saved in OneLake) (We can run notebooks at specified intervals, such that once data gets refreshed in Dataverse, it gets refreshed in Dashboard). Is there any common persona to use, both Replace Feature (From SQL Endpoint) and the New Calculated Column Feature (From Warehouse). I would like to know about the Warehouse Method, where I can get like the below mentioned image. If possible, I would like to know if I should use all the tables in OneLake or is there any way to use the transformed tables alone in OneLake. (I have taken this image from Semantic Model) Below image is for Visual Query in SQL Endpoint Kindly assist me with all the possible ways and the solution of creating measures, calculated fields, etc.. all possible solutions with SQL Endpoint Notebooks and Warehouse methods, since fabric is a fresh perspective. And, if I bring my data into datawarehouse using the "+" options, will that cause storing data into OneLake, or the same will come as a Shortcut tables (I believe data doesn't gets duplicated into OneLake here) Any Assistance on this will be highly helpful. Thanks in Advance.
How can I get Synapse Notebook to only process newly added data?
I have data that I plan on uploading to an Azure storage account. My plan is to create a pipeline in Synapse Studio, which will include an Apache Notebook (Using PySpark). The primary objective is to have the Notebook process the data and then save it to a lake database. The data will be uploaded to an Azure storage container following this format for example: 2022/Week1/Week1.xlsx and 2023/Week10/Week10.xlsx. Initially, I will store and process all historical data in the storage account. After that, the data will be processed and added to the lake database on a weekly basis. Now, the question is, what is the most efficient method to enable the Azure pipeline or the Notebook to identify and process only the newly added data?.
