Materialized Lake View Refresh activity Pipeline bug
Hi all, When using the preview feature refresh mlv activity in the pipeline, is do have some major bugs. After saving, it goes back to default, the pipeline does not want to refresh after, and I cannot parametrize it. Is there any workarround, and is this a known bug? Cause this is not workable.
Mirroring SQL Endpoint Capacity Usage
Hi All, I've been working extensively with the Open Mirroring Database item in Microsoft Fabric, but I'm still trying to fully understand how its capacity (CU) consumption is calculated. From my understanding, the internal operations include components such as: OneLake Write via Redirect OneLake Iterative Read via Proxy OneLake Read via Redirect OneLake Read via Proxy OneLake Write via Proxy OneLake Other Operations My assumption is that these workloads account for activities such as data ingestion, CSV-to-Parquet conversion, data maintenance, vacuum operations, and other background processing required to keep the mirrored database operational. What I do not understand is the Warehouse SQL Endpoint Query Usage metric. On 11 August, this metric increased dramatically in our environment and has remained significantly elevated ever since. For one particular Open Mirroring Database item, usage increased from approximately 7K CU to 34K CU, and I am trying to determine what contributes to this workload and why it changed so suddenly. Before: Screenshot After: Screenshot Another observation is related to the Run By filter in the Capacity Metrics App. Prior to 10 August, I was able to filter consumption by individual user identities and clearly distinguish activity generated by specific users from activity generated by the system. However, from around 10/11 August, all user-generated activity appears to be grouped under a generic "User" identity, and individual users can no longer be distinguished or filtered. Before 10 August: Screenshot showing identifiable users After 10 August: Screenshot showing only "User" I don't understand what this new User identity represents. Is it an aggregation of all user-generated activity, a new classification introduced by the metering changes, or something else entirely? My current theory is that both the increase in Warehouse SQL Endpoint Query Usage and the change in the Run By filter may be related to the recent Fabric capacity metering changes, but I have not been able to find any documentation that clearly explains this behavior. Can anyone provide insight into the following? What specifically contributes to Warehouse SQL Endpoint Query Usage for an Open Mirroring Database? Do any of the internal mirroring processes execute through the SQL Endpoint and therefore appear under this metric? Did the recent Fabric metering changes alter how these operations are classified or billed? What exactly does the "User" identity represent in the Run By filter? Why did individual user identities disappear after 10 August and become a generic "User" value? Is the new User Identity dimension connected to the recent metering changes? Has the attribution model for user-generated workloads changed, resulting in individual users being grouped into a single category? Any clarification would be greatly appreciated, as these changes make it difficult to understand capacity consumption patterns and compare usage before and after 10 August. Thanks!
iifnull() failing on non-null input
The documentation for iifNull() says this: Returns the first not null item when given two or more inputs. We have a dataflow which handles the contents of a number of files, using a specified CSV format. One of the columns is named 'Dealer'. Due to an error in an upstream system, a significant number of files came in with the header misspelled 'Delaer'. There are so many of these files that we cannot simply edit them manually to fix. I have the following expression in a Derived Column transform: iifNull('Dealer',byName('Delaer')) I understand this expression to mean "Use the value of the column 'Dealer' if it is not null, else use the value of the drifted column 'Delaer'." (That name is not in the list set by the dataset, so it is drifted, and leaving out byName() causes validation to fail.) When the file does not contain the spelling error, my expected result is that it simply retains the correct value in 'Dealer' because this is not null. However, when actually run, this expression actually returns null, and causes the column in the Derived Column transform to not be created, resulting in a "Resolved attribute(s) [name] missing from [names]" error.Dataflow Gen2 - failing to load into the lakehouse
Hello. I have a dataflow gen2, fetching data from an azure data factory environment and loading them into a Fabric Lakehouse. It has been working flawlessfly for the last 3 months, but since two weeks, on some of the tables, I get this error: "There was a problem refreshing the dataflow: "Something went wrong, please try again later. If the error persists, please contact support.". Error code: EntityUserFailure. (Request ID: 6d111afc-11ee-459f-9e31-db4d08c32c44)." "XXX - ABC Entity Management_WriteToDataDestination: There was a problem refreshing the dataflow: "Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: Error in creating lakehouse table in a version, InnerException: Tables can only contain uppercase or lowercase letters, numbers or the underscore character., Underlying error: Tables can only contain uppercase or lowercase letters, numbers or the underscore character. Details: Reason = Expression.Error;ErrorCode = Lakehouse036;Message = Tables can only contain uppercase or lowercase letters, numbers or the underscore character.;Message.Format = Tables can only contain uppercase or lowercase letters, numbers or the underscore character.;ErrorCode = Lakehouse019;InnerErrorCode = Lakehouse021,Lakehouse019;Microsoft.Data.Mashup.Error.Context = User GatewayObjectId: 1ee1c1a1-a11v-32bt-a19b-5ff6d874f432". Error code: 104100. (Request ID: 66d111afc-11ee-459f-9e31-db4d08c32c44)." The weird part is that there is no consistency between which tables go to error and which ones proceed to being loaded. Sometimes it works sometimes it does not. Also, there are no issue with the naming, as all the columns are respecting the requirements laid out from the error. And above all, it was working flawlessly until few weeks ago and nothing a
Is it possible to write Copy activity logging to Fabric Lakehouse?
Greetings, community. Have a requirement where we need to log what files get transferred/fail from a Copy activity. I want to use the Logging feature, but this currently only supports ADLS connections. I remember reading somewhere that it is possible to access Lakehouse as ADLS, but I cannot get that working in Fabric. Anyone have ideas?
Semantic Model Refresh failing in Pipeline but not manually
Hello everyone, we have a Fabric F8 environment with a pipeline that triggers multiple pipelines to load data into bronze, silver and gold layer. After the Gold pipeline is succeeded, a Semantic Model Refresh activity should refresh a Semantic Model in another workspace. After it had worked for a few days, the pipeline now fails at the Semantic Model Refresh step. Restarting the pipeline from that step or triggering the Semantic Model refresh from a another pipeline or manually via the Refresh Button all succeed. The error message states, that a column is missing in the rowset. The column is created immediately before the semantic model refresh starts. What I have tried so far: - adding a 15 minutes waiting time to solve any metadata refresh activities - rebuild the whole pipeline from scratch - adding an invoke pipeline step, that starts a pipeline which only contains the Semantic Model Refresh activity None of that has solved the issue. Any suggestion would be highly welcome. Error message: Semantic model refresh execution failed, error message received from semantic model refresh operation - '{"errorCode":"ModelRefresh_ShortMessage_ProcessingError","errorDescription":"Retry attempts for failures while executing the refresh exceeded the retry limit set on the request.\n0xC11200B0: The '<oii>is_anonym<\/oii>' column does not exist in the rowset.\n0xC112001C: The operation was canceled.. The exception was raised by the IDbCommand interface.\n0xC11C0006: The current operation was cancelled because another operation in the transaction failed.\n0xC112001C: The operation was canceled.. The exception was raised by the IDbCommand interface.\n0xC112001C: The operation was canceled.. The exception was raised by the IDbCommand interface.\n0xC14700F0: {\"RootActivityId\":\"46081B1B-9B40-4967-9402-297B0D45D710\"}"}'
AzureDatabricksWorkspace connector doesn't work in Pipeline with On-Prem Gateway?
Hello fellow Fabric users. I'm trying to create a Fabric Pipeline step that would trigger a job in Azure Databricks workspace using the built in activity for that (image #3). Our ADB workspace is accessible only through On-Prem gateway connection, querying from Databricks SQL connector works well through the gateway, but I need to be able to also trigger things when needed from Fabric. BUT, for some reason the activity settings do not "see" the gateway connection I created beforehand (image #1) and also when I try to create a brand new connection using built-in dialog, it completely ignores any gateways when I type in ADB workspace URL (image #2). Please advise on how to solve this 🤔 thank you!
