general question

440 Topics

Will Invoke Pipeline (Legacy) be deprecated?
Is there currently any plan or timeline to deprecate Invoke Pipeline (Legacy) in Microsoft Fabric? For same-workspace pipeline orchestration, the Legacy activity is still very convenient. The new Invoke Pipeline activity provides additional capabilities, such as cross-workspace invocation and child-pipeline monitoring, but it also requires a Connection object. Given the performance concerns that have been reported by multiple users, I’m wondering what the recommended approach is: Is it safe to continue using Invoke Pipeline (Legacy) for now? Does Microsoft recommend migrating existing pipelines to the new Invoke Pipeline activity? Is there any indication that the Legacy activity will eventually be deprecated or removed? Should we already start rebuilding existing pipeline orchestration using the new activity and Connections? It would be great to understand the intended direction, especially before investing time in rebuilding existing pipeline implementations. Thanks!
benitovbreugel
2 days ago Place Pipelines
26Views
0likes
3Comments
Copy job vs pipeline + watermark table for incremental loads from Azure SQL: production experiences?
Hi all, I'm comparing Copy job with a classic pipeline (Lookup + Copy activity + watermark table) for incremental loads from Azure SQL Database into a Lakehouse, for around a few dozen tables. The pipeline version is the pattern most of us know: Lookup: SELECT LastWatermark FROM etl.Watermark WHERE TableName = @{item().name} Copy: SELECT * FROM @{item().name} WHERE ModifiedDate > '@{activity('Lookup').output.firstRow.LastWatermark}' Stored proc: UPDATE etl.Watermark SET LastWatermark = <max ModifiedDate copied> Copy job seems to cover most of this out of the box (incremental column per table, state tracked for you), which is attractive because there is much less to maintain. Before choosing, I'd like to understand a few things from people using Copy job in production: Visibility: when a single table fails, how easy is it to see which table and why, and to rerun only that one? Resetting state: if a table needs a full reload (schema change, data fix), can you reset the watermark for just that table? Deletes: for sources where rows are deleted, how are you handling it with Copy job, or do you fall back to mirroring / a pipeline for those tables? CI/CD: does Copy job work cleanly with Git integration and deployment pipelines, including connection changes between Dev and Prod? Any experience or lessons learned would be really helpful. Thanks!
Solved
jubinsoni
2 days ago Place Copy job
34Views
0likes
3Comments
Can parallel pipeline runs upsert into the same Lakehouse table without commit conflicts?
I want to ingest from 10 DB2 databases into a Lakehouse using one generic pipeline, invoked once per DB. They all write into the same Bronze tables (e.g. .raw_<table>). Every row carries a _db_id, so no two runs ever touch the same record. The Copy activity uses Table action = Upsert with key columns _db_id + the source primary key. The tables are not partitioned or clustered. In DESCRIBE HISTORY the commits show operation = MERGE, engineInfo = datafactorycopy, and operationParameters, operationMetrics, readVersion and isBlindAppend are all NULL. I would like to run the DBs in parallel, but we are unsure about commit conflicts. The concurrency control article says conflicts depend on the files a MERGE reads, and that the partition column must be a literal in the merge condition. It describes Spark, though, and I could not find anything about the Copy sink. Questions: Does the Copy activity upsert read the whole target table during the MERGE, or can it prune files, for example by _db_id? Can two parallel Copy upserts (different DBs, same table) fail with something like ConcurrentAppendException? Does the sink retry internally? Would partitioning the table by _db_id help with the Copy sink, even though we cannot set a literal predicate? Is there a recommended pattern for this, for example append into per-DB staging tables and a single MERGE afterwards? For now we run the DBs sequentially. Any experience or documentation pointers would be appreciated.
Microfyx
3 days ago Place Pipelines
45Views
0likes
3Comments
Edge device detections into Fabric: Eventstream + Eventhouse, or Lakehouse?
Hi all, While working on a computer vision project at DataToBiz, we built the edge side of a real-time system, and I’d like to compare notes with people here on how the data side would work in Fabric. On the edge, the detection runs on a Raspberry Pi. We used YOLOv8n with ByteTrack to track vehicles across frames, plus Depth-Anything-v2 for depth estimation from a single camera so we could apply spacing rules. The model was converted to ONNX, TFLite and NCNN, and we used quantization and threading to get the speed we needed without heavy hardware. A camera, GPS and compass were synced by timestamp, so every detection comes out tagged with its time, location and direction. Post-processing smooths the detections to reduce flickering. That leaves a stream of time-stamped, geo-tagged detection events from many devices, and that’s the part I’d like your input on in Fabric. Would you ingest it through Eventstreams into an Eventhouse and query it with KQL, or land it in a Lakehouse and process it in batches? Where does that line sit for you when the end goal is near real-time dashboards? How do you handle devices that go offline and send their events late or out of order? Do you rely on event time in the Eventhouse, or clean it up downstream? For the geo side, with coordinates and direction on every event, how have you handled spatial queries and map visuals in Real-Time Intelligence or Power BI? Is it good enough, or do you still end up pre-processing the coordinates? And with raw detections arriving continuously, how do you decide what to keep as raw events, and what to aggregate into something like per-location availability for reporting? Would love to hear how you’d approach it.
Solved
kiara-mehra35
5 days ago Place Real-Time Intelligence
81Views
0likes
4Comments
No XML option on Lakehouse destination
Hi, I have a requirement to connect to a third party 'API' that spits out XML only. I notice I dont have the option to save XML to a Lakehouse Destination via a Copy Activity. Documentation specifies XML should be supported (Configure Lakehouse in a copy activity - Microsoft Fabric | Microsoft Learn) Have tried creating a brand new Copy activity explicitly setup with an XML file format so I could rule out any other setup causing this.
Solved
DanielAmbler
10 days ago Place Pipelines
1.7KViews
0likes
8Comments
Session tag restrictions for high concurrency mode in pipelines
Hi all, Our pipeline runs 3 notebooks in succession using high concurrency mode for pipelines with a session tag. The run failed with this error: Failed to run notebook due to invalid request. [Error: Session tag can only contain letters, numbers, and underscores. (Parameter 'HighConcurrencyModeOptions.SessionTag')] The tag we used was dataverse-ingest, so the hyphen seems to be the cause. We will switch to underscores, but I could not find this restriction in the Microsoft documentation. Is the allowed character set for session tags documented anywhere? Are there other limitations (length, reserved words, case sensitivity) we should know about? Thanks!
Solved
AvdBerg
11 days ago Place Pipelines
55Views
0likes
2Comments
Fabric Plan tenant settings for session upgrades control
Hi there, I'm reading through the Fabric Plan documentation, specifically the "Control session upgrades" section. It states that administrators can control which users are allowed to upgrade to Planner and Stakeholder sessions through tenant settings. However, even as an admin, I can't find these settings anywhere in the Fabric Admin Portal. Am I looking in the wrong place? Could someone point me to the exact location of these settings or let me know if there are any other prerequisites for them to appear? Thanks!
miriam_bobbo
13 days ago Place IQ
46Views
0likes
1Comment
Best practice for handling schema evolution in Fabric Eventstream before data reaches Eventhouse?
I have an Eventstream receiving operational events where the schema may evolve over time. For example, the producer initially sends: DeviceId, Timestamp, Temperature, Status but later adds fields such as: Location, FirmwareVersion, ErrorCode I want the pipeline to continue ingesting events without breaking downstream KQL tables, update policies, materialized views, or Real-Time Dashboards. I am trying to understand where schema evolution should ideally be handled in a production Fabric RTI architecture. Would you: enforce the contract upstream using Schema Registry normalize changing fields inside Eventstream before Eventhouse ingestion land the raw payload first and handle schema evolution inside Eventhouse/KQL maintain separate versioned event schemas/tables How are people handling this in production when producers can add fields without notice? I am particularly interested in avoiding a design where every small upstream schema change forces updates across Eventstream, KQL tables, update policies, and downstream dashboards.
Solved
ShivekMaharaj
13 days ago Place Real-Time Intelligence
168Views
0likes
3Comments
Cosmos DB in Fabric as AI agent memory: vector search, partitioning, and the OneLake mirror?
I'm evaluating Cosmos DB in Fabric as the storage layer for an AI agent, for example one built in Microsoft Foundry. It would hold: Conversation history and session state per user Long-term memory: facts or summaries the agent extracts, stored with embeddings for retrieval Agent run logs (tool calls, decisions) that I'd like to analyze in Power BI or KQL My understanding is that Cosmos DB in Fabric supports vector search and automatically mirrors data to OneLake, so the same data could serve the agent at runtime and support analytics afterward. Before committing, I'd like to understand a few things. Questions: Partitioning: for agent memory, is /userId a sensible partition key, or is it better to use hierarchical partition keys (for example /tenantId/userId/sessionId)? Are there hot-partition risks with very active users? Vector search: what are the practical limits in Cosmos DB in Fabric (embedding dimensions, index types, items per container)? Is hybrid search (vector plus full-text) available here, or only in standalone Azure Cosmos DB?
Solved
jubinsoni
20 days ago Place Cosmos DB (NoSQL)
86Views
1like
3Comments
Sharepoint lists mirroring issue
Hello, We currently face an issue with sharepoint lists mirroring in microsoft fabric and we have no clue what can happen because the error message is pretty useless : In fact, we are mirroring more than 20 lists but only half of them succeed. The others failed with the following error : Internal system error occurred. ArtifactId: daa35143-...-5aa6accfec68, SequenceNumber: 9 I have linked an eventhouse but the error message is still the same, no more information: Internal system error occurred. ArtifactId: daa35143-...-5aa6accfec68, SequenceNumber: 9 How can I have more information about the issue ? Thank you very much for your help, Regards
Gllme
20 days ago Place Mirroring
77Views
0likes
3Comments