User Profile
__maca__
Advocate I
Joined 2 years ago
User Widgets
Contributions
Re: Pipeline | REST API | High CU % Usage
Hi v-cboorla-msft, Thanks for the detailed answer and sorry for the late reply. I’ve been busy trying out all the ideas you suggested. Unfortunately, I haven’t seen any improvement yet. The API we’re using limits us to 15 IDs at a time, which makes bulk retrieval somewhat impossible. I’ve set up my pipeline so that all the ForEach loops are triggered one after another and tried using higher batch sizes (the max allowed by Fabric is 50, so I couldn’t go beyond that). Do you think setting the ForEach activities to sequential within the loops could solve the problem? It might take forever, though... There must be another way to make this work. I don’t see how switching to an even higher capacity makes sense for such a basic operation. I’d really appreciate it if you could provide me with other ideas or solutions. Thanks!3.2KViews0likes2CommentsPipeline | REST API | High CU % Usage
Hi, I have set up a simple process to query an external API and retrieve data using Copy Data Activity. The issue I'm facing is that the CU % Usage keeps increasing linearly and eventually reaches 100% (F64), even though the task seems simple and should not require such high usage. Here are the details of my setup: I retrieve a list of 1,000 IDs from the REST API using a Copy Data Activity I then use two ForEach blocks in parallel, each containing a Copy Data Activity, to query two separate endpoints with these IDs in batches of 15 The Intelligent throughput optimization is set to 4, and the Degree of copy parallelism is set to 1 (i.e., all settings to minimum) Each file, between 100KB and 200KB in size, is saved as a JSON file (raw) in a Lakehouse / Files. Hence, the data is not processed in any way. The whole process is contained within an Until activity and repeats itself until no more IDs are available (in total, probably about 20,000 IDs) Despite the really simple nature of these operations, the CU % Usage continues to rise. I expected it to remain stable at a low percentage for the entire run. Is there an auto-scaling function that might be causing this? I believe this Data Pipeline should be able to run efficiently on an F64 capacity for days without issues since the same process can easily run on any laptop, or even a Raspberry Pi. Any help would be greatly appreciated! Cheers!3.3KViews0likes6CommentsNotebook | VS Code (Desktop) | Not working...
Hi everyone, I am new here and to the world of Microsoft Fabric. I am trying to run a notebook in VS Code (Desktop) on macOS (M3), but I am unable to run the notebook using the Fabric instance (Spark). While I can load, open, update, and retrieve all notebooks from my workspace, I can't run cells with code like: df = spark.sql('SELECT * FROM my_lakehouse.my_table') --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) Cell In[3], line 1 ----> 1 spark.sql('SELECT * FROM my_lakehouse.my_table') AttributeError: 'NoneType' object has no attribute 'sql' I’ve noticed that many users have had similar issues and I’ve tried all the proposed fixes, but nothing works. I have reinstalled everything multiple times, ensuring all requirements (conda, java, etc.) are met. I have also experimented with both environments (1-1 & 1-2). Yet, I still get the following error (log): root ERROR Failed to initialize Spark Lighter variables. An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext. : java.net.BindException: Can't assign requested address: Service 'sparkDriver' failed after 16 retries (on a random free port)! Consider explicitly setting the appropriate binding address for the service 'sparkDriver' (for example spark.driver.bindAddress for SparkDriver) to the correct binding address. Has anyone encountered this issue before? I’d really appreciate any help or suggestions. Thanks!1.2KViews0likes2Comments
Data Privacy
Microsoft Fabric Community and Privacy
To learn more about how we manage your data, please review the Microsoft Fabric Community Data Privacy guide.