topic Re: How to decode protobuf data with a Fabric PySpark notebook using the from_protobuf() method? in Data Engineering

How to decode protobuf data with a Fabric PySpark notebook using the from_protobuf() method?

friend1 — Thu, 23 Jan 2025 06:58:15 GMT

Hi there,

I am trying to stream event data from an Azure Event Hub using Spark Structured Streaming from within a Fabric Notebook to a lakehouse. The event data is protobuf and base64 encoded. I wanted to use the "from_protobuf()" method (Protobuf Data Source Guide - Spark 3.5.4 Documentation) to decode the payload.

However, I am getting the following error message:

"Spark Protobuf libraries not found in class path. Try one of the following. 1. Include the Protobuf library and its dependencies with in the spark-submit command as $ bin/spark-submit --packages org.apache.spark:spark-protobuf:3.4.3.5.3.20241016.1 ... 2. Download the JAR of the artifact from Maven Central http://search.maven.org/, Group Id = org.apache.spark, Artifact Id = spark-protobuf, Version = 3.4.3.5.3.20241016.1. Then, include the jar in the spark-submit command as $ bin/spark-submit --jars <spark-protobuf.jar> ..."

Now my question is how do I do this in a Fabric notebook environment? Is there a way to include the mentioned library?

Also: I have two Python modules that contain the classes generated from the .proto schemas which are required to decode the payload. Where do I have to put these so that I can hand them to the from_protobuf() method?

Looking forward to any ideas on this! Thanks a lot and best, flo.

Re: How to decode protobuf data with a Fabric PySpark notebook using the from_protobuf() method?

v-prasare — Mon, 27 Jan 2025 10:25:20 GMT

@friend1, Thanks reaching out to MS Fabric community support.

please refer to below wiki on how to manage apache spark libraries and let me know if this helps?

Manage Apache Spark libraries - Microsoft Fabric | Microsoft Learn

Thanks,

Prashanth Are

MS Fabric community support

Did we answer your question? Mark post as a solution, this will help others!

If our response(s) assisted you in any way, don't forget to drop me a "Kudos"

Re: How to decode protobuf data with a Fabric PySpark notebook using the from_protobuf() method?

v-prasare — Fri, 31 Jan 2025 08:33:15 GMT

@friend1, As we haven’t heard back from you, we wanted to check in to see if the resolution provided helps?

If you’re still facing any issues or have additional questions, please don’t hesitate to let us know.

We’re here to help and would be happy to assist further if needed. Looking forward to your feedback!

Thanks,

Prashanth Are

MS Fabric community support.

Did we answer your question? Mark post as a solution, this will help others!

If our response(s) assisted you in any way, don't forget to drop me a "Kudos"

Re: How to decode protobuf data with a Fabric PySpark notebook using the from_protobuf() method?

v-prasare — Tue, 04 Feb 2025 21:51:47 GMT

@friend1, As we haven’t heard back from you, we wanted to check in to see if the resolution provided helps?

If you’re still facing any issues or have additional questions, please don’t hesitate to let us know.

We’re here to help and would be happy to assist further if needed. Looking forward to your feedback!

Thanks,

Prashanth Are

MS Fabric community support.

Did we answer your question? Mark post as a solution, this will help others!

If our response(s) assisted you in any way, don't forget to drop me a "Kudos"

Re: How to decode protobuf data with a Fabric PySpark notebook using the from_protobuf() method?

v-prasare — Wed, 12 Feb 2025 09:00:52 GMT

@friend1, As we haven’t heard back from you, we wanted to check in to see if the resolution provided helps?

If you’re still facing any issues or have additional questions, please don’t hesitate to let us know.

We’re here to help and would be happy to assist further if needed. Looking forward to your feedback!

Thanks,

Prashanth Are

MS Fabric community support.

Did we answer your question? Mark post as a solution, this will help others!

If our response(s) assisted you in any way, don't forget to drop me a "Kudos"