topic Spark streaming using Eventstream custom endpoint (Azure Event Hub) in Data Engineering

Spark streaming using Eventstream custom endpoint (Azure Event Hub)

chetanhiwale — Tue, 22 Apr 2025 07:42:36 GMT

I have created a Eventstream and using a custom endpoint as a source. Here my source is Azure Event hub. I am trying to read the data using spark streaming in Fabric notebook. Although I am able to start the spark stream, but not able to read any data. Is there setting enabled for custom endpoint like it can only listen to events.

Event stream:

Pyspark code:

from pyspark.sql import SparkSession

from pyspark.sql.functions import from_json, col

from pyspark.sql.types import StructType, StructField, StringType, IntegerType

# Event Hub connection string

event_hub_connection_string = "<Endpoint>"

# Read data from Event Hub

df = spark.readStream.format("eventhubs")\

.options(**ehConf)\

.load()

rawData = df\

.withColumn("bodyAsString", f.col("body").cast("string"))\

.writeStream\

.format("delta")\

.option("checkpointLocation", "Files/checkpoint/stocks")\

.outputMode("append")\

.option("path", 'Tables/stock_events')

handler = rawData.start()

Re: Spark streaming using Eventstream custom endpoint (Azure Event Hub)

v-prasare — Wed, 23 Apr 2025 05:04:31 GMT

Hi chetanhiwale,

Thanks for reaching out MS Fabric communnity support

custom endpoints in Eventstream are output-only, not sources. You cannot use Eventstream to feed Spark as a source. You either:

Connect Spark directly to Event Hub (bypass Eventstream)
Let Eventstream output to a Lakehouse or Delta, and Spark reads from that

Thanks,

Prashanth Are

Re: Spark streaming using Eventstream custom endpoint (Azure Event Hub)

chetanhiwale — Wed, 23 Apr 2025 05:18:43 GMT

Thanks v-prasare for sharing !!