Spark streaming using Eventstream custom endpoint (Azure Event Hub)

[chetanhiwale]

I have created a Eventstream and using a custom endpoint as a source. Here my source is Azure Event hub. I am trying to read the data using spark streaming in Fabric notebook. Although I am able to start the spark stream, but not able to read any data. Is there setting enabled for custom endpoint like it can only listen to events. 

Event stream:

Pyspark code: 

from pyspark.sql import SparkSession

from pyspark.sql.functions import from_json, col

from pyspark.sql.types import StructType, StructField, StringType, IntegerType

# Event Hub connection string

event_hub_connection_string = "<Endpoint>"

# Read data from Event Hub

df = spark.readStream.format("eventhubs")\

  .options(**ehConf)\

  .load()  

rawData = df\

  .withColumn("bodyAsString", f.col("body").cast("string"))\

  .writeStream\

  .format("delta")\

  .option("checkpointLocation", "Files/checkpoint/stocks")\

  .outputMode("append")\

  .option("path", 'Tables/stock_events')

handler = rawData.start()

[chetanhiwale]

Thanks @v-prasare for sharing !!

[v-prasare · Accepted solution]

Hi @chetanhiwale,

Thanks for reaching out MS Fabric communnity support

custom endpoints in Eventstream are output-only, not sources. You cannot use Eventstream to feed Spark as a source. You either:

• Connect Spark directly to Event Hub (bypass Eventstream)

• Let Eventstream output to a Lakehouse or Delta, and Spark reads from that

Thanks,

Prashanth Are