Yes of course.
We are running some queries in parallel threads when processing our data, which works for sparks sql, when for tsql it throws 429 errors. I quickly added an example you can run in notebook so you can see where the issue is:
import com.microsoft.spark.fabric
from com.microsoft.spark.fabric.Constants import Constants
from concurrent.futures import ThreadPoolExecutor
WORKSPACE_ID = "your_workspace_id"
LAKEHOUSE_NAME = "lakehouse_name"
def run_spark_multiple_threads(
target_function, args_list, number_of_threads = 8
):
with ThreadPoolExecutor(number_of_threads) as pool:
futures = [pool.submit(target_function, *args) for args in args_list]
results = [future.result() for future in futures]
return results
def run_tsql(i):
df = (
spark.read.option(
Constants.WorkspaceId,
WORKSPACE_ID,
)
.option(
Constants.DatabaseName,
LAKEHOUSE_NAME,
)
.synapsesql("""SELECT 1 as Temp""")
)
print(i, df.count())
def run_sparksql(i):
df = spark.sql("""SELECT 1 as Temp""")
print(i, df.count())
run_spark_multiple_threads(
run_sparksql,
[(i,) for i in range(100)],
number_of_threads=8
)
print("done with spark sql")
run_spark_multiple_threads(
run_tsql,
[(i,) for i in range(100)],
number_of_threads=8
)
print("done with tsql")
Here you will see that spark sql finishes normally for 100 queries, when tsql stops working after 50 or some queries. I believe there is rate limit 50req/50sec set, but its not mentioned in limitations of Spark connector for Microsoft Fabric Data Warehouse.
If you need more info still please let me know.
