Forum Discussion
Community ChampionReading limitation spark vs sempy
Hi smpa01 ,
Spark limitations include for example ISNULL, IS_NOT_NULL, STARTS_WITH, ENDS_WITH, and CONTAINS.
SemPy has not seen official documentation on these limitations and can bypass some of the limitations faced by Spark.SemPy may have some limitations in terms of data size and query complexity.
In summary, while both Spark and SemPy have their own limitations, SemPy may have more flexibility in executing complex queries directly against Power BI datasets without some of the limitations in Spark.
Best Regards,
Yang
Community Support Team
If there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!
Community ChampionI am more interested in number of rows read for which spark has clear limiations. Can you please confirm if sempy has any limitations in respect to that? Anonymous I am not interested in any other limiations sempy might have ATM. Cause I need to handle that, if it does have any such limitations.
Hi smpa01 ,
I have not seen documentation that indicates sempy is limit on the number of rows that can be read.
The limit I see in the documentation for sempy is:
The amount of data that can be retrieved is limited by the per-query maximum memory of the capacity SKU hosting the semantic model and the Spark driver node running the notebook (see Node Size).
Please see this official documentation for information:
If you have any other questions please feel free to contact me.
Best Regards,
Yang
Community Support TeamIf there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!