Jun 5, 2022
--
It's worth remembering the `pandas` API is available now on PySpark:
https://spark.apache.org/docs/latest/api/python/user_guide/pandas_on_spark/index.html
Or for older versions via the `koalas` library:
https://koalas.readthedocs.io/en/latest/