A collection of useful copy-pasteable standalone PySpark code snippets with corresponding output explaining behavior of commonly used functions.

The source code of snippets is rendered as HTML and hosted at http://isabekov.github.io/pyspark-cookbook/.

Emacs with org-mode is used as a development environment. Compared to Jupyter notebooks, the source code is easier to keep in a version control system since it is just a plain text.

Depending on the operating system, install Python and packages py4j, pyspark, pandas, pyarrow, tabulate using corresponding package manager and pip.

Install the codebook module with pip package manager:

$ pip install  git+https://github.com/ebanner/pynt

On ArchLinux, pip is not allowed to install by default, so pass an extra argument:

$ pip install --break-system-packages  git+https://github.com/ebanner/pynt

Open Emacs. Install pynt in Emacs through MELPA.

M-x package-install RET pynt

where RET is just the “Enter” key.

To fix the following error during evaluation of code blocks:

ModuleNotFoundError: No module named 'notebook.services'

Find the installation of PyNT:

$ grep -i kernelmanager /usr/lib/python3.11/site-packages/codebook/manager.py
from jupyter_server.services.kernels.kernelmanager import MappingKernelManager

which is defined in the source code and change that line in manager.py to

from jupyter_server.services.kernels.kernelmanager import MappingKernelManager

PySpark Cookbook’s recipes were tested in Emacs IDE using Java Runtime environment: 17.0.10.. Set it as default:

$ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
$ sudo ln -s /usr/lib/jvm/java-17-openjdk /usr/lib/jvm/default

Newer versions of Java are not compatible with PySpark v3.4.0.

$ git clone https://github.com/nhoffman/org-export.git
$ cd org-export
$ sudo install -D -m 755 org-export* /usr/local/bin

To produce HTML page with PySpark code snippets, run:

$ make index.html

To render examples of converting PySpark tables displayed in pretty format to orgtbl format (see tabulate package describing the formats), run:

$ make test_ps2org.html

Navigate to any snippet outside “Functions”~ chapter (which is meant to provide only service functions for post-processing the output). Make sure that the cursor is inside a Python code block:

#+begin_src python :post pretty2orgtbl(data=*this*)
  ...
#+end_src

Press C-c C-c (i.e. Ctrl-c twice). Emacs will execute the source code block inside a Python session and display the output.