Introduction

In this tutorial, we will learn how to handle missing or null values in PySpark DataFrames using the fillna() function. Handling missing data is a critical part of data cleaning in data engineering workflows.

Why Use fillna() in PySpark?

• Replace NULL values in DataFrame columns with specific values.
• Apply different replacement values to different columns.
• Clean your dataset before analysis or feeding it into machine learning models.

Step 1: Import SparkSession and Create Spark Session

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("PySparkFillnaFunction") \
    .getOrCreate()

Step 2: Create a Sample DataFrame

data = [
    ("Amir Shahzad", "Engineering", 5000),
    ("Ali", None, 4000),
    ("Raza", "Marketing", None),
    (None, "Sales", 4500),
    ("Ali", None, None)
]

columns = ["Name", "Department", "Salary"]

df = spark.createDataFrame(data, schema=columns)

df.show()

Expected Output

+-------------+-----------+------+
|         Name| Department|Salary|
+-------------+-----------+------+
|Amir Shahzad |Engineering|  5000|
|          Ali|       null|  4000|
|         Raza|  Marketing|  null|
|         null|      Sales|  4500|
|          Ali|       null|  null|
+-------------+-----------+------+

Step 3: Fill All NULL Values

Fill all NULL values with 'Unknown' for string columns and 0 for numeric columns.

df_fill_all = df.fillna("Unknown").fillna(0)

df_fill_all.show()

Expected Output

+-------------+-----------+------+
|         Name| Department|Salary|
+-------------+-----------+------+
|Amir Shahzad |Engineering|  5000|
|          Ali|    Unknown|  4000|
|         Raza|  Marketing|     0|
|      Unknown|      Sales|  4500|
|          Ali|    Unknown|     0|
+-------------+-----------+------+

Step 4: Fill NULLs with Column-Specific Values

df_fill_columns = df.fillna({
    "Department": "NA",
    "Salary": 10000
})

df_fill_columns.show()

Expected Output

+-------------+-----------+------+
|         Name| Department|Salary|
+-------------+-----------+------+
|Amir Shahzad |Engineering|  5000|
|          Ali|         NA|  4000|
|         Raza|  Marketing| 10000|
|         null|      Sales|  4500|
|          Ali|         NA| 10000|
+-------------+-----------+------+

Step 5: Fill NULLs in a Specific Column Only

df_fill_name = df.fillna("No Name", subset=["Name"])

df_fill_name.show()

Expected Output

+-------------+-----------+------+
|         Name| Department|Salary|
+-------------+-----------+------+
|Amir Shahzad |Engineering|  5000|
|          Ali|       null|  4000|
|         Raza|  Marketing|  null|
|      No Name|      Sales|  4500|
|          Ali|       null|  null|
+-------------+-----------+------+

Conclusion

Handling null and missing values is an essential part of data processing in PySpark. The fillna() function provides a simple and flexible way to replace these values, ensuring your data is clean and ready for further analysis or modeling.