Step 1: Configure Spark to Use SAS Token for Authentication

In Azure Blob Storage, SAS (Shared Access Signature) provides secure delegated access to your storage resources. Below is an example SAS token and how you configure Spark to use it.

# SAS token example (for illustration only)
sas_token = "sp=r&st=2025-03-06T17:28:38Z&se=2026-03-07T01:28:38Z&spr=https&sv=2022-11-02&sr=c&sig=VAI..."

Step 2: Define the File Path Using WASBS (Azure Blob Storage)

# Define file path
file_path = "wasbs://<container_name>@<storage_account_name>.blob.core.windows.net/<path_to_your_file>.csv"

Step 3: Configure Spark with SAS Token

# Spark configuration for accessing the blob
spark.conf.set(
    "fs.azure.sas.<container_name>.<storage_account_name>.blob.core.windows.net",
    sas_token
)
        

Step 4: Read the CSV File into a DataFrame

# Read CSV file into DataFrame
df = spark.read.format("csv") \
    .option("header", "true") \
    .option("inferSchema", "true") \
    .load(file_path)
        

Step 5: Show the Data and Print Schema

# Display the DataFrame contents
df.show()

# Print the DataFrame schema
df.printSchema()
        

Conclusion

Using the above steps, you can securely connect to Azure Blob Storage using SAS tokens and read CSV files directly into PySpark DataFrames. This method is essential for data processing workflows in big data and cloud environments.