Generating temp CSV works in Preview but not Build

Ask the Community
1

Facing a weird issue - I can generate a temp csv file within a preview but when I try doing this in a build it doesn’t seem to allow it.

The code looks like this:

    # Create a temporary directory for CSV processing
    temp_dir = tempfile.mkdtemp()
    temp_csv_path = os.path.join(temp_dir, "temp_csv")    

    df.write.csv(temp_csv_path, header=True, sep="|", emptyValue="")

    # Read the CSV file back into a string
    csv_files = glob.glob(f"{temp_csv_path}/part-*.csv")
    if csv_files:
        with open(csv_files[0], 'r') as file:
            csv_string = file.read()
        # Clean up temporary directory
        shutil.rmtree(temp_dir)
    else:
        csv_string = ""
        message = "No CSV file was generated."
        print(message)
        logger.info(message)
        shutil.rmtree(temp_dir)

Then in my logs I get "No CSV file was generated" does anyone know the cause of this? Thanks!

2

My guess would be it’s because you write on the spark executors and try to check for the file on the driver?

Irrespective, can you describe your envisioned workflow? What do you want to achieve ultimately?

3

Thank you, I believe you are correct! The workflow was a little convoluted but in summary it was to submit a binary CSV file via FTPS to an external server.
Thanks again for your help!

1 Like