Facing a weird issue - I can generate a temp csv file within a preview but when I try doing this in a build it doesn’t seem to allow it.
The code looks like this:
# Create a temporary directory for CSV processing
temp_dir = tempfile.mkdtemp()
temp_csv_path = os.path.join(temp_dir, "temp_csv")
df.write.csv(temp_csv_path, header=True, sep="|", emptyValue="")
# Read the CSV file back into a string
csv_files = glob.glob(f"{temp_csv_path}/part-*.csv")
if csv_files:
with open(csv_files[0], 'r') as file:
csv_string = file.read()
# Clean up temporary directory
shutil.rmtree(temp_dir)
else:
csv_string = ""
message = "No CSV file was generated."
print(message)
logger.info(message)
shutil.rmtree(temp_dir)
Then in my logs I get "No CSV file was generated"
does anyone know the cause of this? Thanks!