DuckDB Directly stream inputs and outputs documentation

Ask the Community
,
1

Hi question on the DuckDB API advanced use-cases docs:

Docs say that when streaming outputs to use:

TO '{output_dataset.lazy_write_table_path}'

But that gives me this error:
AttributeError: ‘LightweightOutput’ object has no attribute ‘lazy_write_table_path’

I tried this after some digging and it is working at the moment:
TO '{output_dataset.path_for_object_store_write_table}'

Can anyone confirm what the correct attribute to use will be long term?

Thanks

3

Thanks for the flag!

path_for_object_store_write_table is the intended public prop. I just put up a fix for the docs which should be live soon.

3 Likes
4

What’s the best approach for reading files written to this path in the same transform?

For example:

  1. In a transform, write files to path_for_object_store_write_table
  2. Then, if needed to read the written files in the same transform, would it be:
    1. read_parquet(output_df.path_for_object_store_write_table)
5

Yes, haven’t tested but this should work.(you might need to add a glob *.parquet or **/*.parquet

1 Like