We always use column descriptions. We read them from input datasets and write them to the output.
In Python repositories, when using Spark, Below I provide our currently used functions for reading and writing the column descriptions.
Question. How do we read and write column descriptions when using Polars or DuckDB? We would use these much more, if only we could employ column metadata reading and writing. I would expect answers with code examples.
def get_df_with_metadata(source, incr_mode='current', incr_schema=None):
""" Produces df with columns containing metadata taken from column descriptions of the source dataset. Intended to
be used at the beginning of the compute function so that column metadata is carried over during transformations.
Use together with `@transform` decorator.
Args:
* `source`: Dataset that has column descriptions added.
* `incr_mode`: mode when metadata is taken from class `transforms.api.IncrementalTransformOutput`.
* `incr_schema`: schema when metadata is taken from class `transforms.api.IncrementalTransformOutput`.
Returns: df with columns that contain metadata.
Example:
```
def compute(source):
df = get_df_with_metadata(source)
...
```
"""
# Enabling Incremental and Preview
if 'IncrementalTransformOutput' in str(type(source)):
df = source.dataframe(incr_mode, incr_schema)
else:
df = source.dataframe()
# Enabling transformation testing
if 'DummyTransformInput' in str(type(source)):
return df
descriptions = source.column_descriptions
# Nulls are replaced with empty strings, so that DataFrameWriter, wouldn't fail.
descriptions = {k: ('' if v is None else v) for k, v in descriptions.items()}
df = df.select(*[set_col_metadata(c, descriptions.get(c, '')) for c in df.columns])
return df
def metadata(df, new_metadata: dict = {}) -> dict:
""" Returns a dictionary of df column metadata which is later used to write column descriptions to a dataset.
Use together with `@transform` decorator.
Args:
* `df`: dataframe to take metadata from.
* `new_metadata`: dictionary of new metadata: col names as keys and metadata as values.
Returns: dict of column names (keys) and column descriptions (values).
Example1:
```
out.write_dataframe(df, column_descriptions=metadata(df))
```
Example2:
```
out.write_dataframe(df, column_descriptions=metadata(df, {"new_col": "example col description"}))
```
Example3:
```
df = df.withColumn("new_col", F.lit(1))
meta = metadata(df, {"new_col": "example col description"})
out.write_dataframe(df, column_descriptions=meta)
```
"""
existing_metadata = {c: df.schema[c].metadata.get('description', '') for c in df.columns}
return {**existing_metadata, **new_metadata}