When using PySpark, we can both, read and write the column descriptions from/to Foundry datasets. When using lightweight transforms, in Duck DB, there is a way to write column descriptions, but how can we read them?
Our organization mostly transforms small-medium sized datasets. The probability for the datasets to become large one day is very low. So we are in a perfect position to use lightweight transforms. But it is critical for us to have column descriptions. So it seems, we cannot employ lightweight…
How to read the column descriptions from Foundry datasets when using lightweight transforms?