Hello, I would like to know if there is a way to delete the schema and update it automaticaly after each build. In my case, our input comes from a data connection, and sometimes the schema of our input changes. However, these changes are not taken into account automatically, and we have to manually delete the schema and update it using the “Apply Schema” button.
I’d like to know if there is a way to automate this process.
And also If you have an optimized way to avoid doing it after each build, or to check if the schema has changed
Thank you!
What kind of Source are you using in your Data Connection?
Is it maybe a Source that uploads “raw” files, maybe csv?
If it’s a first party connector like JDBC the schema updates should be triggered automatically. If it’s one of the partner provided connectors that only updates raw files I would recommend to add a downstream transform that does the schema inference in code and outputs a new dataset.