Hi,
I have a question regarding the behavior of Pipeline Builder functions.
The start of the week varies by region, but is it correct to assume that Pipeline Builder’s truncate date function always returns Monday regardless of the user’s timezone when executed with weeks?
Thanks!
Your assumption is correct - per https://spark.apache.org/docs/3.5.3/api/sql/#date_trunc (which is what Pipeline Builder is using under the hood for this function in batch pipelines), the “week” option always truncates to Monday.
It’s also worth noting that all Pipeline Builder code is executed server-side, so the user’s timezone should not matter in any case (and I believe that the server timezone is always UTC, regardless of the physical location of the cloud infrastructure).
Thank you for taking the time to answer my question.
Your explanation, including the reference to the Spark documentation, was incredibly helpful.
I appreciate your expertise and will keep this in mind for future Pipeline Builder projects.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.