What is the max size limit of a row/column of a dataset?

Ask the Community
1

Hey, we have a transform writing very large json objects into some dataset columns and we are encountering the following build failure:

java.lang.IllegalArgumentException: Cannot grow BufferHolder by size 1074387552 because the size after growing exceeds size limitation 2147483632

Can someone confirm if 2GB is the current size limitation of a row/column? or is the 2GB size limit referring to another limit?

2

The 2GB is per column value and is a Spark limitation - https://kb.databricks.com/sql/cannot-grow-bufferholder-exceeds-size

3

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.