# I want to write code using Bucketing in the Foundry code repository

**URL:** <https://community.palantir.com/t/i-want-to-write-code-using-bucketing-in-the-foundry-code-repository/1275>\
**Category:** Ask the Community\
**Tags:** code-repositories\
**Created:** [September 13, 2024, 11:11am UTC](https://community.palantir.com/t/i-want-to-write-code-using-bucketing-in-the-foundry-code-repository/1275 "2024-09-13T11:11:05Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![sandpiper](https://yyz1.discourse-cdn.com/palantir/user_avatar/community.palantir.com/sandpiper/32/273_2.png) [@sandpiper](https://community.palantir.com/u/sandpiper)\
**Post date:** [September 17, 2024, 4:40am UTC](https://community.palantir.com/t/i-want-to-write-code-using-bucketing-in-the-foundry-code-repository/1275/2 "2024-09-17T04:40:38Z")

</div>

Here is a minimal example:

```auto
@transform(
    output=Output("/path/to/bucketed/flights/dataset"),
    source=Input("/path/to/flights/dataset"),
)
def compute(output, source):
    output.write_dataframe(
        source.dataframe().repartition("dep_runway_actual"),
        bucket_cols=["dep_runway_actual"],
        bucket_count=200,
        sort_by=["dep_runway_actual"],
    )

```

In the above example, I include a `sort_by` in addition to specifying `bucket_cols` because in principle this would allow downstream consumers to take advantage of both the bucketing and the sorting when performing a sort-merge join. However, this will only make a difference if the [BUCKET\_SORTED\_SCAN\_ENABLED](https://www.palantir.com/docs/foundry/optimizing-pipelines/spark-profiles-reference/) spark property is enabled on the downstream job (and as noted at [https://www.palantir.com/docs/foundry/optimizing-pipelines/projections-overview/#join-projected-and-non-projected-datasets](https://www.palantir.com/docs/foundry/optimizing-pipelines/projections-overview/#join-projected-and-non-projected-datasets), a bucket sorted scan does not necessarily improve performance).

Also note that in the above example, I call repartition on the dataframe in addition to specifying bucket columns in the write settings - this is recommended because Spark will perform bucketing separately for each partition, so if the same value of the bucket column exists in multiple partitions, you can end up with an excessive amount of output dataset files.

Please see [my earlier answer](https://community.palantir.com/t/pipeline-optimization-dataset-projection-how-do-they-work-and-how-they-relate-to-hive-partitioning-bucketing/375/2) for a discussion of the differences between bucketing and using a join-optimized projection (depending upon details of your use-case, like whether your pipeline is incremental, creating a join-optimized projection may be the correct way to achieve your objectives here).

---

_[View the full topic](https://community.palantir.com/t/i-want-to-write-code-using-bucketing-in-the-foundry-code-repository/1275)._