Foundry Managed Iceberg tables not working

Ask the Community
,
1

So i am trying the foundry managed iceberg tables for the first time but its A) not detecting preview and debug mode at design time and B) failing right at the checks step with “Internal Error”

the code is literally a copy paste of most rudamentory example shown here:

from transforms.api import (
    IncrementalTableTransformInput,
    IncrementalTransformContext,
    incremental,
    transform,
)
from transforms.tables import TableInput, TableOutput, TableTransformOutput

@incremental(v2_semantics=True)
@transform(
    source=TableInput("/maiden7705-c20212/[SANDBOX] Bay Area Transit/data/manual/yellow_tripdata_2012-01"),
    output=TableOutput("/maiden7705-c20212/[SANDBOX] Bay Area Transit/data/manual/yellow_tripdata_2012-01_output"),
)
def incremental_cleanup(
    ctx: IncrementalTransformContext,
    source: IncrementalTableTransformInput,
    output: TableTransformOutput,
):
    # Create a changelog view from the last seen Iceberg snapshot ID.
    changelog_view_df = source.changelog(["id"])

    # Read the changelog view, then merge into the target Iceberg table.
    output.apply_changelog(
        changelog_view_df.filter(...)  # Changelog view with any transformation logic applied
        # ["id"]  # Identifier column(s) # my input has no visible identifier
    )

when asked AIP to explain the error it says:

The error "RemoteException: INTERNAL (Default:Internal)" suggests that this is an internal issue within Foundry that you cannot resolve directly through your code. Such errors typically indicate a configuration or system-level problem on the Foundry platform.

Suggestion:
You should contact Palantir support to report this issue, providing them with details of the error (e.g., task :renderShrinkwrap, instance ID cc8f1e1f-450a-4ca8-b69e-4f3e8019c72d) and the steps that led to it. Apologies for the inconvenience caused!

If you suspect there may be an issue with the dependencies in your meta.yaml, you can attempt to verify that all versions match what is expected or required. However, this error likely pertains to Foundry's internal configuration or processes, and direct resolution on your side may not be possible.

NOTE 1: The data used in this is NY Taxi Cab and this POC is performed on my sandbox, not client’s instance

NOTE 2: Also wierdly your system wont let me make this post until i have changed all occurance of word Im*pl (without astreks in it), so i replaced it with im*pl, such as ld_im*pl_linux-64=2.44=ha97dd6f_2@conda in conda lock file, whats up with that???

2

Hi @maiden7705

A couple first things to check:

  • Do you have Foundry Iceberg beta features enabled on your environment? This is a private beta feature and needs to be proactively enabled via Palantir support to begin use.
  • Noting that the code snippet mentioned requires some user input (the ellipeses in .filter(...))

I’d recommend filing a support issue with Palantir to investigate further.

3

@maiden7705, would you mind sharing the full stacktrace you see in the CI logs? I might find you that way in our logs.

But @sgershkon is right. We don’t have Foundry-managed Iceberg enabled on all stacks yet but are generally happy to enable with some caveats to talk through!

4

so its not available on the AIP sandbox by default? i didnt wanted to do trial and error compute on client’s instance hence i tried on my sandbox first?

5

can it be enabled on my sandbox, as I dont want to do trial and error on client computer, and before i present any architecture or design improvement, i always play around with things on my sandbox. let me know whom to contact for enabling this feature?

6

Hi - currently Foundry Iceberg is in limited Beta release, which means it is disabled by default on all environments unless specifically requested and approved.

You can register interest in the Beta program via your Palantir representative or via your normal Palantir support channels.

Glad you are interested in experimenting with Iceberg!