Hi I’m following the Deep Dive: Creating Your First Data Connection. After following the tutorial I get Configuration error with this error message. I have added egress policy with s3-bucket.sandbox.training.palantir.com.s3.eu-west-1.amazonaws.com (Port 443). However, it is not working. Can someone help me with this?
It looks like this same issue has been posted a couple of times over the past year or so.
The error you’re encountering indicates that the credentials provided do not have sufficient permissions to access the requested S3 resource. Specifically, the user arn:aws:iam::441477072185:user/s3-training-test-user-1 is not authorized to perform the s3:ListBucket action on the specified S3 bucket. This is likely due to a missing or incorrect identity-based policy that allows this action.
Suggested Fix:
- Check IAM Policies: Ensure that the IAM user or role associated with the credentials has a policy attached that allows the
s3:ListBucketaction on the specified bucket. You can modify the policy to include the necessary permissions. - Verify Bucket Policy: If the IAM policy is correct, check the bucket policy to ensure it allows the necessary actions from the user’s account.
- Network Configuration: If the bucket is in a different region or there are network restrictions, ensure that the Foundry egress IPs or VPC endpoints are allowed to access the bucket.
Here is a minimal example of how you might update the IAM policy to include the necessary permissions:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::s3-bucket.sandbox.training.palantir.com" } ] }
If you continue to experience issues, it may be beneficial to contact Palantir support for further assistance, especially if there are specific network configurations or permissions that need to be verified.
For more detailed information, you can refer to the AWS IAM documentationand the Amazon S3 documentation.
The explorer command failed to run:
java.lang.Throwable:RemoteException: CUSTOM_CLIENT (ExplorerCommand:ExplorerCommandFailed) with instance ID caa19286-f51b-4a54-bd95-a6cc11f27858: {stacktrace=com.palantir.magritte.plugin.s3.errors.AwsForbiddenEnhancement.enhanceException(AwsForbiddenEnhancement.java:30)
com.palantir.magritte.plugin.s3.errors.AwsExceptionEnhancer.lambda$enhanceException$0(AwsExceptionEnhancer.java:46)
java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)
java.util.AbstractList$RandomAccessSpliterator.tryAdvance(AbstractList.java:708)
java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:129)
java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:527)
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:513)
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:150)
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:647)
com.palantir.magritte.plugin.s3.errors.AwsExceptionEnhancer.enhanceException(AwsExceptionEnhancer.java:47)
com.palantir.magritte.plugin.s3.errors.AwsExceptionEnhancer.runWithExceptionEnhancement(AwsExceptionEnhancer.java:37)
com.palantir.magritte.plugin.s3.WrappedS3Client.listObjectsV2(WrappedS3Client.java:37)
com.palantir.magritte.plugin.s3.S3Crawler$ListObjectsV2ResultIterator.computeNext(S3Crawler.java:50)
com.palantir.magritte.plugin.s3.S3Crawler$ListObjectsV2ResultIterator.computeNext(S3Crawler.java:30)
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
java.util.Iterator.forEachRemaining(Iterator.java:132)
java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1939)
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
com.palantir.magritte.plugin.s3.S3FileAndDirExplorer.getFileTreeNodes(S3FileAndDirExplorer.java:44)
com.palantir.magritte.plugin.s3.S3BucketReader.getFileTreeNodes(S3BucketReader.java:68)
com.palantir.magritte.plugin.s3.S3DirectSource.getFileTreeNodes(S3DirectSource.java:264)
com.palantir.magritte.source.explore.FileBasedExplorableSource$1.visitGetFileTreeNodesRequest(FileBasedExplorableSource.java:21)
com.palantir.magritte.source.explore.FileBasedExplorableSource$1.visitGetFileTreeNodesRequest(FileBasedExplorableSource.java:18)
com.palantir.magritte.explorer.api.FileBasedExplorationRequest$GetFileTreeNodesRequestWrapper.accept(FileBasedExplorationRequest.java:265)
com.palantir.magritte.explorer.api.FileBasedExplorationRequest.accept(FileBasedExplorationRequest.java:73)
com.palantir.magritte.source.explore.FileBasedExplorableSource.exploreFileBased(FileBasedExplorableSource.java:18)
com.palantir.magritte.api.Source.lambda$explore$7(Source.java:66)
com.palantir.magritte.explorer.api.ExplorationRequest$VisitorBuilder$1.visitFileBased(ExplorationRequest.java:227)
com.palantir.magritte.explorer.api.ExplorationRequest$FileBasedWrapper.accept(ExplorationRequest.java:315)
com.palantir.magritte.explorer.api.ExplorationRequest.accept(ExplorationRequest.java:87)
com.palantir.magritte.api.Source.explore(Source.java:73)
com.palantir.magritte.cloud.explorer.CloudSourceExplorationResource.lambda$executeExplorerCommand$26(CloudSourceExplorationResource.java:355)
com.palantir.magritte.connector.plugin.common.exploration.GetExplorationResponseVisitor.visitFileBased(GetExplorationResponseVisitor.java:26)
com.palantir.magritte.connector.plugin.common.exploration.GetExplorationResponseVisitor.visitFileBased(GetExplorationResponseVisitor.java:16)
com.palantir.magritte.explorer.api.ExplorationRequest$FileBasedWrapper.accept(ExplorationRequest.java:315)
com.palantir.magritte.explorer.api.ExplorationRequest.accept(ExplorationRequest.java:87)
com.palantir.magritte.cloud.explorer.CloudSourceExplorationResource.executeExplorerCommand(CloudSourceExplorationResource.java:368)
com.palantir.magritte.cloud.explorer.CloudSourceExplorationResource.lambda$getExplorationResponse$21(CloudSourceExplorationResource.java:180)
com.palantir.magritte.cloud.explorer.CloudSourceExplorationResource.rethrowRuntimeExceptionsAsExplorerCommandFailures(CloudSourceExplorationResource.java:258)
com.palantir.magritte.cloud.explorer.CloudSourceExplorationResource.getExplorationResponse(CloudSourceExplorationResource.java:180)
com.palantir.magritte.cloud.explorer.CloudSourceExplorationServiceEndpoints$GetExplorationResponseEndpoint.handleRequest(CloudSourceExplorationServiceEndpoints.java:74)
com.palantir.conjure.java.undertow.runtime.ConjureExceptionHandler.handleRequest(ConjureExceptionHandler.java:42)
com.palantir.tracing.undertow.TracedStateHandler.handleRequest(TracedStateHandler.java:44)
com.palantir.conjure.java.undertow.runtime.LoggingContextHandler.handleRequest(LoggingContextHandler.java:40)
io.undertow.server.Connectors.executeRootHandler(Connectors.java:395)
io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:861)
com.palantir.witchcraft.ActiveCountingExecutorService$TaskWrapper.run(ActiveCountingExecutorService.java:84)
com.palantir.nylon.threads.RenamingExecutorService$RenamingRunnable.run(RenamingExecutorService.java:92)
org.jboss.threads.EnhancedViewExecutor$EnhancedViewExecutorRunnable.run(EnhancedViewExecutor.java:496)
org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2651)
org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2630)
org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1622)
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1589)
com.palantir.tritium.metrics.TaggedMetricsThreadFactory$InstrumentedTask.run(TaggedMetricsThreadFactory.java:94)
java.lang.Thread.run(Thread.java:1583), exceptionClass=com.palantir.logsafe.exceptions.SafeRuntimeException, message=The credentials provided have insufficient rights to access the requested resource, please check that the provided credentials are valid and have the required permissions. This might be caused by the bucket not allowing request from the job’s origin. For agents, you will need to allow incoming traffic from the agent host or from the proxy if you have one setup (this can be setup in the source config or in the agent advanced config via JVM properties). For direct connection, if your bucket is in the same region as the Foundry stack please allow incoming traffic from the Foundry vpc endpoint (please contact Palantir to get the right endpoint). If your bucket is in a different region please allow incoming requests from the Foundry egress IPs found in the control panel app.: {exceptionClass=com.amazonaws.services.s3.model.AmazonS3Exception, isRetryable=true, statusCode=403, errorType=Client, serviceName=Amazon S3, errorCode=AccessDenied, requestId=XH54BCEWPFX8200N, errorMessage=User: arn:aws:iam::441477072185:user/s3-training-test-user-1 is not authorized to perform: s3:ListBucket on resource: “arn:aws:s3:::s3-bucket.sandbox.training.palantir.com” because no identity-based policy allows the s3:ListBucket action (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: XH54BCEWPFX8200N; S3 Extended Request ID: jFizTD6SZdl1zQv+rd9HvU1ejxRdZlv4eLKvli7LVpo9Cjzf9GudGHFAO5owQC8lKTtEKMc7OrU=; Proxy: )}}
com.palantir.conjure.java.dialogue.serde.DefaultClients.newRemoteException(DefaultClients.java:148)
com.palantir.conjure.java.dialogue.serde.DefaultClients.block(DefaultClients.java:123)
com.palantir.conjure.java.dialogue.serde.DefaultClients.callBlocking(DefaultClients.java:76)
com.palantir.magritte.cloud.explorer.CloudSourceExplorationServiceBlocking$1.getExplorationResponse(CloudSourceExplorationServiceBlocking.java:128)
com.palantir.magritte.coordinator.command.CloudSourceExplorationRequestMapper$1.subtype(CloudSourceExplorationRequestMapper.java:84)
com.palantir.magritte.coordinator.command.CloudSourceExplorationRequestMapper$1.subtype(CloudSourceExplorationRequestMapper.java:59)
com.palantir.magritte.bridge.command.SourceExplorationCommand.map(SourceExplorationCommand.java:50)
com.palantir.magritte.coordinator.command.CloudSourceExplorationRequestMapper.getCloudSourceExplorationResponse(CloudSourceExplorationRequestMapper.java:59)
com.palantir.magritte.coordinator.command.SourceExplorerCommandRunner.getCloudRunExplorerResponse(SourceExplorerCommandRunner.java:139)
com.palantir.magritte.coordinator.command.SourceExplorerCommandRunner.lambda$getExplorerResponseForSource$1(SourceExplorerCommandRunner.java:111)
com.palantir.magritte.store.source.api.RuntimePlatformResponse$VisitorBuilder$1.visitCloud(RuntimePlatformResponse.java:175)
com.palantir.magritte.store.source.api.RuntimePlatformResponse$CloudWrapper.accept(RuntimePlatformResponse.java:297)
com.palantir.magritte.store.source.api.RuntimePlatformResponse.accept(RuntimePlatformResponse.java:70)
com.palantir.magritte.coordinator.command.SourceExplorerCommandRunner.getExplorerResponseForSource(SourceExplorerCommandRunner.java:95)
com.palantir.magritte.coordinator.resources.FileBasedSourceExplorationResource.getFileTreeNodesInner(FileBasedSourceExplorationResource.java:71)
com.palantir.magritte.coordinator.resources.FileBasedSourceExplorationResource.getFileTreeNodes(FileBasedSourceExplorationResource.java:48)
com.palantir.magritte.coordinator.api.FileBasedSourceExplorationServiceEndpoints$GetFileTreeNodesEndpoint.handleRequest(FileBasedSourceExplorationServiceEndpoints.java:86)
com.palantir.conjure.java.undertow.runtime.ConjureExceptionHandler.handleRequest(ConjureExceptionHandler.java:42)
com.palantir.tracing.undertow.TracedStateHandler.handleRequest(TracedStateHandler.java:44)
com.palantir.conjure.java.undertow.runtime.LoggingContextHandler.handleRequest(LoggingContextHandler.java:40)
io.undertow.server.Connectors.executeRootHandler(Connectors.java:395)
io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:861)
com.palantir.witchcraft.ActiveCountingExecutorService$TaskWrapper.run(ActiveCountingExecutorService.java:84)
com.palantir.nylon.threads.RenamingExecutorService$RenamingRunnable.run(RenamingExecutorService.java:92)
org.jboss.threads.EnhancedViewExecutor$EnhancedViewExecutorRunnable.run(EnhancedViewExecutor.java:496)
org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2651)
org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2630)
org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1622)
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1589)
com.palantir.tritium.metrics.TaggedMetricsThreadFactory$InstrumentedTask.run(TaggedMetricsThreadFactory.java:94)
java.lang.Thread.run(Thread.java:1583)