Hello,
What’s the maximum upload bandwidth (assuming no network limitation) for the basic Data Connection Agent? Is there even a case when a larger agent will actually be needed?
Best,
Aron
My answer is probably only valid for single tenant stacks where secure access to the backing foundry bucket is possible:
If your data connection agent and your foundry stack run on AWS in the same region you can configure it to write files directly to the Foundry Filesystem (the backing object store of datasets).
With a correct setup this goes through an S3 Gateway Endpoint - which offers almost unlimited upload bandwidth to the underlying S3 Bucket. The maximum upload bandwidth will than depend on the EC2 instance type used - there are instance types that have > 200 Gibps bandwidth (e.g. c8gn.16xlarge) or even more.
If the Data Connection Agent connects to the Foundry load balancer your upload speed will be constrained by that component / the data proxy API behind it.
Yeah, same region cloud ingestion is not the problem. Here, we are looking at on-prem to cloud.
Though the answer sounds like “the network bandwidth is always going to be a stricter limit than the agent hardware”.
1 Like
I assume the Foundry front-door has some scaling limits (compared to virtual unlimited scaling of the S3 Front-door) but that’s probably something your infra team can better answer and provide the answer here. It surely depends on the underlying hyperscaler primitives being used (e.g. in AWS’s case NLB)