Can I connect to a HTTP source system via direct connection?

Ask the Community
1

I have a source system, over the internet, that is not a HTTPS source but a HTTP one.

It seems that source creation is limited to HTTPS connections. How can I create an HTTP source ? How can I reach this source from External Transforms ?

Can the direct connection work with HTTP ?
Can an egress be set on an HTTP Source ?

2

You can attach an appropriate egress policy to a Generic source or a Rest source and use the source in external transforms without any issues.

While the UI in the Domain configuration section for the Rest source type implies that the connection must use HTTPS, it seems that there is no such constraint in practice. In any event, configuring a Domain is optional; it’s only needed if you want to use the get_https_connection method on the Source in your transform.

Both of the below code samples work (the first works with either Generic or Rest sources, and the second one only works with a Rest source that has a Domain configured).

@external_systems(
    generic_http_source=Source("...")
)
@transform(
    output=Output("..."),
)
def compute(generic_http_source, output):
    with output.filesystem().open("response.html", "wb") as output_fp:
        resp = requests.get("http://blah.com")
        output_fp.write(resp.content)

@external_systems(
    rest_http_source=Source("...")
)
@transform(
    output=Output("..."),
)
def compute(rest_http_source, output):
    with output.filesystem().open("response.html", "wb") as output_fp:
        resp = rest_http_source.get_https_connection().get_client().get("http://blah.com")
        output_fp.write(resp.content)