httr / corporate proxies


#1

Apologize in advance for the lack a reprex here, I think it’s to situation-specific to really provide a good one.

So I know that you can set proxy settings for httr using use_proxy or by editing the .Renviron file to include proxy information/credentials. This works fine for 99% of websites but for whatever reason the request times out if I point it at a website hosted by my organization.

i.e.

httr::GET('https://ExternalSite.com')
httr::GET('http://ExternalSite.com')

both work fine but

httr::GET('https://MyOrganizationsSite.com')

fails with

Error in curl::curl_fetch_memory(url, handle = handle) :
Timeout was reached: Operation timed out after 10000 milliseconds with 0 out of 0 bytes received

I assume this has something to do with not needing to route through the proxy for websites hosted by my organization?

Does anyone know of a way around this besides repeatedly turning my proxy settings on and off? I could of course include use_proxy in any scripts to dynamically switch back and forth but this seems sloppy…


#2

So does a GET request to your org. work fine without the proxy?


#3

Yep that’s correct.

For some context, I’ve written a package (https://github.com/rywhale/kiwisR) that wraps a data retrieval API. There are a bunch of hubs that use this API framework to provide data access, all hosted by various organizations (one of which is mine). If I do not set my proxy information, I can access my org’s hub fine but cannot send GET requests to any externally hosted site (e.g. Google, the other API hubs). If I enter my proxy info I can send requests to those external hubs but requests to my internal one all fail with timeout.

I suppose I could dynamically change the proxy settings depending on the url being queried…


#4

Hi,

I think you need to set also the environment variable NO_PROXY with your company domain. This variable is taken into account by curl to not use proxy for url of this domain. It should work in your case then.

As proxy configuration is always a pain, I started a :package: to help with that

It will help you set interactively when needed your environment variables for the sessions. It includes the no proxy configuration.

It is just a start and a small idea, so I take every comment if you want to give it a try.
Thanks!


#5

Thanks a lot for this, works perfectly!

+2 points for you sir.