I am running 25 scrapy jobs, each has 50 concurrent requests on a heavy machine (with Ubuntu 12.04.3 LTS). I am using proxies to send requests. After 15 minutes or so, I get this error from almost all of the requests:

Connection was refused by other side: 111: Connection refused.

I contacted the proxy service provider and they say that this problem is from handling sockets on my side.

So my scripts send 1250 concurrent requests. Is there anyway I could find out where the problem is? Is there a connection limit on Ubuntu?

2 Answers

I did a small amount of research on this and i've found a very similar thread on serverfault here. It seems that your kernel might be a limiting factor in this particular case, as such a high amount of concurrent connections is not very common. The accepted answer states it might be due to the ip_conntrack table overflowing.

On Ubuntu (Server) a fix could be setting some custom parameters in /etc/sysctl.conf

net.ipv4.netfilter.ip_conntrack_max = 32768
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 0
net.ipv4.tcp_orphan_retries = 1
net.ipv4.tcp_fin_timeout = 25
net.ipv4.tcp_max_orphans = 8192
net.ipv4.ip_local_port_range = 32768 61000

Please take a look at the thread I linked to, and always be very careful with custom configurations.

I had the same problem. Try running your spider as superuser. I was running the spider directly with the command scrapy runspider, when I just tried executing it with:

sudo scrapy runspider

it worked.

Your Answer

Sign up or log in

Sign up using Google Sign up using Facebook Sign up using Email and Password Submit

Post as a guest

Post Your Answer Discard

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy