I made a bash script to go through every host in a txt file and return if the HTTP response is 200 ok or not. But it is very slow, and I want to make it faster with a timeout, waiting for the host.

I think I can make it wait a few seconds – and if the host dosen't respond, it should go to the next host in the list.

How can I do this with bash scripting, or – if it is a bad idea – can you give me an alternative solution for this?

I'm totally new to Bash.

for host in $(cat 'hackit.txt');
do echo $host curl -o /dev/null --silent --head --write-out '%{http_code}\n' $host
done

2 Answers

from man curl:

-m, --max-time

Maximum time in seconds that you allow the whole operation to take. This is useful for preventing your batch jobs from hanging for hours due to slow networks or links going down. Since 7.32.0, this option accepts decimal values, but the actual timeout will decrease in accu‐ racy as the specified timeout increases in decimal precision.

and:

--connect-timeout

Maximum time in seconds that you allow curl's connection to take. This only limits the connection phase, so if curl connects within the given period it will continue - if not it will exit. Since version 7.32.0, this option accepts decimal values.

This said, you can also combine this with @slhck parallel suggestion to run queries in parallel.

1

You could use GNU parallel for that. It has a timeout option that terminates individual jobs, and you can also process the URLs in parallel, which massively speeds up the job.

For example if you have a list:

$ cat urls.txt

Run the curl command with parallel:

$ parallel --timeout 5 "curl -o /dev/null --silent --head --write-out '%{url_effective} %{http_code}\n' {}" < urls.txt 2> /dev/null 000 200 200 404

Note:

• --timeout 5 kills each individual job after 5 seconds.
• I've modified the --write-out option so that it prints the URL and the status code in one line separated by a space. This way you can better parse the result.
• You can redirect the result to a file with > results.txt.
• The 2> /dev/null is needed to suppress warnings about timed out jobs.
• The jobs are not executed in the order given in the original list, but that should not really matter.

1

Your Answer

Sign up or log in

Sign up using Google Sign up using Facebook Sign up using Email and Password Submit

Post as a guest

Post Your Answer Discard

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy