6

I have seen other instances of 503 errors using Wget, but to no available I cannot solve this.

When I try to download a certain website, I get a 503 Service Unavailable error. This does not happen to any website except for the one in question.

This is what is happening. I enter:

wget -r --no-parent -U Mozilla http://www.teamspeak.com/

And this is the error I get back.:

--2015-03-12 11:57:08--  http://www.teamspeak.com/
Resolving www.teamspeak.com... 104.28.27.53, 104.28.26.53
Connecting to www.teamspeak.com|104.28.27.53|:80... connected.
HTTP request sent, awaiting response... 503 Service Unavailable
2015-03-12 11:57:09 ERROR 503: Service Unavailable.

This site does use CloudFlare protection (when opening the site you have to wait 5 seconds while it “checks your browser.”

Giacomo1968
  • 55,001
Zac Webb
  • 161

4 Answers4

8

CloudFlare protection is based on JavaScript, cookies and http header filtering. If you want to crawl CloudFlare protected site using wget, you first have to enter it in a browser with debugger (eg. Firefox with Firebug), and copy Cookie request header.

Now the hardest part: this cookie is valid for 1 hour only, so you will have to refresh it manually each hour.

Here is the complete command you can use to crawl the site:

wget -U "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0" --header="Accept: text/html" --header="Cookie: __cfduid=xpzezr54v5qnaoet5v2dx1ias5xx8m4faj7d5mfg4og; cf_clearance=0n01f6dkcd31en6v4b234a6d1jhoaqgxa7lklwbj-1438079290-3600" -np -r http://www.teamspeak.com/

Note that __cfduid cookie value is constant, and you only have to change cf_clearance cookie value each hour.

Tomasz Klim
  • 960
  • I was using python request library to do this instead of wget and this worked for me. Note that first you have to open the page in a proper browser, and then when you take the cookie data, you also have to ensure your script is using the same user agent. – Deep-B May 05 '16 at 20:41
  • Note __cfduid cookie value is now deprecated. See https://blog.cloudflare.com/deprecating-cfduid-cookie/ – sunapi386 Jun 20 '22 at 20:42
2

The issue seems to be that TeamSpeak is using CloudFlare’s DDoS protection in place. See the screenshot at the bottom of the answer. More details on what this protection is/means on this official Amazon page on CloudFlare’s security features:

CloudFlare leverages the knowledge of a diverse community of websites to power a new type of security service. Online threats range from nuisances like comment spam and excessive bot crawling to malicious attacks like SQL injection and denial of service (DOS) attacks. CloudFlare provides security protection against all of these types of threats and more to keep your website safe.

More specifics on their advanced DDoS protection methods can be found here:

CloudFlare's advanced DDoS protection, provisioned as a service at the network edge, matches the sophistication and scale of such threats, and can be used to mitigate DDoS attacks of all forms and sizes including those that target the UDP and ICMP protocols, as well as SYN/ACK, DNS amplification and Layer 7 attacks. This document explains the anatomy of each attack method and how the CloudFlare network is designed to protect your web presence from such threats.

Now how does this factor into the “503 Service Temporarily Unavailable” you are seeing? Well, that means that the site you are trying to access is under such a high level of protection from the Amazon CloudFlare DDoS detection/mitigation services that non-standard access via a command line tool like wget or curl is just not possible at this point.

FWIW, I have done a few different curl attempts from the command line and I believe what happens is that CloudFlare’s DDoS protection just acts like a huge web page proxy for sites that opt to use it. And the “real” website exists somewhere other than the IP address the hostname resolves to. Sites like this claim to give you the “real” IP address connected to a CloudFlare hostname, but it doesn’t seem to work at all. Or maybe the IP address that is given is valid, but the way the service is setup just denies you direct access to the real site without jumping through CloudFlare’s loops.

Which simply means, the best you can do is sit and wait and maybe in a few hours or possibly days the security issues that site faced will fade away and standard wget or curl calls can be made. But the reality is if this security protection is in place, and is solid, and the website owner does not disable it, then you can’t do much to get around it.

enter image description here

Giacomo1968
  • 55,001
1

Just to follow up on this answer https://superuser.com/a/946274/755660 - now that __cfduid cookie is deprecated, this works:

wget --header='cookie: cf_chl_2=5f706f217dfec17; cf_chl_prog=x12; cf_clearance=6on.0F8CTI4m4K2dqEx63zQQD62bq63eF8OOITzovsI-1655756823-0-150' \
  --header='user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36' \
  -np -r https://example.com/uploads

This will get all the sub-directories with -np (no-parent) and -r (recursive).

In order to get these values, open up your browser debugger and copy the network as curl and format it to wget. Only the user agent header and cookie headers are needed.

copy as cURL

Here was what it looked like as cURL (replaced with example.com website) before I modified to wget.

curl 'https://example.com/uploads/' \
  --header='authority: example.com' \
  --header='accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
  --header='accept-language: en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7' \
  --header='cache-control: max-age=0' \
  --header='cookie: cf_chl_2=5f706f217dfec17; cf_chl_prog=x12; cf_clearance=6on.0F8CTI4m4K2dqEx63zQQD62bq63eF8OOITzovsI-1655756823-0-150' \
  --header='referer: https://example.com/wp-content/uploads/' \
  --header='sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="102", "Google Chrome";v="102"' \
  --header='sec-ch-ua-mobile: ?0' \
  --header='sec-ch-ua-platform: "macOS"' \
  --header='sec-fetch-dest: document' \
  --header='sec-fetch-mode: navigate' \
  --header='sec-fetch-site: same-origin' \
  --header='sec-fetch-user: ?1' \
  --header='upgrade-insecure-requests: 1' \
  --header='user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36' \
sunapi386
  • 111
0

This may be easier to use.

@echo off
set U=Mozilla/5.0 (Windows NT 6.1; WOW64; rv:9.0) Gecko/20100101 Firefox/9.0
set cf_clearance=
set SaveTo=
set Optional=-q
:If it fails, replace -q with -d -oLog for details.
for %%f in (
http://itorrents.org/torrent/606029c69df51ab29d5275b8ad4d531fa56a450b.torrent
) do wget %%f %Optional% -U "%U%" --header="Accept:text/html" --header="Cookie:__cfduid=dbef4c7a393e2d6a95385ccfadbc46e371591967392;cf_clearance=%cf_clearance%" -np -nd -P%SaveTo%
pause

According to this, cf_clearance may be valid for up to 1h45. This appears to be a solution for automating retrieval of these tokens. It uses Node.js that doesn't run on XP. Couldn't try it.

Masutin
  • 31