|
楼主 |
发表于 2012-5-19 23:40:53
|
显示全部楼层
What about mirroring the whole site?
this will download all pages from a specified site and make their links relative.
- wget --referer="http://www.google.com" \
- --user-agent="Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6" \
- --header="Accept:text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5" \
- --header="Accept-Language: en-us,en;q=0.5" \
- --header="Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7" \
- --header="Keep-Alive: 300" \
- -mk http://www.example.com
复制代码 you could also add -w 10 or whatever number to wait x seconds between requests.
this is the important bit, most of those args are to appear as a regular browser- wget -mk -w 5 http://www.example.com
复制代码 Dont remember for sure if its helping or not, but worth a test |
|