Unix and Linux Stack Exchange
Unix and Linux Stack Exchange

Questions tagged [wget]

wget - command-line utility to download content non-interactively(can be called from scripts, cron jobs , terminals without the X-Windows support, etc.)

The wget utility can retrieve files from the World Wide Web (WWW) through protocols like HTTP, HTTPS and FTP. It is non-interactive command line tool that can be called from scripts, cron jobs, terminals without the X-Windows support, etc. It can also work on the background.

GNU wget has many features to make retrieving large files or mirroring entire web or FTP, including:

  • Can resume aborted downloads, using REST and RANGE

  • NLS-based message files for many different languages

  • Optionally converts absolute links in downloaded documents to relative, so that downloaded documents may link to each other locally
  • Runs on most UNIX-like operating systems as well as Microsoft Windows
  • Supports HTTP proxies
  • Supports HTTP cookies
  • Supports persistent HTTP connections
  • Unattended / background operation
  • Uses local file timestamps to determine whether documents need to be re-downloaded when mirroring
  • GNU Wget is distributed under the GNU General Public License.

Examples

Basic usage:

$ wget https://upload.wikimedia.org/wikipedia/commons/3/35/Tux.svg

Downloading image in the background, saving it in logfile.txt and try to download it up to 45 times. 

$ wget -t 45 -o logfile.txt https://upload.wikimedia.org/wikipedia/commons/3/35/Tux.svg &

Reference

840 questions
612
votes
4 answers

Does curl have a --no-check-certificate option like wget?

I am trying to make a curl request to one of our local development servers running a dev site with a self-signed SSL cert. I am using curl from the command line. I saw some blog posts mentioning that you can add to the list of certificates or…
cwd
  • 44,479
  • 71
  • 146
  • 167
461
votes
5 answers

What is the difference between curl and wget?

I am keen to know the difference between curl and wget. Both are used to get files and documents but what the key difference between them. Why are there two different programs?
lakshmen
  • 6,071
  • 5
  • 17
  • 9
161
votes
6 answers

Adding a self-signed certificate to the "trusted list"

I've generated a self-signed certificate for my build server and I'd like to globally trust the certificate on my machine, as I created the key myself and I'm sick of seeing warnings. I'm on Ubuntu 12.04. How can I take the certificate and globally…
Naftuli Kay
  • 38,686
  • 85
  • 220
  • 311
159
votes
3 answers

Download using wget to a different directory than current directory

I need to use wget to download a file to the directory /var/cache/foobar/ (so, as an example, if I download stackexchange-site-list.txt, it'd be downloaded to /var/cache/foobar/stackexchange-site-list.txt) Is this possible? curl would also be an…
jrg
  • 4,206
  • 3
  • 20
  • 16
136
votes
4 answers

Throttle the download speed of wget or curl while downloading

Is it possible to throttle (limit) the download speed of wget or curl ? Is it possible to change the throttle value while it is downloading ?
Gautam
  • 2,295
  • 5
  • 18
  • 18
119
votes
6 answers

How to download an archive and extract it without saving the archive to disk?

I'd like to download, and extract an archive under a given directory. Here is how I've been doing it so far: wget http://downloads.mysql.com/source/dbt2-0.37.50.3.tar.gz tar zxf dbt2-0.37.50.3.tar.gz mv dbt2-0.37.50.3 dbt2 I'd like instead to…
BenMorel
  • 4,447
  • 8
  • 36
  • 46
95
votes
8 answers

How to download a folder from google drive using terminal?

I want to download a folder from my google drive using terminal? Is there any way to do that? I tried this: $ wget "https://drive.google.com/folderview?id=0B-Zc9K0k9q-WWUlqMXAyTG40MjA&usp=sharing" But it is downloading this text file:…
user22180
  • 1,123
  • 1
  • 8
  • 8
86
votes
5 answers

why would curl and wget result in a 403 forbidden?

I try to download a file with wget and curl and it is rejected with a 403 error (forbidden). I can view the file using the web browser on the same machine. I try again with my browser's user agent, obtained by http://www.whatsmyuseragent.com. I do…
starfry
  • 7,302
  • 6
  • 47
  • 69
85
votes
5 answers

Set a network range in the no_proxy environment variable

I'm in a network using a proxy. I've got machines using lots of scripts here and there accessing each other over HTTP. The network is 10.0.0.0/8. My proxy is 10.1.1.1:81, so I set it up accordingly: export http_proxy=http://10.1.1.1:81/ I want to…
SamK
  • 1,580
  • 2
  • 13
  • 14
82
votes
3 answers

Why doesn't cp have a progress bar like wget?

Please note that I don't ask how. I already know options like pv and rsync -P. I want to ask why doesn't cp implement a progress bar, at least as a flag ?
Lamnk
  • 1,450
  • 1
  • 11
  • 10
77
votes
1 answer

What does 2>&1 in this command mean?

I understand that this command attempts to write to nowhere or a null device but what does 2>&1 mean here? wget -q -O - http://yourwebsite.com/wp-cron.php?doing_wp_cron > /dev/null 2>&1
codecowboy
  • 3,362
  • 7
  • 23
  • 29
73
votes
4 answers

wget with wildcards in http downloads

I need to download a file using wget, however I don't know exactly what the file name will be. https://foo/bar.1234.tar.gz According to the man page, wget lets you turn off and on globbing when dealing with a ftp site, however I have a http url.…
spuder
  • 17,643
  • 36
  • 91
  • 119
66
votes
4 answers

How do I display all URLs in a redirect chain?

I'm looking for a way to show all of the URLs in a redirect chain, preferably from the shell. I've found a way to almost do it with curl, but it only shows the first and last URL. I'd like to see all of them. There must be a way to do this simply,…
felwithe
  • 882
  • 1
  • 7
  • 12
61
votes
3 answers

How do I download from SourceForge with wget?

This question explains how to do it with curl. The accepted answer also points out that If wget is available, that would be far simpler. I looked through man wget but the got lost in there, and didn't find an option to follow redirects.
Vorac
  • 2,957
  • 8
  • 36
  • 53
55
votes
3 answers

How to download files with wget where the page makes you wait for download?

I am trying to download a file from sourceforge using wget, but as we all know we have to click on the download button and then wait for it to auto download. how do you download this type of file using wget? I am trying to download this:…
Patoshi パトシ
  • 1,695
  • 6
  • 24
  • 34
1
2 3
…
55 56