Bulk data downloads using FTP

I’ve been needing to download vast amounts of data from various source recently including from the NSIDC. As with the NSIDC, there are various ways made available to you to download data. One useful area is the http file directory that can be accessed such as here. You can the click and save the files that you need. I however needed to sort out a bulk download - this is where using an FTP client comes in handy such as WinSCP.

However, I found it much simpler - especially as the data was all freely available (i.e. no user or password details requried) - to do this from a terminal. In linux, this can be done directly through bash. In a windows environment, the best approach is to download cygwin, ensure that you have “wget” installed (search the package list as and when you run in the cygwin installation .exe). Then all you’ll need to type is:

> wget -P ~/your_output_folder_destination -m ftp://the_url_of_the_ftp_site/*

In the above snippet “-P” is allowing you to specify where you want to put the data you are acquiring and “-m” allows a recursive download of the specified URL. Get all of that running and leave it in the background to complete and you’re done!

Written on December 16, 2014