download¶

Examples¶

Fetching MODIS

Automated download or “fetching” can be helpful for bulk data retrieval, or for enabling software to pull the most up to date data at runtime. This module has a few fetch functions for different NASA data platforms and products. As an example, the proper syntax for the fetch_MODIS function to download all available data for 2015 (January 1st through December 31st) is

from dnppy import download
from datetime import datetime

product = "MYD11A1"                  # land surface temperature
version = "041"                      # version 041
tiles = ["h11v05", "h12v05"]         # two tiles of interest (SouthEast US)
outdir = r"C:\Users\jwely\test"      # local path to save files
start_dto = datetime(2015,1,1)       # datetime object of start date
end_dto = datetime(2015,12,31)       # datetime object of end date

download.fetch_MODIS(product, version, tiles, outdir, start_dto, end_dto)

The other fetching functions work similarly.

Code Help¶

Auto-documentation for functions and classes within this module is generated below!

download_filelist(ftp_texts, file_type=None, outdir=None)[source]¶

Reads text file of download links, downloads them.

This script reads a text file with urls such as those output from ECHO REVERB and outputs them to an output directory. It will retry failed links 20 times before giving up and outputting a warning to the user.

Parameters:	ftp_texts – array of txt files ordered from reverb containing ftp links file_type – file extension of the desired files, leave blank or False to grab all types. outdir – folder where files are to be placed after download
Return list failed:
	list of files which failed to download after the end of the script.

download_url(url, outname, username=None, password=None, force_overwrite=False)[source]¶

Download a single file. input source url and output filename

Parameters:

url – a string url to file for download outname – filepath to write location of file username – use if url leads to file on ftp server with username password – use if url leads to file on ftp server with password force_overwrite – by default, this function will not overwrite existing files on drive. This is to avoid accidental overwrite of critical files as well as repeated download of the same files.

Return outname:

returns filepath to locally created file after download

download_urls(url_list, outdir, file_types=None)[source]¶

Downloads a list of files. Retries failed downloads

This script downloads a list of files and places it in the output directory. It was built to be nested within “Download_filelist” to allow loops to continuously retry failed files until they are successful or a retry limit is reached.

Parameters:	url_list – array of urls, probably as read from a text file file_types – list of file types to download. Useful for excluding extraneous metadata by only downloading ‘hdf’ or ‘tif’ for example. Please note that often times, you actually NEED the metadata. outdir – folder where files are to be placed after download
Return failed:	list of files which failed download

fetch_GPM_IMERG(start_dto, end_dto, outdir, product='gis', time_res='1day')[source]¶

Fetches 30 minute resolution GPM IMERG data from an ftp server. Several restrictions exist for this relatively new dataset, please read in the input section carefully.

http://pps.gsfc.nasa.gov/Documents/GPM_Data_Info_140616.pdf

Parameters:

start_dto – datetime object for starting time of study boundary end_dto – datetime object for ending time of study boundary outdir – output directory to save the data product – either “early” , “late” or “final” for full HDF5 data stacks of the respective runs, which are all at 30minute resolutions. OR product can be set equal to “gis” (default) to find only tif averages of the precipitation estimates. This gis tif data is ONLY provided for data less than one year old. time_res – if “product” is set to “gis”, specify what time average period you want. options are “30min”, “3hr”, “1day”, “3day”, “7day”. Defaults to “1day”

Returns:

Returns a list of filepaths to freshly downloaded files

learn more at [http://pmm.nasa.gov/data-access/downloads/gpm]

fetch_Landsat8(path_row_pairs, start_dto, end_dto, outdir, max_cloud_cover=100, bands=None)[source]¶

This function downloads all landsat 8 tiles for the input path_row_pairs and within the bounds of the start_dto and the end_dto, and saves them to the output directory. It uses the amazon web service at [https://aws.amazon.com/public-data-sets/landsat/]

Parameters:	path_row_pairs – tupled integer values of path,row coordinates of tile. may be a list of several tuples. example: [(1,1),(1,2)] start_dto – python datetime object of start date of range end_dto – python datetime object of end date of range outdir – the folder to save the output landsat files in max_cloud_cover – maximum percent cloud cover that is acceptable to download the file.
Return output_filelist:
	A list of tile names downloaded by this function.

fetch_Landsat_WELD(product, tiles, years, outdir)[source]¶

Fetch WELD data from the server at [http://e4ftl01.cr.usgs.gov/WELD]. Weld data is corrected and processed Landsat 5 and 7 data that is distributed in the MODIS sinusoidal projection and grid format. Read more about WELD data. https://landsat.usgs.gov/WELD.php http://globalmonitoring.sdstate.edu/projects/weldglobal/

Parameters:	product – WELD product to download such as ‘USWK’,’USMO’,’USYR’ tiles – list of tiles to grab such as [‘h11v12’,’h11v11’] years – list of years to grab such as range(2001,2014) outdir – output directory to save downloaded files
Return output_filelist:
	A list of full filepaths to files fetched be this function

fetch_MODIS(product, version, tiles, outdir, start_dto, end_dto, force_overwrite=False)[source]¶

Fetch MODIS Land products from one of two servers. If this function runs and downloads 0 files, check that your inputs are consistent with the naming convention at the appropriate server address.

http://e4ftl01.cr.usgs.gov ftp://n5eil01u.ecs.nsidc.org

Parameters:	product – MODIS product to download such as ‘MOD10A1’ or ‘MYD11A1’ version – version number, usually ‘004’ or ‘041’ or ‘005’ tiles – list of tiles to grab such as [‘h11v12’,’h11v11’] NOTE: for some MODIS products, the h and v are omitted.

:param outdir : output directory to save downloaded files :param start_dto: datetime object, the starting date of the range of data to download :param end_dto: datetime object, the ending date of the range of data to download :param force_overwrite: will re-download files even if they already exist

Return out_filepaths:
	a list of filepaths to all files created by this function

fetch_MPE(start_dto, end_dto, outdir, area=None)[source]¶

Fetches Multisensor Precipitation Estimates data from weather/noaa server at [http://water.weather.gov/precip/p_download_new/]

Parameters:	start_dto – datetime object for start date of desired range end_dto – datetime object for end date of desired range outdir – output directory where files should be saved (str) area – area of interest, either “conus”, “ak” or “pr” for continental us, alaska, or Puerto Rico respectively
Return output_files:
	list of output files fetched by this function

fetch_SRTM(ll_lat, ll_lon, ur_lat, ur_lon, product, outdir=None, mosaic=None)[source]¶

downloads data from the Shuttle Radar Topography Mission (SRTM) [http://e4ftl01.cr.usgs.gov/SRTM/]

This data can be used to create DEMs of a variety of resolutions.

Parameters:	ll_lat – latitude of lower left corner ll_lon – longitude of lower left corner ur_lat – latitude of upper right corner ur_lon – longitude of upper right corner product – short name of product you want. See http://e4ftl01.cr.usgs.gov/SRTM/ . do not include the version number. Example: “SRTMGL1”. Note that version “002” data of .DEM format does not support mosaicing. outdir – local directory to save downloaded files mosaic – Set to TRUE to mosaic all downloaded DEM tiles as “SRTM_mosaic.tif”
Return tile_list:
	a list of all successfully downloaded tif filepaths for further manipulation

NOTE: arcmap will open the output hgt files ONLY if they are not renamed. turns out arcmap does some funky things when interpreting these files.

fetch_TRMM(start_dto, end_dto, outdir, product_string)[source]¶

Fetches TRMM data from an FTP server.

ftp://trmmopen.gsfc.nasa.gov/trmmdata/ByDate/V07/

Parameters:

start_dto – datetime object for start date of desired range end_dto – datetime object for end date of desired range outdir – output directory where files should be saved (str) product_string – the string for the desired product, options include 1B11, 1B21, 1CTMI, 2A12, 2A21, 2A23, 2A25, 2B31, 3B42, 3G25, 3G31. The usual precip product of interest is the well known 3B42 data product. output_files – a list of new filepaths created by this function

list_http_e4ftl01(site)[source]¶

Lists contents of http download site at [http://e4ftl01.cr.usgs.gov] which hosts select MODIS products, landsat WELD, and SRTM data.

Parameters:	site – a url to somewhere on the server at http://e4ftl01.cr.usgs.gov
Return file_urls:
	returns a list of urls to files on that http page.

list_http_waterweather(site)[source]¶

Lists contents of http directories at [http://water.weather.gov/precip/p_download_new/] which hosts MPE data.

Parameters:	site – url to somewhere on the server at http://water.weather.gov/precip/p_download_new/
Return file_urls:
	returns a list of file urls at input site.

list_ftp(site, username=None, password=None, dir=None)[source]¶

lists contents of typical FTP download site. returns two lists, the first is of filenames, the second is of full filepaths (including filenames) that one could patch through to the “download_url” function. returns False if the server has rejected our connection

Parameters:	site – url to ftp website root, does not need to include “ftp://“ username – username to log in with, if required password – password to log in with, if required dir – specific ftp directory we wish to access
Returns filenames:
	a list of filenames
Returns filepaths:
	a list of full filepaths