The textio module, read like “text I/O”, is a repository of functions for reading specific text data formats as they are served up from their respective DAACs. Custom text file formats are common in historical weather data and other ground based data collection networks. This module aims to convert them to something more standardized. Some limited json writing capabilities exist.
Text reader for DS3505 data (space delimited) with some fixes
Weather data downloaded from the following website has a peculiarity [http://gis.ncdc.noaa.gov/map/viewer/#app=cdo&cfg=cdo&theme=hourly&layers=1&node=gi] in that it has some upercase T’s that are rarely needed, but ruin otherwise uniform space formatting.
- filepath – filepath to DS3505 data
- has_headers – set False if filepath does not have headers. This doesn’t seem to ever happen for DS3505 data.
returns a text data object with DS3505 data in it.
A text data object is a very simple template structure for passing text type data (usually lists of weather or climate data entries) around between functions.
Builds column wise data dictionary structure out of row_data. (row_data is a list of rows, where each row is a list, so row_data is a list of lists). Col data is a single dictionary, where the keys are the “headers” and the value is the list of all values in that column from the top down.
Builds row wise data from existing column data. The opposite of _build_col_data.
Appends digits to duplicate items in a list. Used to ensure each header is a unique string so a column-wise dictionary can be built.
For example, a text file with headers
["name", "tag", "tag", "tag"]will be changed to have headers
["name", "tag1", "tag2", "tag3"].
Parameters: headers – list of string elements intended for use as headers.
read_csv(text_filepath, delim=', ', has_headers=True)¶
simple default reader of a delimited file. Does not read fixed-width
- text_filepath – csv filepath to read from
- delim – delimiter to use, defaults to comma
- has_headers – Set “False” if csv file has no headers (this is bad, you should give your file headers)
read_json(json_filepath, row_wise=None, col_wise=None)¶
Reads the contents of this tdo from a json file created by the
text_data.write_json()function. Please note that this text_data class is designed for use with tabular type data, so this should function will not read ALL json files in a satisfactory manner. Users wishing to read json files in a general sense should simply use the
jsonmodule and invoke
json.dumpsdirectly on their data.
- json_filepath – json filepath to read from
- row_wise – read json file objects in as rows
- col_wise – read json file objects in as columns
write_csv(text_filepath, delim=', ')¶
writes the contents of this text file object as a CSV
- text_filepath – output filepath to write csv .txt
- delim – delimiter to use. defaults to comma
write_json(json_filepath, row_wise=None, col_wise=None)¶
Writes the contents of this text data object to a json file. Note that json format does not support complex numbers with imaginary components. Also note that json formats are dictionary like, in that they preserve relationships, but do not display the list of relationships in any particular order.
- json_filepath – output filepath to write json
- row_wise – set to TRUE to save each row as its own structure
- col_wise – set to True to save each col as its own structure
An ioconfig object is an extension to the text_data_class it has the same methods of text_data_class, plus an add_param function and simplified
read()methods for making visually formatted csv files so users can edit config files directly.
It is used to generate “config” files, lists of important inputs to a set of complex functions. Allows you to save inputs and call them over and over again to avoid tedious input definitions, also facilitates good record keeping by keeping a hard file with a summary of all inputs alongside any outputs that might have been generated.
When a param has been added or imported, its value may be accessed with
Parameters: input_filepath – Optional filepath to read/write this ioconfig object to/from. If you do not provide a filepath here, you can provide one when invoking the
allows interpretation of config values based on type
Adds parameters to the ioconfig object
- param_names – A string, the name of the parameter
- param_values – the value of the parameter. May be any built in datatype. (bool, int, str, list, long, etc)
when a param has been added or imported, its value may be accessed with
Reads the contents of a csv file generated by
ioconfig.write(), interprets and formats each of the variables to the state it was before exporting. Once finished, this the ioconfig object will have all its standard attributes including
Parameters: filepath – filepath to a csv or txt file generated by the
Prepares the row data with nice visual formatting then calls the
Parameters: filepath – filepath to write csv to. (.txt, .csv, etc)