Time Series Data File Format

The Time Series Data file format has been developed to provide a flexible way of storing large quantities of time series data from different sources so that the data can be imported into InfoWorks.

Contact us to:

• let us know that you are interested in using the format
• get the latest format definitions, in case they have changed from those detailed here
• tell us of any special requirements you may have from the format

 

The following sections describe the format of the Time Series Data Files:

1. TSD File
2. DAT File

The format descriptions are correct at the time of issue of this help file.

TSD File

TSD files contain a header section followed by a data section. Any lines that start with a semi-colon are comment lines and will be ignored by the file reader. An example is given below:

 

[TSD_VERSION=3.0]

[SYSTEM_TYPE=Radcom logger]

FO120716,FO12 STATION FLOW,Flow,m3/h,USED,0,500

FO120717,FO12 BOREHOLE FLOW,Flow,m3/h,USED,0,500

Header Section

The header contains miscellaneous parameters and their values.

Each line has a parameter (one of a set of enumerated parameter types) followed by = and a value (see below). Each parameter-value pair is enclosed in a pair of square braces. Only one parameter-value pair can appear on a line.

Data Section

This is the most important section of the TSD file.

It consists of a number of comma-separated lines that contain the measurement points or locations used in the system. The comma-separated fields also contain other important information such as the reading type and units.

The first field of each comma-separated line is a unique key. This key is used to relate the information in the other fields on the line to the values (the actual readings) in the DAT files.

The values contained in each field respectively are:

Field	Description	Format	Example
Key	The unique identifier used to relate the other data on the line to the values in the DAT file	8 character string containing characters [0-9] and [A-Z] Note that: Key comparisons are case insensitive Leading and trailing spaces are ignored	FO120716
Location	The location of the values	String of any combination of alphanumeric characters except for comma and double quotes Note that: All string comparisons are case insensitive Leading and trailing spaces are ignored	FO12 STATION FLOW
Data Type	The type of measurement	One of the enumerated types in the set Data Types. (See Data Types and Units below)	Flow
Units	The units of measurement	One of the enumerated types in the set Units. (See Data Types and Units below)	m3/h
Status	Reserved field: Set value to USED	Enumerated type	USED
Minimum Valid Value	The lower and upper acceptable limits of values respectively. These optional fields can be used to filter or flag incongruous values that appear in the DAT files. For example, this may happen if an instrument becomes faulty. How these values are dealt with will depend on the specific implementation of the system reading the TSD file	Number Leave these fields blank or white space if unused Floating Point or Integer Signed or unsigned Do not use exponents e.g. 1.0234e-01	0
Maximum Valid Value			500

Data Types & Units

Acceptable units for time series data files are:

• Units present in the dropdown list of the Display Units column of the Units tab that can be found on the Options dialog (available under Tools | Options) for the corresponding data type.
• Units detailed in the table below:


Please Note

If using any of the units below, please type in the exact strings, as they appear inside the quotes.

Data Type	Acceptable Units
FLOW	'l/d', 'l/hr', 'l/min', 'l/s', 'Litres per sec' 'Ml/d', 'Million litres per day' 'm3/h', 'm3/hr', 'm3/hour', 'cum/hr', 'm3/s', 'cfs'
PRESSURE	'm', 'mHd', 'metre', 'Metres' 'cm', 'mm', 'mmH2O' 'bar', 'psi', 'ft', 'KPa'
DEPTH	'm' 'cm' 'mm'
CONCENTRATION	'mg/l', 'microg/l' 'ppb', 'ppm'
PUMP_RUNNING	None (Any non-zero value from the DAT file is read as ON)
PC_VOLUME	'% full' (Values from the DAT file are read as percentages)
OPENING	'% open' (Values from the DAT file are read as percentage opening)

DAT Files

The DAT files are a set of date stamped files that contain a number of time stamped sections.  Each section contains a set of key-value-flag records. An example is given below:

 

_00:00

FO120716,238.0952, 1

FO120717,102.3199, 1

_00:21

FO120716,236.0195, 1

FO120717,102.3199, 2

FO120718,3.2451, 1

FO120719,2.2073, 1

_00:33

FO120716,236.3858, 1

FO120717,102.0757, 2

FO120718,3.2599, 1

FO120719,2.2073, 1

 

DAT files have a specific naming convention representing the date of the data it contains. The format is:

YYYY-MM-DD.dat

where:

YYYY is a 4-digit year

MM is a 2-digit month (01 to 12)

DD is a 2-digit day (01 to 31)

Leading zeroes are required. The date components must be separated by a minus.

Examples are:

2001-01-20.dat

1998-12-31.dat

Sections and Key-Value-Flag Records

Each section contains a time stamp and a series of values recording at this time. The time stamped sections appear in ascending chronological order. Each value appears as a key-value-flag record. The key is used to relate the value to the data stored in the TSD file.

Row	Format	Examples
Timestamp	_hh:mm where: hh is a 2 digit hour (00 to 23). mm is a 2 digit minute (00 to 59). Note that: Leading zeroes are required A colon must separate the time components The time must be preceded by an underscore	_00:00   _00:21
Key-Value-Flag Records	key,value,flag where: key is a key that appears in the TSD file value is the value (see Note 3 above) flag is an optional integer field that can be used to indicate the reliability, or quality of the data. See the Live Data Flags topic for more information on displaying flag information.	FO120716,238.0952, 1 FO120717,102.3199, 1