5.3 QA/QC Parameters

 

This page provides the functionality to see and edit QA/QC parameters for all sites within the system. The page is able to show all sites and parameters together or can be filtered to show only those relating to one site, dataset type or parameter type. "Admin" privileges are required to make changes on this page.

 

 

 

There are many factors that contribute to the quality of monitoring data, and how to measure it.  All of these factors combined are known as Quality Assurance.  These factors include the performance of each instrument, how it is installed and whether and how it is regularly maintained.  They also include the reliability of data transmission and whether missing data records can be recovered when there are errors in communication.  It is also includes being able to assess sensor i.e. raw and processed data against a set of known criteria and to identify or flag any erroneous or suspect data records.  This latter aspect of assessment is commonly known as Quality Control. 

 

The QA/QC system in Port-Log is based on QARTOD - Quality Assurance / Quality Control of Real-Time Oceanographic Data (Ref.) and allows site or station specific quality criteria to be updated based on experience.  Port-Log users can and should regularly assess the quality of the data being identified and adjust the QA/QC criteria accordingly.

 

Data quality auditing and control is undertaken by direct trigger within the Port-Log database, or in some circumstances by stored procedures running at pre-defined intervals within the Port-Log scheduler service or Port-log ETL.  Often, the data is either rejected, or a quality flag set on the record in the database – so that data of a lower quality can be marked or filtered out.  Parameters failing QA/QC are marked accordingly and are not displayed or used in post analysis.

 

The QA/QC tests which are undertaken are:

 

a)     Validation of the incoming data stream by checksum.  This is undertaken by the Port-Log ETL and the data record rejected on failure (if enabled on the station data logger/modem e.g. ip.buffer a missing acknowledgement from the server means the data will be resent).

b)     Data Timeout:  Undertaken by Port-log Scheduler.  Email dispatched.

c)     Validation of data: Undertaken by triggers within the database. Quality flag set.

i.     Missing/Null values

ii.     Exceedance of permitted Minimum/Maximum values.

iii.     Rate of Change compared to previous good value.

d)     Analysis of Data: Undertaken by external or stored procedures – mainly where records have to be checked periodically in comparison to other records within the database:

i.     Calculation of Tidal Prediction – added by Port-Log ETL as data is received, gathered in real-time from the Tide Prediction Data Tables

ii.     Comparison with Tidal Prediction – record quality marked on failure within tolerance.

iii.     Comparison of Standard Deviation of neighbouring records – record quality marked on failure within tolerance.

iv.     Calculation of Mean Sea Level (MSL).  Undertaken by stored procedure periodically as data allows.  Doodson MSL filter applied.

v.     Calculation of observed HW/LW times/heights – undertaken periodically to assess the maxima/minima between zero-crossing times of MSL.  Temporary HW/LW times are added after a peak/trough until such time as MSL is crossed.

 

In most cases a quality rating (0-100) is assigned and can be used on the displays to indicate quality of information and respond accordingly.