dqe - data quality engine

Data Quality refers to the quality of data. Data are of high quality "if they are fit for their intended uses in operations, decision making and planning" (J.M. Juran).

To meet the demands of the most discerning Financial Institutions DSAL have developed DQE (Data Quality Engine), which is specifically de-signed to validate the quality of market data.

Data Stream Analysis

DQE has been engineered to provide clients with an application that can be configured to process:

  •   Any financial market data type - transactions, trades, quotes, corporate actions or instrument reference or static data.
  •   Any frequency of update - from irregular data, updating hundreds of thousands of times per second to updating once a year, to regular data, updating daily or weekly or monthly or even annually.

DQE provides configurable validation modules to execute the various tests:

  •   Logical - this module can be configured to execute logical validation of data records, such as; data typing, not null, greater than, less than, within range, exists in, etc.
  •   Historical - this module can be configured to execute historical validation of data records, such as; stale (no change in value for time = t), tolerance to previous value (percentage, stan-dard deviation, etc.), tolerance to aggregate (such as moving average over time = t) value, etc.
  •   Comparison - this module can be configured to execute comparison validation of data re-cords, such as; source 1 = source 2 or source 1 = source 2 = source 3 or any two from three, etc.

DQE can be configured to filter records as soon as they fail a single test or to allow them to be proc-essed through all tests first in order to determine all possible failures in a single pass. These failures can then be routed for repair or dropped from the data stream. Once repaired the data can be re-submitted for validation via a separate configurable set of tests.

DQE provides a full audit trail of:

  •   Configuration Data - the rules that are available within each module, the rules that have been configured to execute, the parameters utilized by each rule, etc.
  •   Meta Data - data that defines the instrument classes and individual instruments, which is used to determine whether or not a rule should be executed for an update.
  •   Tests Executed and Results - the tests executed and the result achieved for every update passed through the application.

DQE is currently available for Sun Solaris 9, Red Hat & SuSe Linux and Microsoft Windows 2003 & XP.