Specification of the Ntuple Query Processor in PAW (since version 2.07)

• Introduction
• Basic types supported
• Basic operators
• Basic expressions
• The concept of arrays and shape matching
• Arrays
• Shape matching
• Builtin functions
• Constants
• Math functions
• Bit handling functions
• Logical operations
• Shift operations
• Bit testing
• Bit subfields
• Conversion
• Vector reduction operations
• Strings manipulations
• Predefined cuts
• COMIS functions
• Kuip vectors

See also the PAW 2.07 release notes.

Introduction

The interactive analysis of high energy physics event data using HBOOK ntuples has always been one of the major strengths of the PAW system. With the design and implementation of the PIAF system and the advent of Column Wise Ntuples (CWNs) the quantity of event data which can be analysed interactively has been significantly increased. Other improvements due to the use of CWNs include the support for more data types and new possibilities for structuring the data. CWNs can store boolean, bit field, integer, real, double precision and character string data types. Data can be structured in arrays of one or more dimensions, with the last dimension of each array being of variable length. This allows for efficient storage of lists of values, lists of vectors etc. The actual length of the array in a given event is defined by another integer variable in the ntuple.

The component of the PAW system responsible for executing queries like ntuple/plot or ntuple/project was previously only partially adapted to handle these new features. Furthermore, this part of PAW lacked stability and robustness, and was hindered by many built-in limitations.

To overcome these deficiencies a new query processor has been developed which removes most limitations of the old system. More importantly, it offers a regular syntax which allows much greater freedom to the user when expressing queries. It also supports all the data types offered by the CWNs in combination with automatic type promotion. The new query language is largely backward compatible, with more powerful alternatives being offered in the few incompatible cases. Non scalar ntuple variables like vectors and multi dimensional arrays, possibly of variable length, are fully integrated. Other new features include access to KUIP vectors, and the use of arrays that index other arrays, which allows for more complex structures in ntuples.

The structure of the query compiler allows for optimisations, like static sub-expression evaluation, and reordering of terms based on their complexity. It is foreseen to upgrade the MASK mechanism and to use this to cache the results of cuts (macros).

Basic types supported

The variable type supported by the new query processor are those defined in the Column Wise Ntuple:

• Boolean (B).
• Unsigned int 32 bits integers (U32).
• unsigned int 64 bits integers (U64).
• 32 bits integers (I32).
• 64 bits integers (I64).
• 32 bits floating point (F32).
• 64 bits floating point (F64).
• Character strings (S).

The Ntuple variables can be simple variables (RWN and CWN) or array variables (CWN only).

Basic operators

In the table below, the key letters L,R and N refer to the operator precedence as being Left, Right or Non associative respectively. The table begins with the operator of highest precedence .

+------------+---------------+------------+-------------------------------+---------------------------------+
| Precedence |   Operator    |  Operator  |    Description                |   Type                          |
|   level    |               | precedence |                               |                                 |
+------------+---------------+------------+-------------------------------+---------------------------------+
|      1     | **            |     R      | exponential                   |                                 |
+------------+---------------+------------+-------------------------------+---------------------------------+
|      2     | -             |     L      | unary minus                   | U32,U64,I32,I64,F32,F64         |
+------------+---------------+------------+-------------------------------+---------------------------------+
|      3     | *             |     L      | multiply                      | U32,U64,I32,I64,F32,F64         |
|            | /             |     L      | divide                        | U32,U64,I32,I64,F32,F64         |
+------------+---------------+------------+-------------------------------+---------------------------------+
|      4     | +             |     L      | addition                      | U32,U64,I32,I64,F32,F64 (S ??)  |
|            | -             |     L      | subtracton                    | U32,U64,I32,I64,F32,F64         |
+------------+---------------+------------+-------------------------------+---------------------------------+
|            | = == .eq.     |     N      | equality                      | U32,U64,I32,I64,F32,F64,S       |
|            | < >  != .ne.  |     N      | non-equality                  | U32,U64,I32,I64,F32,F64,S       |
|            | # .ct.        |     N      | close to (*specify*)          | U32,U64,I32,I64,F32,F64,S       |
|            | <  .lt.       |     N      | less then                     | U32,U64,I32,I64,F32,F64,S       |
|            | <  = .le.     |     N      | less then or equal            | U32,U64,I32,I64,F32,F64,S       |
|            | >  .gt.       |     N      | greater than                  | U32,U64,I32,I64,F32,F64,S       |
|            | > = .ge.      |     N      | greater than or equal         | U32,U64,I32,I64,F32,F64,S       |
|            | a <  b <  c   |     N      | dbl comp.(also with .lt. etc) | U32,U64,I32,I64,F32,F64,S       |
|      5     | a <  b < = c  |     N      |                               |                                 |
|            | a < = b <  c  |     N      |                               |                                 |
|            | a > = b > = c |     N      |                               |                                 |
|            | a >  b >  c   |     N      |                               |                                 |
|            | a >  b > = c  |     N      |                               |                                 |
|            | a > = b >  c  |     N      |                               |                                 |
|            | a > = b > = c |     N      |                               |                                 |
+------------+---------------+------------+-------------------------------+---------------------------------+
|            | !  .not.      |     R      |  unary boolean not            | B                               |
|      6     | && .and.      |     L      |  boolean and                  | B                               |
|            | || .or.       |     L      |  boolean or                   | B                               |
+------------+---------------+------------+-------------------------------+---------------------------------+

Basic expressions

The basic expression types supported by the new query processor appear below:

  boolean_expression       :       boolean_term
                                   boolean_expression '.or.' boolean_term

  boolean_term             :       boolean_factor
                                   boolean_term '.or.' boolean_factor

  boolean_factor           :       comparison
                                   .not. boolean_factor

  comparison               :       expression
                                   expression CMP_OP expression
                                   expression CMP_OP expression CMP_OP expression

  expression               :       literal
                                   name
                                   expression NUM_OP expression
                                   '(' boolean_expression ')'

  name                     :       nt-variable
                                   builtin-function
                                   comis-function
                                   kuip-vector
                                   cut

The concept of arrays and shape matching

Arrays

Ntuple variables can support upto seven dimensions, the last one can be variable length.

KUIP vectors can be three dimensional.

Arrays can be indexed by signed or unsigned integer expresions. Each dimension should have one index, which can be either scalar or a range.

A range can be i:j, :j, i: or :. The missing parameter is taken at its extreme (1 or the length of the dimension) If the array name is not followed by ( the full array is used, otherwise the reference should look like for example 'SomeArray(index1,index2)' for a two dimensional array.

If one or more indices are ranges the resultant object is a smaller array, possibly of a lower dimensionality.

Shape matching

When arrays are used in expressions the shapes of the arrays, that is the number of dimensions and the length of each dimension, should follow the following rules.

Assume a operator O with n scalar arguments

r = O(a1...an)

This operator is then defined for one or more non scalar arguments as (in quasi tensor notation)

r(i) = O(a1...ak(i)..al(i)..an)

This implies that all non scalar arguments have the same shape and that the result r will have this shape. This automatially defines e.g. scalar times array as componentwise multiplication.

For the builtin functions of more then one argument it has been defined how they interact with non scalar arguments.

Builtin functions

Constants

Math functions

• The math functions available are the usual one: sin(x), cos(x), sqrt(x), exp(x), log(x), atan(x), abs(x), log10(x), tanh(x), acos(x), asin(x), tan(x), sinh(x) and cosh(x)
• Also functions like mod(x,y), atan2(x,y) and sign(x,y).
• frac(x) gives the fractional part of a real ie: x-int(x) (x is real or double)
• fint(x) returns real(int(x)) or double(int(x)) depending if x is real or double.

Bit handling functions

The operand(s) and result are both bit strings . In the discussion below, m and n, can be an Ntuple variable, a bit string constant or an expression with a bit string result. Bits are numbered from left to right, starting from zero.

Logical operations

• IOR(m,n) inclusive OR of bit strings m and n.
• IAND(m,n) logical AND of bit strings m and n.
• NOT(m) logical complement of bit string m.
• IEOR(m,n) exclusive OR of bit strings m and n.

An example of a bit string function is:

   
    NTUPLE/SCAN 30 IAND(X,B'1010').EQ.B'0101'

Shift operations

The shift count is represented by k. The absolute value of k specifies the number of positions to shift, while its sign specifies the direction of the shift, i.e. a left shift when k>0, no shift when k=0 and a right shift when k<0.

• ISHFT(m,k) provides the value of the argument m with the bits shifted. Bits shifted out to the left or right are lost, and zeros are shifted in from the opposite end.
• ISHFTC(m,k,ic) provides the value of the argument m with the rightmost ic bits shifted and the remaining bits untouched. The shift is circular; i.e. no bits are lost.

Bit testing

• IBSET(n,i) has the value of argument n with bit i set to 1.
• IBCLR(n,i) has the value of argument n with bit i set to 0.
• BTEST(n,i) has the value 'TRUE' if bit i of argument n is set, and 'FALSE' otherwise.

Bit subfields

• IBITS(m,i,len) provides, right justified, the value of the len bits of the argument m, starting from position i.
• MVBITS(m,i,len,n,j) moves len bits of argument m, starting at position i, to the argument n, starting at position j. All other bits of n are left untouched. Arguments m and n may refer to the same numeric storage unit.

Conversion

for any numeric or boolean arg:

• BOOL(arg) Convert the argument to a boolean.
• LOGICAL(arg) Convert the argument to a boolean (same as BOOL).
• UINT(arg) Convert the argument to an unsigned integer.
• ULONG(arg) Convert the argument to an unsigned long.
• INT(arg) Convert the argument to an integer.
• LONG(arg) Convert the argument to a long.
• REAL(arg) Convert the argument to a real
• FLOAT(arg) Convert the argument to a real (same as REAL)
• DOUBLE(arg) Convert the argument

Vector reduction operations

• VADD : Return the sum of all the components of an array variable.

                PAW > NT/PLOT 10.VADD(X)   | where X is an array variable

• VMUL : Return the product of all the components of an array variable.

                PAW > NT/PLOT 10.VMUL(X)   | where X is an array variable

• VMIN : Return the minimum of all the components of an array variable.

                PAW > NT/PLOT 10.VMIN(X)   | where X is an array variable

• VMAX : Return the maximum of all the components of an array variable.

                PAW > NT/PLOT 10.VMAX(X)   | where X is an array variable

• VAND : Return the logical AND of all the components of an array variable (bolean).

                PAW > NT/PLOT 10.VAND(X)   | where X is an array variable

• VOR : Return the logical OR of all the components of an array variable (bolean).

                PAW > NT/PLOT 10.VOR(X)   | where X is an array variable

These functions uses all element they get. One can give a slice of the 2D array like vmin(p(3,:)), each array index can either be an integer selecting a specific row/column/... or a range. One can specifiy the full range by giving :, You can also do thing like 2:3'or istart: (see before).

Strings manipulations

Two functions are available:

• SUBSTR(string,start,len) : return a substring of a string

                PAW > NT/PLOT 10.SUBSTR(NAME,1,4)   | Plot the first 4 letters of NAME

• MATCH(match_string,string) : Return a boolean value TRUE if match_string is found in string, FALSE otherwise.

                PAW > NT/PLOT 10.NAME MATCH('F*',NAME)  | Plot all the names starting with F
                PAW > NT/PLOT 10.NAME MATCH('*F*',NAME) | Plot all the names with F inside
                PAW > NT/PLOT 10.NAME MATCH('???',NAME) | Plot all the names with 3 letters

Predefined cuts

Predefined cuts are a kind of aliases. A cut is referenced by a number between 1 and 99 inclusive and can contain any valid ``boolean_expression'' (see above). Inside an expression a cut is referenced by giving a $ followed by the number of the cut. Cuts are defined by the CUT command, which can also be used to list them. Cuts can also be written to file and read back later on.

A special type of cut exists which is called a graphical cut, these are defined by the (new) command GCUT. The command GCUT is like NT/PLOT, but after the command the user can select a polygon interactively. If a graphical cut is applied, the expressions used to define the cut ar reevaluated and the cut returns true if the resulting point is inside the polygon. In the case of a one dimensional plot the graphical cut selects a range.

COMIS functions

Function known to the COMIS interpreter can be used in expressions. There are three possibilities.

The types of the arguments and the return type of the function are checked as far as possible. The number of arguments for a COMIS function is not limited.

Sometimes, a file name can be ambigous with other construct. For example on VMS characters like [,] can be used or / in UNIX file names which can be interpreted as a divide operation. To escape from those kind of ambiguties, file names can be enclose with double quotes "...". Note that for simple file names like myfile.f the double quotes can be ommited.

Kuip vectors

KUIP vectors can be used in selection mechanism or as arguments to a COMIS function. Real or integer arrays with maximum three dimensions. The KUIP vector should not be changed during the Ntuple processing.

Release Notes

Known bugs

FAQs

Contributions

Tutorial

Reference manual

Down load

Miscellaneous