Release notes for DAQ/HLT-I S/W Release tdaq-01-09-00


General notes from s/w librarian

General info

This is a general DAQ/HLT-I release which is intended to be used for the ATLAS detector commissioning and first data runs starting from April 2008.

Since tdaq-01-08-04 the externals were changed, in order to get latest patches for LCG packages: it is compatible with LCG54d s/w and offline-14.0.X releases branch.

Supported platforms and compilers


System and compiler
 CMTCONFIG
Compatibility list
Linux 2.6.9 (SLC4), gcc-3.4.x i686-slc4-gcc34-opt
SLC4.x, RHEL4
Linux 2.6.9 (SLC4), gcc-3.4.x i686-slc4-gcc34-dbg - ~ -

External s/w and run-time environment tuning.

 external package & version
 tdaq-common-01-09-03 (NB: API changes in ers and eformat!)
 dqm-common-00-04-02
 LCG 54d
 Java Runtime Environment 1.6.0

Release distribution

This release is distributed in RPM format, along with all required dependencies.

Development environment

Versions and paths of used external s/w are defined in TDAQExternal package.

Tools needed for development:

The default compiler is gcc-3.4.x on SLC4, no need to install it additionally.
CMT v1r20p20070720 (installed with RPM in <inst_root>/CMT/v1r20p20070720)
JDK 1.6.0 (installed with RPM in <inst_root>/sw/lcg/external/Java/JDK/1.6.0)

Important changes overview (see detailed notes below)


Packages and tags used in the release

ac v4r3p1
AccessManager AccessManager-00-05-10
clips clips-06-10-06
cmdl cmdl-01-04-14
cmem_rcc v2r0p23
coca coca-01-05-00
config config-01-16-02
coral_auth coral_auth-01-02-00
dal tdaq-01-09-00_patches-01
DataflowPolicy v1r6p0
dccommon v1r1p12
dcmessages tdaq-01-09-00_patches_01
ddc ddc-05-04-021
DFConfiguration v8r5p18
DFDebug v2r0p19
DFExceptions v3r0p0
DFInfo v2r2p21
DFM v2r18p13
DFMessage v2r0p0
DFRelease DFnightly-00-00-43
DFSubSystemItem v6r3p4
DFTests v2r1p6
DFThreads v2r4p0
dqm_config dqm_config-00-01-02
dqmf dqmf-00-01-07
dvs dvs-00-31-11
ed ed-00-02-16
efd efd-01-13-11
efio v2r6p9
emon tdaq-01-09-00_patches_01
errorRecovery v3r2p16
ErrorReporting v3r0p1
FarmTools FarmTools-01-05-03
file_sampler tdaq-01-09-00-p3
gatherer v8r5p81
genconfig genconfig-02-08-01
gnam gnam-010900-03-02-06
gnamDummyLib gnamDummyLib-01-01-01
histmon histmon-00-02-20
igui tdaq-01-09-00_patches_03
instrumentation v1r3p6
io_rcc v2r0p41
ipc ipc-04-07-01
is is-05-04-00
ispy ispy-00-00-11
ktidbexplorer HEAD
l2dummy v1r8p10
l2pu tdaq-01-09-00_patch_03
l2rh tdaq-01-09-00_patch_03
L2streamTest
l2sv v1r14p6
ls ls-01-02-02
mda mda-03-00-00
MonaIsa tdaq-01-09-00-p5
MonGatherer MonGatherer-02-07-00
mrs mrs-01-09-04
msg v1r1p7
msgconf v1r2p24
msginput v1r2p2
msgsctp v1r0p3
msgtcp v1r1p5
msgudp v1r0p48
oh oh-00-00-78
ohp ohp-02-04-04
ohpplugins ohpplugins-00-00-08
oks tdaq-01-09-00_patches_01
oks2cool oks2cool_tdaq-01-09-00_01
oks2coral tdaq-01-09-00_patches_01
oksconfig oksconfig-02-02-00
OMD tdaq-01-09-00_p5
omni omni-04-08-07
onasic onasic_tdaq-01-08-09_00
OnlinePolicy online-00-22-23
OnlineRecovery OnlineRecovery-02-01-11
OnlineRelease nightly-00-00-53
opmon opmon-00-00-07
owl owl-00-00-26
PackageID v1r0p19
PartitionMaker PartitionMaker-06-04-23
PmgGui PmgGui-00-00-06
ProcessManager ProcessManager-01-02-26
pt v4r4p5
ptdummy v3r10p5
PTIO PTIO-03-08-01
queues v1r0p16
rcc_corbo v2r0p4
rcc_error v2r0p5
rcc_rodbusy v2r0p8
rcc_time_stamp v2r0p6
rcdal rcdal-00-01-11
RCDBitString v1r4p1
RCDExampleConfiguration v2r1p1
RCDExampleModules v2r3p3
RCDExampleTriggers v0r3p1
RCDJtagChain v1r2p0
RCDLtp v1r9p2
RCDLtpi v1r9p3
RCDLtpiModule v1r9p4
RCDLTPModule v1r9p2
RCDMenu v1r3p0
RCDModuleDesign v2r2p2
RCDTtc v1r4p0
RCDUtilities v1r3p0
RCDVme v1r6p1
RCInfo RCInfo-00-01-00
RCUtils tdaq-01-09-00_05
rdb tdaq-01-09-00_patches-01
rdbconfig tdaq-01-09-00_patches_01
rm rm-02-02-04
rn rn-01-02-05
robin_ppc v0r0p77
RobinTestSuite v2r1p21
RODBusy v1r9p2
RODBusyModule v1r9p2
roib v2r7p2
ROSApplication tdaq-01-09-00_patch01
ROSBufferManagement v2r3p0
ROSCore tdaq-01-09-00_patch01
rose tdaq-01-09-00_patch_03
ROSEventFragment v2r1p20
ROSEventInputManager v2r1p3
ROSfilar v2r0p35
ROSGetInput v2r0p4
ROSInterruptScheduler v0r0p7
ROSIO tdaq-01-09-00_patch01
ROSMemoryPool v2r2p0
ROSModules v2r7p17
ROSMonitor v1r1p2
ROSObjectAllocation v2r0p1
ROSRCDdrivers ROSRCDdrivers-00-00-36
ROSRobin v0r1p56
ROSslink v2r0p11
ROSsolar v2r0p48
ROSUtilities v2r4p0
RunController RunController-01-06-18
setup
SFI v4r7p32
SFIOEmulators SFIOEmulators-00-06-04
SFO v2r12p24
siom v1r2p6
sysmon v2r2p13
sysmonapps tdaq-01-09-00_patch_03
system system-00-00-12
TDAQExternal TDAQExternal-00-10-00
TDAQPolicy TDAQPolicy-00-07-08
thread_allocator v1r0p7
tidb2 tidb2_tdaq-01-09-00_01
tmgr tmgr-01-04-11
training training-00-04-05
transport v1r0p57
TriP TriP-00-00-12
ttcpr v2r2p0
TTCviModule v1r9p2
vme_rcc v2r0p48
wmi wmi-00-01-07
xmext xmext-01-02-09

Changes in packages (in ABC order)

AccessManager |  config |  dal |  ddc |  DFConfiguration |  dqm_config |  dqmf |  dvs |  errorRecovery |  gatherer |  gnam |  histmon |  igui |  is |  l2pu |  l2rh |  ls |  MonGatherer |  mrs |  msg |  msginput |  oh |  ohp |  ohpplugins |  oks |  OMD |  OnlineRecovery |  PmgGui |  ProcessManager |  ptdummy |  PTIO |  queues |  RCDExampleModules |  RCDLtpi |  RCDLtpiModule |  RCDLTPModule |  RCInfo |  RCUtils |  rdbconfig |  rm |  rn |  RODBusyModule |  ROSApplication |  ROSCore |  rose |  ROSEventFragment |  ROSfilar |  ROSInterruptScheduler |  ROSIO |  ROSModules |  ROSRCDdrivers |  ROSRobin |  ROSslink |  ROSsolar |  RunController |  setup |  SFI |  sysmon |  sysmonapps |  system |  tmgr |  training |  TriP |  TTCviModule |  vme_rcc |  wmi | 

AccessManager

Roles

The roles used to be defined in LDAP using a special LDAP schema developed for this purpose. The drawback of this approach was that operating system security mechanism (PAM) could not access easily the roles information. The NIS netgroups defined in LDAP are used by several PAM modules (pam_access for example) and, in the same time, they have the aggregation ability which may be used to implement the roles hierarchy. In this TDAQ release the roles are defined using NIS netgroups in LDAP.

Policies

The Policy Access Point code has been reviewed and one of the changes is that the roles hierarchy is looked up in LDAP. Changes and additions:

Server

Client API (Java and C++)

The API accepts a list of AM servers to be contacted: the servers provided in the environment variables are checked one after the other until one of them responds with a valid answer. If a server responds with "server busy" message and provides a list of servers to be contacted, then this list is used before continuing to iterate through the client list of am servers.

Tools


config

New ConfigActions Class

An action is called on database load/reload/unload operation or config object modification.

As an example it is used to implement DAL's Component::disabled() algorithm in a way transparent to users (the disabled() algorithm holds static set, that needs to be updated in case of database modification).

To add an action user has to:

Unread All Template Objects (C++)

Add method Configuration::unread_all_objects() to unread all template objects.

As an example it is used when attribute string converter reads many template objects (partition, segments, resources, applications, sw repositories) while builds conversion map. All such objects have to be unread after conversion map has been built, since their attributes may have variables also need to be converted.

Existing unread_objects() method was replaced by _unread_objects(CacheBase*) to be effectively used by the new unread_all_objects().

Thread Safe (C++)

As required by new RunControl accessing DAL objects simultaneously from several threads, the config and generated code has been made thread-safe.

Performance Improvements and API Changes (C++)

Improve performance of most DAL methods replacing STL set and map by GNU hash_set and hash_map.

Above requires changes in code using Configuration superclasses() and subclasses() methods:

Old methods:
New methods:
See config/map.h and config/set.h for more information about config::map and config::set classes.

Note, new methods return unordered data.

dal

Visit DAL package TWiki page https://twiki.cern.ch/twiki/bin/view/Atlas/DaqHltDal

Algorithms

Schema Changes

Changes in dal_create_sw_repository

Changes in dal_dump_apps



ddc

Introduction

The complete documentation for the DAQ - DCS Communication package may be found at CERN/EDMS as https://edms.cern.ch/document/684955/5.3

General changes

The DIM library version 17 release 1installed, where a series of DIM problems is fixed.

Extention of the Command Transfer facilities

The DDC Commander is moved to new control interface of TDAQ Run Controller (this does not affect the DDC users).
The facility of sending different non-transition commands (i.e. addressed to different PVSS datapoints) in parallel is introduced. In order to enable this facility
  - The additional command line option -B YES of ddc_ct_dim  application should be used. The default option is NO.
  -
The optional last parameter (bool parallelizm) of the ddcExecCommand() call must be set by true.
  Take into account that with these settings an NT-command will be sent to DCS for execution independently of both the NT-commands addressed to other PVSS  DIM RPC services AND any transition command. If the required service is occupied by other command of the same client (user process) the coming command will be queued. The command of other client will be rejected in this case.

Extention of DDC Message Transfer facility

The DDC text variables declared as messages for TDAQ by default will not be passed to TDAQ just after starting DDC-MT application - only on change. In order to keep passing data just after DDC started, one has to define in the DDC configuration (the schema of DdcTextMessage class is extended) the string type attribute SendWhileConnection by "YES".

No changes in the Data Transfer application



DFConfiguration

Configuration Schema Changes since tdaq-01-08-04

SFO

EFD

ptdummy

Information Service Info Description Changes since tdaq-01-08-04

SFO

SFI

DFM

Configuration Data Changes (examples) since tdaq-01-08-04



dqm_config

New features:

                                Ex: ${TDAQ_INST_PATH}/share/data/ReferenceHistograms.root:CaloClusterVecMon/EtaPhi/EtaPhiCut0
Ex: ${TDAQ_INST_PATH}/share/data/ReferenceHistograms.root:
Ex: OH#Histogramming.LAr._EtaEcut0

Known bugs/To do:


dqmf

New features:

Known bugs/To do:


dvs

New features:

          in Point1: /var/log/pmg.log      The .err file is not produced
          elsewhere: InitialPartition_LogRoot/initial/pmg_agent_MachineName_Timestamp.out
                            InitialPartition_LogRoot/initial/pmg_agent_MachineName_Timestamp.err
          Partition_LogRoot/Partition_Name/ApplicationName_MachineName_Timestamp.out
          Partition_LogRoot/Partition_Name/ApplicationName_MachineName_Timestamp.err

errorRecovery

General changes




gatherer

The ResetAfterPublish parameter is removed from schema. Now resetting is done after each iteration.



gnam

Changes from tdaq-01-08-04 to tdaq-01-09-00

New plugin API
  1. Added a new entry point: This function will be called at the stopGathering transition. Users should not update the histograms in this function because the changes will no be published. The function has been added mainly to support a clean, final, IS gathering by MonaIsa.

histmon

General changes

Updated ERS macros just to compile the package. No new functionality.

The implementation of the THistRegister update to reflect changes in hltinterfaces/ITHistRegister. This means that the rgister can now accept all ROOT objects. It does reasonable things with TH1 (whole faimilly of ROOT histograms) TGrap and TGraph2D which are supported by offline and OH already.


igui

Introduction

The IGUI (Integrated Graphical User Interface) web page is at:
   http://atlas-onlsw.web.cern.ch/Atlas-onlsw/components/igui/welcome.html

General changes

Advices
Bug fixes

is

Using infomation object tags

IS repository provides a way of keeping several versions of an infomation object in the IS repository. Each version has a unique 32-bit tag associated with it. Using tags it is possible to update, remove and read value of a specific version of an infomation object. The IS API, which provides this functionality, has been slightly changed with respect to the tdaq01-08-03(04) releases where it was itroduced for the first time.

Automatic tags assignment

When an infomation object is updated using the keep_history mode a new tag is automatically created and assigned to the new version of the object. The new tag is equal to the maximum tag value for the given object plus 1. This mode can be used by calling one of the following functions with the true value for the keep_history parameter:
ISInfoDictionary::update( const std::string & name, ISInfo & info, bool keep_history = false );
ISInfoDictionary::checkin( const std::string & name, ISInfo & info, bool keep_history = false );

ISNamedInfo::chekin( bool keep_history = false );
Note that the default value of the keep_history parameter is false which means that the old object version will be overwritten by a new one and the tag will remain unchanged.

Setting tags explicitely

It is possible to set new tags explicitely by using another form of the update (checkin) functions:
ISInfoDictionary::update( const std::string & name, int tag, ISInfo & info );
ISInfoDictionary::checkin( const std::string & name, int tag, ISInfo & info );

ISNamedInfo::chekin( int tag );
If the object version associated with the given tag alreasy exists in the repositopry then it will be replaced with the new one. Otherwise a new version will be created and associated with the given tag.

Reading objects with tags

There are two groups of functions which can be used for reading objects from the IS repostiory. The first group contains function which do not require tag value to be provided:
ISInfoDictionary::getValue( const std::string & name, ISInfo & info );
ISNamedInfo::chekout( );
These functions return the most recent value of the given infomation object.

Another group of functions require object tags to be explicitely provided:
ISInfoDictionary::getValue( const std::string & name, int tag, ISInfo & info );
ISNamedInfo::chekout( int tag );
These functions return the object value which is associated with the given tag. If there is no value for the given tag in the IS repository then daq::is::InfoNotFound exception will be thrown.


l2pu

General changes




l2rh

General changes




ls

Introduction

This package substitues the obsolete logService. A new requirement whereby database technologies other than ORACLE had to be dropped came in around spiring 2007. This new requirement meant the re-writing of the logService package, which was higly dependent on MySQL. This opportunity was taken to refactorize the code, especially the log manager, which was never very user friendly. The database access in C++ is done using the CORAL interface, which hides the underlying technology. For the log manager, JAVA was the language chosen, since it brings in the flexibility requiered to make this tool more intuitive. The resulting java application can be run from the console, or remotely using the Java Web Start technology.

Known issues/bugs

None known.

To be implemented

Add an option to display statistics, internal and from IS.

Changes from previous release

Not many changes have been introduced. A few bugs have been fixed (like the SIGTERM not being caught) and a new utility has been added: logGetPartitionNames which retrieves the name of the partitions currently stored in the database.

Example applications

None exist at the moment.

Applications

Log Manager

Usage: log_manager 
Instruction on how to use to be written.
The Log Manager can also be run using Java Web Start technology from the link:

If possible, it is preferrable to launch this tool as a Java application rather than from the link above.

lsReceiver

Description: This application subscribes to the MRS service to receive and log on a database messages produced by TDAQ applications.
Usage: lsReceiver [-p partition-name] [-u user-name] [-n IS-server-name] [-s threshold-size] -c connect-string [-S subscribe-expression]
Options/Arguments:
        -p partitionName     Partition name
        -u userName          User name
        -n ISserverName      Name of the Information Service to publish the message rate into.
        -c connectionString  Database connection string. 

Test units

logTest

Description: Test binary for the Log Receiver application.
Usage: logTest -c connect-string [-p partition-name] [-l complexity-level]
Options/Arguments:
        -p partitionName      Partition name
        -l level           Level of Complexity of the test [1: open/close - 2: tests the Log Service Infrastructure].
        -c connectionString  Database connection string. 

Utilities

logSelect

Description: Application to retrieve log messages for a given partition according to the search criteria specified. By default, messages are dumped on std::cout.
Usage: logSelect -c connect-string -p partition-name [-i message-name] [-m machine-name] 
                 [-a application-name] [-l time-low] [-u time-up] [-s severity]  
                 [-x text] [-r parameters] [-d order-list] [-e max-rows] [-f offset-row]
Options/Arguments:
        -p partitionName      Partition name
        -c connectionString   Database connection string. 
        -i message-name       Message name or ID.
        -m machine-name       Machine name where the message was issued.
        -a application-name   Application name where the message was issued.
        -l time-low           Lower time threshold.
        -u time-up            Upper time threshold.
        -s severity           Message severity:
                                 1 - FATAL
                                 2 - ERROR
                                 3 - WARNING
                                 4 - DEBUG
                                 5 - INFORMATION
                                 6 - SUCCESS
        -x text               Text in the message body.
        -r parameters         Message parameters.
        -d order-list         Parameter to sort the messages by.
        -e max-rows           Maximum number of rows to retrieve from the database; 100 by default. If 0, all entries are retrieved.
        -f offset-row         Offset in the table to retrieve the messages from.

logDelete

Description: Application to remove log messages for a given partition according to the search criteria specified.
Usage: logDelete -c connect-string -p partition-name [-i message-name] [-m machine-name] 
                 [-a application-name] [-l time-low] [-u time-up] [-s severity]  
                 [-x text] [-r parameters]
Options/Arguments:
        -p partitionName      Partition name
        -c connectionString   Database connection string. 
        -i message-name       Message name or ID.
        -m machine-name       Machine name where the message was issued.
        -a application-name   Application name where the message was issued.
        -l time-low           Lower time threshold.
        -u time-up            Upper time threshold.
        -s severity           Message severity:
                                 1 - FATAL
                                 2 - ERROR
                                 3 - WARNING
                                 4 - DEBUG
                                 5 - INFORMATION
                                 6 - SUCCESS
        -x text               Text in the message body.
        -r parameters         Message parameters.

logGetPartitionNames

Description: Application to retrieve the list of partition names.
Usage: logGetPartitionNames -c connectionString 
Options/Arguments:
        -c connectionString  Database connection string. 

logCleanDatabase

Description: Application to clean the database by removing all the existing tables.
Usage: logCleanDatabase -c connectionString 
Options/Arguments:
        -c connectionString  Database connection string. 


MonGatherer

Implemented summation of arbitrary IS objects with ISInfoDynAny class.

If an object has some composite structure, the respective fields will be summed if they are numbers or vectors of numbers.
Other fields are not summed, instead the first observed value is assigned to the sum.
A basic check of types and dimensionality of arrays is performed

Corrected averaging of profile histograms.

For the profile histograms in the 'average' mode histograms are considered as independent measurements, so the error of average is less, than the error for each histogram.


mrs

General changes

  • New implementation of algorithm to handle messages when internal queue on private server (worker) is full.
  • New implementation of subscribe and is_subscribe_active methods to fix possible inconsistency between private and public servers if private server has connectivity problems during subscription/subscription_checking operation.
  • New IS information provided by public and private servers
  • MRS server (public and private component)



    MRS API

  • The sender
  • The receiver/li>
  • The commander:
  • Documentation


    msg

    A new default buffer manager is used that is simpler than the old version with thread specific data, but more efficient that the alternatives (which were never meant for production use).

    class MultipleBufferManager

    This implemenation guarantees that the just received message is contained in a single memory area (manipulation of the Buffer object may still lead to multiple internal memory areas), and that there are no more size limits on the message - the old default version had an upper limit due to the internal structure and the maximum number of iovec structures that can be used in a recvmsg call. The old default buffer manager can be selected by chosing threaded as the DFParameters.BufferManager entry. The simple and single options should no longer be used, they were initially only meant for testing.

    msginput

    Package msginput

    The 'msginput' package contains various utility functions which are used throughout the dataflow software.

    class MessageHeader

    The 'MessageHeader' class can be used to decode the generic values in every message buffer. A typical use is to initialize it with a buffer that has just been received. The various methods return the field values:

       MessagePassing::Buffer *buf = ...
    
       MessageInput::MessageHeader header(buf);
    
       if(header.valid()) {
           switch (header.type()) {
              case ...:
            
           }
       } else {
           // error message
       }
    

    class MessageDispatcher

    The 'MessageDispatcher' class can be used to register callback handlers for different message types and handle them centrally. There are two different ways to do this: a backward compatible 'old style', where every callback function has to inherit from the 'InputHandler' base class and implement two methods:

    
           void InputHandler::message(Buffer *, void *);
           void InputHandler::timeout(unsigned int xid, void *);
    
    When registering an InputHandler, an additional 'void *' cookie parameter can be passed, which will in turn be passed back when the callbacks are executed (this is the second 'void*' parameter in the signature above).

    The new-style callbacks use the boost::function and boost::bind packages to provide much more flexibility. Any free function, an arbitrary method or an object with the function call operator can be used as a callback. All it has to do is be compatible with the boost::function taking one parameter (a 'Buffer *') for the message handlers, and no parameter for the timeout callbacks. Note there are no more 'cookie' parameters, and the timeout callback does not even get the 'xid' parameter it used to get in the old scheme. The reason is that all these additional parameters can now be specified via boost::bind(), providing much more flexibility than before. Some examples: