Running a simple DQR workflow

This guide will show how to install, configure, and run the DQR with a set of example tasks. In this guide, two tasks are included: stationarity and gwdetchar-omega. This DQR example was run at Virgo, but should work on any IGWN computing location.

Installing the DQR and dependencies

$ conda create --name o4_dqr_proto python=3.10
$ conda activate o4_dqr_proto
$ conda install python-ldas-tools-framecpp=2.6.14
$ pip install git+ssh://git@git.ligo.org/o4-dqr/o4-dqr-configuration.git@0.1.12
$ pip install git+ssh://git@git.ligo.org/detchar/dqrtasks.git@v0.1.5

At the time of writing, the o4-dqr-configuration and dqrtasks packages require IGWN authentication to be downloaded. Instructions on how to set up the required ssh key can be found in the gitlab documentation.

You also need htcondor installed. If you do not have access to a condor cluster, you can set up one up locally using the “as user” directions in the HTCondor documentation.

Writing configuration files

To run this DQR, we need to configure each of the tasks we will run in addition to the DQR task manager itself. In this example, we will make a number of different configuration files, but these options can be listed in a single configuration file or spread out in even more files. These configuration options listed in the minimum number of files are shown on the following page

The first configuration file is defaults.ini. This file contains global variables that will be used throughout the other configuration files. The values for output_dir and output_url should be set to the relevant output locations, while task_config_dir is the location of these configuration files.

[DEFAULT]
output_dir = /users/derek.davis/public_html/detchar/preO4/dqr_test_pages
output_url = https://scientists.virgo-gw.eu/DataAnalysis/DetCharDev/users/derek.davis/detchar/preO4/dqr_test_pages
gracedb_url = https://gracedb.ligo.org/api/

[extra_variables]
config_dir = /users/derek.davis/detchar/preO4/dqr_tests/ex_configs

The second file is condor.ini. This file contains the condor options.

[condor]
accounting_group = ligo.dev.o4.detchar.transient.dqr
accounting_group_user = derek.davis
getenv = True
request_memory = 500MB
request_disk = 500MB
request_cpus = 1
universe=local

The third file is iterate.ini. This file contains interferometer-specific information that will be used by tasks. For now, let’s only set up an interator for a single interferomenter. .. code-block:: ini

[v1] iterator = 1 ifo = V1 channel = V1:Hrec_hoft_16384Hz frame-type = V1Online

Iterator sections must contain the line iterator = 1. Otherwise, any variables can be specified. The extra variables can be used by any task that is using this iterator.

Setting up the task-specific config files

We now need to configure the individual tasks. The first is stationarity.ini. Many of the options in this file are generic for most executables. The main options that would need to be changed for other executables are the executable and arguments options. The executable option is the path to the relevant executable, either absolute or relative. The arguments option is the full set of options that will be run with this executable. Note that there are a number of bracketed variables (${}$) in the arguments option. These correspond to variables that are set when the configuration file is parsed by the DQR task manager. For example, ${t_0} is the event time, which is read in from gracedb. There is also the line iterator = v1. This allows the task to use all the variables specified in the relevant iterator section. Note that you can have a task run over multiple iterators. For example, the line iterator = h1 l1 v1 will run 3 different versions of this task, each time using the variables specified in the relevant iterator section.

[stationarity]
description = stationarity test
librarian = derek.davis@ligo.org
include_in_dag = True
tier = 1
question = Are known sources of noise without auxiliary witnesses active?
executable = dqr-stationarity
iterate = v1
arguments = "${t_0} -v --output-dir ${outdir}   --channel ${channel}   --frame-type ${frame-type}   --threshold 2   --long-seg-len 512   --short-seg-len 8   --fft-len 4   --overlap 2   --psd-var-stride 0.1   --psd-method median   --psd-window hann   --f-low 20   --f-high 480   --resample_rate 4096   --window 500   --padding 4   --data-crop-window 32   --q-low 5   --q-high 6   --v-min 0   --v-max 16   --plot-window-before 10   --plot-window-after 2   --plot-f-low 10   --plot-f-high 1000"

The next configuration file is omega.ini for the gwdetchar-omega executable. This file is very similar to stationarity.ini. One note is that this file references an additional configuration file, V1-O3-llhoft-omegascan.ini. This file includes additional information about how the task is run.

[omegascan]
description = stationarity test
librarian = derek.davis@ligo.org
include_in_dag = True
tier = 1
question = Are known sources of noise without auxiliary witnesses active?
executable = gwdetchar-omega
iterate = v1
arguments = "${t_0} -i V1 -f ${task_config_dir}/V1-O3-llhoft-omegascan.ini -o ${outdir} -c viridis"

The file V1-O3-llhoft-omegascan.ini is shown below:

[GW]
name = Gravitational Wave Strain
q-range = 3.3166,150.0
frequency-range = 10.0,inf
resample = 4096
frametype = V1Online
duration = 64
fftlength = 8
max-mismatch = 0.2
snr-threshold = 5
always-plot = True
plot-time-durations = 1,4,16
channels = V1:Hrec_hoft_16384Hz

Setting up the results page configuration

While not required, there are a number of additional configuration options that can be specified so that a results html page is generated. In the DQR, the results pages are generated with the dqr-make-html executable. This runs the same way as any other task, so let’s set up a configuration file html.ini to list these options

[html]
description = make html
librarian = detchar@ligo.org
include_in_dag = True
tier = 1
executable = dqr-make-html
arguments = "  --output-dir ${outbase} -n ${graceid} --time-out 1800 --refresh 30 --gpstime ${t_0} --pvalue-thresh 0.05 --config ${task_config_dir}/html_config.ini --verbose --gracedb-url https://gracedb.ligo.org"

Note that this configuration file references another configuration file (just like the omegascan task) called html_config.ini. For our DQR workflow, the required configuration file looks like this:

Note that the omegascan task does not produce a data.json file like other DQR tasks. Therefore, this task will not show up on the results page without additional work. This can be done by setting up a child task that creates the relevant file (e.g., with touch) or by using an HTCondor POST job. Details about how to do this will be provided in a tutorial in a later update.

Generating the DAG file and submitting

Now that we’ve set up the configuration files, it’s time to run the DQR task manager. The chosen event, S190924h, is a public alert, so the required information is available to anyone without authenticating. (To access data from gracedb that is not public, you must first authenticate with ligo-proxy-init) One could also simply provide a GPS time instead of a graceid, or a Gracedb-formatted json file that contains the relevant data. The following command will create the dag for the event S190924h by connecting to Gracedb:

$ dqr-create-dag --graceid S190924h --no-globus -v -c ex_configs/defaults.ini ex_configs/condor.ini ex_configs/stationarity.ini ex_configs/omega.ini

Note that this command was run from /users/derek.davis/detchar/preO4/dqr_tests/, one level above ex_configs.

You should now have a DQR dag written. This can be submitted by entering

$ condor_submit_dag /users/derek.davis/public_html/detchar/preO4/dqr_test_pages/S190924h/data_quality_report_S190924h.dag

In general, the dag can be found at ${output_dir}/${event_id}/data_quality_report_${event_id}.dag The results will be stored at ``${output_dir}/${event_id}/${task_name}

The results for this example can be found at the output directory at EGO. Note that EGO credentials are required to view these example results.

Running this example at different locations

Versions of the configuration files are included as part of the dqrtasks repository in the etc folder.

Assuming that the installation steps at the beginning of this tutorial have been completed, the following command (after replacing the relevant variables) can be used to generate a DQR workflow at either a EGO or LIGO computing site:

$ dqr-create-dag --graceid S190924h --no-globus -v \
    --config-files ${dqrtasks_dir}/etc/virgo_configs/* \
    --config-overrides DEFAULT:output_dir=${output_dir} DEFAULT:output_url=${output_url}  DEFAULT:task_config_dir=${task_config_dir}

The following variables in the above commands must be specified:

  • ${output_dir}: The path to the desired directory for the DQR dag and task outputs.

  • ${output_url}: The web path to the location specified in ${output_dir}}.

  • ${task_config_dir}: The path to the etc/task_configs folder in your installation of dqrtasks or a custom location.

  • ${user}: your albert.einstein username

After the DQR dag is generated, it can be submitted as above with condor_submit_dag