Running your own production DQR

In cases where you may want to run a personal version of the Data Quality Report using the same configuration as is done for the production DQR, users with access to IGWN computing resources at Caltech can do this easily by taking advantage of the --config-overrides option in the dqr-create-dag script.

Situations where a user may want to do this include testing of a new task or DQR configuration, or creating a personal rerun of the DQR when the production DQR fails or does not generate for some reason. In both these examples, users should generate DQR results in their own personal directories. An example of how to do this can be seen in the follwoing example.

Running the low-latency version of the DQR

As previously mentioned, rerunning the DQR with slightly different settings is straighforward if you are planning to run at a location that already has a production DQR environment installed. At Caltech (CIT), the most recent production DQR conda environment is located at /home/dqr/mambaforge/envs/dqr-igwn310. The production configuration files are located at /home/dqr/src/dqr-configuration-files/O4/online/CIT/. To rerun the DQR, all a user needs to do is source this environment and run the dqr-create-dag script while overriding a few variables. For convenience, a script has been created to just this.

Users at Caltech can launch new DQRs in their personal directories by running:

$ bash /home/dqr/src/dqr-configuration-files/O4/online/CIT/bin/rrt_command.sh ${graceid} ${output_dir}

  • ${output_dir}: The ID of the superevent on https://gracedb.ligo.org that you would like to analyze

  • ${output_dir}: The path to the desired directory for the DQR dag and task outputs.

Full code details

The rrt_command.sh script contains the following code blocks:

#!/bin/bash

source /cvmfs/oasis.opensciencegrid.org/ligo/sw/conda/etc/profile.d/conda.sh
conda activate /home/dqr/mambaforge/envs/dqr-igwn310

event=${1}
dqr_path=${2}

mkdir -p $dqr_path
pushd $dqr_path

output_dir=$(pwd)
rel_path=$( echo ${output_dir##*public_html/} )
output_url=https://ldas-jobs.ligo.caltech.edu/~${USER}/${rel_path}

ligo-proxy-init ${USER}

dqr-create-dag \
    -v \
    -g ${event} \
    -c /home/dqr/src/dqr-configuration-files/O4/online/CIT/config/taskmgr-alert-dev.ini \
    --no-globus \
    --config-overrides 'annotate:arguments="--help"' 'validation:arguments-high_sig="--help"' \
    general:output_dir=${output_dir} general:output_url=${output_url} condor:accounting_group_user=${USER}

popd

This script creates the required directory, authenticates, and then launches a DQR based on information from GraceDB. In addition to overriding the variables associated with the output directory, output url, and accounting group user, note that the arguments to the annotate and validation tasks are also overriden. This is to prevent custom runs of the DQR from interacting with the production GraceDB and event validation instances.

Ensure that you do not annotate GraceDB or add information into the event validation database unless you intend to do so

Once the DAG for the DQR has been generated, follow the on-screen directions to submit the DAG using condor_submit_dag.

Note that the above script will only launch tasks that are configured to run at CIT. Other tasks that are configured to run at other sites would need to be separately produced at the relevant DQR sites.

Triggering the production DQR to run automatically

These directions will trigger the DQR to run at all sites and annotate the production GraceDB instance. Users should instead follow the above directions on how to run the DQR in their personal directories unless they are members of the RRT, there is not already a DQR running, and they intend to run production DQRs at all IGWN sites!

If there is an instance where the DQR was not run on a superevent in GraceDB, but there is a desire to launch the entire DQR at all sites and annotate GraceDB, this can be accomplished using superevent labels.

The O4 version of the DQR is automatically launched when the label DQR_REQUEST is added to a superevent in GraceDB. This generally happens at the time of the second preliminary alert, which in turn happens about 5 minutes after a significant candidate is uploaded to GraceDB. If you are viewing a superevent within 5 minutes of when the superevent was made public, the DQR will likely be generated shortly without any intervention. If there is a case where a DQR is needed but the DQR_REQUEST label is not added automatically at this stage, the label can be added manually.

Users should test the following directions using the playground instance of GraceDB and a playground superevent to confirm that they are working before interacting with the production GraceDB instance.

To interact with GraceDB, LVK members first need to to activate and usable conda environment and then authenticate using ligo-proxy-init using their albert.einstein credential. This can be done (at CIT) using the following commands:

$ conda activate /home/dqr/mambaforge/envs/dqr-igwn310
$ ligo-proxy-init albert.einstein

where albert.einstein has been replaced with your username.

Once this is done, a label can be added to GraceDB using the gracedb executable. For example, to add the DQR_REQUEST label to superevent S230524a in the playground instance of GraceDB, they would run the following:

$ gracedb add label S230524a DQR_REQUEST --service-url https://gracedb-playground.ligo.org/api/

To add a label to a superevent in GraceDB, the superevent ID and the service url would need to be changed to the correct ID and url in production gracedb. The production GraceDB service url is https://gracedb-playground.ligo.org/api/