===============================
Running your own production DQR
===============================

In cases where you may want to run a personal version of the Data Quality Report
using the same configuration as is done for the production DQR, 
users with access to IGWN computing resources at Caltech can 
do this easily by taking advantage of the 
``--config-overrides`` option in the ``dqr-create-dag`` script. 

Situations where a user may want to do this include testing of a new task
or DQR configuration, or creating a personal rerun of the DQR when 
the production DQR fails or does not generate for some reason. 
In both these examples, users should generate DQR results in their
own personal directories. 
An example of how to do this can be seen in the follwoing example. 

Running the low-latency version of the DQR
------------------------------------------

As previously mentioned, rerunning the DQR with slightly different settings
is straighforward if you are planning to run at a location that already
has a production DQR environment installed. 
At Caltech (CIT), the most recent production DQR conda environment is located at
``/home/dqr/mambaforge/envs/dqr-igwn310``. 
The production configuration files are located at ``/home/dqr/src/dqr-configuration-files/O4/online/CIT/``. 
To rerun the DQR, all a user needs to do is source this environment and run the ``dqr-create-dag`` script
while overriding a few variables. 
For convenience, a script has been created to just this. 

Users at Caltech can launch new DQRs in their personal directories by running: 

.. code-block:: console

    $ bash /home/dqr/src/dqr-configuration-files/O4/online/CIT/bin/rrt_command.sh ${graceid} ${output_dir}

* ``${output_dir}``: The ID of the superevent on https://gracedb.ligo.org that you would like to analyze 
* ``${output_dir}``: The path to the desired directory for the DQR dag and task outputs.

Full code details
-----------------

The ``rrt_command.sh`` script contains the following code blocks:

.. code-block:: shell

    #!/bin/bash
    
    source /cvmfs/oasis.opensciencegrid.org/ligo/sw/conda/etc/profile.d/conda.sh
    conda activate /home/dqr/mambaforge/envs/dqr-igwn310
    
    event=${1}
    dqr_path=${2}
    
    mkdir -p $dqr_path
    pushd $dqr_path
    
    output_dir=$(pwd)
    rel_path=$( echo ${output_dir##*public_html/} )
    output_url=https://ldas-jobs.ligo.caltech.edu/~${USER}/${rel_path}
    
    ligo-proxy-init ${USER}
    
    dqr-create-dag \
    	-v \
    	-g ${event} \
    	-c /home/dqr/src/dqr-configuration-files/O4/online/CIT/config/taskmgr-alert-dev.ini \
    	--no-globus \
    	--config-overrides 'annotate:arguments="--help"' 'validation:arguments-high_sig="--help"' \
    	general:output_dir=${output_dir} general:output_url=${output_url} condor:accounting_group_user=${USER}
    
    popd

This script creates the required directory, authenticates, and then launches a DQR based on information from GraceDB. 
In addition to overriding the variables associated with the output directory, output url, and accounting group user, 
note that the arguments to the ``annotate`` and ``validation`` tasks are also overriden. 
This is to prevent custom runs of the DQR from interacting with the production GraceDB and event validation instances. 

**Ensure that you do not annotate GraceDB or add information into the event validation database unless
you intend to do so**

Once the DAG for the DQR has been generated, follow the on-screen directions to submit the DAG using 
``condor_submit_dag``. 

Note that the above script will only launch tasks that are configured to run at CIT. 
Other tasks that are configured to run at other sites would need to be separately 
produced at the relevant DQR sites. 

Triggering the production DQR to run automatically
-------------------------------------------------

**These directions will trigger the DQR to run at all sites and annotate the production GraceDB
instance. Users should instead follow the above directions on how to run 
the DQR in their personal directories unless they are members of the RRT,
there is not already a DQR running, and they intend 
to run production DQRs at all IGWN sites!**

If there is an instance where the DQR was not run on a superevent in GraceDB, 
but there is a desire to launch the entire DQR at all sites and annotate
GraceDB, this can be accomplished using superevent labels. 

The O4 version of the DQR is automatically launched when the label ``DQR_REQUEST``
is added to a superevent in GraceDB. 
This generally happens at the time of the second preliminary alert, 
which in turn happens about 5 minutes after a significant candidate
is uploaded to GraceDB. 
If you are viewing a superevent within 5 minutes of when the
superevent was made public,
the DQR will likely be generated shortly without any intervention.
If there is a case where a DQR is needed but the ``DQR_REQUEST`` label is 
not added automatically at this stage, the label can be added manually. 

**Users should test the following directions using the
playground instance of GraceDB and a playground superevent 
to confirm that they are working
before interacting with the production GraceDB instance.**

To interact with GraceDB, LVK members first need to to activate 
and usable conda environment and then authenticate using
``ligo-proxy-init`` using their ``albert.einstein`` credential. 
This can be done (at CIT) using the following commands:

.. code-block:: console

    $ conda activate /home/dqr/mambaforge/envs/dqr-igwn310 
    $ ligo-proxy-init albert.einstein

where ``albert.einstein`` has been replaced with your username. 

Once this is done, a label can be added to GraceDB using the 
``gracedb`` executable. 
For example, to add the ``DQR_REQUEST`` label to superevent
``S230524a`` in the playground instance of GraceDB, 
they would run the following: 

.. code-block:: console

    $ gracedb add label S230524a DQR_REQUEST --service-url https://gracedb-playground.ligo.org/api/

To add a label to a superevent in GraceDB, the superevent ID and the service url 
would need to be changed to the correct ID and url in production gracedb. 
The production GraceDB service url is ``https://gracedb-playground.ligo.org/api/``