============================= Running a simple DQR workflow ============================= This guide will show how to install, configure, and run the DQR with a set of example tasks. In this guide, two tasks are included: ``stationarity`` and ``gwdetchar-omega``. This DQR example was run at Virgo, but should work on any IGWN computing location. ----------------------------------- Installing the DQR and dependencies ----------------------------------- .. code-block:: console $ conda create --name o4_dqr_proto python=3.10 $ conda activate o4_dqr_proto $ conda install python-ldas-tools-framecpp=2.6.14 $ pip install git+ssh://git@git.ligo.org/o4-dqr/o4-dqr-configuration.git@0.1.12 $ pip install git+ssh://git@git.ligo.org/detchar/dqrtasks.git@v0.1.5 At the time of writing, the `o4-dqr-configuration` and `dqrtasks` packages require IGWN authentication to be downloaded. Instructions on how to set up the required ssh key can be found in the `gitlab documentation `_. You also need htcondor installed. If you do not have access to a condor cluster, you can set up one up locally using the "as user" directions in the `HTCondor documentation `_. --------------------------- Writing configuration files --------------------------- To run this DQR, we need to configure each of the tasks we will run in addition to the DQR task manager itself. In this example, we will make a number of different configuration files, but these options can be listed in a single configuration file or spread out in even more files. These configuration options listed in the minimum number of files are shown on the `following page `_ The first configuration file is ``defaults.ini``. This file contains global variables that will be used throughout the other configuration files. The values for `output_dir` and `output_url` should be set to the relevant output locations, while `task_config_dir` is the location of these configuration files. .. code-block:: ini [DEFAULT] output_dir = /users/derek.davis/public_html/detchar/preO4/dqr_test_pages output_url = https://scientists.virgo-gw.eu/DataAnalysis/DetCharDev/users/derek.davis/detchar/preO4/dqr_test_pages gracedb_url = https://gracedb.ligo.org/api/ [extra_variables] config_dir = /users/derek.davis/detchar/preO4/dqr_tests/ex_configs The second file is ``condor.ini``. This file contains the condor options. .. code-block:: ini [condor] accounting_group = ligo.dev.o4.detchar.transient.dqr accounting_group_user = derek.davis getenv = True request_memory = 500MB request_disk = 500MB request_cpus = 1 universe=local The third file is ``iterate.ini``. This file contains interferometer-specific information that will be used by tasks. For now, let's only set up an interator for a single interferomenter. .. code-block:: ini [v1] iterator = 1 ifo = V1 channel = V1:Hrec_hoft_16384Hz frame-type = V1Online Iterator sections must contain the line ``iterator = 1``. Otherwise, any variables can be specified. The extra variables can be used by any task that is using this iterator. ----------------------------------------- Setting up the task-specific config files ----------------------------------------- We now need to configure the individual tasks. The first is `stationarity.ini`. Many of the options in this file are generic for most executables. The main options that would need to be changed for other executables are the ``executable`` and ``arguments`` options. The ``executable`` option is the path to the relevant executable, either absolute or relative. The ``arguments`` option is the full set of options that will be run with this executable. Note that there are a number of bracketed variables (``${}$``) in the arguments option. These correspond to variables that are set when the configuration file is parsed by the DQR task manager. For example, ``${t_0}`` is the event time, which is read in from gracedb. There is also the line ``iterator = v1``. This allows the task to use all the variables specified in the relevant iterator section. Note that you can have a task run over multiple iterators. For example, the line ``iterator = h1 l1 v1`` will run 3 different versions of this task, each time using the variables specified in the relevant iterator section. .. code-block:: ini [stationarity] description = stationarity test librarian = derek.davis@ligo.org include_in_dag = True tier = 1 question = Are known sources of noise without auxiliary witnesses active? executable = dqr-stationarity iterate = v1 arguments = "${t_0} -v --output-dir ${outdir} --channel ${channel} --frame-type ${frame-type} --threshold 2 --long-seg-len 512 --short-seg-len 8 --fft-len 4 --overlap 2 --psd-var-stride 0.1 --psd-method median --psd-window hann --f-low 20 --f-high 480 --resample_rate 4096 --window 500 --padding 4 --data-crop-window 32 --q-low 5 --q-high 6 --v-min 0 --v-max 16 --plot-window-before 10 --plot-window-after 2 --plot-f-low 10 --plot-f-high 1000" The next configuration file is ``omega.ini`` for the ``gwdetchar-omega`` executable. This file is very similar to ``stationarity.ini``. One note is that this file references an additional configuration file, ``V1-O3-llhoft-omegascan.ini``. This file includes additional information about how the task is run. .. code-block:: ini [omegascan] description = stationarity test librarian = derek.davis@ligo.org include_in_dag = True tier = 1 question = Are known sources of noise without auxiliary witnesses active? executable = gwdetchar-omega iterate = v1 arguments = "${t_0} -i V1 -f ${task_config_dir}/V1-O3-llhoft-omegascan.ini -o ${outdir} -c viridis" The file ``V1-O3-llhoft-omegascan.ini`` is shown below: .. code-block:: ini [GW] name = Gravitational Wave Strain q-range = 3.3166,150.0 frequency-range = 10.0,inf resample = 4096 frametype = V1Online duration = 64 fftlength = 8 max-mismatch = 0.2 snr-threshold = 5 always-plot = True plot-time-durations = 1,4,16 channels = V1:Hrec_hoft_16384Hz ----------------------------------------- Setting up the results page configuration ----------------------------------------- While not required, there are a number of additional configuration options that can be specified so that a results html page is generated. In the DQR, the results pages are generated with the `dqr-make-html` executable. This runs the same way as any other task, so let's set up a configuration file `html.ini` to list these options .. code-block:: ini [html] description = make html librarian = detchar@ligo.org include_in_dag = True tier = 1 executable = dqr-make-html arguments = " --output-dir ${outbase} -n ${graceid} --time-out 1800 --refresh 30 --gpstime ${t_0} --pvalue-thresh 0.05 --config ${task_config_dir}/html_config.ini --verbose --gracedb-url https://gracedb.ligo.org" Note that this configuration file references another configuration file (just like the omegascan task) called `html_config.ini`. For our DQR workflow, the required configuration file looks like this: .. code-block:: ini [general] sites = V1 primary-site = CIT ifos = V1 Network [site_urls] cit = https://ldas-jobs.ligo.caltech.edu/~dqr/ [omegascan] ifos = V1 sites = V1 [stationarity] ifos = V1 sites = V1 Note that the omegascan task does not produce a ``data.json`` file like other DQR tasks. Therefore, this task will not show up on the results page without additional work. This can be done by setting up a child task that creates the relevant file (e.g., with ``touch``) or by using an HTCondor ``POST`` job. Details about how to do this will be provided in a tutorial in a later update. -------------------------------------- Generating the DAG file and submitting -------------------------------------- Now that we've set up the configuration files, it's time to run the DQR task manager. The chosen event, S190924h, is a public alert, so the required information is available to anyone without authenticating. (To access data from gracedb that is not public, you must first authenticate with ``ligo-proxy-init``) One could also simply provide a GPS time instead of a graceid, or a Gracedb-formatted json file that contains the relevant data. The following command will create the dag for the event S190924h by connecting to Gracedb: .. code-block:: console $ dqr-create-dag --graceid S190924h --no-globus -v -c ex_configs/defaults.ini ex_configs/condor.ini ex_configs/stationarity.ini ex_configs/omega.ini Note that this command was run from ``/users/derek.davis/detchar/preO4/dqr_tests/``, one level above ``ex_configs``. You should now have a DQR dag written. This can be submitted by entering .. code-block:: console $ condor_submit_dag /users/derek.davis/public_html/detchar/preO4/dqr_test_pages/S190924h/data_quality_report_S190924h.dag In general, the dag can be found at ``${output_dir}/${event_id}/data_quality_report_${event_id}.dag`` The results will be stored at ````${output_dir}/${event_id}/${task_name}`` The results for this example can be found at `the output directory at EGO `_. Note that EGO credentials are required to view these example results. ------------------------------------------- Running this example at different locations ------------------------------------------- Versions of the configuration files are included as part of the ``dqrtasks`` repository in the ``etc`` folder. Assuming that the installation steps at the beginning of this tutorial have been completed, the following command (after replacing the relevant variables) can be used to generate a DQR workflow at either a EGO or LIGO computing site: .. tabs:: .. tab:: EGO .. code-block:: console $ dqr-create-dag --graceid S190924h --no-globus -v \ --config-files ${dqrtasks_dir}/etc/virgo_configs/* \ --config-overrides DEFAULT:output_dir=${output_dir} DEFAULT:output_url=${output_url} DEFAULT:task_config_dir=${task_config_dir} .. tab:: LIGO .. code-block:: console $ dqr-create-dag --graceid S190924h --no-globus -v \ --config-files ${dqrtasks_dir}/etc/ligo_configs/* \ --config-overrides DEFAULT:output_dir=${output_dir} DEFAULT:output_url=${output_url} DEFAULT:config_dir=${task_config_dir} condor:accounting_group_user=${user} The following variables in the above commands must be specified: * ``${output_dir}``: The path to the desired directory for the DQR dag and task outputs. * ``${output_url}``: The web path to the location specified in ``${output_dir}}``. * ``${task_config_dir}``: The path to the ``etc/task_configs`` folder in your installation of ``dqrtasks`` or a custom location. * ``${user}``: your `albert.einstein` username After the DQR dag is generated, it can be submitted as above with ``condor_submit_dag``