Aaron: Created page with "{{DISPLAYTITLE:cctbx.xfel GUI/Jobs tab}} The '''Jobs tab''' lists every job the GUI has submitted, with its status in the queuing system, and provides the controls for stopping, deleting and resubmitting jobs. While the tab is showing, the '''Job Monitor''' thread queries the queuing system every 10 seconds and updates the statuses. Back to cctbx.xfel GUI. == The job table == {| class="wikitable" ! Column !! Meaning |- | '''Job''' || The job's id in the database. A..."

2026-10-09T21:34:48Z

Created page with "{{DISPLAYTITLE:cctbx.xfel GUI/Jobs tab}} The '''Jobs tab''' lists every job the GUI has submitted, with its status in the queuing system, and provides the controls for stopping, deleting and resubmitting jobs. While the tab is showing, the '''Job Monitor''' thread queries the queuing system every 10 seconds and updates the statuses. Back to cctbx.xfel GUI. == The job table == {| class="wikitable" ! Column !! Meaning |- | '''Job''' || The job's id in the database. A..."

New page

{{DISPLAYTITLE:cctbx.xfel GUI/Jobs tab}}
The '''Jobs tab''' lists every job the GUI has submitted, with its status in the queuing system, and provides the controls for stopping, deleting and resubmitting jobs. While the tab is showing, the '''Job Monitor''' thread queries the queuing system every 10 seconds and updates the statuses.

Back to [[cctbx.xfel GUI]].

== The job table ==
{| class="wikitable"
! Column !! Meaning
|-
| '''Job''' || The job's id in the database. Also used in log and error messages.
|-
| '''Type''' || Empty (<code>-</code>) for indexing jobs submitted by a trial. For dataset jobs, the task type: <code>ensemble_refinement</code>, <code>scaling</code>, <code>merging</code> or <code>phenix</code>.
|-
| '''Dataset''' || The dataset the job belongs to, for dataset jobs.
|-
| '''Trial''', '''Run''', '''Block''' || The trial number (<code>t003</code>), run (<code>r0012</code>) and run block (<code>rg005</code>). These make up the job's output folder, <code>r0012/003_rg005/</code>. Merging and Phenix jobs have no run or block.
|-
| '''Task''' || The task id for dataset jobs; its results go in <code>task<id>/</code> under the run folder.
|-
| '''Version''' || The dataset version a merging or Phenix job produced.
|-
| '''Subm ID''' || The id assigned by the queuing system (or the process id for local jobs). Ensemble refinement jobs that run as several chunks list several ids.
|-
| '''Status''' || See below.
|}

Click a column header to sort by it; click again to reverse. Selections survive the periodic refresh.

=== Job statuses ===
{| class="wikitable"
! Status !! Meaning
|-
| <code>SUBMIT</code> || Submitted; not yet seen in the queue.
|-
| <code>PEND</code> / <code>HOLD</code> || Waiting in the queue.
|-
| <code>RUN</code> || Running.
|-
| <code>DONE</code> || Finished normally.
|-
| <code>EXIT</code> || Finished with an error, was cancelled, or ran out of memory. Look at the job's log.
|-
| <code>TIMEOUT</code> || Killed for exceeding the wall time.
|-
| <code>SUSP</code> || Suspended or pre-empted by the scheduler.
|-
| <code>ERR</code> || The queue reported an error for the job.
|-
| <code>S_FAIL</code> || The submission command itself failed; the message is in the terminal.
|-
| <code>UNKWN</code> || The queue does not know the job (not yet registered, or expired from the queue's history). Jobs in this state keep being polled.
|-
| <code>DELETED</code> || Its results were deleted from disk and the database with '''Delete job'''.
|}

In the dataset pipeline the next task for a run is only submitted when the previous one is <code>DONE</code>, and a merging job only includes runs whose last local task is <code>DONE</code>.

== Filters ==
* '''Filter by''': ''All jobs'' or the jobs of a single trial.
* '''Only display jobs from active trials/blocks''': ticked by default, so finished experiments and abandoned trials do not clutter the list. Untick it to see everything.

== Actions ==
Select one or more rows first (shift-click and control-click work).

{| class="wikitable"
! Button !! What it does
|-
| '''Stop job''' || Cancels the selected jobs in the queuing system (<code>scancel</code>, <code>bkill</code>, <code>qdel</code>, <code>condor_rm</code>, or killing the process for local jobs). The job's status becomes <code>EXIT</code> once the monitor sees it. The job's database record and results stay.
|-
| '''Delete job''' || Deletes the selected jobs' results from disk (the job folder) and from the database (the per-image logs), and marks them <code>DELETED</code>. Only finished jobs can be deleted; stop running jobs first. For a merging job, the version's folder is removed. You are asked to confirm.
|-
| '''Restart job''' || Deletes the job as above and then removes it from the job table, so the job sentinel sees the work as not done and submits it again on its next cycle (with ''Auto-submit jobs'' on). Use this after fixing a problem (a bad mask, a full disk, a crashed node). Stop the job first if it is still running.
|}

A job that has failed (<code>EXIT</code>) is '''not''' retried automatically; the job sentinel treats every submitted job, failed or not, as done. ''Restart job'' is the way to retry it.

== Finding a job's log ==
Indexing jobs log to <code>stdout/log.out</code> in their output folder, for example <code><output folder>/r0012/003_rg005/stdout/log.out</code>; the submission script and queue output are in the same folder. Ensemble refinement logs are under <code>task<id>/combine_experiments_t<trial>/intermediates/</code>, scaling under <code>task<id>/</code>, and merging and Phenix under <code><dataset>/v<version>/</code>. The queue's own error output (<code>submit.err</code> or similar) lives next to the log.

Cctbx.xfel GUI/Jobs tab - Revision history