cctbx.xfel GUI/Jobs tab
The Jobs tab lists every job the GUI has submitted, with its status in the queuing system, and provides the controls for stopping, deleting and resubmitting jobs. While the tab is showing, the Job Monitor thread queries the queuing system every 10 seconds and updates the statuses.
Back to cctbx.xfel GUI.
The job table
| Column | Meaning |
|---|---|
| Job | The job's id in the database. Also used in log and error messages. |
| Type | Empty (-) for indexing jobs submitted by a trial. For dataset jobs, the task type: ensemble_refinement, scaling, merging or phenix.
|
| Dataset | The dataset the job belongs to, for dataset jobs. |
| Trial, Run, Block | The trial number (t003), run (r0012) and run block (rg005). These make up the job's output folder, r0012/003_rg005/. Merging and Phenix jobs have no run or block.
|
| Task | The task id for dataset jobs; its results go in task<id>/ under the run folder.
|
| Version | The dataset version a merging or Phenix job produced. |
| Subm ID | The id assigned by the queuing system (or the process id for local jobs). Ensemble refinement jobs that run as several chunks list several ids. |
| Status | See below. |
Click a column header to sort by it; click again to reverse. Selections survive the periodic refresh.
Job statuses
| Status | Meaning |
|---|---|
SUBMIT |
Submitted; not yet seen in the queue. |
PEND / HOLD |
Waiting in the queue. |
RUN |
Running. |
DONE |
Finished normally. |
EXIT |
Finished with an error, was cancelled, or ran out of memory. Look at the job's log. |
TIMEOUT |
Killed for exceeding the wall time. |
SUSP |
Suspended or pre-empted by the scheduler. |
ERR |
The queue reported an error for the job. |
S_FAIL |
The submission command itself failed; the message is in the terminal. |
UNKWN |
The queue does not know the job (not yet registered, or expired from the queue's history). Jobs in this state keep being polled. |
DELETED |
Its results were deleted from disk and the database with Delete job. |
In the dataset pipeline the next task for a run is only submitted when the previous one is DONE, and a merging job only includes runs whose last local task is DONE.
Filters
- Filter by: All jobs or the jobs of a single trial.
- Only display jobs from active trials/blocks: ticked by default, so finished experiments and abandoned trials do not clutter the list. Untick it to see everything.
Actions
Select one or more rows first (shift-click and control-click work).
| Button | What it does |
|---|---|
| Stop job | Cancels the selected jobs in the queuing system (scancel, bkill, qdel, condor_rm, or killing the process for local jobs). The job's status becomes EXIT once the monitor sees it. The job's database record and results stay.
|
| Delete job | Deletes the selected jobs' results from disk (the job folder) and from the database (the per-image logs), and marks them DELETED. Only finished jobs can be deleted; stop running jobs first. For a merging job, the version's folder is removed. You are asked to confirm.
|
| Restart job | Deletes the job as above and then removes it from the job table, so the job sentinel sees the work as not done and submits it again on its next cycle (with Auto-submit jobs on). Use this after fixing a problem (a bad mask, a full disk, a crashed node). Stop the job first if it is still running. |
A job that has failed (EXIT) is not retried automatically; the job sentinel treats every submitted job, failed or not, as done. Restart job is the way to retry it.
Finding a job's log
Indexing jobs log to stdout/log.out in their output folder, for example <output folder>/r0012/003_rg005/stdout/log.out; the submission script and queue output are in the same folder. Ensemble refinement logs are under task<id>/combine_experiments_t<trial>/intermediates/, scaling under task<id>/, and merging and Phenix under <dataset>/v<version>/. The queue's own error output (submit.err or similar) lives next to the log.