cctbx.xfel GUI/Troubleshooting

What to check when something does not behave. The GUI prints errors and progress to the terminal it was started from, so that terminal is always the first place to look.

Back to cctbx.xfel GUI.

A status light is red

A red light means that thread hit an exception and stopped; the traceback is in the terminal. Common causes: the database connection dropped (the server was restarted or the job running it ended), the queuing system command failed (squeue or bjobs not on the path), or a run block refers to a run that no longer exists. Fix the cause, then restart the thread: toggle Watch for new runs or Auto-submit jobs off and on, or switch away from and back to the tab for the tab-specific sentinels. If the database itself is gone, restart the GUI.

No runs appear

• Watch for new runs must be on (the Run Sentinel light green).
• LCLS: check the experiment name and the XTC stream location in the credentials dialog, and that the run database web service is reachable from this machine.
• Standalone: check the folder, the template (it is a glob, so *.h5, not .h5), and the completion criteria; a file younger than Minimum time since last modified or smaller than Minimum file size is not picked up yet. Folders need the status file or the required number of files.

Jobs are not submitted

• Auto-submit jobs must be on, the trial must be active, and it must have a run block containing the runs.
• In local mode only one job runs at a time; the next is submitted when it finishes.
• mp.max_queued in the settings file limits the number of queued and running jobs.
• For datasets: the dataset must be active and must have at least one tag; all its tags must be in use on the trial's runs; the runs' indexing jobs must be DONE; and each stage waits for the previous stage of the same run to be DONE. A failed stage (EXIT) blocks the rest of the pipeline for that run until it is restarted.
• Watch the terminal: the job sentinel prints what it submits and why it is waiting.

A job fails immediately (S_FAIL or EXIT)

Open the job's log: <output folder>/r<run>/<trial>_rg<block>/stdout/log.out for indexing jobs, and the queue's error file next to it. Typical causes are a wrong detector address, a mask or reference geometry file that the compute nodes cannot read, an environment script that does not set up cctbx, or a PHIL parameter that the processing program rejects. After fixing, use Restart job in the Jobs tab (or, for a run block change, let the new block's jobs run).

Images index poorly

• Check the unit cell and space group in the trial, and the detector distance and beam centre in the run block (an average image from the Runs tab helps with the latter).
• In the Run Stats tab, many grey points high in the strong spots panel are hits that did not index: look at some of those images.
• Too many or too few spots: adjust the spot finding thresholds in a new trial and compare.

The Run Stats plot is empty

A trial must be selected, and runs ticked (or Auto plot last five runs pressed). Only runs whose jobs have logged results show up; a run whose job is still pending has nothing to plot yet. If Auto update is unticked the plot does not refresh.

The cosym page says there is no plot

The embedding plot is only written when the merging stage runs modify_cosym, and only for a single selected version. Check the merging stage's step list under Edit PHIL and that the merging job is DONE.

The GUI refuses to load a project

While the GUI is connected, a project with a different database connection or experiment tag cannot be loaded. Quit and start the GUI again; the login dialog can load any project.

Resetting an experiment

To start again with an empty database but the same experiment tag, tick Delete and regenerate all tables in the credentials dialog at login. This erases every run, trial, job and logged result for that tag (the files on disk are untouched, so delete or move the output folder as well if you want a clean slate). Using a new experiment tag instead keeps the old records available.

Getting help

The About entry in the Help menu lists the authors. Questions and bug reports go to the cctbx.xfel developers; include the terminal output and the relevant job log.