Progress monitoring

If configured to do so, cctbx.xfel can use a mysql database during hit finding to log the hit rate and other useful parameters, enabling real-time monitoring via a graphical display.

Configuring trial monitoring

Add to the mod_hitfind section of your pyana config file these lines:

# cxi.lsf will replace this with its deterimined id, otherwise
# set it to the ID you wish.  0 is the default.
trial_id        = 0
db_logging      = True
sql_buffer_size = 5

These three parameters turn on and configure database-backed logging of hitfinding results:

• trial_id: we find it useful to organize processing in numbered trials. Whenever we go back and change the parameters of our processing, adjusting the target cell or spotfinder parameters for example, we re-process under a different trial ID. When you submit your processing job using cxi.lsf, you can specify the trial ID on the command line with –t. What you specify will overwrite what is listed here in the cfg file. If you use bsub, then what is in the cfg file is what’s used. Note, it isn’t a good idea to re-use the trial_id when re-prossing the data if database logging is being used, simply because the hits will be logged twice which will clutter future analysis. Finally, if you want to see which trials you have used for which runs, execute cxi.list_db_metadata
• db_logging: Switch for turning on and off logging
• sql_buffer_size: a queue size for regulating how often to add data to the database. In our experience, 5 has been just fine.

Monitoring hit finding in real time

After submitting a job with db logging enabled, use cxi.monitor_trials <trial number> to display the hit rate during processing. Other command line options and their defaults are listed below, including the ability to display multiple runs at once. The default is to display the 5 latest runs by timestamp.

• t_wait = 8000: amount of time in ms between queries to the database
• hit_cutoff = 16: the number of bragg spots before a frame is considered a hit. If this is different than what’s in your config file, then the monitor will be using a different cutoff to display which frames are hits.
• average_window = 1000: how many frames are in the sliding averaging window shown in the hit % graph
• n_points = 1000: how many total points to display on the graph. Set it to 0 to display all the data points (can be computationally intensive).
• display_time = 1800: number of seconds of experiment time to display. Displays this number of seconds unless that would add up to more than 5 runs.
• run_num = display only this run
• run_min = start at this run
• run_max = end with this run