LB67 Thermolysin

A short tutorial for processing thermolysin data from LB67 with a gain mask applied. The data for this tutorial is at /reg/d/psdm/cxi/cxi84914/xtc/e350. Config files and phil files are available at /reg/d/psdm/cxi/cxi84914/scratch/cctbx-tutorial/lb67thermolysin.

IMPORTANT. Throughout this tutorial and in the config and phil files referenced, we are using $USER to represent your username. This will substitute your username in the command line, but not in the phil or config files. Be sure to replace $USER with your username in the config and phil files.

Dark average

First off, use mod_average to make a dark from LB67 run 24. Copy dark.cfg from /reg/d/psdm/cxi/cxi84914/scratch/cctbx-tutorial/lb67thermolysin into your myrelease folder. Then, from the myrelease folder, submit the averaging job:

cxi.lsf -c dark.cfg -o /reg/d/psdm/cxi/cxi84914/scratch/$USER/averages -x cxi84914 -r 24 -s -q psanacsq -p 6 -i /reg/d/psdm/cxi/cxi84914/xtc/e350

You can track the progress of your job using a command like this:

tail -f /reg/d/psdm/cxi/cxi84914/scratch/$USER/averages/r0024/000/stdout/sNN.out

When the average is complete, you can inspect them thusly:

cctbx.image_viewer /reg/d/psdm/cxi/cxi84914/scratch/$USER/averages/r0024/000/out/*

Nine images are created, 3 for each of the front CSPAD (Ds1), the back CSPAD (Dsd) and the XES spectrometer that was attached to this experiment (Sc1). For each detector the average (avg), standard deviation (stddev) and maximum projection (max) are created.

Gain map

As is likely clear from examining the averages for the Ds1 detector, during this experiment a gain mask was applied. At high resolution, the individual pixels were set in high gain mode (more sensitive to photons) and the low resolution pixels were set to low gain mode (low sensitivity to photons). This allows more accurate recording of low resolution reflections without saturating the pixels, while preserving the ability to record weak data at high resolution.

We treat this data by multiplying the low gain pixels by a constant factor to put them on the same scale as the high gain pixels. To do this we apply a mask. To generate this mask, we need to convert the raw mask used by the beamline operator to a format usable to cctbx.xfel. First, let's extract some useful information from one of the dark averages:

cxi.print_pickle /reg/d/psdm/cxi/cxi84914/scratch/$USER/averages/r0024/000/out/avg-Ds1-r0024.pickle

Detector format version: CXI 8.2
DISTANCE 378.0042
DETECTOR_ADDRESS CxiDs1-0|Cspad-0
SIZE1 1765
SIZE2 1765
TIMESTAMP 2014-03-06T00:05Z19.502
CCD_IMAGE_SATURATION 16383
SATURATED_VALUE 16383
64 active areas, first one:  [715, 439, 909, 624]
PIXEL_SIZE 0.11
BEAM_CENTER_Y 96.965825
BEAM_CENTER_X 96.99305
WAVELENGTH 1.75860511344
xtal_target None
DATA len=3115225 max=16383.000000 min=0.000000
SEQUENCE_NUMBER 0

We will use some of the information here to convert the gain mask to cctbx.xfel format. Copy the file cxib6714_1.gain from /reg/d/psdm/cxi/cxi84914/scratch/cctbx-tutorial/lb67thermolysin to your myrelease folder, then run:

cxi.gain_map -o cxib6714_1.pickle -d 378 -w 1.75860511344 cxib6714_1.gain -f "CXI 8.2"

Here -o is the output file name, -d is the distance, -w is the wavelength and -f is the detector format version. You can view the resultant cxib6714_1.pickle with cctbx.image_viewer

Finally, we test the map is working using a light average. Copy average.cfg from /reg/d/psdm/cxi/cxi84914/scratch/cctbx-tutorial/lb67thermolysin to your myrelease directory. Make sure the paths to the dark average and standard deviation are correct, as well as the gain maak pickle file. Submit the averaging job:

cxi.lsf -c cxi84914/average.cfg -o /reg/d/psdm/cxi/cxi84914/scratch/$USER/averages -x cxi84914 -r 25 -s -q psanacsq -p 8 -i /reg/d/psdm/cxi/cxi84914/xtc/e350

Again, you can follow the log file and view the result as describe above.

Indexing

Now that the darks and gain mask are in place, the data is indexed straightforwardly. Copy all the phil files and LB67-thermolysin.cfg to your myrelease folder, then submit the indexing job:

cxi.lsf -c LB67-thermolysin.cfg -o /reg/d/psdm/cxi/cxi84914/scratch/$USER/thermolysin -p 16 -q psanacsq -r 25 -x cxi84914 -i /reg/d/psdm/cxi/cxi84914/xtc/e350

You can follow the log of one of the streams using

tail -f /reg/d/psdm/cxi/cxi84914/scratch/$USER/thermolysin/r0024/000/stdout/s00.out

Individual indexed frames will be deposited in /reg/d/psdm/cxi/cxi84914/scratch/$USER/thermolysin/r0024/000/out and the integrated results will be in /reg/d/psdm/cxi/cxi84914/scratch/$USER/thermolysin/r0024/000/integration. For the parameters provided, around 1923 indexed images are expected.

Discovery

UNDER CONSTRUCTION

The indexing results above present already optimized processing parameters. The user will need to explore their data, testing a variety of parameters to extract the best reduced signal. Over the next few sections, techniques for discovering necessary parameters are presented.

Detector distance

The best way to be sure your detz offset is correct is through a series of trials, testing different values. Create a dist_trials directory in your myrelease folder. cd into it, then generate config files with different candidate detz offsets, thusly:

for i in `seq 565 580`; do vi -c "%s/578/$i/g" -c "w LB67-thermolysin_$i.cfg" \
-c q\! ../LB67-thermolysin.cfg ; done

Here, a vi command is executed repeatedly that searches for the number 578 in your config file and replaces it with a number from 565 to 580, then writes out the new file with an appropriate file name. If this seems like a poor window to search, note that the number was originally 572, and has already been optimized to 578 by the authors.

Next, submit indexing jobs for each candidate detz offset from your myrelease folder:

for i in `seq 565 580`; do cxi.lsf -c dist_trials/LB67-thermolysin_$i.cfg \
-o /reg/d/psdm/cxi/cxib6714/ftc/$USER/dist_trials/ -x cxib6714 -r 30 -q psanacsq -p 8 -t $i; done

When complete, go to your results directory:

cd /reg/d/psdm/cxi/cxib6714/scratch/$USER/dist_trials/r0025

Then determine which detz offset is best:

for i in `ls`; do echo -n "$i "; ls $i/out | wc -l; done