Preparatory steps pyana version

This tutorial covers averaging dark and light runs, and creating mask images.

Setup your scratch area

Create a scratch folder and populate it with some directories that will be used during this tutorial, changing <username> to your SLAC account name:

$ cd /reg/g/cctbx/tutorials/scratch/
$ mkdir <username>
$ cd <username>
$ mkdir darks
$ mkdir averages
$ mkdir results

Or you may use a directory in your home folder.

Create a dark average

The CSPAD detector at LCLS is sensitive to temperature distortions and, unlike images collected at a synchrotron source, its output is not automatically dark-subtracted. Because of this it is necessary to periodically record a dark run with the shutters closed during an LCLS experiment. The dark run should contain on the order of at least 1000 images. Before an experimental run can be processed, this dark run must be averaged so that subsequent experimental data can have the dark subtracted from each image. Furthermore, events during collection can necessitate a new dark, such as ice in the beam over-saturating creating dead pixels or a change in temperature.

To create the average, start by copying the dark.cfg file to your pyana directory:

$ cd ~/myrelease
$ sit_setup
$ cp /reg/g/cctbx/tutorials/preprocessing/dark.cfg .

Open dark.cfg. It will look something like this:

[pyana]
modules = my_ana_pkg.mod_average
num-cpu = 8

[my_ana_pkg.mod_average]
calib_dir       = /reg/g/cctbx/sources/cctbx_project/xfel/metrology/CSPad/run4/CxiDs1.0_Cspad.0
avg_dirname     = /reg/g/cctbx/tutorials/scratch/<username>/darks
stddev_dirname  = /reg/g/cctbx/tutorials/scratch/<username>/darks
max_dirname     = /reg/g/cctbx/tutorials/scratch/<username>/darks
detz_offset     = 581
address         = CxiDs1-0|Cspad-0
avg_basename    = Ds1-r0002-avg
stddev_basename = Ds1-r0002-stddev
max_basename    = Ds1-r0002-max

The configuration file is broken into two sections, the pyana section and the modules section.

pyana section

The pyana section specifies which modules to run. In this case we are running cctbx.xfel's mod_average module. If we were running multiple modules, we could chain them together here, and each module would be executed in sequence using the parameters specified in the corresponding module section. We also request a number of processes to parallelize the job with. Even though averaging must be done on a single host, on that host the processing can be split up among multiple cores. Note, other aspects of cctbx.xfel, such as indexing and integration, do not have this restriction.

Module section

The module section specifies parameters specific to averaging:

calib_dir

this directory is provided by LCLS and includes the layout information of the 64 tiles of the CSPAD detector

avg_dirname

directory to place the average image

stddev_dirname

directory to place the standard deviation image

max_dirname

directory to place the maximum projection image

detz_offset

experiment-specific number used to calculate the detector distance. This is the distance in millimeters between the sample interaction region and the far end of the detector stage. The XTC stream contains the distance from the detector stage to the detector. These two numbers are subtracted to calculate the distance between sample interaction region and the detector. We determine this number experimentally using a grid search, testing a range of potential detz_offsets and choosing the one that shows the greatest success at indexing images

address

the name of the detector in the XTC stream

avg_basename

this name is prepended to a date stamp to name the average image

stddev_basename

this name is prepended to a date stamp to name the standard deviation image

max_basename

this name is prepended to a date stamp to name the maximum projection image image

Running and testing the command

Before starting the averaging job the command, replace <username> in the configuration file with your user name. To average run 2 using the modified configuration file, execute

$ cxi.lsf -c ~/myrelease/dark.cfg -i /reg/d/ana11/cxi/data/Mar2013calib/xtc \
    -o /reg/g/cctbx/tutorials/scratch/<username> -p 6 -q psanaq -r 2 -s

This will request 6 processors, overriding the num-cpu parameter set in the configuration file. The directory specified in the -i option will be searched for XTC streams, and output will be arranged in the directory defined by the value of the -o option. The -s option tells cxi.lsf to process all the streams on the same host—this is a requirement for averaging. The job will be scheduled in the queue called psanaq, which is one of several available queues. Note that the job must be submitted from the ~/myrelease directory.

Type bjobs to see if the job is running. When it is, follow the output thusly:

$ tail -f /reg/g/cctbx/tutorials/scratch/<username>/r0002/000/stdout/sNN.out

Note the presence of the 000 directory. cxi.lsf places the output of each job in a separate trial directory, and prints this directory just before exiting. By default, trial numbers start at zero and increase by one with each subsequent analysis of the same run. To use any other, previously unused, trial number, use cxi.lsf's -t option.

When the average has been completed, the output can be inspected using the cctbx image viewer:

$ cctbx.image_viewer /reg/g/cctbx/tutorials/scratch/<username>/r0002/000/out/Ds1-r0002-*.pickle

Averaging an experimental light run

Once the dark average is completed, experimental, lighted, runs can be averaged to see the extent of diffraction and potential pathologies in the run. Copy the average.cfg file from the tutorials directory to your pyana directory:

$ cd ~/myrelease
$ cp /reg/g/cctbx/tutorials/preprocessing/average.cfg .

The configuration file will look like this:

[pyana]
modules = my_ana_pkg.mod_average
num-cpu = 8

[my_ana_pkg.mod_average]
calib_dir      = /reg/g/cctbx/sources/cctbx_project/xfel/metrology/CSPad/run4/CxiDs1.0_Cspad.0
dark_path      = /reg/g/cctbx/tutorials/scratch/<username>/r0002/000/out/<name of dark average>.pickle
dark_stddev    = /reg/g/cctbx/tutorials/scratch/<username>/r0002/000/out/<name of dark stddev>.pickle
avg_dirname    = /reg/g/cctbx/tutorials/scratch/<username>/averages
stddev_dirname = /reg/g/cctbx/tutorials/scratch/<username>/averages
max_dirname    = /reg/g/cctbx/tutorials/scratch/<username>/averages
detz_offset    = 581
address         = CxiDs1-0|Cspad-0
avg_basename    = Ds1-r0003-avg
stddev_basename = Ds1-r0003-stddev
max_basename    = Ds1-r0003-max

The only difference is the addition of two parameters:

dark_path

path to the average dark image created before

dark_stddev

path to the standard deviation dark image created before

The job is ran as before using cxi.lsf, again remembering to change all instances of <username> to your own SLAC user name. Here you must also change <name of dark average> and <name of dark stddev> to the names of the average and standard deviation images created in the previous step. The result of the run will be three new images that can be viewed with cctbx.image_viewer in the same manner as the darks were.

Information contained in the light average

Firstly, the light and dark images can be used to mask of any bad pixels. Inactive and non-bonded pixels, as well as hypersensitive pixels and any beam stop (if used) should be masked out. The light images (maximum in particular) also serve as a virtual, dark-subtracted, powder pattern that can used to inspect the data for diffraction. The light maximum should also flag up obvious errors in metrology, because rings will be non-continuous if the quadrants are misaligned. Intensity values from the corners of the light average or maximum are also useful to estimate background when taking first guesses at thresholds for hit-finding and integration.

Creating a mask image

cctbx.xfel uses three images to create a mask. Those images, and the purposes of them are listed below:

• Average image from a dark run: pixels ≤ 0 are considered dead, pixels > 2000 are too sensitive
• Standard deviation from a dark run: pixels ≤ 0 are considered dead, pixels ≥ 10 are too uncertain
• Maximum projection from a lighted run (i.e. an experimental run): pixels < 300 are considered non-bonded or in a shadow. [N.B.–this assumes a lot–300 may be appropriate for XPP data collected at atmosphere; not CXI data in vacuo–NKS.] The presence of diffracting data is not needed, but also will not interfere.

To create the mask, execute these commands:

$ cd /reg/g/cctbx/tutorials/scratch/<username>/
$ mkdir masks
$ cd masks
$ cxi.make_mask -m 10 <name of dark average>.pickle <name of dark standard deviation>.pickle \
    <name of light maximum projection>.pickle

A new image is created where each pixel has one of two values: -2 (masked out pixels), and 0 (good pixels). Note, the maximum projection minimum pixel value is overridden here to a value that is more sensible for the tutorial's lysozyme data. You will likely need to adjust this parameter based on your data (see below for details and for further overrides).

Use the resultant mask image during processing with cctbx.xfel by adding this flag to your pyana config file:

mask_path = <path to mask file>

In addition, a polygon and a circle can be specified using the --poly_mask and --circle_mask parameters:

$ cxi.make_mask --poly_mask=17,854,76,967,760,982,762,879 --circle_mask=855,939,103 Cspad-avg.pickle Cspad-stddev.pickle Cspad-max.pickle

This example creates a mask image with a polygon [with vertices (17,854), (76,967), (760,982), and (762,879)] masked out, and a circle with center (855,939) and radius 103 masked out. Only one polygon and circle can be specified at the moment.

The mask can be examined with cctbx.image_viewer.

Applying the mask

Once the mask is created, it needs to be referenced by your configuration file:

mask_path = <absolute path to mask file>

Further, the indexing software needs to know what value to use as the masked-out value. Add the following to your phil files for indexing:

integration.mask_pixel_value=-2

Where -2 is the default mask value, but it can be whatever you specify using the -x option cxi.make_mask (see below).

Options

Some additional options are available for specifying the details of the masking operation:

-a, --avg_max (default 2000)

specifies the maximum value in ADU a pixel in the average image can have before it's masked out. The minimum is always zero.

-s, --stddev_max (default 10)

specifies the maximum value in ADU a pixel in the standard deviation image can have before it's masked out. The minimum is always zero.

-m, --maxproj_min (default 300)

specifies the minimum value in ADU a pixel in the maximum projection image can have before it's masked out. Of the three parameters for controlling cutoffs in cxi.make_mask, the -m option is unique from the -a and -s options in that it will likely vary the according to your sample's background. Carefully examine the corners of your lighted maximum projection and choose a value lower than the ADU values displayed. Also note that there is no cutoff on the high end of the maximum projection image specified in the mask, as that is defined as the saturation value for the detector.

-x, --mask_pix_val (default -2)

specifies the value to use when masking out a pixel. In other words, bad pixels in images in the XTC stream will be replaced with this value.

-o, --output (default mask_.pickle)

specifies the output file path, should be *.pickle, can include directory.