Difference between revisions of "Preparatory steps"

From cctbx_xfel
Jump to: navigation, search
(Running and testing the command)
 
(One intermediate revision by the same user not shown)
Line 1: Line 1:
 
This tutorial covers averaging dark and light runs, and creating mask images.
 
This tutorial covers averaging dark and light runs, and creating mask images.
  
UNDER CONSTRUCTION
+
== Setup your scratch area ==
  
 +
Create a scratch folder and populate it with some directories that will be used during this tutorial.  Usually you will work within the ftc directory of your experiment, for example:
  
== Setup your scratch area ==
+
/reg/d/psdm/cxi/cxi49812/ftc
  
Create a scratch folder and populate it with some directories that will be used during this tutorial, changing <code><b><i><username></i></b></code> to your SLAC account name:
+
In that directory, add a directory for your username, changing <code><b><I>expname</i></b></code> to your experiment name and <code><b><i><username></i></b></code> to your SLAC account name:
  
  $ cd /reg/g/cctbx/tutorials/scratch/
+
  $ cd /reg/d/psdm/cxi/<b><i><expname></i></b>/ftc
 
  $ mkdir <b><i><username></i></b>
 
  $ mkdir <b><i><username></i></b>
 
  $ cd <b><i><username></i></b>
 
  $ cd <b><i><username></i></b>
Line 15: Line 16:
 
  $ mkdir results
 
  $ mkdir results
  
Or you may use a directory in your home folder.
+
You may use a directory in your home folder for saving results but this is not advised as it will fill quickly.
  
 +
Finally, create a directory in your myrelease folder to hold config files and phil files needed during processing:
 +
$ cd ~/myrelease
 +
$ mkdir <b><i><expname></i></b>
  
 
== Create a dark average ==
 
== Create a dark average ==
Line 22: Line 26:
 
The CSPAD detector at LCLS is sensitive to temperature distortions and, unlike images collected at a synchrotron source, its output is not automatically dark-subtracted.  Because of this it is necessary to periodically record a dark run with the shutters closed during an LCLS experiment.  The dark run should contain on the order of at least 1000 images.  Before an experimental run can be processed, this dark run must be averaged so that subsequent experimental data can have the dark subtracted from each image.  Furthermore, events during collection can necessitate a new dark, such as ice in the beam over-saturating creating dead pixels or a change in temperature.
 
The CSPAD detector at LCLS is sensitive to temperature distortions and, unlike images collected at a synchrotron source, its output is not automatically dark-subtracted.  Because of this it is necessary to periodically record a dark run with the shutters closed during an LCLS experiment.  The dark run should contain on the order of at least 1000 images.  Before an experimental run can be processed, this dark run must be averaged so that subsequent experimental data can have the dark subtracted from each image.  Furthermore, events during collection can necessitate a new dark, such as ice in the beam over-saturating creating dead pixels or a change in temperature.
  
To create the average, start by copying the <code>dark.cfg</code> file to your pyana directory:
+
To create the average, you will need this config file: [[dark.cfg]].  Save it to your myrelease/<b><i><expname></i></b> folder.
$ cd ~/myrelease
+
$ sit_setup
+
$ cp /reg/g/cctbx/tutorials/preprocessing/dark.cfg .
+
  
Open <code>dark.cfg</code>.  It will look something like this:
+
The configuration file is broken into two sections, the [[#psana section | ''psana'' section]] and the [[#Modules section | modules section]].
[pyana]
+
modules = my_ana_pkg.mod_average
+
num-cpu = 8
+
+
[my_ana_pkg.mod_average]
+
calib_dir      = /reg/g/cctbx/sources/cctbx_project/xfel/metrology/CSPad/run4/CxiDs1.0_Cspad.0
+
avg_dirname    = /reg/g/cctbx/tutorials/scratch/<b><i><username></i></b>/darks
+
stddev_dirname  = /reg/g/cctbx/tutorials/scratch/<b><i><username></i></b>/darks
+
max_dirname    = /reg/g/cctbx/tutorials/scratch/<b><i><username></i></b>/darks
+
detz_offset    = 581
+
address        = CxiDs1-0|Cspad-0
+
avg_basename    = Ds1-r0002-avg
+
stddev_basename = Ds1-r0002-stddev
+
max_basename    = Ds1-r0002-max
+
  
The configuration file is broken into two sections, the [[#''pyana'' section | ''pyana'' section]] and the [[#Modules section | modules section]].
+
=== ''psana'' section ===
 +
The ''psana'' section specifies which modules to run.  In this case we are running the built in module <code>CSPadPixCoords.CSPadNDArrProducer</code>.  This module is fully documented in [https://confluence.slac.stanford.edu/display/PSDM/psana+-+Module+Catalog#psana-ModuleCatalog-ModuleCSPadPixCoords::CSPadNDArrProducer SLAC's module catalog].  If we were running multiple modules, we could chain them together here, and each module would be executed in sequence using the parameters specified in the corresponding module section.  
  
 +
=== Module section ===
 +
The module section specifies parameters specific to the CSPadPixCoords.CSPadNDArrProducer.  The user should only ever need to change the source, which dictates which detector is going to be averaged.  This string can be obtained from the beamline operator.
  
=== ''pyana'' section ===
+
=== Running and testing the command ===
The ''pyana'' section specifies which modules to run.  In this case we are running ''cctbx.xfel'''s <code>mod_average</code> module.  If we were running multiple modules, we could chain them together here, and each module would be executed in sequence using the parameters specified in the corresponding module section.  We also request a number of processes to parallelize the job with.  Even though averaging must be done on a single host, on that host the processing can be split up among multiple cores.  Note, other aspects of ''cctbx.xfel'', such as indexing and integration, do not have this restriction.
+
  
 +
To average a dark run, the user can use cxi.mpi_average.  Run it with no parameters to get a full help listing with examples.  To test it on 10 events before using the queuing system, run it as follows:
  
=== Module section ===
+
  $ cxi.mpi_average -c '''''expname'''''/dark.cfg -x '''''expname''''' -r '''''runnumber''''' -a '''''address''''' -d '''''detzoffset''''' -n 10 -p
The module section specifies parameters specific to averaging:
+
;calib_dir
+
: this directory is provided by LCLS and includes the layout information of the 64 tiles of the CSPAD detector
+
;avg_dirname
+
: directory to place the average image
+
;stddev_dirname
+
: directory to place the standard deviation image
+
;max_dirname
+
: directory to place the maximum projection image
+
;detz_offset
+
: experiment-specific number used to calculate the detector distance. This is the distance in millimeters between the sample interaction region and the far end of the detector stage. The XTC stream contains the distance from the detector stage to the detector. These two numbers are subtracted to calculate the distance between sample interaction region and the detector.  We determine this number experimentally using a grid search, testing a range of potential detz_offsets and choosing the one that shows the greatest success at indexing images
+
;address
+
: the name of the detector in the XTC stream
+
;avg_basename
+
: this name is prepended to a date stamp to name the average image
+
;stddev_basename
+
: this name is prepended to a date stamp to name the standard deviation image
+
;max_basename
+
: this name is prepended to a date stamp to name the maximum projection image image
+
  
 +
The options are described as follows:
  
=== Running and testing the command ===
+
* -c: path to the dark config file
 +
* -x: your experiment name
 +
* -r: the dark run number
 +
* -a: the address string for the detector. Will be something like CxiDs1.0:Cspad.0
 +
* -d: The detz offset.  This is the offset (in mm) from the sample interaction region to the back the CSPAD detector rail at CXI.  Typically it's on the order of between 560 and 580 mm.  The beamline operator will provide you with a value.  At XPP you must specify the detector distance directly as their CSPAD is on a robot arm.
 +
* -p: write the output as image pickle files.  Currently the standard file format for ''cctbx.xfel''.
  
Before starting the averaging job the command, replace <code><b><i><username></i></b></code> in the configuration file with your user nameTo average run 2 using the modified configuration file, execute
+
When complete, three new .pickle files will be present in your directoryThe dark average will be named '''''expname'''''_avg-r'''''runnum'''''.pickle. A similar stddev and max file will be found as well. Use the -o option to put these averages somewhere else, such as a location available for all users to access for your experiment, e.g. /reg/d/psdm/'''''instrumentname'''''/'''''expname'''''/res/darks
  $ cxi.lsf -c ~/myrelease/dark.cfg -i /reg/d/ana11/cxi/data/Mar2013calib/xtc \
+
 
    -o /reg/g/cctbx/tutorials/scratch/<b><i><username></i></b> -p 6 -q psanacsq -r 2 -s
+
Now, to submit the averaging job to the processing queue using many processor and all the events, use bsub:
This will request 6 processors, overriding the <code>num-cpu</code> parameter set in the configuration file.  The directory specified in the <code>-i</code> option will be searched for XTC streams, and output will be arranged in the directory defined by the value of the <code>-o</code> option. The <code>-s</code> option tells ''cxi.lsf'' to process all the streams on the same host—this is a requirement for averaging.  The job will be scheduled in the queue called <code>psanacsq</code>, which is one of several available queues.  Note that the job must be submitted from the <code>~/myrelease</code> directory.
+
 
 +
$ bsub -a mympi -n 100 -q psanacsq -o dark_average.log cxi.mpi_average -c '''''expname'''''/dark.cfg -x '''''expname''''' -r '''''runnumber''''' -a '''''address''''' -d '''''detzoffset''''' -p
 +
 
 +
The job will be scheduled in the queue called <code>psanacsq</code>, which is one of several available queues, using 100 cores.  Note that the job must be submitted from the <code>~/myrelease</code> directory.
  
 
Type <code>bjobs</code> to see if the job is running.  When it is, follow the output thusly:
 
Type <code>bjobs</code> to see if the job is running.  When it is, follow the output thusly:
  $ tail -f /reg/g/cctbx/tutorials/scratch/<b><i><username></i></b>/r0002/000/stdout/sNN.out
+
  $ tail -f dark_average.log
Note the presence of the <code>000</code> directory.  ''cxi.lsf'' places the output of each job in a separate <i>trial</i> directory, and prints this directory just before exiting.  By default, trial numbers start at zero and increase by one with each subsequent analysis of the same run.  To use any other, previously unused, trial number,  use ''cxi.lsf'''s <code>-t</code> option.
+
  
 
When the average has been completed, the output can be inspected using the ''cctbx'' image viewer:
 
When the average has been completed, the output can be inspected using the ''cctbx'' image viewer:
  $ cctbx.image_viewer /reg/g/cctbx/tutorials/scratch/<b><i><username></i></b>/r0002/000/out/Ds1-r0002-*.pickle
+
  $ cctbx.image_viewer <path to pickle files>
 
+
  
 
== Averaging an experimental light run ==
 
== Averaging an experimental light run ==
  
Once the dark average is completed, experimental, lighted, runs can be averaged to see the extent of diffraction and potential pathologies in the run.  Copy the <code>average.cfg</code> file from the tutorials directory to your pyana directory:
+
Experimental, lighted, runs can be averaged to see the extent of diffraction and potential pathologies in the run.  Here, you will need this config file [[average.cfg]].  Copy the file from to your myrelease/'''''expname''''' directory.
$ cd ~/myrelease
+
$ cp /reg/g/cctbx/tutorials/preprocessing/average.cfg .
+
 
+
The configuration file will look like this:
+
[pyana]
+
modules = my_ana_pkg.mod_average
+
num-cpu = 8
+
+
[my_ana_pkg.mod_average]
+
calib_dir      = /reg/g/cctbx/sources/cctbx_project/xfel/metrology/CSPad/run4/CxiDs1.0_Cspad.0
+
dark_path      = /reg/g/cctbx/tutorials/scratch/<b><i><username></i></b>/r0002/000/out/<b><i><name of dark average></i></b>.pickle
+
dark_stddev    = /reg/g/cctbx/tutorials/scratch/<b><i><username></i></b>/r0002/000/out/<b><i><name of dark stddev></i></b>.pickle
+
avg_dirname    = /reg/g/cctbx/tutorials/scratch/<b><i><username></i></b>/averages
+
stddev_dirname = /reg/g/cctbx/tutorials/scratch/<b><i><username></i></b>/averages
+
max_dirname    = /reg/g/cctbx/tutorials/scratch/<b><i><username></i></b>/averages
+
detz_offset    = 581
+
address        = CxiDs1-0|Cspad-0
+
avg_basename    = Ds1-r0003-avg
+
stddev_basename = Ds1-r0003-stddev
+
max_basename    = Ds1-r0003-max
+
The only difference is the addition of two parameters:
+
;dark_path
+
: path to the average dark image created before
+
;dark_stddev
+
: path to the standard deviation dark image created before
+
  
The job is ran as before using ''cxi.lsf'', again remembering to change all instances of <code><b><i><username></i></b></code> to your own SLAC user name. Here you must also change <code><b><i><name of dark average></i></b></code> and <code><b><i><name of dark stddev></i></b></code> to the names of the average and standard deviation images created in the previous stepThe result of the run will be three new images that can be viewed with ''cctbx.image_viewer'' in the same manner as the darks were.
+
The configuration file uses two new modules, [https://confluence.slac.stanford.edu/display/PSDM/psana+-+Module+Catalog#psana-ModuleCatalog-Modulecspad_mod.CsPadCalib cspad_mod.CsPadCalib] and pyimgalgos.cspad_gainmap (currently undocumented).  The former uses darks prepared by [[calibman]] for dark subtraction and performs common mode correction.  The latter applies any high/low gain pixel mask applied during data collection.  Note how the data from one module is passed to the next module using inkey and outkeycxi.mpi_average looks for the final data using the outkey ''image0''.
  
 +
Submit the job and view the results exactly as before, with the exception of using this average.cfg file instead.
  
 
== Information contained in the light average ==
 
== Information contained in the light average ==
Line 130: Line 84:
 
* Maximum projection from a lighted run (''i.e.'' an experimental run): pixels < 300 are considered non-bonded or in a shadow.  [N.B.–this assumes a lot–300 may be appropriate for XPP data collected at atmosphere; not CXI data <i>in vacuo</i>–NKS.] The presence of diffracting data is not needed, but also will not interfere.
 
* Maximum projection from a lighted run (''i.e.'' an experimental run): pixels < 300 are considered non-bonded or in a shadow.  [N.B.–this assumes a lot–300 may be appropriate for XPP data collected at atmosphere; not CXI data <i>in vacuo</i>–NKS.] The presence of diffracting data is not needed, but also will not interfere.
  
To create the mask, execute these commands:
+
To create the mask, execute this command:
$ cd /reg/g/cctbx/tutorials/scratch/<b><i><username></i></b>/
+
$ mkdir masks
+
$ cd masks
+
 
  $ cxi.make_mask -m 10 <b><i><name of dark average></i></b>.pickle <b><i><name of dark standard deviation></i></b>.pickle \
 
  $ cxi.make_mask -m 10 <b><i><name of dark average></i></b>.pickle <b><i><name of dark standard deviation></i></b>.pickle \
 
     <b><i><name of light maximum projection></i></b>.pickle
 
     <b><i><name of light maximum projection></i></b>.pickle

Latest revision as of 03:43, 19 December 2014

This tutorial covers averaging dark and light runs, and creating mask images.

Setup your scratch area

Create a scratch folder and populate it with some directories that will be used during this tutorial. Usually you will work within the ftc directory of your experiment, for example:

/reg/d/psdm/cxi/cxi49812/ftc

In that directory, add a directory for your username, changing expname to your experiment name and <username> to your SLAC account name:

$ cd /reg/d/psdm/cxi/<expname>/ftc
$ mkdir <username>
$ cd <username>
$ mkdir darks
$ mkdir averages
$ mkdir results

You may use a directory in your home folder for saving results but this is not advised as it will fill quickly.

Finally, create a directory in your myrelease folder to hold config files and phil files needed during processing:

$ cd ~/myrelease
$ mkdir <expname>

Create a dark average

The CSPAD detector at LCLS is sensitive to temperature distortions and, unlike images collected at a synchrotron source, its output is not automatically dark-subtracted. Because of this it is necessary to periodically record a dark run with the shutters closed during an LCLS experiment. The dark run should contain on the order of at least 1000 images. Before an experimental run can be processed, this dark run must be averaged so that subsequent experimental data can have the dark subtracted from each image. Furthermore, events during collection can necessitate a new dark, such as ice in the beam over-saturating creating dead pixels or a change in temperature.

To create the average, you will need this config file: dark.cfg. Save it to your myrelease/<expname> folder.

The configuration file is broken into two sections, the psana section and the modules section.

psana section

The psana section specifies which modules to run. In this case we are running the built in module CSPadPixCoords.CSPadNDArrProducer. This module is fully documented in SLAC's module catalog. If we were running multiple modules, we could chain them together here, and each module would be executed in sequence using the parameters specified in the corresponding module section.

Module section

The module section specifies parameters specific to the CSPadPixCoords.CSPadNDArrProducer. The user should only ever need to change the source, which dictates which detector is going to be averaged. This string can be obtained from the beamline operator.

Running and testing the command

To average a dark run, the user can use cxi.mpi_average. Run it with no parameters to get a full help listing with examples. To test it on 10 events before using the queuing system, run it as follows:

$ cxi.mpi_average -c expname/dark.cfg -x expname -r runnumber -a address -d detzoffset -n 10 -p

The options are described as follows:

  • -c: path to the dark config file
  • -x: your experiment name
  • -r: the dark run number
  • -a: the address string for the detector. Will be something like CxiDs1.0:Cspad.0
  • -d: The detz offset. This is the offset (in mm) from the sample interaction region to the back the CSPAD detector rail at CXI. Typically it's on the order of between 560 and 580 mm. The beamline operator will provide you with a value. At XPP you must specify the detector distance directly as their CSPAD is on a robot arm.
  • -p: write the output as image pickle files. Currently the standard file format for cctbx.xfel.

When complete, three new .pickle files will be present in your directory. The dark average will be named expname_avg-rrunnum.pickle. A similar stddev and max file will be found as well. Use the -o option to put these averages somewhere else, such as a location available for all users to access for your experiment, e.g. /reg/d/psdm/instrumentname/expname/res/darks

Now, to submit the averaging job to the processing queue using many processor and all the events, use bsub:

$ bsub -a mympi -n 100 -q psanacsq -o dark_average.log cxi.mpi_average -c expname/dark.cfg -x expname -r runnumber -a address -d detzoffset -p

The job will be scheduled in the queue called psanacsq, which is one of several available queues, using 100 cores. Note that the job must be submitted from the ~/myrelease directory.

Type bjobs to see if the job is running. When it is, follow the output thusly:

$ tail -f dark_average.log

When the average has been completed, the output can be inspected using the cctbx image viewer:

$ cctbx.image_viewer <path to pickle files>

Averaging an experimental light run

Experimental, lighted, runs can be averaged to see the extent of diffraction and potential pathologies in the run. Here, you will need this config file average.cfg. Copy the file from to your myrelease/expname directory.

The configuration file uses two new modules, cspad_mod.CsPadCalib and pyimgalgos.cspad_gainmap (currently undocumented). The former uses darks prepared by calibman for dark subtraction and performs common mode correction. The latter applies any high/low gain pixel mask applied during data collection. Note how the data from one module is passed to the next module using inkey and outkey. cxi.mpi_average looks for the final data using the outkey image0.

Submit the job and view the results exactly as before, with the exception of using this average.cfg file instead.

Information contained in the light average

Firstly, the light and dark images can be used to mask of any bad pixels. Inactive and non-bonded pixels, as well as hypersensitive pixels and any beam stop (if used) should be masked out. The light images (maximum in particular) also serve as a virtual, dark-subtracted, powder pattern that can used to inspect the data for diffraction. The light maximum should also flag up obvious errors in metrology, because rings will be non-continuous if the quadrants are misaligned. Intensity values from the corners of the light average or maximum are also useful to estimate background when taking first guesses at thresholds for hit-finding and integration.


Creating a mask image

cctbx.xfel uses three images to create a mask. Those images, and the purposes of them are listed below:

  • Average image from a dark run: pixels ≤ 0 are considered dead, pixels > 2000 are too sensitive
  • Standard deviation from a dark run: pixels ≤ 0 are considered dead, pixels ≥ 10 are too uncertain
  • Maximum projection from a lighted run (i.e. an experimental run): pixels < 300 are considered non-bonded or in a shadow. [N.B.–this assumes a lot–300 may be appropriate for XPP data collected at atmosphere; not CXI data in vacuo–NKS.] The presence of diffracting data is not needed, but also will not interfere.

To create the mask, execute this command:

$ cxi.make_mask -m 10 <name of dark average>.pickle <name of dark standard deviation>.pickle \
    <name of light maximum projection>.pickle

A new image is created where each pixel has one of two values: -2 (masked out pixels), and 0 (good pixels). Note, the maximum projection minimum pixel value is overridden here to a value that is more sensible for the tutorial's lysozyme data. You will likely need to adjust this parameter based on your data (see below for details and for further overrides).

Use the resultant mask image during processing with cctbx.xfel by adding this flag to your pyana config file:

mask_path = <path to mask file>

In addition, a polygon and a circle can be specified using the --poly_mask and --circle_mask parameters:

$ cxi.make_mask --poly_mask=17,854,76,967,760,982,762,879 --circle_mask=855,939,103 Cspad-avg.pickle Cspad-stddev.pickle Cspad-max.pickle

This example creates a mask image with a polygon [with vertices (17,854), (76,967), (760,982), and (762,879)] masked out, and a circle with center (855,939) and radius 103 masked out. Only one polygon and circle can be specified at the moment.

The mask can be examined with cctbx.image_viewer.

Applying the mask

Once the mask is created, it needs to be referenced by your configuration file:

mask_path = <absolute path to mask file>

Further, the indexing software needs to know what value to use as the masked-out value. Add the following to your phil files for indexing:

integration.mask_pixel_value=-2

Where -2 is the default mask value, but it can be whatever you specify using the -x option cxi.make_mask (see below).

Options

Some additional options are available for specifying the details of the masking operation:

-a, --avg_max (default 2000)
specifies the maximum value in ADU a pixel in the average image can have before it's masked out. The minimum is always zero.
-s, --stddev_max (default 10)
specifies the maximum value in ADU a pixel in the standard deviation image can have before it's masked out. The minimum is always zero.
-m, --maxproj_min (default 300)
specifies the minimum value in ADU a pixel in the maximum projection image can have before it's masked out. Of the three parameters for controlling cutoffs in cxi.make_mask, the -m option is unique from the -a and -s options in that it will likely vary the according to your sample's background. Carefully examine the corners of your lighted maximum projection and choose a value lower than the ADU values displayed. Also note that there is no cutoff on the high end of the maximum projection image specified in the mask, as that is defined as the saturation value for the detector.
-x, --mask_pix_val (default -2)
specifies the value to use when masking out a pixel. In other words, bad pixels in images in the XTC stream will be replaced with this value.
-o, --output (default mask_.pickle)
specifies the output file path, should be *.pickle, can include directory.