cctbx_xfel - User contributions [en]

Ha14 installation

2014-04-23T18:40:12Z

Hattne: Added instructions for obtaining CBFlib and DIALS sources.

== Prerequisite: install and set up the PSDM suite ==

It is assumed that the [[Set up PSDM software | PSDM software distribution has been set up]]. Developers may additionally want to [[Set up ssh-agent | set up an ssh-agent]]. Note again that several sites already have ''cctbx.xfel'' installed, and that this step can be skipped for regular users not involved in the development of the software. Once ''cctbx.xfel'' has been installed it must be [[Setup | set up]] before it can be used.

== Download and extract a source tree ==

This step has to be performed on a host with Internet access. Not all hosts at SLAC have that, but the members of e.g. psexport pool do. In what follows, the absolute path <code>/path/to/cctbx/sources</code> should be replaced with something more appropriate (a site-wide install will benefit from a location on a shared disk, whereas <code>~/projects/phenix-src</code> is a common choice among developers). The directory must accessible everywhere ''cctbx.xfel'' is to be run.

=== Regular users ===
$ mkdir -p /path/to/cctbx/sources
$ cd /path/to/cctbx/sources
$ wget http://cci.lbl.gov/cctbx_build/results/current/cctbx_bundle.tar.gz
$ wget http://adder.lbl.gov/cctbx.xfel/downloads/labelit_bundle_current.tar.gz
$ mkdir cctbx_project
$ tar -xpvzf cctbx_bundle.tar.gz -C cctbx_project --strip-components=1
$ tar -xpvzf labelit_bundle_current.tar.gz





=== Developers ===
A user account on cci.lbl.gov is required, the name of which is to be substituted for <code>user</code> below.
$ mkdir -p /path/to/cctbx/sources
$ cd /path/to/cctbx/sources
$ svn export svn+ssh://user@cci.lbl.gov/phenix_regression/trunk/phenix_svn_getting_started.csh
$ ./phenix_svn_getting_started.csh user
$ rm phenix_svn_getting_started.csh

Developers may also want to check out the bleeding edge [http://www.bernstein-plus-sons.com/software/CBF CBFlib] sources, as well as sources and auxiliary data for the [http://dials.sourceforge.net DIALS] project.
$ svn checkout "http://svn.code.sf.net/p/cbflib/code-0/trunk/CBFlib_bleeding_edge" cbflib
$ svn checkout "http://svn.code.sf.net/p/dials/code/trunk" dials
$ svn checkout "svn+ssh://user@cci.lbl.gov/dials_regression/trunk" dials_regression

== Create the <code>cxi_user</code> directory ==

Add the <code>cxi_user</code> directory and its <code>__init__.py</code> module. This defines the default integration algorithm used by ''cctbx.xfel''.
$ mkdir cxi_user
$ cat > cxi_user/__init__.py << EOF
from xfel.mono_simulation.mono_treatment import post_outlier_rejection
from xfel.mono_simulation.mono_treatment import pre_get_predictions
EOF

== Configure the source tree ==
To avoid problems with run-time dynamic linking of Python extensions, the Python interpreter required for the next step must be the one provided by the PSDM software distribution. That interpreter can be located using find, e.g.
$ find $SIT_ROOT/sw/external/python -perm /0111 -type f -wholename "*/$SIT_ARCH/*/python" 2> /dev/null
At SLAC this interpreter is located somewhere under <code>/reg/g/psdm/sw/external/python</code>. Then create and initialize the build directory, <code>/path/to/cctbx/build</code> below, using the Python interpreter located using the above find command. The build directory is often located next to the source directory (<code>~/projects/phenix-build</code> is a common developer choice).

=== Regular users ===
$ mkdir -p /path/to/cctbx/build
$ cd /path/to/cctbx/build
$ python /path/to/cctbx/sources/cctbx_project/libtbx/configure.py xfel

=== Developers ===
$ mkdir -p /path/to/cctbx/build
$ cd /path/to/cctbx/build
$ python /path/to/cctbx/sources/cctbx_project/libtbx/configure.py cxi_xdr_xes labelit_regression phenix xfel


== Compile the sources ==

Initialise the running shell using the newly created configuration files. bash-users should
$ . setpaths.sh
while csh-users will instead need to run
% source setpaths.csh

The next step requires a C++ compiler. At SLAC, the members of the psexport pool do not have any compilers installed, but the interactive nodes, reachable via psanacs, do.
$ cd /path/to/cctbx/build
$ make
$ make
Please be sure to run make twice, as shown. On SLAC's interactive nodes, this takes just over 6 minutes. To finalize the installation, see [[Setup]].



== External links ==

* [http://cctbx.sourceforge.net/current_cvs/installation.html#manually-building-from-sources-under-unix Manually building from sources under Unix]

Ha14 installation

2014-04-15T15:12:31Z

Hattne: Added note for developers regarding dials and dials_regression.

== Prerequisite: install and set up the PSDM suite ==

It is assumed that the [[Set up PSDM software | PSDM software distribution has been set up]]. Developers may additionally want to [[Set up ssh-agent | set up an ssh-agent]]. Note again that several sites already have ''cctbx.xfel'' installed, and that this step can be skipped for regular users not involved in the development of the software. Once ''cctbx.xfel'' has been installed it must be [[Setup | set up]] before it can be used.

== Download and extract a source tree ==

This step has to be performed on a host with Internet access. Not all hosts at SLAC have that, but the members of e.g. psexport pool do. In what follows, the absolute path <code>/path/to/cctbx/sources</code> should be replaced with something more appropriate (a site-wide install will benefit from a location on a shared disk, whereas <code>~/projects/phenix-src</code> is a common choice among developers). The directory must accessible everywhere ''cctbx.xfel'' is to be run.

=== Regular users ===
$ mkdir -p /path/to/cctbx/sources
$ cd /path/to/cctbx/sources
$ wget http://cci.lbl.gov/cctbx_build/results/current/cctbx_bundle.tar.gz
$ wget http://adder.lbl.gov/cctbx.xfel/downloads/labelit_bundle_current.tar.gz
$ mkdir cctbx_project
$ tar -xpvzf cctbx_bundle.tar.gz -C cctbx_project --strip-components=1
$ tar -xpvzf labelit_bundle_current.tar.gz





=== Developers ===
A user account on cci.lbl.gov is required, the name of which is to be substituted for <code>user</code> below.
$ mkdir -p /path/to/cctbx/sources
$ cd /path/to/cctbx/sources
$ svn export svn+ssh://user@cci.lbl.gov/phenix_regression/trunk/phenix_svn_getting_started.csh
$ ./phenix_svn_getting_started.csh user
$ rm phenix_svn_getting_started.csh



== Create the <code>cxi_user</code> directory ==

Add the <code>cxi_user</code> directory and its <code>__init__.py</code> module. This defines the default integration algorithm used by ''cctbx.xfel''.
$ mkdir cxi_user
$ cat > cxi_user/__init__.py << EOF
from xfel.mono_simulation.mono_treatment import post_outlier_rejection
from xfel.mono_simulation.mono_treatment import pre_get_predictions
EOF

== Configure the source tree ==
To avoid problems with run-time dynamic linking of Python extensions, the Python interpreter required for the next step must be the one provided by the PSDM software distribution. That interpreter can be located using find, e.g.
$ find $SIT_ROOT/sw/external/python -perm /0111 -type f -wholename "*/$SIT_ARCH/*/python" 2> /dev/null
At SLAC this interpreter is located somewhere under <code>/reg/g/psdm/sw/external/python</code>. Then create and initialize the build directory, <code>/path/to/cctbx/build</code> below, using the Python interpreter located using the above find command. The build directory is often located next to the source directory (<code>~/projects/phenix-build</code> is a common developer choice).

=== Regular users ===
$ mkdir -p /path/to/cctbx/build
$ cd /path/to/cctbx/build
$ python /path/to/cctbx/sources/cctbx_project/libtbx/configure.py xfel

=== Developers ===
$ mkdir -p /path/to/cctbx/build
$ cd /path/to/cctbx/build
$ python /path/to/cctbx/sources/cctbx_project/libtbx/configure.py cxi_xdr_xes labelit_regression phenix xfel


== Compile the sources ==

Initialise the running shell using the newly created configuration files. bash-users should
$ . setpaths.sh
while csh-users will instead need to run
% source setpaths.csh

The next step requires a C++ compiler. At SLAC, the members of the psexport pool do not have any compilers installed, but the interactive nodes, reachable via psanacs, do.
$ cd /path/to/cctbx/build
$ make
$ make
Please be sure to run make twice, as shown. On SLAC's interactive nodes, this takes just over 6 minutes. To finalize the installation, see [[Setup]].



== External links ==

* [http://cctbx.sourceforge.net/current_cvs/installation.html#manually-building-from-sources-under-unix Manually building from sources under Unix]

Preparatory steps pyana version

2014-03-14T20:05:02Z

Hattne: Updated and reorganized averaging to cover cxi.lsf's -s option

This tutorial covers averaging dark and light runs, and creating mask images.

== Setup your scratch area ==

Create a scratch folder and populate it with some directories that will be used during this tutorial, changing <code><username></code> to your SLAC account name:

$ cd /reg/g/cctbx/tutorials/scratch/
$ mkdir <username>
$ cd <username>
$ mkdir darks
$ mkdir averages
$ mkdir results

Or you may use a directory in your home folder.

== Create a dark average ==

The CSPAD detector at LCLS is sensitive to temperature distortions and, unlike images collected at a synchrotron source, its output is not automatically dark-subtracted. Because of this it is necessary to periodically record a dark run with the shutters closed during an LCLS experiment. The dark run should contain on the order of at least 1000 images. Before an experimental run can be processed, this dark run must be averaged so that subsequent experimental data can have the dark subtracted from each image. Furthermore, events during collection can necessitate a new dark, such as ice in the beam over-saturating creating dead pixels or a change in temperature.

To create the average, start by copying the <code>dark.cfg</code> file to your pyana directory:
$ cd ~/myrelease
$ sit_setup
$ cp /reg/g/cctbx/tutorials/preprocessing/dark.cfg .

Open <code>dark.cfg</code>. It will look something like this:
[pyana]
modules = my_ana_pkg.mod_average
num-cpu = 8

[my_ana_pkg.mod_average]
calib_dir = /reg/g/cctbx/sources/cctbx_project/xfel/metrology/CSPad/run4/CxiDs1.0_Cspad.0
avg_dirname = /reg/g/cctbx/tutorials/scratch/<username>/darks
stddev_dirname = /reg/g/cctbx/tutorials/scratch/<username>/darks
max_dirname = /reg/g/cctbx/tutorials/scratch/<username>/darks
detz_offset = 581
address = CxiDs1-0|Cspad-0
avg_basename = Ds1-r0002-avg
stddev_basename = Ds1-r0002-stddev
max_basename = Ds1-r0002-max

The configuration file is broken into two sections, the [[#''pyana'' section | ''pyana'' section]] and the [[#Modules section | modules section]].

=== ''pyana'' section ===
The ''pyana'' section specifies which modules to run. In this case we are running ''cctbx.xfel'''s <code>mod_average</code> module. If we were running multiple modules, we could chain them together here, and each module would be executed in sequence using the parameters specified in the corresponding module section. We also request a number of processes to parallelize the job with. Even though averaging must be done on a single host, on that host the processing can be split up among multiple cores. Note, other aspects of ''cctbx.xfel'', such as indexing and integration, do not have this restriction.

=== Module section ===
The module section specifies parameters specific to averaging:
;calib_dir
: this directory is provided by LCLS and includes the layout information of the 64 tiles of the CSPAD detector
;avg_dirname
: directory to place the average image
;stddev_dirname
: directory to place the standard deviation image
;max_dirname
: directory to place the maximum projection image
;detz_offset
: experiment-specific number used to calculate the detector distance. This is the distance in millimeters between the sample interaction region and the far end of the detector stage. The XTC stream contains the distance from the detector stage to the detector. These two numbers are subtracted to calculate the distance between sample interaction region and the detector. We determine this number experimentally using a grid search, testing a range of potential detz_offsets and choosing the one that shows the greatest success at indexing images
;address
: the name of the detector in the XTC stream
;avg_basename
: this name is prepended to a date stamp to name the average image
;stddev_basename
: this name is prepended to a date stamp to name the standard deviation image
;max_basename
: this name is prepended to a date stamp to name the maximum projection image image

=== Running and testing the command ===

Before starting the averaging job the command, replace <code><username></code> in the configuration file with your user name. To average run 2 using the modified configuration file, execute
$ cxi.lsf -c ~/myrelease/dark.cfg -i /reg/d/ana11/cxi/data/Mar2013calib/xtc \
-o /reg/g/cctbx/tutorials/scratch/<username> -p 6 -q psanacsq -r 2 -s
This will request 6 processors, overriding the <code>num-cpu</code> parameter set in the configuration file. The directory specified in the <code>-i</code> option will be searched for XTC streams, and output will be arranged in the directory defined by the value of the <code>-o</code> option. The <code>-s</code> option tells ''cxi.lsf'' to process all the streams on the same host—this is a requirement for averaging. The job will be scheduled in the queue called <code>psanacsq</code>, which is one of several available queues. Note that the job must be submitted from the <code>~/myrelease</code> directory.

Type <code>bjobs</code> to see if the job is running. When it is, follow the output thusly:
$ tail -f /reg/g/cctbx/tutorials/scratch/<username>/r0002/000/stdout/sNN.out
Note the presence of the <code>000</code> directory. ''cxi.lsf'' places the output of each job in a separate trial directory, and prints this directory just before exiting. By default, trial numbers start at zero and increase by one with each subsequent analysis of the same run. To use any other, previously unused, trial number, use ''cxi.lsf'''s <code>-t</code> option.

When the average has been completed, the output can be inspected using the ''cctbx'' image viewer:
$ cctbx.image_viewer /reg/g/cctbx/tutorials/scratch/<username>/r0002/000/out/Ds1-r0002-*.pickle

== Averaging an experimental light run ==

Once the dark average is completed, experimental, lighted, runs can be averaged to see the extent of diffraction and potential pathologies in the run. Copy the <code>average.cfg</code> file from the tutorials directory to your pyana directory:
$ cd ~/myrelease
$ cp /reg/g/cctbx/tutorials/preprocessing/average.cfg .

The configuration file will look like this:
[pyana]
modules = my_ana_pkg.mod_average
num-cpu = 8

[my_ana_pkg.mod_average]
calib_dir = /reg/g/cctbx/sources/cctbx_project/xfel/metrology/CSPad/run4/CxiDs1.0_Cspad.0
dark_path = /reg/g/cctbx/tutorials/scratch/<username>/r0002/000/out/<name of dark average>.pickle
dark_stddev = /reg/g/cctbx/tutorials/scratch/<username>/r0002/000/out/<name of dark stddev>.pickle
avg_dirname = /reg/g/cctbx/tutorials/scratch/<username>/averages
stddev_dirname = /reg/g/cctbx/tutorials/scratch/<username>/averages
max_dirname = /reg/g/cctbx/tutorials/scratch/<username>/averages
detz_offset = 581
address = CxiDs1-0|Cspad-0
avg_basename = Ds1-r0003-avg
stddev_basename = Ds1-r0003-stddev
max_basename = Ds1-r0003-max
The only difference is the addition of two parameters:
;dark_path
: path to the average dark image created before
;dark_stddev
: path to the standard deviation dark image created before

The job is ran as before using ''cxi.lsf'', again remembering to change all instances of <code><username></code> to your own SLAC user name. Here you must also change <code><name of dark average></code> and <code><name of dark stddev></code> to the names of the average and standard deviation images created in the previous step. The result of the run will be three new images that can be viewed with ''cctbx.image_viewer'' in the same manner as the darks were.

== Information contained in the light average ==

Firstly, the light and dark images can be used to mask of any bad pixels. Inactive and non-bonded pixels, as well as hypersensitive pixels and any beam stop (if used) should be masked out. The light images (maximum in particular) also serve as a virtual, dark-subtracted, powder pattern that can used to inspect the data for diffraction. The light maximum should also flag up obvious errors in metrology, because rings will be non-continuous if the quadrants are misaligned. Intensity values from the corners of the light average or maximum are also useful to estimate background when taking first guesses at thresholds for hit-finding and integration.

== Creating a mask image ==
''cctbx.xfel'' uses three images to create a mask. Those images, and the purposes of them are listed below:
* Average image from a dark run: pixels ≤ 0 are considered dead, pixels > 2000 are too sensitive
* Standard deviation from a dark run: pixels ≤ 0 are considered dead, pixels ≥ 10 are too uncertain
* Maximum projection from a lighted run (''i.e.'' an experimental run): pixels < 300 are considered non-bonded or in a shadow. [N.B.–this assumes a lot–300 may be appropriate for XPP data collected at atmosphere; not CXI data in vacuo–NKS.] The presence of diffracting data is not needed, but also will not interfere.

To create the mask, execute these commands:
$ cd /reg/g/cctbx/tutorials/scratch/<username>/
$ mkdir masks
$ cd masks
$ cxi.make_mask -m 10 <name of dark average>.pickle <name of dark standard deviation>.pickle \
<name of light maximum projection>.pickle

A new image is created where each pixel has one of two values: -2 (masked out pixels), and 0 (good pixels). Note, the maximum projection minimum pixel value is overridden here to a value that is more sensible for the tutorial's lysozyme data. You will likely need to adjust this parameter based on your data (see below for details and for further overrides).

Use the resultant mask image during processing with ''cctbx.xfel'' by adding this flag to your pyana config file:
mask_path = <path to mask file>
In addition, a polygon and a circle can be specified using the <code>--poly_mask</code> and <code>--circle_mask</code> parameters:
$ cxi.make_mask --poly_mask=17,854,76,967,760,982,762,879 --circle_mask=855,939,103 Cspad-avg.pickle Cspad-stddev.pickle Cspad-max.pickle
This example creates a mask image with a polygon [with vertices (17,854), (76,967), (760,982), and (762,879)] masked out, and a circle with center (855,939) and radius 103 masked out. Only one polygon and circle can be specified at the moment.

The mask can be examined with ''cctbx.image_viewer''.

=== Options ===

Some additional options are available for specifying the details of the masking operation:
;-a, --avg_max (default 2000)
: specifies the maximum value in ADU a pixel in the average image can have before it's masked out. The minimum is always zero.
;-s, --stddev_max (default 10)
: specifies the maximum value in ADU a pixel in the standard deviation image can have before it's masked out. The minimum is always zero.
;-m, --maxproj_min (default 300)
: specifies the minimum value in ADU a pixel in the maximum projection image can have before it's masked out. Of the three parameters for controlling cutoffs in ''cxi.make_mask'', the -m option is unique from the -a and -s options in that it will likely vary the according to your sample's background. Carefully examine the corners of your lighted maximum projection and choose a value lower than the ADU values displayed. Also note that there is no cutoff on the high end of the maximum projection image specified in the mask, as that is defined as the saturation value for the detector.
;-x, --mask_pix_val (default -2)
: specifies the value to use when masking out a pixel. In other words, bad pixels in images in the XTC stream will be replaced with this value.

Preparatory steps pyana version

2014-03-14T18:11:33Z

Hattne: Changed CS-PAD to CSPAD for consistency.

The CSPAD detector at LCLS is sensitive to temperature distortions and is not automatically dark-subtracted, unlike images from a synchrotron source. Because of this it is necessary to periodically collect a dark run with the shutters closed during an LCLS experiment. The dark run should contain on the order of at least 1000 images. Before an experimental run can be processed, this dark run must be averaged so that subsequent experimental data can have the dark subtracted from each image. Further, events during collection can necessitate a new dark, such as ice in the beam oversaturating creating dead pixels or a change in temperature.

In addition, averaging a lighted run in general is a useful tool for examining diffraction throughout a data collection run.

Finally, inactive and non-bonded pixels, as well as hypersensitive pixels and any beam stop (if used) should be masked out. This tutorial covers averaging dark and light runs, and creating mask images.

== Setup your scratch area ==

Create a scratch folder and populate it with some directories that will be used during this tutorial, changing <username> to your SLAC account name:

$ cd /reg/g/cctbx/tutorials/scratch/
$ mkdir <username>
$ cd <username>
$ mkdir darks
$ mkdir averages
$ mkdir results

Or you may use a directory in your home folder.

== Create a dark average ==

To create the average, start by copying the dark.cfg file to your pyana directory:

$ cd ~/myrelease
$ sit_setup
$ cp /reg/g/cctbx/tutorials/preprocessing/dark.cfg .

Open dark.cfg. It will look something like this:


<pre>
# -*- Mode: Conf -*-
#
# Since mod_average must be run on a single host, it cannot be
# submitted using cxi.lsf.
#
# bsub -n 6 -J r0002 -o /reg/g/cctbx/tutorials/scratch/<username>/dark_r0002.out -q psanacsq -R "span[hosts=1]" \
# "cxi.pyana -c ~/myrelease/dark.cfg /reg/d/ana11/cxi/data/Mar2013calib/xtc/e236-r0002-s0*.xtc"
#

[pyana]
modules = my_ana_pkg.mod_average
num-cpu = 8

[my_ana_pkg.mod_average]
calib_dir = /reg/g/cctbx/sources/cctbx_project/xfel/metrology/CSPad/run4/CxiDs1.0_Cspad.0
avg_dirname = /reg/g/cctbx/tutorials/scratch/<username>/darks
stddev_dirname = /reg/g/cctbx/tutorials/scratch/<username>/darks
max_dirname = /reg/g/cctbx/tutorials/scratch/<username>/darks
detz_offset = 581
address = CxiDs1-0|Cspad-0
avg_basename = Ds1-r0002-avg
stddev_basename = Ds1-r0002-stddev
max_basename = Ds1-r0002-max
</pre>

The config file is broken into three sections, the header, the pyana section and the modules section.

=== Header Section ===
The comments at the top of the file include a bsub command. This is the command to be entered from the ~/myrelease directory to queue the job for processing. Bsub will submit the job, asking for 6 CPUs (-n), naming the job r0002 (-J), placing a log file in the -o location, and so forth. There are several queues available for use, the psanacsq is just one of them. The -R option restricts the job to running on a single host, which is required for averaging. Finally, the cxi.pyana command in quotes is the command that will actually be executed on the host computer, and specifies the config file and the streams to use. Note the asterisk, which allows averaging all the streams from a single run at one time.

'''Before executing the command, replace <username> in the command and in the config file with your user name.'''

=== Pyana Section ===
Pyana operates by having a root section in the config file which specifies which modules to run. In this case we are running ''cctbx.xfel's'' mod_average module. If we were running multiple modules, we could chain them together here, and each module would be executed in sequence using the parameters specified below. We also request a number of cpus to parallelize the job with. Even though averaging must be done on a single host, on that host the processing can be split up among multiple cores. Note, other aspects of ''cctbx.xfel'', such as indexing and integration, do not have this restriction.

=== Module Section ===
The final section specifies parameters specific to averaging:
* calib_dir: this directory is provided by LCLS and includes the layout information of the 64 tiles of the CSPAD detector
* avg_dirname: directory to place the average image
* stddev_dirname: directory to place the standard deviation image
* max_dirname: directory to place the maximum projection image
* detz_offset: experiment-specific number used to calculate the detector distance. This is the distance in millimeters between the sample interaction region and the far end of the detector stage. The XTC stream contains the distance from the detector stage to the detector. These two numbers are subtracted to calculate the distance between sample interaction region and the detector. We determine this number experimentally using a grid search, testing a range of potential detz_offsets and choosing the one that shows the greatest success at indexing images
* address: the name of the detector in the XTC stream
* avg_basename: this name is prepended to a datestamp to name the average image
* stddev_basename: this name is prepended to a datestamp to name the standard deviation image
* max_basename: this name is prepended to a datestamp to name the maximum projection image image

=== Running and testing the command ===
Submit the job as specified above, using the bsub command in the header:
$ bsub -n 6 -J r0002 -o /reg/g/cctbx/tutorials/scratch/<username>/dark_r0002.out -q psanacsq -R "span[hosts=1]"
"cxi.pyana -c ~/myrelease/dark.cfg /reg/d/ana11/cxi/data/Mar2013calib/xtc/e236-r0002-s0*.xtc"

Type bjobs to see if the job is running. When it is, follow the output thusly:
$ tail -f /reg/g/cctbx/tutorials/scratch/<username>/dark_r0002.out

When the average has been completed, the output can be inspected using the "cctbx" image viewer:
$ cctbx.image_viewer /reg/g/cctbx/tutorials/scratch/<username>/darks/Ds1-r0002-*.pickle

== Averaging an experimental light run ==

Once the dark run is made, experimental runs can be averaged to see the extent of diffraction and potential pathologies in the run. Copy the average.cfg file from the tutorials directory to your pyana directory:

$ cd ~/myrelease
$ cp /reg/g/cctbx/tutorials/preprocessing/average.cfg .

The cfg will look like this:
<pre>
# -*- Mode: Conf -*-
#
# Since mod_average must be run on a single host, it cannot be
# submitted using cxi.lsf.
#
# bsub -n 6 -J r0003 -o /reg/g/cctbx/tutorials/scratch/<username>/average_r0003.out -q psanacsq -R "span[hosts=1]" \
# "cxi.pyana -c average.cfg /reg/d/ana11/cxi/data/Mar2013calib/xtc/e236-r0003-s0*.xtc"
#

[pyana]
modules = my_ana_pkg.mod_average
num-cpu = 8

[my_ana_pkg.mod_average]
calib_dir = /reg/g/cctbx/sources/cctbx_project/xfel/metrology/CSPad/run4/CxiDs1.0_Cspad.0
dark_path = /reg/g/cctbx/tutorials/scratch/<username>/darks/<name of dark average>.pickle
dark_stddev = /reg/g/cctbx/tutorials/scratch/<username>/darks/<name of dark stddev>.pickle
avg_dirname = /reg/g/cctbx/tutorials/scratch/<username>/averages
stddev_dirname = /reg/g/cctbx/tutorials/scratch/<username>/averages
max_dirname = /reg/g/cctbx/tutorials/scratch/<username>/averages
detz_offset = 581
address = CxiDs1-0|Cspad-0
avg_basename = Ds1-r0003-avg
stddev_basename = Ds1-r0003-stddev
max_basename = Ds1-r0003-max
</pre>

The only difference is the addition of two parameters:
* dark_path: path to the average dark image created before
* dark_stddev: path to the standard deviation dark image created before

The job is ran as before using the bsub command in the header, again remembering to change all instances of <username> to your own SLAC user name. Here you must also change <name of dark average> and <name of dark stddev> to the names of the average and standard deviation images created in the previous step. The result of the run will be three new images that can be viewed with cctbx.image_viewer in the same manner as the darks were.

== Information contained in the light average ==

Firstly, the light and dark images can be used to mask of any bad pixels, as detailed below. The light images (maximum in particular) also serve as a virtual, dark-subtracted, powder pattern that can used to inspect the data for diffraction. The light maximum should also flag up obvious errors in metrology, because rings will be non-continuous if the quadrants are misaligned. Intensity values from the corners of the light average or maximum are also useful to estimate background when taking first guesses at thresholds for hit-finding and integration.

== Creating a mask image ==
''cctbx.xfel'' uses three images to create a mask. Those images, and the purposes of them are listed below:
* Average image from a dark run: pixels ≤ 0 are considered dead, pixels > 2000 are too sensitive
* Standard deviation from a dark run: pixels ≤ 0 are considered dead, pixels ≥ 10 are too uncertain
* Maximum projection from a lighted run (''i.e.'' an experimental run): pixels < 300 are considered non-bonded or in a shadow. [N.B.--this assumes a lot--300 may be appropriate for XPP data collected at atmosphere; not CXI data in vacuo--NKS.] The presence of diffracting data is not needed, but also will not interfere.

To create the mask, execute these commands:
$ cd /reg/g/cctbx/tutorials/scratch/<username>/
$ mkdir masks
$ cd masks
$ cxi.make_mask -m 10 ../darks/<name of dark average>.pickle ../darks/<name of dark standard deviation>.pickle
../averages/<name of light maximum projection>.pickle

A new image is created where each pixel has one of two values: -2 (masked out pixels), and 0 (good pixels). Note, the maximum projection minimum pixel value is overridden here to a value that is more sensible for the tutorial's lysozyme data. You will likely need to adjust this parameter based on your data (see below for details and for further overrides).

Use the resultant mask image during processing with ''cctbx.xfel'' by adding this flag to your pyana config file:

mask_path = <path to mask file>

In addition, a polygon and a circle can be specified using the --poly_mask and --circle_mask parameters:

$ cxi.make_mask --poly_mask=17,854,76,967,760,982,762,879 --circle_mask=855,939,103 Cspad-avg.pickle Cspad-stddev.pickle Cspad-max.pickle

This example creates a mask image with a polygon [with vertices (17,854), (76,967), (760,982), and (762,879)] masked out, and a circle with center (855,939) and radius 103 masked out. Only one polygon and circle can be specified at the moment.

The mask can be examined with cctbx.image_viewer.

=== Options ===

Some additional options are available for specifying the details of the masking operation:
;-a, --avg_max (default 2000)
: specifies the maximum value in ADU a pixel in the average image can have before it's masked out. The minimum is always zero.
;-s, --stddev_max (default 10)
: specifies the maximum value in ADU a pixel in the standard deviation image can have before it's masked out. The minimum is always zero.
;-m, --maxproj_min (default 300)
: specifies the minimum value in ADU a pixel in the maximum projection image can have before it's masked out. Of the three parameters for controlling cutoffs in ''cxi.make_mask'', the -m option is unique from the -a and -s options in that it will likely vary the according to your sample's background. Carefully examine the corners of your lighted maximum projection and choose a value lower than the ADU values displayed. Also note that there is no cutoff on the high end of the maximum projection image specified in the mask, as that is defined as the saturation value for the detector.
;-x, --mask_pix_val (default -2)
: specifies the value to use when masking out a pixel. In other words, bad pixels in images in the XTC stream will be replaced with this value.

Metrology refinement

2014-03-14T18:10:58Z

Hattne: Changed CS-PAD to CSPAD for consistency.

Indexing requires highly precise knowledge of pixel positions in laboratory space. Due to the physics of the CSPAD detector and the brevity of the incident XFEL pulses, a parallelized integrating detector was necessary to build to be able to operate at the framing rates available at the LCLS. 64 asics are arranged in a tetragonal pattern, 2 asics per sensor, 8 sensors per quadrant and 4 quadrants per detector. The placement of these tiles in physical space is measured by the beamline operators optically using electron microscopy. The four quadrants are movable at CXI, necessitating pixel-level adjustments to individual quadrants. ''cctbx.xfel'' then refines tile positions to whole pixel, and then sub-pixel precision by calculating spot positions on a highly diffracting dataset and refining tile locations versus actual spot positions for each tile. This creates four levels of metrology information and corrections. These levels and the method of measuring and refining them are detailed below.

Of the operations of ''cctbx.xfel'', calculating and refining metrology is the operation most suited to beam line staff and the authors of the software. Please don't hesitate to ask for assistance regarding any aspect of this procedure.

=== Optical Metrology ===

LCLS provides initial tile placements. Periodically the detector is disassembled and re-built, necessitating new calibration information. The optical measurements are included in the ''cctbx.xfel'' in the source code and are reference by your ''pyana'' config file. See [preparatory steps] for more information.

=== Quadrant positioning ===

The CXI CSPAD detectors have quadrants on rails to allow re-sizing of the central hole. The second-level of metrology corrections consists of unit pixel corrections to entire quadrants using the cxi.view quadrant calibration tool. First, create a virtual powder pattern using averaging (see [preparatory steps]). Usually the standard deviation or maximum projection will show virtual powder ring. An image with strong rings to moderate resolution is desired. Once obtained, start the calibration tool with this command:

cxi.view <virtual powder pattern image path> viewer.calibrate_silver=true distl.detector_format_version="CXI 7.1"

Note, the flag distl.detector_format_version is a legacy requirement that will be phased out.

Once loaded, you will see your image and many red rings, incidentally corresponding to the diffraction pattern of silver behenate. Additionally, a separate settings panel will be present with 8 input boxes corresponding to the x and y offsets of each of the four quadrants, UL, UR, LL, and LR (upper and lower, left and right). Manually adjust the offsets until the powder rings are circular through the whole image, using the red rings as guides. These corrections will but put into your phil parameters file in the distl quad_translations entry in this form:

distl {
quad_translations = 2 -6 3 -6 -7 0 -1 -4
}

Where the numbers are listed in this order: ULX ULY, URX URY, LLX LLY, LRX, LRY

=== Unit pixel metrology corrections ===

=== Sub-pixel metrology corrections ===

Processing L498 thermolysin

2014-03-14T18:09:00Z

Hattne: Cleanups, spell checks, etc.

This page contains instructions for reproducing the results reported in the 2014 ''cctbx.xfel'' paper<ref name="Hattne:2014">[http://dx.doi.org/10.1038/nmeth.2887 Hattne, J et al. Accurate macromolecular structures using minimal measurements from X-ray free-electron lasers. Nat. Methods In press (2014).]</ref>. The data are available from the [http://cxidb.org Coherent X-ray Imaging Data Bank] ([http://cxidb.org/id-23 CXIDB ID 23]; [http://cxidb.org/id-23-raw.html raw XTC files], 4.0 TiB), and must be downloaded to a local disk prior to starting the analysis. Furthermore, the [https://confluence.slac.stanford.edu/display/PSDM/Software+Distribution PSDM Software Distribution] must be [[Set up PSDM software | set up]], along with a test release and an empty analysis package. A [http://www.mysql.com MySQL] database is required to merge the integrated diffraction intensities. In the interest of keeping this guide as general as possible, no particular queuing system is assumed to be available. Processing time depends strongly on the computational resources available; using 48 processors on a 64-core 1.4 GHz Opteron-based computer, the analysis takes around 24 hours. The instructions assume some familiarity with ''cctbx.xfel'' and its installation and configuration procedure.

== Installing a ''cctbx.xfel'' snapshot from March 28, 2013 ==

The thermolysin data for the ''cctbx.xfel'' paper was processed around March 28, 2013. Unfortunately, regular nightly releases are not available for that time, but a custom source-code bundle has been prepared. This bundle differs from the regular ''cctbx'' bundles in that [http://adder.lbl.gov/labelit ''LABELIT''] is included, and that the directory layout is identical to that of a developer installation. To download and unpack the bundle in the current directory:
$ wget http://adder.lbl.gov/cctbx.xfel/downloads/cctbx.xfel-20130328.tar.gz
$ tar -xpvzf cctbx.xfel-20130328.tar.gz
Next, create a build directory (called <code>phenix-build-20130328</code> below, but that is an arbitrary choice), in which the sources are configured and compiled. The Python interpreter required to complete this step must be the one supplied by the PSDM Software Distribution. Once the PSDM software has been [[Set up PSDM software | set up]], this interpreter can be located using
$ find $SIT_ROOT/sw/external/python -perm /0111 -type f -wholename "*/$SIT_ARCH/*/python"
To prepare the build directory using this interpreter
$ mkdir phenix-build-20130328
$ cd phenix-build-20130328
$ python ../phenix-src-20130328/cctbx_project/libtbx/configure.py cxi_xdr_xes xfel
$ . setpaths.sh
where <code>python</code> is the path to the Python interpreter located using the previous find-command. Note that csh-users should run <code>source setpaths.csh</code> instead of <code>. setpaths.sh</code>. Then compile the sources:
$ make
$ make
Note that the <code>make</code> command may need to be run twice in order to complete the build. Once <code>make</code> does not produce any output from the compiler, the build is complete.

To make the ''cctbx.xfel'' analysis modules available to PSDM's ''pyana'',
$ cd /path/to/test/release
$ sit_setup
$ cd my_ana_pkg
$ ln -fns /path/to/phenix-src-20130328/cctbx_project/xfel/cxi/cspad_ana src
$ cd ..
$ scons
where <code>/path/to/test/release</code> and <code>my_ana_pkg</code> are the path to the test release and the name of the analysis package chosen while [[Set up PSDM software | setting up the PSDM software distribution]], respectively. <code>/path/to</code> denotes the path to the directory containing the unpacked ''cctbx.xfel'' sources. The last step compiles the ''cctbx.xfel'' analysis modules.

== Creating a dark image ==

To meaningfully process the thermolysin diffraction data, an average of all the images in a dark run—a run without any X-rays impinging on the detector—must be subtracted from the individual diffraction images. The configuration file below can be used to produce such an average, as well as an image of the standard deviation of all the pixels over the course of the run.
[pyana]
modules = my_ana_pkg.mod_average
num-cpu = 4

[my_ana_pkg.mod_average]
address = CxiDs1-0|Cspad-0
calib_dir = /path/to/phenix-src-20130328/cctbx_project/xfel/metrology/CSPad/run4/CxiDs1.0_Cspad.0
avg_basename = Ds1-avg
avg_dirname = r0031
stddev_basename = Ds1-stddev
stddev_dirname = r0031
/path/to refers to the directory containing the unpacked ''cctbx.xfel'' sources. All other options with values set in italics can be modified without adversely affecting averaging. The above file will use four simultaneous processes, and write the average and standard deviation images to files whose names start with <code>Ds1-avg</code> and <code>Ds1-stddev</code>, respectively, both in a directory called <code>r0031</code>.

The data deposited at the [http://cxidb.org/id-23 CXIDB] contains a dark run, <code>r0031</code>. To average the images in that run, save the above configuration file to disk, e.g. <code>L498-dark.cfg</code>, apply modifications as necessary, and execute
$ cxi.pyana -c L498-dark.cfg /path/to/xtc/files/e157-r0031-*.xtc
where /path/to/xtc/files is the path to the directory containing the raw XTC files downloaded from [http://cxidb.org/id-23 CXIDB]. The files written to the <code>r0031</code> directory will have a current date stamp appended, which can safely be removed to simplify subsequent configuration files.
$ cd r0031
$ mv Ds1-avg20140308104336073.pickle Ds1-avg.pickle
$ mv Ds1-stddev20140308104336371.pickle Ds1-stddev.pickle
$ cd ..
Note that the date stamp on the generated files above depends on the time of their creation. Further details are available on the [[Preparatory steps#Create a dark average|Create a dark image]] page of the [[Tutorials|tutorials]].

== Indexing the thermolysin data ==

A configuration file for processing the primary lattices in the thermolysin data is shown below.
[pyana]
modules = my_ana_pkg.mod_hitfind:threshold \
my_ana_pkg.mod_hitfind:index
num-cpu = 48

[my_ana_pkg.mod_hitfind]
address = CxiDs1-0|Cspad-0
calib_dir = /path/to/phenix-src-20130328/cxi_xdr_xes/cftbx/metrology/CSPad/run4/CxiDs1.0:Cspad.0
dark_path = r0031/Ds1-avg.pickle
dark_stddev = r0031/Ds1-stddev.pickle
db_logging = False
detz_offset = 575

[my_ana_pkg.mod_hitfind:threshold]
dispatch = nop
distl_flags = permissive
distl_min_peaks = 16
threshold = 450
xtal_target = hitfind

[my_ana_pkg.mod_hitfind:index]
dispatch = index
integration_dirname = integration-first-lattice
integration_basename = int-
xtal_target = thermolysin27
The configuration file above instructs <code>mod_hitfind</code> to use 48 processes. It disables all image output, which reduces the amount of disk space required to perform the analysis to about 3.4 GiB. /path/to again refers to the directory containing the unpacked ''cctbx.xfel'' sources, and <code>dark_path</code> as well as <code>dark_stddev</code> may have to be changed to reflect the location of the previously generated dark images. Integration results will be written to the directory integration-first-lattice.

Due to particularities of the thermolysin measurement, processing needs to proceed in two batches. To analyze the first batch, runs 16 through 27, save the above configuration file to disk, e.g. <code>L498-indexigrate.cfg</code>, apply modifications as necessary, and execute
$ cxi.pyana -c L498-indexigrate.cfg /path/to/xtc/files/e157-r0016-*.xtc
$ cxi.pyana -c L498-indexigrate.cfg /path/to/xtc/files/e157-r0017-*.xtc
$ …
$ cxi.pyana -c L498-indexigrate.cfg /path/to/xtc/files/e157-r0027-*.xtc
where /path/to/xtc/files is the path to the directory containing the raw XTC files. The second batch, runs 71 through 73, was recorded using a different distance between the interaction region and the detector. Whilst the changes to the detector position are automatically handled by ''cctbx.xfel'', the resulting difference in shadowing is not. Different areas of the detector should be ignored at the different distances, and this is accounted for by the value of the <code>xtal_target</code> option in the configuration file. To analyze this set of runs, edit <code>L498-indexigrate.cfg</code>, change <code>thermolysin27</code> to <code>thermolysin73</code>, and
$ cxi.pyana -c L498-indexigrate.cfg /path/to/xtc/files/e157-r0071-*.xtc
$ cxi.pyana -c L498-indexigrate.cfg /path/to/xtc/files/e157-r0072-*.xtc
$ cxi.pyana -c L498-indexigrate.cfg /path/to/xtc/files/e157-r0073-*.xtc
On successful completion, the number of files in the integration-first-lattice directory corresponds to the number of successfully integrated images. Owing to variations in hardware and compiler internals, it may deviate slightly from 11,583, the number reported in the ''cctbx.xfel'' paper<ref name="Hattne:2014"/>. Further details are available on the [[Indexing and integration|indexing and integration]] page of the [[Tutorials|tutorials]].

=== Indexing the secondary lattice ===

Indexing the secondary lattice is very similar to indexing the primary lattice, but requires a change to the source code. Edit <code>/path/to/phenix-src-20130328/labelit_regression/xfel/xfel_targets.py</code>, and uncomment (i.e. remove the leading <code>#</code> character) <code>"outlier_detection_switch=True"</code> on line 25. Then edit the configuration file, <code>L498-indexigrate.cfg</code> above, and change integration-first-lattice to integration-second-lattice in order not to overwrite the results of the previous analysis of the primary lattice. Before reanalyzing the first batch, ensure that <code>xtal_target</code> is set to <code>thermolysin27</code>.
$ cxi.pyana -c L498-indexigrate.cfg /path/to/xtc/files/e157-r0016-*.xtc
$ cxi.pyana -c L498-indexigrate.cfg /path/to/xtc/files/e157-r0017-*.xtc
$ …
$ cxi.pyana -c L498-indexigrate.cfg /path/to/xtc/files/e157-r0027-*.xtc
Then set <code>xtal_target</code> is set to <code>thermolysin73</code> in the configuration file, and process the second batch.
$ cxi.pyana -c L498-indexigrate.cfg /path/to/xtc/files/e157-r0071-*.xtc
$ cxi.pyana -c L498-indexigrate.cfg /path/to/xtc/files/e157-r0072-*.xtc
$ cxi.pyana -c L498-indexigrate.cfg /path/to/xtc/files/e157-r0073-*.xtc
The number of integrated secondary lattices should be close to 2,021, the number reported in the ''cctbx.xfel'' paper<ref name="Hattne:2014"/>.

== Merging all integrated images ==

The phil-file below defines values suitable for merging the primary and secondary lattices previously integrated.
data = integration-first-lattice
data = integration-second-lattice
d_min = 2.10
merge_anomalous = True
min_corr = -1
model = 2tli.pdb
nproc = 16
plot_single_index_histograms = False
raw_data.sdfac_auto = True
rescale_with_average_cell = True
significance_filter.apply = True
set_average_unit_cell = True
mysql {
database = db_name
passwd = db_passwd
runtag = L498_thermolysin
user = db_user
}
output {
n_bins = 10
prefix = L498_thermolysin
}
scaling {
algorithm = mark0
mtz_file = 2tli.mtz
show_plots = False
log_cutoff = 0.0
}
<code>integration-first-lattice</code> and <code>integration-second-lattice</code> may need to be adjusted to point to the directories where the [[#Indexing and integration | indexing and integration step]] left its results. <code>db_name</code><code>, db_user</code>, and <code>db_passwd</code> must be substituted with the database name and access credentials to a [http://www.mysql.com MySQL] database. Databases on hosts other than the one used to merge the thermolysin data can be accessed by additionally specifying the <code>mysql.host</code> and <code>mysql.port</code> options. The model and structure factors for the scaling reference, <code>model</code> and <code>scaling.mtz_file</code> above, are both available for download from the [http://www.rcsb.org RCSB Protein Data Bank] ([http://www.rcsb.org/pdb/explore/explore.do?structureId=2tli PDB ID 2tli]). If the [http://www.phenix-online.org ''PHENIX''] suite is installed, these are conveniently obtained at the command line using
$ phenix.fetch_pdb --mtz 2tli

To merge the thermolysin data, save the suitably modified configuration file to e.g. <code>L498-merge.phil</code>, and run
$ cxi.merge L498-merge.phil
$ cxi.xmerge L498-merge.phil
Merging statistics are printed on standard output. The merged MTZ-file is written to a file whose name is determined by the value of <code>output.prefix</code> in the configuration file (with the values shown above, the output file would be <code>L498_thermolysin.mtz</code>). Note that the version of merging programs from 28 March, 2013 do not not report the Rsplit statistic.

== References ==

<references/>

Processing L498 thermolysin

2014-03-10T10:54:28Z

Hattne: Created instructions to reproduce L498 thermolysin processing results.

This page contains instructions for reproducing the results reported in the 2014 ''cctbx.xfel'' paper<ref>[http://dx.doi.org/10.1038/nmeth.2887 Hattne, J et al. Accurate macromolecular structures using minimal measurements from X-ray free-electron lasers. Nat. Methods In press (2014).]</ref>. The data are available from the [http://cxidb.org/id-23 Coherent X-ray Imaging Data Bank (CXIDB)] (raw XTC files, 4.0 TiB), and must be downloaded to a local disk prior to starting the analysis. Furthermore, the PSDM Software Distribution must be [[Set up PSDM software | set up]], along with a test release and an empty analysis package. A [http://www.mysql.com MySQL] database is required to merge the integrated diffraction images. Processing time depends strongly on the computational resources available; using 48 processors on a 64-core 1.4 GHz Opteron-based computer, the analysis takes around 24 hours. The instructions assume some familiarity with ''cctbx.xfel'' and its installation and configuration procedure.

== Installing a ''cctbx.xfel'' snapshot from March 28, 2013 ==

The thermolysin data for the ''cctbx.xfel'' paper was processed around March 28, 2013. Unfortunately, regular nightly releases are not available for that time, but a specially prepared source-code bundle has been made available. This bundle differs from the regular ''cctbx'' bundles in that ''LABELIT'' is included, and that the directory layout is identical to that of a developer installation. To download and unpack the bundle in the current directory:
$ wget http://adder.lbl.gov/cctbx.xfel/downloads/cctbx.xfel-20130328.tar.gz
$ tar -xpvzf cctbx.xfel-20130328.tar.gz
Next, create a build directory (called <code>phenix-build-20130328</code> below, but that is an arbitrary choice), in which the sources are configured and compiled. The Python interpreter required to complete this step, must be the one supplied by the PSDM Software Distribution. Once the PSDM software has been [[Set up PSDM software | set up], this interpreter can be located using
$ find $SIT_ROOT/sw/external/python -perm /0111 -type f -wholename "*/$SIT_ARCH/*/python"
To prepare the build directory using this interpreter
$ mkdir phenix-build-20130328
$ cd phenix-build-20130328
$ python ../phenix-src-20130328/cctbx_project/libtbx/configure.py cxi_xdr_xes xfel
$ . setpaths.sh
where <code>python</code> is the path to the Python interpreter located using the previous find-command. Note that csh-users should run <code>source setpaths.csh</code> instead of <code>. setpaths.sh</code>. Then compile the sources
$ make
$ make
Note that the <code>make</code> command may need to be run twice in order to complete the build. Once <code>make</code> does not produce any output from the compiler, the build is complete.

To make the ''cctbx.xfel'' analysis modules available to PSDM's ''pyana'',
$ cd /path/to/test/release
$ sit_setup
$ cd my_ana_pkg
$ ln -fns /path/to/phenix-src-20130328/cctbx_project/xfel/cxi/cspad_ana src
$ cd ..
$ scons
where <code>/path/to/test/release</code> and <code>my_ana_pkg</code> are the path to the test release and the name of the analysis package chosen while [[Set up PSDM software | setting up the PSDM software distribution]]. <code>/path/to</code> denotes the path to the directory containing the unpacked ''cctbx.xfel'' sources. The last step compiles the ''cctbx.xfel'' analysis modules.

== Create a dark image ==

To meaningfully process the thermolysin diffraction data, an average of all the images in a dark run—a run without any X-rays impinging on the detector—must be subtracted from the individual diffraction images. The configuration file below can be used to produce such an average, as well as an image of the standard deviation of all the pixels over the course of the run.
# -*- mode: conf -*-

[pyana]
modules = my_ana_pkg.mod_average
num-cpu = 4

[my_ana_pkg.mod_average]
address = CxiDs1-0|Cspad-0
calib_dir = /path/to/phenix-src-20130328/cctbx_project/xfel/metrology/CSPad/run4/CxiDs1.0_Cspad.0
avg_basename = Ds1-avg
avg_dirname = r0031
stddev_basename = Ds1-stddev
stddev_dirname = r0031
/path/to refers to the directory containing the unpacked ''cctbx.xfel'' sources. All other options with values set in italics can be modified without adversely affecting averaging. The above file will use four simultaneous processes, and write the average and standard deviation images to files whose names start with <code>Ds1-avg</code> and <code>Ds1-stddev</code>, respectively, both in a directory called <code>r0031</code>.

The data deposited at the CXIDB contains a dark run, <code>r0031</code>. To average the images in that run, save the above configuration file to disk, e.g. <code>L498-dark.cfg</code>, apply modifications as necessary, and execute
$ cxi.pyana -c L498-dark.cfg /path/to/xtc/files/e157-r0031-*.xtc
where /path/to/xtc/files is the path to the directory containing the raw XTC files dowloaded from [http://cxidb.org/id-23 CXIDB]. The files written to the <code>r0031</code> directory will have a current datestamp appended, which can safely be removed to simplify subsequent configuration files.
$ cd r0031
$ mv Ds1-avg20140308104336073.pickle Ds1-avg.pickle
$ mv Ds1-stddev20140308104336371.pickle Ds1-stddev.pickle
$ cd ..
Further details are available on the [[Preparatory steps#Create a dark average|Create a dark image]] page of the [[Tutorials|tutorials]].

== Index the thermolysin data ==

A configuration file for processing the primary lattices in the thermolysin data is shown below.
# -*- mode: conf -*-

[pyana]
modules = my_ana_pkg.mod_hitfind:threshold \
my_ana_pkg.mod_hitfind:index
num-cpu = 48

[my_ana_pkg.mod_hitfind]
address = CxiDs1-0|Cspad-0
calib_dir = /path/to/phenix-src-20130328/cxi_xdr_xes/cftbx/metrology/CSPad/run4/CxiDs1.0:Cspad.0
dark_path = r0031/Ds1-avg.pickle
dark_stddev = r0031/Ds1-stddev.pickle
db_logging = False
detz_offset = 575

[my_ana_pkg.mod_hitfind:threshold]
dispatch = nop
distl_flags = permissive
distl_min_peaks = 16
threshold = 450
xtal_target = hitfind

[my_ana_pkg.mod_hitfind:index]
dispatch = index
integration_dirname = integration-first-lattice
integration_basename = int-
xtal_target = thermolysin27
The configuration file above instructs <code>mod_hitfind</code> to use 48 processes. It disables all image output, which reduces the amout of disk space required to perform the analysis to about 3.4 GiB. /path/to again refers to the directory containing the unpacked ''cctbx.xfel'' sources, and <code>dark_path</code> as well as <code>dark_stddev</code> may have to be changed to reflect the location of the previously generated dark images. Integration results will be written to the directory integration-first-lattice.

Due to particularities of the thermolysin measurement, processing proceeds in two batches. To analyse the first batch, runs 16 through 27, save the above configuration file to disk, e.g. <code>L498-indexigrate.cfg</code>, apply modifications as necessary, and execute
$ cxi.pyana -c L498-indexigrate.cfg /path/to/xtc/files/e157-r0016-*.xtc
$ cxi.pyana -c L498-indexigrate.cfg /path/to/xtc/files/e157-r0017-*.xtc
$ …
$ cxi.pyana -c L498-indexigrate.cfg /path/to/xtc/files/e157-r0027-*.xtc
where /path/to/xtc/files is the path to the directory containing the raw XTC files. The second batch, runs 71 through 73, was recorded using a different distance between the interaction region and the detector. This necessitates different areas of the detector to be ignored due to different shadowing, and this is accounted for by a different value of the <code>xtal_target</code> option in the configuration file. To analyse this set of runs, edit <code>L498-indexigrate.cfg</code>, change <code>thermolysin27</code> to <code>thermolysin73</code>, and
$ cxi.pyana -c L498-indexigrate.cfg /path/to/xtc/files/e157-r0071-*.xtc
$ cxi.pyana -c L498-indexigrate.cfg /path/to/xtc/files/e157-r0072-*.xtc
$ cxi.pyana -c L498-indexigrate.cfg /path/to/xtc/files/e157-r0073-*.xtc
On successful completion, the number of files in the integration-first-lattice directory corresponds to the number of successfully integrated images. Owing to variations in hardware and compiler internals, it may deviate slightly from 11,583, the number reported in the ''cctbx.xfel'' paper.

=== Index the secondary lattice ===

Indexing the secondary lattice is very similar to indexing the primary lattice, but requires a change to the source code. Edit <code>/path/to/phenix-src-20130328/labelit_regression/xfel/xfel_targets.py</code>, and uncomment (i.e. remove the leading <code>#</code> character) <code>"outlier_detection_switch=True"</code> on line 25. Then edit the configuration file, <code>L498-indexigrate.cfg</code> above, and change integration-first-lattice to integration-second-lattice in order not to overwrite the results of the previous analysis of the primary lattice. Before re-analysing the first batch, ensure that <code>xtal_target</code> is set to <code>thermolysin27</code>.
$ cxi.pyana -c L498-indexigrate.cfg /path/to/xtc/files/e157-r0016-*.xtc
$ cxi.pyana -c L498-indexigrate.cfg /path/to/xtc/files/e157-r0017-*.xtc
$ …
$ cxi.pyana -c L498-indexigrate.cfg /path/to/xtc/files/e157-r0027-*.xtc
Then set <code>xtal_target</code> is set to <code>thermolysin73</code> in the configuration file, and process the second batch.
$ cxi.pyana -c L498-indexigrate.cfg /path/to/xtc/files/e157-r0071-*.xtc
$ cxi.pyana -c L498-indexigrate.cfg /path/to/xtc/files/e157-r0072-*.xtc
$ cxi.pyana -c L498-indexigrate.cfg /path/to/xtc/files/e157-r0073-*.xtc
The number of integrated secondary lattices should be close to 2,021, the number reported in the ''cctbx.xfel'' paper.

== Merge all integrated images ==

The phil-file below defines values suitable for merging the primary and secondary lattices previously integrated.
data = integration-first-lattice
data = integration-second-lattice
d_min = 2.10
merge_anomalous = True
min_corr = -1
model = 2tli.pdb
nproc = 16
plot_single_index_histograms = False
raw_data.sdfac_auto = True
rescale_with_average_cell = True
significance_filter.apply = True
set_average_unit_cell = True
mysql {
database = db_name
passwd = db_passwd
runtag = L498_thermolysin
user = db_user
}
output {
n_bins = 10
prefix = L498_thermolysin
}
scaling {
algorithm = mark0
mtz_file = 2tli.mtz
show_plots = False
log_cutoff = 0.0
}
<code>integration-first-lattice</code> and <code>integration-second-lattice</code> may need to be adjusted to point to the directories where <code>mod_hitfind</code> left the integration results. <code>db_name</code><code>, db_user</code>, and <code>db_passwd</code> must be substituted with the database name and access credentials to a MySQL database. Databases on hosts other than the one used to merge the thermolysin data can be accessed by additionally specifing the <code>mysql.host</code> and <code>mysql.port</code> options. The model and structure factors for the scaling reference, <code>model</code> and <code>scaling.mtz_file</code> above, are both available for download from the [http://www.rcsb.org RCSB Protein Data Bank (PDB)]. If the [http://www.phenix-online.org ''PHENIX''] suite is installed, this is conveniently achieved from the command line using
$ phenix.fetch_pdb --mtz 2tli

To merge the thermolysin data, save the suitably modified configuration to e.g. <code>L498-merge.phil</code>, and run
$ cxi.merge L498-merge.phil
$ cxi.xmerge L498-merge.phil
Merging statistics are printed on standard output. The merged MTZ-file is written to a file whose name is determined by the value of <code>output.prefix</code> in the configuration file (<code>L498_thermolysin.mtz</code> with the values shown above). Note that the version of merging programs from 28 March, 2013 do not not report the Rsplit statistic.

== References ==

<references/>

Ha14 Tutorials

2014-03-10T06:44:51Z

Hattne: Added link to L498 thermolysin reprocessing page

These tutorials assumes the steps outlined in [[setup]] have been followed first. Processing xfel data at LCLS consists of four steps:

* [[Preparatory steps]]
* [[Metrology refinement]]
* [[Indexing and integration]]. Optional steps:
** Real-time [[progress monitoring]].
** Configuring specific spotfinder, indexing and integration parameters using ''[[phil]]''.
* [[Merging]]
* [[Advanced Merging]]: Use cases for isomorphous replacement and de novo structure
* [[Resolving an Indexing Ambiguity]]

Additionally, ''cctbx.xfel'' utilizes ''[[phil]]'' files to manage its configuration settings.

[[Processing L498 thermolysin]] details instructions to reproduce the results published in the forthcoming ''cctbx.xfel'' paper.

Setup

2014-03-03T21:10:03Z

Hattne: Add NERSC to the table.

== Prerequisities ==

''cctbx.xfel'' must be [[Installation | installed]] before it can be used. The installation root directories affect the absolute paths used in the following instructions and differ from site to site. Some common installation root directories are listed below.
{| class="wikitable sortable"
|-
! Site
!class="unsortable"| Installation build directory
!class="unsortable"| Installation source directory
!class="unsortable"| Notes
|-
| BNL
| <code>/usr/local/crys_test/cctbx/cctbx_build</code>
| <code>/usr/local/crys_test/cctbx</code>
|
|-
| CCI, LBNL
|
|
| No central, shared installation available
|-
| NERSC
|
|
| No central, shared installation available
|-
| SLAC
| <code>/reg/g/cctbx/build</code>
| <code>/reg/g/cctbx/sources</code>
|
|}
''cctbx.xfel'' can be installed anywhere on the filesystem where write access is granted. Developers will generally have ''cctbx.xfel'' installed in their home directory. Even if ''cctbx.xfel'' is already installed, the environment must first be [[Set up PSDM software | prepared for the PSDM software distribution]].

== ''cctbx.xfel'' setup ==

To start using ''cctbx.xfel'', the environment has to be configured. The details depend on the ''cctbx.xfel'' installation root directories and the user's shell.
In what follows <code>/path/to/cctbx/build</code> and <code>/path/to/cctbx/sources</code> should be replaced with the actual build and source directories of the ''cctbx.xfel'' installation. bash-users can prepare the environment for ''cctbx.xfel'' using
$ . /path/to/cctbx/build/setpaths.sh
From a csh, the corresponding command is
% source /path/to/cctbx/build/setpaths.csh
Sourcing <code>setpaths</code> only modifies the environment of the current shell, and would have to be repeated every time a new shell is started. To make the changes persistent, add
test -r /reg/g/cctbx/build/setpaths.sh && \
. /reg/g/cctbx/build/setpaths.sh
to <code>~/.bashrc</code>, or
test -r /reg/g/cctbx/build/setpaths.csh && \
source /reg/g/cctbx/build/setpaths.csh
to <code>~/.cshrc</code> as appropriate.

To make the ''cctbx.xfel'' analysis modules available to PSDM's ''pyana'',
$ cd /path/to/test/release
$ sit_setup
$ cd my_ana_pkg
$ ln -fns /path/to/cctbx/sources/cctbx_project/xfel/cxi/cspad_ana src
$ cd ..
$ scons
where <code>/path/to/test/release</code> and <code>my_ana_pkg</code> are the path to the test release and the name of the analysis package chosen while [[Set up PSDM software | setting up the PSDM software distribution]]. The last step compiles the ''cctbx.xfel'' analysis modules. These steps need only be performed once.

Set up PSDM software

2014-03-03T21:09:03Z

Hattne: Add NERSC to the table.

{{ mbox
| type = notice
| text = Note that throughout the instructions <code>$</code> denotes the prompt of any unprivileged user on the system. Certain shells may be configured to use <code>%</code> or <code>></code> for that purpose. The prompt should not be included in the commands typed to the shell.
}}

== Prerequisites ==

The [https://confluence.slac.stanford.edu/display/PCDS/Data+Management Photon Science Data Management (PSDM)] software distribution must be installed. The PSDM software distribution provides ''pyana'', which is a critical dependency of ''cctbx.xfel''. Its installation root directory affects the absolute paths used in the following instructions and differs from site to site. Some common installation root directories are listed below.
{| class="wikitable sortable"
|-
! Site
! class="unsortable"| Installation root directory
! class="unsortable"| Notes
|-
| BNL
| <code>/usr/local/crys/psdm</code>
|
|-
| CCI, LBNL
| <code>/net/viper/raid1/psdm</code>
| Only available on viper.lbl.gov
|-
| NERSC
| <code>/global/project/projectdirs/lcls/psdm-carver</code>
| Only to be used on the carver system
|-
| SLAC
| <code>/reg/g/psdm</code>
|
|}
The PSDM software distribution is supported on the most popular flavors of Linux and is installed at several sites worldwide. Separate installation and maintenance instructions are provided by the PSDM project at [https://confluence.slac.stanford.edu/display/PSDM/Software+Distribution PSDM software distribution]. The package can be installed anywhere on the filesystem, and it is generally possible to install it without superuser privileges.

== Create a test release and an analysis package ==

A test release, which refers to a particular release of the PSDM distribution, is represented by a directory in the file system, often called <code>analysis-rel</code> or <code>myrelease</code>. An analysis package within the test release in turn refers to the files implementing the analysis modules of the package. Details about test releases and analysis packages are covered in [https://confluence.slac.stanford.edu/display/PCDS/Analysis+Workbook.+Quick+Tour Analysis Workbook. Quick Tour]. To prepare the environment for using the PSDM software distribution, bash-users should source <code>ana_env.sh</code>,
$ . /path/to/psdm/etc/ana_env.sh
where <code>/path/to/psdm</code> should be replaced with the path to the installation root directory. csh-users should source <code>ana_env.csh</code> instead,
% source /path/to/psdm/etc/ana_env.csh
Sourcing <code>ana_env.sh</code> or <code>ana_env.csh</code> only modifies the environment of the current shell, and would have to be repeated every time a new shell is started. To make the changes persistent, add
test -r /path/to/psdm/etc/ana_env.sh && . /path/to/psdm/etc/ana_env.sh
to <code>~/.bashrc</code>, or
test -r /path/to/psdm/etc/ana_env.csh && source /path/to/psdm/etc/ana_env.csh
to <code>~/.cshrc</code> as appropriate. This ensures the environment is properly prepared for non-interactive shells, e.g. the shells that are used to run jobs submitted to the cluster. To also be able to use the PSDM software from interactive shells, it is recommended that bash-users put
test -r ~/.bashrc && . ~/.bashrc
near the top of <code>~/.bash_profile</code>. There is no need to edit any additional files for csh-users, because csh always reads <code>~/.cshrc</code>.

A test released based on the most current PSDM distribution is then set up in the current working directory using
$ newrel ana-current myrelease
$ cd myrelease
$ sit_setup
$ newpkg my_ana_pkg
Note that the names for the test release and the analysis package, myrelease and my_ana_pkg above, are common, but nevertheless arbitrary, choices.

These steps only need to be performed once.

== External links ==

* [https://confluence.slac.stanford.edu/display/PCDS/Pyana+User+Manual Pyana User Manual]
* [https://confluence.slac.stanford.edu/display/PCDS/Analysis+Workbook.+Packages+and+Releases Analysis Workbook. Packages and Releases]

Maintaining an installation

2014-02-28T23:45:37Z

Hattne: Updated link to PSDM installation instructions.

== Keeping up to date with PSDM releases==



Since the ''cctbx.xfel'' modules do not critically depend on any particular version of ''pyana'', it is generally safe to closely track the latest version of the PSDM software distribution. To upgrade to the current PSDM release,
$ cd /path/to/test/release
$ relupgrade ana-current
where <code>/path/to/test/release</code> must be substituted for path of the test release. The ''cctbx.xfel'' modules should then be rebuilt using
$ scons -c
$ scons
It is possible to manually edit <code>/path/to/test/release/.sit_release</code> to contain <code>ana-current</code> instead of a particular version number. This ensures that the latest version of the PSDM software distribution is used during each session.

== Keeping a developer installation up to date ==

Given the rapid development of the ''PHENIX'' suite, it is recommended to update the sources and rebuild often. Certain changes may require the configure command (or <code>libtbx.refresh</code>) to be rerun.
$ cd /path/to/cctbx/sources
$ ./allsvn update
$ cd /path/to/cctbx/build
$ make

Set up PSDM software

2014-02-28T23:44:01Z

Hattne: Updated link to PSDM installation instructions.

{{ mbox
| type = notice
| text = Note that throughout the instructions <code>$</code> denotes the prompt of any unprivileged user on the system. Certain shells may be configured to use <code>%</code> or <code>></code> for that purpose. The prompt should not be included in the commands typed to the shell.
}}

== Prerequisites ==

The [https://confluence.slac.stanford.edu/display/PCDS/Data+Management Photon Science Data Management (PSDM)] software distribution must be installed. The PSDM software distribution provides ''pyana'', which is a critical dependency of ''cctbx.xfel''. Its installation root directory affects the absolute paths used in the following instructions and differs from site to site. Some common installation root directories are listed below.
{| class="wikitable sortable"
|-
! Site
! class="unsortable"| Installation root directory
! class="unsortable"| Notes
|-
| BNL
| <code>/usr/local/crys/psdm</code>
|
|-
| CCI, LBNL
| <code>/net/viper/raid1/psdm</code>
| Only available on viper.lbl.gov
|-
| SLAC
| <code>/reg/g/psdm</code>
|
|}
The PSDM software distribution is supported on the most popular flavors of Linux and is installed at several sites worldwide. Separate installation and maintenance instructions are provided by the PSDM project at [https://confluence.slac.stanford.edu/display/PSDM/Software+Distribution PSDM software distribution]. The package can be installed anywhere on the filesystem, and it is generally possible to install it without superuser privileges.

== Create a test release and an analysis package ==

A test release, which refers to a particular release of the PSDM distribution, is represented by a directory in the file system, often called <code>analysis-rel</code> or <code>myrelease</code>. An analysis package within the test release in turn refers to the files implementing the analysis modules of the package. Details about test releases and analysis packages are covered in [https://confluence.slac.stanford.edu/display/PCDS/Analysis+Workbook.+Quick+Tour Analysis Workbook. Quick Tour]. To prepare the environment for using the PSDM software distribution, bash-users should source <code>ana_env.sh</code>,
$ . /path/to/psdm/etc/ana_env.sh
where <code>/path/to/psdm</code> should be replaced with the path to the installation root directory. csh-users should source <code>ana_env.csh</code> instead,
% source /path/to/psdm/etc/ana_env.csh
Sourcing <code>ana_env.sh</code> or <code>ana_env.csh</code> only modifies the environment of the current shell, and would have to be repeated every time a new shell is started. To make the changes persistent, add
test -r /path/to/psdm/etc/ana_env.sh && . /path/to/psdm/etc/ana_env.sh
to <code>~/.bashrc</code>, or
test -r /path/to/psdm/etc/ana_env.csh && source /path/to/psdm/etc/ana_env.csh
to <code>~/.cshrc</code> as appropriate. This ensures the environment is properly prepared for non-interactive shells, e.g. the shells that are used to run jobs submitted to the cluster. To also be able to use the PSDM software from interactive shells, it is recommended that bash-users put
test -r ~/.bashrc && . ~/.bashrc
near the top of <code>~/.bash_profile</code>. There is no need to edit any additional files for csh-users, because csh always reads <code>~/.cshrc</code>.

A test released based on the most current PSDM distribution is then set up in the current working directory using
$ newrel ana-current myrelease
$ cd myrelease
$ sit_setup
$ newpkg my_ana_pkg
Note that the names for the test release and the analysis package, myrelease and my_ana_pkg above, are common, but nevertheless arbitrary, choices.

These steps only need to be performed once.

== External links ==

* [https://confluence.slac.stanford.edu/display/PCDS/Pyana+User+Manual Pyana User Manual]
* [https://confluence.slac.stanford.edu/display/PCDS/Analysis+Workbook.+Packages+and+Releases Analysis Workbook. Packages and Releases]

Ha14 installation

2014-02-25T11:49:22Z

Hattne: Always use current bundles.

== Prerequisite: install and set up the PSDM suite ==

It is assumed that the [[Set up PSDM software | PSDM software distribution has been set up]]. Developers may additionally want to [[Set up ssh-agent | set up an ssh-agent]]. Note again that several sites already have ''cctbx.xfel'' installed, and that this step can be skipped for regular users not involved in the development of the software. Once ''cctbx.xfel'' has been installed it must be [[Setup | set up]] before it can be used.

== Download and extract a source tree ==

This step has to be performed on a host with Internet access. Not all hosts at SLAC have that, but the members of e.g. psexport pool do. In what follows, the absolute path <code>/path/to/cctbx/sources</code> should be replaced with something more appropriate (a site-wide install will benefit from a location on a shared disk, whereas <code>~/projects/phenix-src</code> is a common choice among developers). The directory must accessible everywhere ''cctbx.xfel'' is to be run.

=== Regular users ===
$ mkdir -p /path/to/cctbx/sources
$ cd /path/to/cctbx/sources
$ wget http://cci.lbl.gov/cctbx_build/results/current/cctbx_bundle.tar.gz
$ wget http://adder.lbl.gov/cctbx.xfel/downloads/labelit_bundle_current.tar.gz
$ mkdir cctbx_project
$ tar -xpvzf cctbx_bundle.tar.gz -C cctbx_project --strip-components=1
$ tar -xpvzf labelit_bundle_current.tar.gz





=== Developers ===
A user account on cci.lbl.gov is required, the name of which is to be substituted for <code>user</code> below.
$ mkdir -p /path/to/cctbx/sources
$ cd /path/to/cctbx/sources
$ svn export svn+ssh://user@cci.lbl.gov/phenix_regression/trunk/phenix_svn_getting_started.csh
$ ./phenix_svn_getting_started.csh user
$ rm phenix_svn_getting_started.csh

== Create the <code>cxi_user</code> directory ==

Add the <code>cxi_user</code> directory and its <code>__init__.py</code> module. This defines the default integration algorithm used by ''cctbx.xfel''.
$ mkdir cxi_user
$ cat > cxi_user/__init__.py << EOF
from xfel.mono_simulation.mono_treatment import post_outlier_rejection
from xfel.mono_simulation.mono_treatment import pre_get_predictions
EOF

== Configure the source tree ==
To avoid problems with run-time dynamic linking of Python extensions, the Python interpreter required for the next step must be the one provided by the PSDM software distribution. That interpreter can be located using find, e.g.
$ find $SIT_ROOT/sw/external/python -perm /0111 -type f -wholename "*/$SIT_ARCH/*/python" 2> /dev/null
At SLAC this interpreter is located somewhere under <code>/reg/g/psdm/sw/external/python</code>. Then create and initialize the build directory, <code>/path/to/cctbx/build</code> below, using the Python interpreter located using the above find command. The build directory is often located next to the source directory (<code>~/projects/phenix-build</code> is a common developer choice).

=== Regular users ===
$ mkdir -p /path/to/cctbx/build
$ cd /path/to/cctbx/build
$ python /path/to/cctbx/sources/cctbx_project/libtbx/configure.py xfel

=== Developers ===
$ mkdir -p /path/to/cctbx/build
$ cd /path/to/cctbx/build
$ python /path/to/cctbx/sources/cctbx_project/libtbx/configure.py cxi_xdr_xes labelit_regression phenix xfel


== Compile the sources ==

Initialise the running shell using the newly created configuration files. bash-users should
$ . setpaths.sh
while csh-users will instead need to run
% source setpaths.csh

The next step requires a C++ compiler. At SLAC, the members of the psexport pool do not have any compilers installed, but the interactive nodes, reachable via psanacs, do.
$ cd /path/to/cctbx/build
$ make
On SLAC's interactive nodes, this takes just over 6 minutes. To finalize the installation, see [[Setup]].



== External links ==

* [http://cctbx.sourceforge.net/current_cvs/installation.html#manually-building-from-sources-under-unix Manually building from sources under Unix]

Ha14 installation

2014-02-20T00:07:11Z

Hattne: LABELIT bundle is now publicly available.

== Prerequisite: install and set up the PSDM suite ==

It is assumed that the [[Set up PSDM software | PSDM software distribution has been set up]]. Developers may additionally want to [[Set up ssh-agent | set up an ssh-agent]]. Note again that several sites already have ''cctbx.xfel'' installed, and that this step can be skipped for regular users not involved in the development of the software. Once ''cctbx.xfel'' has been installed it must be [[Setup | set up]] before it can be used.

== Download and extract a source tree ==

This step has to be performed on a host with Internet access. Not all hosts at SLAC have that, but the members of e.g. psexport pool do. In what follows, the absolute path <code>/path/to/cctbx/sources</code> should be replaced with something more appropriate (a site-wide install will benefit from a location on a shared disk, whereas <code>~/projects/phenix-src</code> is a common choice among developers). The directory must accessible everywhere ''cctbx.xfel'' is to be run.

=== Regular users ===
$ mkdir -p /path/to/cctbx/sources
$ cd /path/to/cctbx/sources
$ wget http://cci.lbl.gov/cctbx_build/results/2013_08_13_0005/cctbx_bundle.tar.gz
$ wget http://adder.lbl.gov/cctbx.xfel/downloads/labelit_bundle_20140106.tar.gz
$ mkdir cctbx_project
$ tar -xpvzf cctbx_bundle.tar.gz -C cctbx_project --strip-components=1
$ tar -xpvzf labelit_bundle_20130814.tar.gz

{{ mbox
| type = speedy
| text = New bundles are automatically prepared nightly. It is, however, not advisable to automatically update from the latest bundle. Please check with the program authors before upgrading a critical installation.
}}



=== Developers ===
A user account on cci.lbl.gov is required, the name of which is to be substituted for <code>user</code> below.
$ mkdir -p /path/to/cctbx/sources
$ cd /path/to/cctbx/sources
$ svn export svn+ssh://user@cci.lbl.gov/phenix_regression/trunk/phenix_svn_getting_started.csh
$ ./phenix_svn_getting_started.csh user
$ rm phenix_svn_getting_started.csh

== Create the <code>cxi_user</code> directory ==

Add the <code>cxi_user</code> directory and its <code>__init__.py</code> module. This defines the default integration algorithm used by ''cctbx.xfel''.
$ mkdir cxi_user
$ cat > cxi_user/__init__.py << EOF
from xfel.mono_simulation.mono_treatment import post_outlier_rejection
from xfel.mono_simulation.mono_treatment import pre_get_predictions
EOF

== Configure the source tree ==
To avoid problems with run-time dynamic linking of Python extensions, the Python interpreter required for the next step must be the one provided by the PSDM software distribution. That interpreter can be located using find, e.g.
$ find $SIT_ROOT/sw/external/python -perm /0111 -type f -wholename "*/$SIT_ARCH/*/python" 2> /dev/null
At SLAC this interpreter is located somewhere under <code>/reg/g/psdm/sw/external/python</code>. Then create and initialize the build directory, <code>/path/to/cctbx/build</code> below, using the Python interpreter located using the above find command. The build directory is often located next to the source directory (<code>~/projects/phenix-build</code> is a common developer choice).

=== Regular users ===
$ mkdir -p /path/to/cctbx/build
$ cd /path/to/cctbx/build
$ python /path/to/cctbx/sources/cctbx_project/libtbx/configure.py xfel

=== Developers ===
$ mkdir -p /path/to/cctbx/build
$ cd /path/to/cctbx/build
$ python /path/to/cctbx/sources/cctbx_project/libtbx/configure.py cxi_xdr_xes labelit_regression phenix xfel


== Compile the sources ==

Initialise the running shell using the newly created configuration files. bash-users should
$ . setpaths.sh
while csh-users will instead need to run
% source setpaths.csh

The next step requires a C++ compiler. At SLAC, the members of the psexport pool do not have any compilers installed, but the interactive nodes, reachable via psanacs, do.
$ cd /path/to/cctbx/build
$ make
On SLAC's interactive nodes, this takes just over 6 minutes. To finalize the installation, see [[Setup]].



== External links ==

* [http://cctbx.sourceforge.net/current_cvs/installation.html#manually-building-from-sources-under-unix Manually building from sources under Unix]

Ha14 installation

2014-02-19T22:56:56Z

Hattne: Updated download link for LABELIT bundle

== Prerequisite: install and set up the PSDM suite ==

It is assumed that the [[Set up PSDM software | PSDM software distribution has been set up]]. Developers may additionally want to [[Set up ssh-agent | set up an ssh-agent]]. Note again that several sites already have ''cctbx.xfel'' installed, and that this step can be skipped for regular users not involved in the development of the software. Once ''cctbx.xfel'' has been installed it must be [[Setup | set up]] before it can be used.

== Download and extract a source tree ==

This step has to be performed on a host with Internet access. Not all hosts at SLAC have that, but the members of e.g. psexport pool do. In what follows, the absolute path <code>/path/to/cctbx/sources</code> should be replaced with something more appropriate (a site-wide install will benefit from a location on a shared disk, whereas <code>~/projects/phenix-src</code> is a common choice among developers). The directory must accessible everywhere ''cctbx.xfel'' is to be run.

=== Regular users ===
$ mkdir -p /path/to/cctbx/sources
$ cd /path/to/cctbx/sources
$ wget http://cci.lbl.gov/cctbx_build/results/2013_08_13_0005/cctbx_bundle.tar.gz
$ wget http://adder.lbl.gov/cctbx.xfel/downloads/labelit_bundle_20140106.tar.gz
$ mkdir cctbx_project
$ tar -xpvzf cctbx_bundle.tar.gz -C cctbx_project --strip-components=1
$ tar -xpvzf labelit_bundle_20130814.tar.gz

{{ mbox
| type = speedy
| text = New bundles are automatically prepared nightly. It is, however, not advisable to automatically update from the latest bundle. Please check with the program authors before upgrading a critical installation. Also note that the labelit_bundle cannot currently be downloaded.
}}



=== Developers ===
A user account on cci.lbl.gov is required, the name of which is to be substituted for <code>user</code> below.
$ mkdir -p /path/to/cctbx/sources
$ cd /path/to/cctbx/sources
$ svn export svn+ssh://user@cci.lbl.gov/phenix_regression/trunk/phenix_svn_getting_started.csh
$ ./phenix_svn_getting_started.csh user
$ rm phenix_svn_getting_started.csh

== Create the <code>cxi_user</code> directory ==

Add the <code>cxi_user</code> directory and its <code>__init__.py</code> module. This defines the default integration algorithm used by ''cctbx.xfel''.
$ mkdir cxi_user
$ cat > cxi_user/__init__.py << EOF
from xfel.mono_simulation.mono_treatment import post_outlier_rejection
from xfel.mono_simulation.mono_treatment import pre_get_predictions
EOF

== Configure the source tree ==
To avoid problems with run-time dynamic linking of Python extensions, the Python interpreter required for the next step must be the one provided by the PSDM software distribution. That interpreter can be located using find, e.g.
$ find $SIT_ROOT/sw/external/python -perm /0111 -type f -wholename "*/$SIT_ARCH/*/python" 2> /dev/null
At SLAC this interpreter is located somewhere under <code>/reg/g/psdm/sw/external/python</code>. Then create and initialize the build directory, <code>/path/to/cctbx/build</code> below, using the Python interpreter located using the above find command. The build directory is often located next to the source directory (<code>~/projects/phenix-build</code> is a common developer choice).

=== Regular users ===
$ mkdir -p /path/to/cctbx/build
$ cd /path/to/cctbx/build
$ python /path/to/cctbx/sources/cctbx_project/libtbx/configure.py xfel

=== Developers ===
$ mkdir -p /path/to/cctbx/build
$ cd /path/to/cctbx/build
$ python /path/to/cctbx/sources/cctbx_project/libtbx/configure.py cxi_xdr_xes labelit_regression phenix xfel


== Compile the sources ==

Initialise the running shell using the newly created configuration files. bash-users should
$ . setpaths.sh
while csh-users will instead need to run
% source setpaths.csh

The next step requires a C++ compiler. At SLAC, the members of the psexport pool do not have any compilers installed, but the interactive nodes, reachable via psanacs, do.
$ cd /path/to/cctbx/build
$ make
On SLAC's interactive nodes, this takes just over 6 minutes. To finalize the installation, see [[Setup]].



== External links ==

* [http://cctbx.sourceforge.net/current_cvs/installation.html#manually-building-from-sources-under-unix Manually building from sources under Unix]

File:Cctbx logo small.png

2014-02-19T22:15:53Z

Hattne: Initial attempt at replacing the default logo.

Initial attempt at replacing the default logo.

Maintaining an installation

2014-02-14T00:47:41Z

Hattne: Noted missing bits.

== Keeping up to date with PSDM releases==



Since the ''cctbx.xfel'' modules do not critically depend on any particular version of ''pyana'', it is generally safe to closely track the latest version of the PSDM software distribution. To upgrade to the current PSDM release,
$ cd /path/to/test/release
$ relupgrade ana-current
where <code>/path/to/test/release</code> must be substituted for path of the test release. The ''cctbx.xfel'' modules should then be rebuilt using
$ scons -c
$ scons
It is possible to manually edit <code>/path/to/test/release/.sit_release</code> to contain <code>ana-current</code> instead of a particular version number. This ensures that the latest version of the PSDM software distribution is used during each session.

== Keeping a developer installation up to date ==

Given the rapid development of the ''PHENIX'' suite, it is recommended to update the sources and rebuild often. Certain changes may require the configure command (or <code>libtbx.refresh</code>) to be rerun.
$ cd /path/to/cctbx/sources
$ ./allsvn update
$ cd /path/to/cctbx/build
$ make

Setup

2014-02-11T00:37:01Z

Hattne: Removed creating of ./lsf.sh symbolic link.

== Prerequisities ==

''cctbx.xfel'' must be [[Installation | installed]] before it can be used. The installation root directories affect the absolute paths used in the following instructions and differ from site to site. Some common installation root directories are listed below.
{| class="wikitable sortable"
|-
! Site
!class="unsortable"| Installation build directory
!class="unsortable"| Installation source directory
!class="unsortable"| Notes
|-
| BNL
| <code>/usr/local/crys_test/cctbx/cctbx_build</code>
| <code>/usr/local/crys_test/cctbx</code>
|
|-
| CCI, LBNL
|
|
| No central, shared installation available
|-
| SLAC
| <code>/reg/g/cctbx/build</code>
| <code>/reg/g/cctbx/sources</code>
|
|}
''cctbx.xfel'' can be installed anywhere on the filesystem where write access is granted. Developers will generally have ''cctbx.xfel'' installed in their home directory. Even if ''cctbx.xfel'' is already installed, the environment must first be [[Set up PSDM software | prepared for the PSDM software distribution]].

== ''cctbx.xfel'' setup ==

To start using ''cctbx.xfel'', the environment has to be configured. The details depend on the ''cctbx.xfel'' installation root directories and the user's shell.
In what follows <code>/path/to/cctbx/build</code> and <code>/path/to/cctbx/sources</code> should be replaced with the actual build and source directories of the ''cctbx.xfel'' installation. bash-users can prepare the environment for ''cctbx.xfel'' using
$ . /path/to/cctbx/build/setpaths.sh
From a csh, the corresponding command is
% source /path/to/cctbx/build/setpaths.csh
Sourcing <code>setpaths</code> only modifies the environment of the current shell, and would have to be repeated every time a new shell is started. To make the changes persistent, add
test -r /reg/g/cctbx/build/setpaths.sh && \
. /reg/g/cctbx/build/setpaths.sh
to <code>~/.bashrc</code>, or
test -r /reg/g/cctbx/build/setpaths.csh && \
source /reg/g/cctbx/build/setpaths.csh
to <code>~/.cshrc</code> as appropriate.

To make the ''cctbx.xfel'' analysis modules available to PSDM's ''pyana'',
$ cd /path/to/test/release
$ sit_setup
$ cd my_ana_pkg
$ ln -fns /path/to/cctbx/sources/cctbx_project/xfel/cxi/cspad_ana src
$ cd ..
$ scons
where <code>/path/to/test/release</code> and <code>my_ana_pkg</code> are the path to the test release and the name of the analysis package chosen while [[Set up PSDM software | setting up the PSDM software distribution]]. The last step compiles the ''cctbx.xfel'' analysis modules. These steps need only be performed once.

Preparatory steps pyana version

2014-02-11T00:36:10Z

Hattne: Replaced ./lsf.sh with cxi.lsf.

The CS-PAD detector at LCLS is sensitive to temperature distortions and is not automatically dark-subtracted, unlike images from a synchrotron source. Because of this it is necessary to periodically collect a dark run with the shutters closed during an LCLS experiment. The dark run should contain on the order of at least 1000 images. Before an experimental run can be processed, this dark run must be averaged so that subsequent experimental data can have the dark subtracted from each image. Further, events during collection can necessitate a new dark, such as ice in the beam oversaturating creating dead pixels or a change in temperature.

In addition, averaging a lighted run in general is a useful tool for examining diffraction throughout a data collection run.

Finally, inactive and non-bonded pixels, as well as hypersensitive pixels and any beam stop (if used) should be masked out. This tutorial covers averaging dark and light runs, and creating mask images.

== Setup your scratch area ==

Create a scratch folder and populate it with some directories that will be used during this tutorial, changing <username> to your SLAC account name:

$ cd /reg/g/cctbx/tutorials/scratch/
$ mkdir <username>
$ cd <username>
$ mkdir darks
$ mkdir averages
$ mkdir results

Or you may use a directory in your home folder.

== Create a dark average ==

To create the average, start by copying the dark.cfg file to your pyana directory:

$ cd ~/myrelease
$ sit_setup
$ cp /reg/g/cctbx/tutorials/preprocessing/dark.cfg .

Open dark.cfg. It will look something like this:


<pre>
# -*- Mode: Conf -*-
#
# Since mod_average must be run on a single host, it cannot be
# submitted using cxi.lsf.
#
# bsub -n 6 -J r0002 -o /reg/g/cctbx/tutorials/scratch/<username>/dark_r0002.out -q psanacsq -R "span[hosts=1]" \
# "cxi.pyana -c ~/myrelease/dark.cfg /reg/d/ana11/cxi/data/Mar2013calib/xtc/e236-r0002-s0*.xtc"
#

[pyana]
modules = my_ana_pkg.mod_average
num-cpu = 8

[my_ana_pkg.mod_average]
calib_dir = /reg/g/cctbx/sources/cctbx_project/xfel/metrology/CSPad/run4/CxiDs1.0_Cspad.0
avg_dirname = /reg/g/cctbx/tutorials/scratch/<username>/darks
stddev_dirname = /reg/g/cctbx/tutorials/scratch/<username>/darks
max_dirname = /reg/g/cctbx/tutorials/scratch/<username>/darks
detz_offset = 581
address = CxiDs1-0|Cspad-0
avg_basename = Ds1-r0002-avg
stddev_basename = Ds1-r0002-stddev
max_basename = Ds1-r0002-max
</pre>

The config file is broken into three sections, the header, the pyana section and the modules section.

=== Header Section ===
The comments at the top of the file include a bsub command. This is the command to be entered from the ~/myrelease directory to queue the job for processing. Bsub will submit the job, asking for 6 CPUs (-n), naming the job r0002 (-J), placing a log file in the -o location, and so forth. There are several queues available for use, the psanacsq is just one of them. The -R option restricts the job to running on a single host, which is required for averaging. Finally, the cxi.pyana command in quotes is the command that will actually be executed on the host computer, and specifies the config file and the streams to use. Note the asterisk, which allows averaging all the streams from a single run at one time.

'''Before executing the command, replace <username> in the command and in the config file with your user name.'''

=== Pyana Section ===
Pyana operates by having a root section in the config file which specifies which modules to run. In this case we are running ''cctbx.xfel's'' mod_average module. If we were running multiple modules, we could chain them together here, and each module would be executed in sequence using the parameters specified below. We also request a number of cpus to parallelize the job with. Even though averaging must be done on a single host, on that host the processing can be split up among multiple cores. Note, other aspects of ''cctbx.xfel'', such as indexing and integration, do not have this restriction.

=== Module Section ===
The final section specifies parameters specific to averaging:
* calib_dir: this directory is provided by LCLS and includes the layout information of the 64 tiles of the CS-PAD detector
* avg_dirname: directory to place the average image
* stddev_dirname: directory to place the standard deviation image
* max_dirname: directory to place the maximum projection image
* detz_offset: experiment-specific number used to calculate the detector distance. This is the distance in millimeters between the sample interaction region and the far end of the detector stage. The XTC stream contains the distance from the detector stage to the detector. These two numbers are subtracted to calculate the distance between sample interaction region and the detector. We determine this number experimentally using a grid search, testing a range of potential detz_offsets and choosing the one that shows the greatest success at indexing images
* address: the name of the detector in the XTC stream
* avg_basename: this name is prepended to a datestamp to name the average image
* stddev_basename: this name is prepended to a datestamp to name the standard deviation image
* max_basename: this name is prepended to a datestamp to name the maximum projection image image

=== Running and testing the command ===
Submit the job as specified above, using the bsub command in the header:
$ bsub -n 6 -J r0002 -o /reg/g/cctbx/tutorials/scratch/<username>/dark_r0002.out -q psanacsq -R "span[hosts=1]"
"cxi.pyana -c ~/myrelease/dark.cfg /reg/d/ana11/cxi/data/Mar2013calib/xtc/e236-r0002-s0*.xtc"

Type bjobs to see if the job is running. When it is, follow the output thusly:
$ tail -f /reg/g/cctbx/tutorials/scratch/<username>/dark_r0002.out

When the average has been completed, the output can be inspected using the "cctbx" image viewer:
$ cctbx.image_viewer /reg/g/cctbx/tutorials/scratch/<username>/darks/Ds1-r0002-*.pickle

== Averaging an experimental light run ==

Once the dark run is made, experimental runs can be averaged to see the extent of diffraction and potential pathologies in the run. Copy the average.cfg file from the tutorials directory to your pyana directory:

$ cd ~/myrelease
$ cp /reg/g/cctbx/tutorials/preprocessing/average.cfg .

The cfg will look like this:
<pre>
# -*- Mode: Conf -*-
#
# Since mod_average must be run on a single host, it cannot be
# submitted using cxi.lsf.
#
# bsub -n 6 -J r0003 -o /reg/g/cctbx/tutorials/scratch/<username>/average_r0003.out -q psanacsq -R "span[hosts=1]" \
# "cxi.pyana -c average.cfg /reg/d/ana11/cxi/data/Mar2013calib/xtc/e236-r0003-s0*.xtc"
#

[pyana]
modules = my_ana_pkg.mod_average
num-cpu = 8

[my_ana_pkg.mod_average]
calib_dir = /reg/g/cctbx/sources/cctbx_project/xfel/metrology/CSPad/run4/CxiDs1.0_Cspad.0
dark_path = /reg/g/cctbx/tutorials/scratch/<username>/darks/<name of dark average>.pickle
dark_stddev = /reg/g/cctbx/tutorials/scratch/<username>/darks/<name of dark stddev>.pickle
avg_dirname = /reg/g/cctbx/tutorials/scratch/<username>/averages
stddev_dirname = /reg/g/cctbx/tutorials/scratch/<username>/averages
max_dirname = /reg/g/cctbx/tutorials/scratch/<username>/averages
detz_offset = 581
address = CxiDs1-0|Cspad-0
avg_basename = Ds1-r0003-avg
stddev_basename = Ds1-r0003-stddev
max_basename = Ds1-r0003-max
</pre>

The only difference is the addition of two parameters:
* dark_path: path to the average dark image created before
* dark_stddev: path to the standard deviation dark image created before

The job is ran as before using the bsub command in the header, again remembering to change all instances of <username> to your own SLAC user name. Here you must also change <name of dark average> and <name of dark stddev> to the names of the average and standard deviation images created in the previous step. The result of the run will be three new images that can be viewed with cctbx.image_viewer in the same manner as the darks were.

== Information contained in the light average ==

Firstly, the light and dark images can be used to mask of any bad pixels, as detailed below. The light images (maximum in particular) also serve as a virtual, dark-subtracted, powder pattern that can used to inspect the data for diffraction. The light maximum should also flag up obvious errors in meteorology, because rings will be non-continuous if the quadrants are misaligned. Intensity values from the corners of the light average or maximum are also useful to estimate background when taking first guesses at thresholds for hit-finding and integration.

== Creating a mask image ==
''cctbx.xfel'' uses three images to create a mask. Those images, and the purposes of them are listed below:
* Average image from a dark run: pixels ≤ 0 are considered dead, pixels > 2000 are too sensitive
* Standard deviation from a dark run: pixels ≤ 0 are considered dead, pixels ≥ 10 are too uncertain
* Maximum projection from a lighted run (''i.e.'' an experimental run): pixels < 300 are considered non-bonded or in a shadow. The presence of diffracting data is not needed, but also will not interfere.

To create the mask, execute these commands:
$ cd /reg/g/cctbx/tutorials/scratch/<username>/
$ mkdir masks
$ cd masks
$ cxi.make_mask -m 10 ../darks/<name of dark average>.pickle ../darks/<name of dark standard deviation>.pickle
../averages/<name of light maximum projection>.pickle

A new image is created where each pixel has one of two values: -2 (masked out pixels), and 0 (good pixels). Note, the maximum projection minimum pixel value is overridden here to a value that is more sensible for the tutorial's lysozyme data. You will likely need to adjust this parameter based on your data (see below for details and for further overrides).

Use the resultant mask image during processing with ''cctbx.xfel'' by adding this flag to your pyana config file:

mask_path = <path to mask file>

In addition, a polygon and a circle can be specified using the --poly_mask and --circle_mask parameters:

$ cxi.make_mask --poly_mask=17,854,76,967,760,982,762,879 --circle_mask=855,939,103 Cspad-avg.pickle Cspad-stddev.pickle Cspad-max.pickle

This example creates a mask image with the polygon with points (17,854), (76,967), (760,982), and (762,879) masked out, and the circle with center (855,939) and radius 103 masked out. Only one polygon and circle can be specified at the moment.

The mask can be examined with cctbx.image_viewer.

=== Options ===

Some additional options are available for specifying the details of the masking operation:
;-a, --avg_max (default 2000)
: specifies the maximum value in ADU a pixel in the average image can have before it's masked out. The minimum is always zero.
;-s, --stddev_max (default 10)
: specifies the maximum value in ADU a pixel in the standard deviation image can have before it's masked out. The minimum is always zero.
;-m, --maxproj_min (default 300)
: specifies the minimum value in ADU a pixel in the maximum projection image can have before it's masked out. Of the three parameters for controlling cutoffs in ''cxi.make_mask'', the -m option is unique from the -a and -s options in that it will likely vary the according to your sample's background. Carefully examine the corners of your lighted maximum projection and choose a value lower than the ADU values displayed. Also note that there is no cutoff on the high end of the maximum projection image specified in the mask, as that is defined as the saturation value for the detector.
;-x, --mask_pix_val (default -2)
: specifies the value to use when masking out a pixel. In other words, bad pixels in images in the XTC stream will be replaced with this value.

Progress monitoring

2014-02-11T00:33:25Z

Hattne: Replaced ./lsf.sh with cxi.lsf.

If configured to do so, ''cctbx.xfel'' can use a mysql database during hit finding to log the hit rate and other useful parameters, enabling real-time monitoring via a graphical display.

== Configuring trial monitoring ==
Add to the mod_hitfind section of your pyana config file these lines:
<pre>
# cxi.lsf will replace this with its deterimined id, otherwise
# set it to the ID you wish. 0 is the default.
trial_id = 0
db_logging = True
sql_buffer_size = 5
</pre>

These three parameters turn on and configure database-backed logging of hitfinding results:
* trial_id: we find it useful to organize processing in numbered trials. Whenever we go back and change the parameters of our processing, adjusting the target cell or spotfinder parameters for example, we re-process under a different trial ID. When you submit your processing job using cxi.lsf, you can specify the trial ID on the command line with –t. What you specify will overwrite what is listed here in the cfg file. If you use bsub, then what is in the cfg file is what’s used. Note, it isn’t a good idea to re-use the trial_id when re-prossing the data if database logging is being used, simply because the hits will be logged twice which will clutter future analysis. Finally, if you want to see which trials you have used for which runs, execute '''cxi.list_db_metadata'''
* db_logging: Switch for turning on and off logging
* sql_buffer_size: a queue size for regulating how often to add data to the database. In our experience, 5 has been just fine.

== Monitoring hit finding in real time ==
After submitting a job with db logging enabled, use cxi.monitor_trials <trial number> to display the hit rate during processing. Other command line options and their defaults are listed below, including the ability to display multiple runs at once. The default is to display the 5 latest runs by timestamp.
* t_wait = 8000: amount of time in ms between queries to the database
* hit_cutoff = 16: the number of bragg spots before a frame is considered a hit. If this is different than what’s in your config file, then the monitor will be using a different cutoff to display which frames are hits.
* average_window = 1000: how many frames are in the sliding averaging window shown in the hit % graph
* n_points = 1000: how many total points to display on the graph. Set it to 0 to display all the data points (can be computationally intensive).
* display_time = 1800: number of seconds of experiment time to display. Displays this number of seconds unless that would add up to more than 5 runs.
* run_num = display only this run
* run_min = start at this run
* run_max = end with this run

Indexing and integration

2014-02-11T00:31:31Z

Hattne: Replaced ./lsf.sh with cxi.lsf.

In principle, one could attempt to index and integrate every single image recorded during a "diffract-and-destroy" experiment. Typically only a small fraction of the recorded images exhibit an indexable lattice, and in practice a ''hit-finder'' is needed to triage the stream of images, and preserve computational resources.

Indexing and integration are tightly coupled; it is impossible to integrate an image without first obtaining an indexing solution. The result of successfully indexing an image is essentially a model of crystal–beam interaction. The model not only describes how the crystal is oriented with respect to the direction of the X-ray pulse, but also its degree of imperfection (''i.e.'' mosaicity), and qualities of the illuminating pulse (''e.g.'' bandwidth). The integration algorithm uses this model to predict where Bragg spots appear on the image, and integrates the intensities around those predications. The integration algorithm thus exploits the regularity of the underlying crystal lattice to recover signal where it was too weak to be detected.

== <code>pyana</code> section ==

For historical reasons, hit-finding and indexing/integration are both implemented in the same analysis module, called <code>mod_hitfind</code>. To chain the two functions together, ''pyana'' must be configured to pass each event through two different instances of the same module. The two instances are differentiated by their name following the colon. In the example below, the first instance is called <code>hitfind</code>, and the second is <code>index</code>.
<pre>
[pyana]
modules = my_ana_pkg.mod_hitfind:hitfind \
my_ana_pkg.mod_hitfind:index
</pre>
Any instance of an analysis module can signal ''pyana'' to skip further processing of the event. In the case above, the <code>hitfind</code> instance of <code>mod_hitfind</code> would signal ''pyana'' to skip the event if it determines that the image associated with the event is not a hit. In that case, the second instance, <code>index</code>, will never see the event. Otherwise, the event is fed to the second instance, which will attempt indexing and integration on the associated image.

The <code>pyana</code>-section of the processing configuration used for this part of the tutorial has one more feature.
<pre>
[pyana]
modules = my_ana_pkg.mod_hitfind:hitfind \
my_ana_pkg.mod_dump:hit \
my_ana_pkg.mod_hitfind:index \
my_ana_pkg.mod_dump:indexed
</pre>
Each instance of <code>mod_hitfind</code> is followed by an instance of <code>mod_dump</code>. <code>mod_dump</code> is a simple module that just outputs the image associated with the event to the file system. In the above configuration, images are written to disk if they are determined to be hits, and then written again—possibly using a different name—if indexing and integration succeeds.

== Hit-finding==

Hit-finding may be viewed as a pattern-recognition problem, separate from whatever methods are used for actual processing of a diffraction image. The hit-finder in ''cctbx.xfel'', however, is based on the same algorithms as are used for data analysis. In particular, the hit-finder implemented in mod_hitfind uses the ''Spotfinder'' (Zhang ''et al.'', 2006) algorithm to detect strong, low-resolution peaks.

The configuration for the <code>hitfind</code> instance of <code>mod_hitfind</code> is shown below.
<pre>
[my_ana_pkg.mod_hitfind:hitfind]
address = CxiDs1-0|Cspad-0
calib_dir = /reg/g/cctbx/sources/cctbx_project/xfel/metrology/CSPad/run4/CxiDs1.0_Cspad.0
dark_path = /reg/g/cctbx/tutorials/darks/Ds1-r0002-avg.pickle
dark_stddev = /reg/g/cctbx/tutorials/darks/Ds1-r0002-stddev.pickle
detz_offset = 581
dispatch = nop
distl_flags = permissive
distl_min_peaks = 16
threshold = 450
xtal_target = /reg/g/cctbx/tutorials/indexing/hitfind-7.1.phil
</pre>

; <code>address</code>
: Full data source address of the DAQ device, see [https://confluence.slac.stanford.edu/display/PCDS/Pyana+User+Manual#PyanaUserManual-DataSourceAddress Data Source Address] in the [https://confluence.slac.stanford.edu/display/PCDS/Pyana+User+Manual Pyana User Manual]. This identifies the detector from which images are to be extracted.
; <code>calib_dir</code>
: This value defines the initial placement of the detector elements. ''cctbx.xfel'' implements its metrology corrections with respect to these initial definitions. As a result, the value of this option should not be changed.
; <code>dark_path</code>
: Path to an average image from a dark run (''i.e.'' pedestal) to use for dark-subtraction.
; <code>dark_stddev</code>
: Path to a standard-deviation image from a dark run.
; <code>detz_offset</code>
: The distance from the interaction region, where the X-ray pulse and the sample jet intersect, to the back of the detector stage, in mm.
; <code>dispatch</code>
: What action <code>mod_hitfind</code> is to take. The value should be <code>nop</code> for hit-finding.
; <code>distl_flags</code>
: General behavior of the hit-finder, either <code>permissive</code> or <code>restrictive</code>.
; <code>distl_min_peaks</code>
: How many ''strong'', low-resolution peaks are required to classify the image as a hit.
; <code>threshold</code>
: How high a peak must be above the background to be classified as ''strong'', in analog-to-digital units (ADU).
; <code>xtal_target</code>
: ''phil''-file containing further configuration options for the hit-finder.

== Image output ==
The configuration section for image output is given below.
<pre>
[my_ana_pkg.mod_dump:hit]
address = CxiDs1-0|Cspad-0
calib_dir = /reg/g/cctbx/sources/cctbx_project/xfel/metrology/CSPad/run4/CxiDs1.0_Cspad.0
out_dirname = out
out_basename = shot-
</pre>

The <code>address</code> and <code>calib_dir</code> options were explained in [[#Hit-finding|Hit-finding]] above.

; <code>out_dirname</code>
: Directory portion of output image pathname.
; <code>out_basename</code>
: Filename prefix of output image pathname.

The actual pathname of the image as it is written to the file system is determined by joining <code>out_dirname</code> and <code>out_basename</code> using the directory separator (<code>/</code> on Unix), and appending a textual representation of the timestamp when the image was recorded.

== Indexing and integration ==
<pre>
address = CxiDs1-0|Cspad-0
calib_dir = /reg/g/cctbx/sources/cctbx_project/xfel/metrology/CSPad/run4/CxiDs1.0_Cspad.0
dark_path = /reg/g/cctbx/tutorials/darks/Ds1-r0002-avg.pickle
dark_stddev = /reg/g/cctbx/tutorials/darks/Ds1-r0002-stddev.pickle
detz_offset = 581
dispatch = index
integration_dirname = integration
integration_basename = int-
xtal_target = Ls04-lysozyme.phil
</pre>

The options not already explained in [[#Hit-finding|Hit-finding]] are:
; <code>dispatch</code>
: What action <code>mod_hitfind</code> is to take. The value should be <code>index</code> for indexing.
; <code>integration_dirname</code>
: Directory portion of output integration file pathname.
; <code>integration_basename</code>
: Filename prefix of output integration file pathname.
; <code>xtal_target</code>
: ''[[phil]]''-file containing further configuration options for the indexing and integration algorithms. Descriptions of these parameters are found on the ''[[phil]]'' section of the tutorial.

The integration file is not an image file, but a list of Miller indices, their integrated intensities, and estimated uncertainties. The actual pathname of the integration file as it is written to the file system is determined by joining out_dirname and out_basename using the directory separator (/ on Unix), and appending a textual representation of the timestamp when the corresponding image was recorded.

== Processing the tutorial data ==
To submit a processing job to the cluster, execute the following commands:
<pre>
$ cd ~/myrelease
$ sit_setup
$ cp /reg/g/cctbx/tutorials/indexing/Ls04-lysozyme.cfg .
$ cxi.lsf \
-c Ls04-lysozyme.cfg -i /reg/d/ana11/cxi/data/Mar2013calib/xtc \
-o /reg/g/cctbx/tutorials/scratch/<username>/lysozyme \
-p 6 -q psanacsq -r 3
</pre>

To minimize the load on the cluster during the tutorial, please do not submit more than one run.

Ha14 installation

2014-02-07T11:02:13Z

Hattne: Cleanup.

== Prerequisite: install and set up the PSDM suite ==

It is assumed that the [[Set up PSDM software | PSDM software distribution has been set up]]. Developers may additionally want to [[Set up ssh-agent | set up an ssh-agent]]. Note again that several sites already have ''cctbx.xfel'' installed, and that this step can be skipped for regular users not involved in the development of the software. Once ''cctbx.xfel'' has been installed it must be [[Setup | set up]] before it can be used.

== Download and extract a source tree ==

This step has to be performed on a host with Internet access. Not all hosts at SLAC have that, but the members of e.g. psexport pool do. In what follows, the absolute path <code>/path/to/cctbx/sources</code> should be replaced with something more appropriate (a site-wide install will benefit from a location on a shared disk, whereas <code>~/projects/phenix-src</code> is a common choice among developers). The directory must accessible everywhere ''cctbx.xfel'' is to be run.

=== Regular users ===
$ mkdir -p /path/to/cctbx/sources
$ cd /path/to/cctbx/sources
$ wget http://cci.lbl.gov/cctbx_build/results/2013_08_13_0005/cctbx_bundle.tar.gz
$ wget http://cci.lbl.gov/~hattne/labelit/labelit_bundle_20130814.tar.gz
$ mkdir cctbx_project
$ tar -xpvzf cctbx_bundle.tar.gz -C cctbx_project --strip-components=1
$ tar -xpvzf labelit_bundle_20130814.tar.gz

{{ mbox
| type = speedy
| text = New bundles are automatically prepared nightly. It is, however, not advisable to automatically update from the latest bundle. Please check with the program authors before upgrading a critical installation. Also note that the labelit_bundle cannot currently be downloaded.
}}



=== Developers ===
A user account on cci.lbl.gov is required, the name of which is to be substituted for <code>user</code> below.
$ mkdir -p /path/to/cctbx/sources
$ cd /path/to/cctbx/sources
$ svn export svn+ssh://user@cci.lbl.gov/phenix_regression/trunk/phenix_svn_getting_started.csh
$ ./phenix_svn_getting_started.csh user
$ rm phenix_svn_getting_started.csh

== Create the <code>cxi_user</code> directory ==

Add the <code>cxi_user</code> directory and its <code>__init__.py</code> module. This defines the default integration algorithm used by ''cctbx.xfel''.
$ mkdir cxi_user
$ cat > cxi_user/__init__.py << EOF
from xfel.mono_simulation.mono_treatment import post_outlier_rejection
from xfel.mono_simulation.mono_treatment import pre_get_predictions
EOF

== Configure the source tree ==
To avoid problems with run-time dynamic linking of Python extensions, the Python interpreter required for the next step must be the one provided by the PSDM software distribution. That interpreter can be located using find, e.g.
$ find $SIT_ROOT/sw/external/python -perm /0111 -type f -wholename "*/$SIT_ARCH/*/python" 2> /dev/null
At SLAC this interpreter is located somewhere under <code>/reg/g/psdm/sw/external/python</code>. Then create and initialize the build directory, <code>/path/to/cctbx/build</code> below, using the Python interpreter located using the above find command. The build directory is often located next to the source directory (<code>~/projects/phenix-build</code> is a common developer choice).

=== Regular users ===
$ mkdir -p /path/to/cctbx/build
$ cd /path/to/cctbx/build
$ python /path/to/cctbx/sources/cctbx_project/libtbx/configure.py xfel

=== Developers ===
$ mkdir -p /path/to/cctbx/build
$ cd /path/to/cctbx/build
$ python /path/to/cctbx/sources/cctbx_project/libtbx/configure.py cxi_xdr_xes labelit_regression phenix xfel


== Compile the sources ==

Initialise the running shell using the newly created configuration files. bash-users should
$ . setpaths.sh
while csh-users will instead need to run
% source setpaths.csh

The next step requires a C++ compiler. At SLAC, the members of the psexport pool do not have any compilers installed, but the interactive nodes, reachable via psanacs, do.
$ cd /path/to/cctbx/build
$ make
On SLAC's interactive nodes, this takes just over 6 minutes. To finalize the installation, see [[Setup]].



== External links ==

* [http://cctbx.sourceforge.net/current_cvs/installation.html#manually-building-from-sources-under-unix Manually building from sources under Unix]

Maintaining an installation

2014-02-07T10:56:23Z

Hattne: Unified maintenance for regular users and developers.i

== Keeping up to date with PSDM releases==

Since the ''cctbx.xfel'' modules do not critically depend on any particular version of ''pyana'', it is generally safe to closely track the latest version of the PSDM software distribution. To upgrade to the current PSDM release,
$ cd /path/to/test/release
$ relupgrade ana-current
where <code>/path/to/test/release</code> must be substituted for path of the test release. The ''cctbx.xfel'' modules should then be rebuilt using
$ scons -c
$ scons
It is possible to manually edit <code>/path/to/test/release/.sit_release</code> to contain <code>ana-current</code> instead of a particular version number. This ensures that the latest version of the PSDM software distribution is used during each session.

== Keeping a developer installation up to date ==

Given the rapid development of the ''PHENIX'' suite, it is recommended to update the sources and rebuild often. Certain changes may require the configure command (or <code>libtbx.refresh</code>) to be rerun.
$ cd /path/to/cctbx/sources
$ ./allsvn update
$ cd /path/to/cctbx/build
$ make

Ha14 installation

2014-02-07T10:39:23Z

Hattne: Unified prerequisites for regular users and developers.i

== Prerequisite: install and set up the PSDM suite ==

It is assumed that the [[Set up PSDM software | PSDM software distribution has been set up]]. Developers may additionally want to [[Set up ssh-agent | set up an ssh-agent]]. Note again that several sites already have ''cctbx.xfel'' installed, and that this step can be skipped for regular users not involved in the development of the software. Once ''cctbx.xfel'' has been installed it must be [[Setup | set up]] before it can be used.

== Download and extract a source tree ==

This step has to be performed on a host with Internet access. Not all hosts at SLAC have that, but the members of e.g. psexport pool do. In what follows, the absolute path <code>/path/to/cctbx/sources</code> should be replaced with something more appropriate (a site-wide install will benefit from a location on a shared disk, whereas <code>~/projects/phenix-src</code> is a common choice among developers). The directory must accessible everywhere ''cctbx.xfel'' is to be run.

=== Regular users ===
$ mkdir -p /path/to/cctbx/sources
$ cd /path/to/cctbx/sources
$ wget http://cci.lbl.gov/cctbx_build/results/2013_08_13_0005/cctbx_bundle.tar.gz
$ wget http://cci.lbl.gov/~hattne/labelit/labelit_bundle_20130814.tar.gz
$ mkdir cctbx_project
$ tar -xpvzf cctbx_bundle.tar.gz -C cctbx_project --strip-components=1
$ tar -xpvzf labelit_bundle_20130814.tar.gz



{{ mbox
| type = speedy
| text = New bundles are automatically prepared nightly. It is, however, not advisable to automatically update from the latest bundle. Please check with the program authors before upgrading a critical installation. Also note that the labelit_bundle cannot currently be downloaded.
}}

=== Developers ===

A user account on cci.lbl.gov is required, the name of which is to be substituted for <code>user</code> below.
$ mkdir -p /path/to/cctbx/sources
$ cd /path/to/cctbx/sources
$ svn export svn+ssh://user@cci.lbl.gov/phenix_regression/trunk/phenix_svn_getting_started.csh
$ ./phenix_svn_getting_started.csh user
$ rm phenix_svn_getting_started.csh

== Create the <code>cxi_user</code> directory ==

Add the <code>cxi_user</code> directory and its <code>__init__.py</code> module. This defines the default integration algorithm used by ''cctbx.xfel''.
$ mkdir cxi_user
$ cat > cxi_user/__init__.py << EOF
from xfel.mono_simulation.mono_treatment import post_outlier_rejection
from xfel.mono_simulation.mono_treatment import pre_get_predictions
EOF

== Configure the source tree ==
To avoid problems with run-time dynamic linking of Python extensions, the Python interpreter required for the next step must be the one provided by the PSDM software distribution. That interpreter can be located using find, e.g.
$ find $SIT_ROOT/sw/external/python -perm /0111 -type f -wholename "*/$SIT_ARCH/*/python" 2> /dev/null
At SLAC this interpreter is located somewhere under <code>/reg/g/psdm/sw/external/python</code>. Then create and initialize the build directory, <code>/path/to/cctbx/build</code> below, using the Python interpreter located using the above find command. The build directory is often located next to the source directory (<code>~/projects/phenix-build</code> is a common developer choice).

=== Regular users ===
$ mkdir -p /path/to/cctbx/build
$ cd /path/to/cctbx/build
$ python /path/to/cctbx/sources/cctbx_project/libtbx/configure.py xfel

=== Developers ===
$ mkdir -p /path/to/cctbx/build
$ cd /path/to/cctbx/build
$ python /path/to/cctbx/sources/cctbx_project/libtbx/configure.py cxi_xdr_xes labelit_regression phenix xfel


== Compile the sources ==

Initialise the running shell using the newly created configuration files. bash-users should
$ . setpaths.sh
while csh-users will instead need to run
% source setpaths.csh

The next step requires a C++ compiler. At SLAC, the members of the psexport pool do not have any compilers installed, but the interactive nodes, reachable via psanacs, do.
$ cd /path/to/cctbx/build
$ make
On SLAC's interactive nodes, this takes just over 6 minutes. To finalize the installation, see [[Setup]].



== External links ==

* [http://cctbx.sourceforge.net/current_cvs/installation.html#manually-building-from-sources-under-unix Manually building from sources under Unix]

Ha14 installation

2014-02-07T10:37:57Z

Hattne: Unified compilation for regular users and developers.

== Prerequisite: install and set up the PSDM suite ==

It is assumed that the [[Set up PSDM software | PSDM software distribution has been set up]]. Note again that several sites already have ''cctbx.xfel'' installed, and that this step can be skipped for regular users not involved in the development of the software. Once ''cctbx.xfel'' has been installed it must be [[Setup | set up]] before it can be used.

== Download and extract a source tree ==

This step has to be performed on a host with Internet access. Not all hosts at SLAC have that, but the members of e.g. psexport pool do. In what follows, the absolute path <code>/path/to/cctbx/sources</code> should be replaced with something more appropriate (a site-wide install will benefit from a location on a shared disk, whereas <code>~/projects/phenix-src</code> is a common choice among developers). The directory must accessible everywhere ''cctbx.xfel'' is to be run.

=== Regular users ===
$ mkdir -p /path/to/cctbx/sources
$ cd /path/to/cctbx/sources
$ wget http://cci.lbl.gov/cctbx_build/results/2013_08_13_0005/cctbx_bundle.tar.gz
$ wget http://cci.lbl.gov/~hattne/labelit/labelit_bundle_20130814.tar.gz
$ mkdir cctbx_project
$ tar -xpvzf cctbx_bundle.tar.gz -C cctbx_project --strip-components=1
$ tar -xpvzf labelit_bundle_20130814.tar.gz



{{ mbox
| type = speedy
| text = New bundles are automatically prepared nightly. It is, however, not advisable to automatically update from the latest bundle. Please check with the program authors before upgrading a critical installation. Also note that the labelit_bundle cannot currently be downloaded.
}}

=== Developers ===

A user account on cci.lbl.gov is required, the name of which is to be substituted for <code>user</code> below.
$ mkdir -p /path/to/cctbx/sources
$ cd /path/to/cctbx/sources
$ svn export svn+ssh://user@cci.lbl.gov/phenix_regression/trunk/phenix_svn_getting_started.csh
$ ./phenix_svn_getting_started.csh user
$ rm phenix_svn_getting_started.csh

== Create the <code>cxi_user</code> directory ==

Add the <code>cxi_user</code> directory and its <code>__init__.py</code> module. This defines the default integration algorithm used by ''cctbx.xfel''.
$ mkdir cxi_user
$ cat > cxi_user/__init__.py << EOF
from xfel.mono_simulation.mono_treatment import post_outlier_rejection
from xfel.mono_simulation.mono_treatment import pre_get_predictions
EOF

== Configure the source tree ==
To avoid problems with run-time dynamic linking of Python extensions, the Python interpreter required for the next step must be the one provided by the PSDM software distribution. That interpreter can be located using find, e.g.
$ find $SIT_ROOT/sw/external/python -perm /0111 -type f -wholename "*/$SIT_ARCH/*/python" 2> /dev/null
At SLAC this interpreter is located somewhere under <code>/reg/g/psdm/sw/external/python</code>. Then create and initialize the build directory, <code>/path/to/cctbx/build</code> below, using the Python interpreter located using the above find command. The build directory is often located next to the source directory (<code>~/projects/phenix-build</code> is a common developer choice).

=== Regular users ===
$ mkdir -p /path/to/cctbx/build
$ cd /path/to/cctbx/build
$ python /path/to/cctbx/sources/cctbx_project/libtbx/configure.py xfel

=== Developers ===
$ mkdir -p /path/to/cctbx/build
$ cd /path/to/cctbx/build
$ python /path/to/cctbx/sources/cctbx_project/libtbx/configure.py cxi_xdr_xes labelit_regression phenix xfel


== Compile the sources ==

Initialise the running shell using the newly created configuration files. bash-users should
$ . setpaths.sh
while csh-users will instead need to run
% source setpaths.csh

The next step requires a C++ compiler. At SLAC, the members of the psexport pool do not have any compilers installed, but the interactive nodes, reachable via psanacs, do.
$ cd /path/to/cctbx/build
$ make
On SLAC's interactive nodes, this takes just over 6 minutes. To finalize the installation, see [[Setup]].



== External links ==

* [http://cctbx.sourceforge.net/current_cvs/installation.html#manually-building-from-sources-under-unix Manually building from sources under Unix]

Ha14 installation

2014-02-07T10:31:45Z

Hattne: Unified configuration for regular users and developers.

== Prerequisite: install and set up the PSDM suite ==

It is assumed that the [[Set up PSDM software | PSDM software distribution has been set up]]. Note again that several sites already have ''cctbx.xfel'' installed, and that this step can be skipped for regular users not involved in the development of the software. Once ''cctbx.xfel'' has been installed it must be [[Setup | set up]] before it can be used.

== Download and extract a source tree ==

This step has to be performed on a host with Internet access. Not all hosts at SLAC have that, but the members of e.g. psexport pool do. In what follows, the absolute path <code>/path/to/cctbx/sources</code> should be replaced with something more appropriate (a site-wide install will benefit from a location on a shared disk, whereas <code>~/projects/phenix-src</code> is a common choice among developers). The directory must accessible everywhere ''cctbx.xfel'' is to be run.

=== Regular users ===
$ mkdir -p /path/to/cctbx/sources
$ cd /path/to/cctbx/sources
$ wget http://cci.lbl.gov/cctbx_build/results/2013_08_13_0005/cctbx_bundle.tar.gz
$ wget http://cci.lbl.gov/~hattne/labelit/labelit_bundle_20130814.tar.gz
$ mkdir cctbx_project
$ tar -xpvzf cctbx_bundle.tar.gz -C cctbx_project --strip-components=1
$ tar -xpvzf labelit_bundle_20130814.tar.gz



{{ mbox
| type = speedy
| text = New bundles are automatically prepared nightly. It is, however, not advisable to automatically update from the latest bundle. Please check with the program authors before upgrading a critical installation. Also note that the labelit_bundle cannot currently be downloaded.
}}

=== Developers ===

A user account on cci.lbl.gov is required, the name of which is to be substituted for <code>user</code> below.
$ mkdir -p /path/to/cctbx/sources
$ cd /path/to/cctbx/sources
$ svn export svn+ssh://user@cci.lbl.gov/phenix_regression/trunk/phenix_svn_getting_started.csh
$ ./phenix_svn_getting_started.csh user
$ rm phenix_svn_getting_started.csh

== Create the <code>cxi_user</code> directory ==

Add the <code>cxi_user</code> directory and its <code>__init__.py</code> module. This defines the default integration algorithm used by ''cctbx.xfel''.
$ mkdir cxi_user
$ cat > cxi_user/__init__.py << EOF
from xfel.mono_simulation.mono_treatment import post_outlier_rejection
from xfel.mono_simulation.mono_treatment import pre_get_predictions
EOF

== Configure the source tree ==
To avoid problems with run-time dynamic linking of Python extensions, the Python interpreter required for the next step must be the one provided by the PSDM software distribution. That interpreter can be located using find, e.g.
$ find $SIT_ROOT/sw/external/python -perm /0111 -type f -wholename "*/$SIT_ARCH/*/python" 2> /dev/null
At SLAC this interpreter is located somewhere under <code>/reg/g/psdm/sw/external/python</code>. Then create and initialize the build directory, <code>/path/to/cctbx/build</code> below, using the Python interpreter located using the above find command. The build directory is often located next to the source directory (<code>~/projects/phenix-build</code> is a common developer choice).

=== Regular users ===
$ mkdir -p /path/to/cctbx/build
$ cd /path/to/cctbx/build
$ python /path/to/cctbx/sources/cctbx_project/libtbx/configure.py xfel

=== Developers ===
$ mkdir -p /path/to/cctbx/build
$ cd /path/to/cctbx/build
$ python /path/to/cctbx/sources/cctbx_project/libtbx/configure.py cxi_xdr_xes labelit_regression phenix xfel


== Compile the sources ==

Initialise the running shell using the newly created configuration files. bash-users should
$ . setpaths.sh
while csh-users will instead need to run
% source setpaths.csh

The next step requires a C++ compiler. At SLAC, the members of the psexport pool do not have any compilers installed, but the interactive nodes, reachable via psanacs, do.
$ make
On SLAC's interactive nodes, this takes just over 6 minutes. To finalize the installation, see [[Setup]].

== External links ==

* [http://cctbx.sourceforge.net/current_cvs/installation.html#manually-building-from-sources-under-unix Manually building from sources under Unix]

Ha14 installation

2014-02-07T10:18:02Z

Hattne: Unified source setup for regular users and developers.

== Prerequisite: install and set up the PSDM suite ==

It is assumed that the [[Set up PSDM software | PSDM software distribution has been set up]]. Note again that several sites already have ''cctbx.xfel'' installed, and that this step can be skipped for regular users not involved in the development of the software. Once ''cctbx.xfel'' has been installed it must be [[Setup | set up]] before it can be used.

== Download and extract a source tree ==

This step has to be performed on a host with Internet access. Not all hosts at SLAC have that, but the members of e.g. psexport pool do. In what follows, the absolute path <code>/path/to/cctbx/sources</code> should be replaced with something more appropriate (a site-wide install will benefit from a location on a shared disk, whereas <code>~/projects/phenix-src</code> is a common choice among developers). The directory must accessible everywhere ''cctbx.xfel'' is to be run.

=== Regular users ===
$ mkdir -p /path/to/cctbx/sources
$ cd /path/to/cctbx/sources
$ wget http://cci.lbl.gov/cctbx_build/results/2013_08_13_0005/cctbx_bundle.tar.gz
$ wget http://cci.lbl.gov/~hattne/labelit/labelit_bundle_20130814.tar.gz
$ mkdir cctbx_project
$ tar -xpvzf cctbx_bundle.tar.gz -C cctbx_project --strip-components=1
$ tar -xpvzf labelit_bundle_20130814.tar.gz



{{ mbox
| type = speedy
| text = New bundles are automatically prepared nightly. It is, however, not advisable to automatically update from the latest bundle. Please check with the program authors before upgrading a critical installation. Also note that the labelit_bundle cannot currently be downloaded.
}}

=== Developers ===


A user account on cci.lbl.gov is required, the name of which is to be substituted for <code>user</code> below.
$ mkdir -p /path/to/cctbx/sources
$ cd /path/to/cctbx/sources
$ svn export svn+ssh://user@cci.lbl.gov/phenix_regression/trunk/phenix_svn_getting_started.csh
$ ./phenix_svn_getting_started.csh user
$ rm phenix_svn_getting_started.csh

== Create the <code>cxi_user</code> directory ==

Add the <code>cxi_user</code> directory and its <code>__init__.py</code> module. This defines the default integration algorithm used by ''cctbx.xfel''.
$ mkdir cxi_user
$ cat > cxi_user/__init__.py << EOF
from xfel.mono_simulation.mono_treatment import post_outlier_rejection
from xfel.mono_simulation.mono_treatment import pre_get_predictions
EOF

== Configure the source tree ==
The Python interpreter required for the next step must be the same interpreter provided by the PSDM software distribution. That interpreter can be located using find, e.g.
$ find $SIT_ROOT/sw/external/python -perm /0111 -type f -wholename "*/$SIT_ARCH/*/python" 2> /dev/null
At SLAC this interpreter is located somewhere under <code>/reg/g/psdm/sw/external/python</code>. Then create and initialize the build directory using the appropriate Python interpreter. Developers may want to use slightly different configuration options, see [[Installation for developers]].
$ mkdir -p /path/to/cctbx_build
$ cd /path/to/cctbx_build
$ python /path/to/cctbx/directory/cctbx_project/libtbx/configure.py xfel
where <code>python</code> should be substituted with output of the previous find command. The name and location of the build directory is arbitrary; <code>/path/to/cctbx/directory/cctbx_build</code> is a common choice.

== Compile the sources ==

Initialise the running shell using the newly created configuration files. bash-users should
$ . setpaths.sh
while csh-users will instead need to run
% source setpaths.csh

The next step requires a C++ compiler. At SLAC, the members of the psexport pool do not have any compilers installed, but the interactive nodes, reachable via psanacs, do.
$ make
On SLAC's interactive nodes, this takes just over 6 minutes. To finalize the installation, see [[Setup]].

== External links ==

* [http://cctbx.sourceforge.net/current_cvs/installation.html#manually-building-from-sources-under-unix Manually building from sources under Unix]

Set up PSDM software

2014-02-07T09:43:41Z

Hattne: Migrated note about prompt.

{{ mbox
| type = notice
| text = Note that throughout the instructions <code>$</code> denotes the prompt of any unprivileged user on the system. Certain shells may be configured to use <code>%</code> or <code>></code> for that purpose. The prompt should not be included in the commands typed to the shell.
}}

== Prerequisites ==

The [https://confluence.slac.stanford.edu/display/PCDS/Data+Management Photon Science Data Management (PSDM)] software distribution must be installed. The PSDM software distribution provides ''pyana'', which is a critical dependency of ''cctbx.xfel''. Its installation root directory affects the absolute paths used in the following instructions and differs from site to site. Some common installation root directories are listed below.
{| class="wikitable sortable"
|-
! Site
! class="unsortable"| Installation root directory
! class="unsortable"| Notes
|-
| BNL
| <code>/usr/local/crys/psdm</code>
|
|-
| CCI, LBNL
| <code>/net/viper/raid1/psdm</code>
| Only available on viper.lbl.gov
|-
| SLAC
| <code>/reg/g/psdm</code>
|
|}
The PSDM software distribution is supported on the most popular flavors of Linux and is installed at several sites worldwide. Separate installation and maintenance instructions are provided by the PSDM project at [https://confluence.slac.stanford.edu/display/PCDS/PSDM+software+distribution PSDM software distribution]. The package can be installed anywhere on the filesystem, and it is generally possible to install it without superuser privileges.

== Create a test release and an analysis package ==

A test release, which refers to a particular release of the PSDM distribution, is represented by a directory in the file system, often called <code>analysis-rel</code> or <code>myrelease</code>. An analysis package within the test release in turn refers to the files implementing the analysis modules of the package. Details about test releases and analysis packages are covered in [https://confluence.slac.stanford.edu/display/PCDS/Analysis+Workbook.+Quick+Tour Analysis Workbook. Quick Tour]. To prepare the environment for using the PSDM software distribution, bash-users should source <code>ana_env.sh</code>,
$ . /path/to/psdm/etc/ana_env.sh
where <code>/path/to/psdm</code> should be replaced with the path to the installation root directory. csh-users should source <code>ana_env.csh</code> instead,
% source /path/to/psdm/etc/ana_env.csh
Sourcing <code>ana_env.sh</code> or <code>ana_env.csh</code> only modifies the environment of the current shell, and would have to be repeated every time a new shell is started. To make the changes persistent, add
test -r /path/to/psdm/etc/ana_env.sh && . /path/to/psdm/etc/ana_env.sh
to <code>~/.bashrc</code>, or
test -r /path/to/psdm/etc/ana_env.csh && source /path/to/psdm/etc/ana_env.csh
to <code>~/.cshrc</code> as appropriate. This ensures the environment is properly prepared for non-interactive shells, e.g. the shells that are used to run jobs submitted to the cluster. To also be able to use the PSDM software from interactive shells, it is recommended that bash-users put
test -r ~/.bashrc && . ~/.bashrc
near the top of <code>~/.bash_profile</code>. There is no need to edit any additional files for csh-users, because csh always reads <code>~/.cshrc</code>.

A test released based on the most current PSDM distribution is then set up in the current working directory using
$ newrel ana-current myrelease
$ cd myrelease
$ sit_setup
$ newpkg my_ana_pkg
Note that the names for the test release and the analysis package, myrelease and my_ana_pkg above, are common, but nevertheless arbitrary, choices.

These steps only need to be performed once.

== External links ==

* [https://confluence.slac.stanford.edu/display/PCDS/Pyana+User+Manual Pyana User Manual]
* [https://confluence.slac.stanford.edu/display/PCDS/Analysis+Workbook.+Packages+and+Releases Analysis Workbook. Packages and Releases]

Setup

2014-02-07T09:40:06Z

Hattne: Cleanup.

== Prerequisities ==

''cctbx.xfel'' must be [[Installation | installed]] before it can be used. The installation root directories affect the absolute paths used in the following instructions and differ from site to site. Some common installation root directories are listed below.
{| class="wikitable sortable"
|-
! Site
!class="unsortable"| Installation build directory
!class="unsortable"| Installation source directory
!class="unsortable"| Notes
|-
| BNL
| <code>/usr/local/crys_test/cctbx/cctbx_build</code>
| <code>/usr/local/crys_test/cctbx</code>
|
|-
| CCI, LBNL
|
|
| No central, shared installation available
|-
| SLAC
| <code>/reg/g/cctbx/build</code>
| <code>/reg/g/cctbx/sources</code>
|
|}
''cctbx.xfel'' can be installed anywhere on the filesystem where write access is granted. Developers will generally have ''cctbx.xfel'' installed in their home directory. Even if ''cctbx.xfel'' is already installed, the environment must first be [[Set up PSDM software | prepared for the PSDM software distribution]].

== ''cctbx.xfel'' setup ==

To start using ''cctbx.xfel'', the environment has to be configured. The details depend on the ''cctbx.xfel'' installation root directories and the user's shell.
In what follows <code>/path/to/cctbx/build</code> and <code>/path/to/cctbx/sources</code> should be replaced with the actual build and source directories of the ''cctbx.xfel'' installation. bash-users can prepare the environment for ''cctbx.xfel'' using
$ . /path/to/cctbx/build/setpaths.sh
From a csh, the corresponding command is
% source /path/to/cctbx/build/setpaths.csh
Sourcing <code>setpaths</code> only modifies the environment of the current shell, and would have to be repeated every time a new shell is started. To make the changes persistent, add
test -r /reg/g/cctbx/build/setpaths.sh && \
. /reg/g/cctbx/build/setpaths.sh
to <code>~/.bashrc</code>, or
test -r /reg/g/cctbx/build/setpaths.csh && \
source /reg/g/cctbx/build/setpaths.csh
to <code>~/.cshrc</code> as appropriate.

To make the ''cctbx.xfel'' analysis modules available to PSDM's ''pyana'',
$ cd /path/to/test/release
$ sit_setup
$ cd my_ana_pkg
$ ln -fns /path/to/cctbx/sources/cctbx_project/xfel/cxi/cspad_ana src
$ cd ..
$ ln -s /path/to/cctbx/sources/cctbx_project/xfel/cxi/lsf.sh .
$ scons
where <code>/path/to/test/release</code> and <code>my_ana_pkg</code> are the path to the test release and the name of the analysis package chosen while [[Set up PSDM software | setting up the PSDM software distribution]]. The last step compiles the ''cctbx.xfel'' analysis modules. These steps need only be performed once.

Maintaining an installation

2014-02-07T08:31:47Z

Hattne: Migrated text from PSDM page.

Set up PSDM software

2014-02-07T08:21:40Z

Hattne: Simplification.

== Prerequisites ==

The [https://confluence.slac.stanford.edu/display/PCDS/Data+Management Photon Science Data Management (PSDM)] software distribution must be installed. The PSDM software distribution provides ''pyana'', which is a critical dependency of ''cctbx.xfel''. Its installation root directory affects the absolute paths used in the following instructions and differs from site to site. Some common installation root directories are listed below.
{| class="wikitable sortable"
|-
! Site
! class="unsortable"| Installation root directory
! class="unsortable"| Notes
|-
| BNL
| <code>/usr/local/crys/psdm</code>
|
|-
| CCI, LBNL
| <code>/net/viper/raid1/psdm</code>
| Only available on viper.lbl.gov
|-
| SLAC
| <code>/reg/g/psdm</code>
|
|}
The PSDM software distribution is supported on the most popular flavors of Linux and is installed at several sites worldwide. Separate installation and maintenance instructions are provided by the PSDM project at [https://confluence.slac.stanford.edu/display/PCDS/PSDM+software+distribution PSDM software distribution]. The package can be installed anywhere on the filesystem, and it is generally possible to install it without superuser privileges.

== Create a test release and an analysis package ==

A test release, which refers to a particular release of the PSDM distribution, is represented by a directory in the file system, often called <code>analysis-rel</code> or <code>myrelease</code>. An analysis package within the test release in turn refers to the files implementing the analysis modules of the package. Details about test releases and analysis packages are covered in [https://confluence.slac.stanford.edu/display/PCDS/Analysis+Workbook.+Quick+Tour Analysis Workbook. Quick Tour]. To prepare the environment for using the PSDM software distribution, bash-users should source <code>ana_env.sh</code>,
$ . /path/to/psdm/etc/ana_env.sh
where <code>/path/to/psdm</code> should be replaced with the path to the installation root directory. csh-users should source <code>ana_env.csh</code> instead,
% source /path/to/psdm/etc/ana_env.csh
Sourcing <code>ana_env.sh</code> or <code>ana_env.csh</code> only modifies the environment of the current shell, and would have to be repeated every time a new shell is started. To make the changes persistent, add
test -r /path/to/psdm/etc/ana_env.sh && . /path/to/psdm/etc/ana_env.sh
to <code>~/.bashrc</code>, or
test -r /path/to/psdm/etc/ana_env.csh && source /path/to/psdm/etc/ana_env.csh
to <code>~/.cshrc</code> as appropriate. This ensures the environment is properly prepared for non-interactive shells, e.g. the shells that are used to run jobs submitted to the cluster. To also be able to use the PSDM software from interactive shells, it is recommended that bash-users put
test -r ~/.bashrc && . ~/.bashrc
near the top of <code>~/.bash_profile</code>. There is no need to edit any additional files for csh-users, because csh always reads <code>~/.cshrc</code>.

A test released based on the most current PSDM distribution is then set up in the current working directory using
$ newrel ana-current myrelease
$ cd myrelease
$ sit_setup
$ newpkg my_ana_pkg
Note that the names for the test release and the analysis package, myrelease and my_ana_pkg above, are common, but nevertheless arbitrary, choices.

These steps only need to be performed once.

== External links ==

* [https://confluence.slac.stanford.edu/display/PCDS/Pyana+User+Manual Pyana User Manual]
* [https://confluence.slac.stanford.edu/display/PCDS/Analysis+Workbook.+Packages+and+Releases Analysis Workbook. Packages and Releases]

Set up PSDM software

2014-02-07T08:18:55Z

Hattne: Cleanup

== Prerequisites ==

The [https://confluence.slac.stanford.edu/display/PCDS/Data+Management Photon Science Data Management (PSDM)] software distribution must be installed. The PSDM software distribution provides ''pyana'', which is a critical dependency of ''cctbx.xfel''. Its installation root directory affects the absolute paths used in the following instructions and differs from site to site. Some common installation root directories are listed below.
{| class="wikitable sortable"
|-
! Site
! class="unsortable"| Installation root directory
! class="unsortable"| Notes
|-
| BNL
| <code>/usr/local/crys/psdm</code>
|
|-
| CCI, LBNL
| <code>/net/viper/raid1/psdm</code>
| Only available on viper.lbl.gov
|-
| SLAC
| <code>/reg/g/psdm</code>
|
|}
The PSDM software distribution is supported on the most popular flavors of Linux and is installed at several sites worldwide. Separate installation and maintenance instructions are provided by the PSDM project at [https://confluence.slac.stanford.edu/display/PCDS/PSDM+software+distribution PSDM software distribution]. The package can be installed anywhere on the filesystem, and it is generally possible to install it without superuser privileges.

== Create a test release and an analysis package ==

A test release, which refers to a particular release of the PSDM distribution, is represented by a directory in the file system, often called <code>analysis-rel</code> or <code>myrelease</code>. An analysis package within the test release in turn refers to the files implementing the analysis modules of the package. Details about test releases and analysis packages are covered in [https://confluence.slac.stanford.edu/display/PCDS/Analysis+Workbook.+Quick+Tour Analysis Workbook. Quick Tour]. To prepare the environment for using the PSDM software distribution, bash-users should source <code>ana_env.sh</code>,
$ . /path/to/psdm/etc/ana_env.sh
where <code>/path/to/psdm</code> should be replaced with the path to the installation root directory. csh-users should source <code>ana_env.csh</code> instead,
% source /path/to/psdm/etc/ana_env.csh
Sourcing <code>ana_env.sh</code> or <code>ana_env.csh</code> only modifies the environment of the current shell, and would have to be repeated every time a new shell is started. To make the changes persistent, add
test -r /path/to/psdm/etc/ana_env.sh && . /path/to/psdm/etc/ana_env.sh
to <code>~/.bashrc</code>, or
test -r /path/to/psdm/etc/ana_env.csh && source /path/to/psdm/etc/ana_env.csh
to <code>~/.cshrc</code> as appropriate. This ensures the environment is properly prepared for non-interactive shells, e.g. the shells that are used to run jobs submitted to the cluster. To also be able to use the PSDM software from interactive shells, it is recommended that bash-users put
test -r ~/.bashrc && . ~/.bashrc
near the top of <code>~/.bash_profile</code>. There is no need to edit any additional files for csh-users, because csh always reads <code>~/.cshrc</code>.

A test released based on the most current PSDM distribution is then set up in the current working directory using
$ newrel ana-current myrelease
$ cd myrelease
$ sit_setup
$ newpkg my_ana_pkg
Note that the names for the test release and the analysis package, myrelease and my_ana_pkg above, as well as the location of the myrelease directory are common, but nevertheless arbitrary, choices.

These steps only need to be performed once.

== External links ==

* [https://confluence.slac.stanford.edu/display/PCDS/Pyana+User+Manual Pyana User Manual]
* [https://confluence.slac.stanford.edu/display/PCDS/Analysis+Workbook.+Packages+and+Releases Analysis Workbook. Packages and Releases]

Ha14 installation

2014-02-07T07:22:24Z

Hattne: Typo.

This document aims to be a step-by-step installation guide for ''cctbx.xfel''. Note that in this text <code>$</code> denotes the prompt of any unprivileged user on the system. Certain shells may be configured to use <code>%</code> or <code>></code> for that purpose. The user prompt is distinct from <code>#</code>, which denotes the prompt of the superuser. The prompt should not be included in the command typed to the shell. This installation can be performed by any regular user—no superuser privileges are required.

{{ mbox
| type = notice
| text = Installing ''cctbx.xfel'' is not necessary for most users, as ''cctbx.xfel'' is installed at SLAC already and is generally accessible. General users should instead go to [[setup]] to configure their unix account at SLAC to run ''cctbx.xfel.''
}}

== Prerequisite: install and set up the PSDM suite ==

See [[Set up PSDM software]]. It is assumed that <code>ana_env.sh</code> or <code>ana_env.csh</code> has been sourced, a release directory has been configured in <code>~/myrelease</code> or <code>~/analisys-rel</code>, and that an empty analysis package has been set up in <code>~/myrelease/my_ana_pkg</code> or <code>~/analisys-rel/my_ana_pkg</code>



== Download and extract ''cctbx'' and ''LABELIT'' source bundles ==





Note that developers should follow the instructions in [[Installation for developers]] instead. This step has to be performed on a host with Internet access. Not all hosts at SLAC have that, but the members of e.g. psexport pool do. In what follows, the absolute path <code>/path/to/cctbx/directory</code> should be replaced with something more appropriate. The directory should accessible everywhere ''cctbx.xfel'' is to be run.
$ mkdir -p /path/to/cctbx/directory
$ cd /path/to/cctbx/directory
$ wget http://cci.lbl.gov/cctbx_build/results/2013_08_13_0005/cctbx_bundle.tar.gz
$ wget http://cci.lbl.gov/~hattne/labelit/labelit_bundle_20130814.tar.gz

{{ mbox
| type = speedy
| text = New bundles are automatically prepared nightly. It is, however, not advisable to automatically update from the latest bundle. Please check with the program authors before upgrading a critical installation. Also note that the labelit_bundle cannot currently be downloaded.
}}

Extract the sources:
$ mkdir cctbx_project
$ tar -xpvzf cctbx_bundle.tar.gz -C cctbx_project --strip-components=1
$ tar -xpvzf labelit_bundle_20130814.tar.gz


Add the <code>cxi_user</code> directory and its <code>__init__.py</code> module. This defines the default integration algorithm used by ''cctbx.xfel''.
$ mkdir cxi_user
$ cat > cxi_user/__init__.py << EOF
from xfel.mono_simulation.mono_treatment import post_outlier_rejection
from xfel.mono_simulation.mono_treatment import pre_get_predictions
EOF

== Configure the source tree ==
This next two steps require access to a C++ compiler and a Python interpreter. At SLAC, the members of the psexport pool do not have any compilers installed, but the interactive nodes, reachable via psanacs, do. The Python interpreter used to configure the source tree must be the same interpreter the PSDM suite uses. That interpreter can be located using find, e.g.
$ find $SIT_ROOT/sw/external/python -perm /0111 -type f -wholename "*/$SIT_ARCH/*/python" 2> /dev/null
At SLAC this interpreter is located somewhere under <code>/reg/g/psdm/sw/external/python</code>. Then create and initialize the build directory using the appropriate Python interpreter. Developers may want to use slightly different configuration options, see [[Installation for developers]].
$ mkdir -p /path/to/cctbx_build
$ cd /path/to/cctbx_build
$ python /path/to/cctbx/directory/cctbx_project/libtbx/configure.py xfel
where <code>python</code> should be substituted with output of the previous find command. The name and location of the build directory is arbitrary; <code>/path/to/cctbx/directory/cctbx_build</code> is a common choice.

Now, initialise the running shell using the newly created configuration files. bash-users should do
$ . setpaths.sh
while csh-users should instead run
% source setpaths.csh

== Compile the sources ==
If building at SLAC, log into one of the psanacs nodes. Then compile:
$ make
On SLAC's interactive nodes, this takes just over 6 minutes. To finalize the installation, see [[Setup]].

== External links ==

* [http://cctbx.sourceforge.net/current_cvs/installation.html#manually-building-from-sources-under-unix Manually building from sources under Unix]

Ha14 installation

2014-02-07T07:18:23Z

Hattne: Used mbox for leading note.

This document aims to be a step-by-step installation guide for ''cctbx.xfel''. Note that in this text <code>$</code> denotes the prompt of any unprivileged user on the system. Certain shells may be configured to use <code>%</code> or <code>></code> for that purpose. The user prompt is distinct from <code>#</code>, which denotes the prompt of the superuser. The prompt should not be included in the command typed to the shell. This installation can be performed by any regular user—no superuser privileges are required.

{{ mbox
| type = notice
| text = installing ''cctbx.xfel'' is not necessary for most users, as ''cctbx.xfel'' is installed at SLAC already and is generally accessible. General users should instead go to [[setup]] to configure their unix account at SLAC to run ''cctbx.xfel.''
}}

== Prerequisite: install and set up the PSDM suite ==

See [[Set up PSDM software]]. It is assumed that <code>ana_env.sh</code> or <code>ana_env.csh</code> has been sourced, a release directory has been configured in <code>~/myrelease</code> or <code>~/analisys-rel</code>, and that an empty analysis package has been set up in <code>~/myrelease/my_ana_pkg</code> or <code>~/analisys-rel/my_ana_pkg</code>



== Download and extract ''cctbx'' and ''LABELIT'' source bundles ==





Note that developers should follow the instructions in [[Installation for developers]] instead. This step has to be performed on a host with Internet access. Not all hosts at SLAC have that, but the members of e.g. psexport pool do. In what follows, the absolute path <code>/path/to/cctbx/directory</code> should be replaced with something more appropriate. The directory should accessible everywhere ''cctbx.xfel'' is to be run.
$ mkdir -p /path/to/cctbx/directory
$ cd /path/to/cctbx/directory
$ wget http://cci.lbl.gov/cctbx_build/results/2013_08_13_0005/cctbx_bundle.tar.gz
$ wget http://cci.lbl.gov/~hattne/labelit/labelit_bundle_20130814.tar.gz

{{ mbox
| type = speedy
| text = New bundles are automatically prepared nightly. It is, however, not advisable to automatically update from the latest bundle. Please check with the program authors before upgrading a critical installation. Also note that the labelit_bundle cannot currently be downloaded.
}}

Extract the sources:
$ mkdir cctbx_project
$ tar -xpvzf cctbx_bundle.tar.gz -C cctbx_project --strip-components=1
$ tar -xpvzf labelit_bundle_20130814.tar.gz


Add the <code>cxi_user</code> directory and its <code>__init__.py</code> module. This defines the default integration algorithm used by ''cctbx.xfel''.
$ mkdir cxi_user
$ cat > cxi_user/__init__.py << EOF
from xfel.mono_simulation.mono_treatment import post_outlier_rejection
from xfel.mono_simulation.mono_treatment import pre_get_predictions
EOF

== Configure the source tree ==
This next two steps require access to a C++ compiler and a Python interpreter. At SLAC, the members of the psexport pool do not have any compilers installed, but the interactive nodes, reachable via psanacs, do. The Python interpreter used to configure the source tree must be the same interpreter the PSDM suite uses. That interpreter can be located using find, e.g.
$ find $SIT_ROOT/sw/external/python -perm /0111 -type f -wholename "*/$SIT_ARCH/*/python" 2> /dev/null
At SLAC this interpreter is located somewhere under <code>/reg/g/psdm/sw/external/python</code>. Then create and initialize the build directory using the appropriate Python interpreter. Developers may want to use slightly different configuration options, see [[Installation for developers]].
$ mkdir -p /path/to/cctbx_build
$ cd /path/to/cctbx_build
$ python /path/to/cctbx/directory/cctbx_project/libtbx/configure.py xfel
where <code>python</code> should be substituted with output of the previous find command. The name and location of the build directory is arbitrary; <code>/path/to/cctbx/directory/cctbx_build</code> is a common choice.

Now, initialise the running shell using the newly created configuration files. bash-users should do
$ . setpaths.sh
while csh-users should instead run
% source setpaths.csh

== Compile the sources ==
If building at SLAC, log into one of the psanacs nodes. Then compile:
$ make
On SLAC's interactive nodes, this takes just over 6 minutes. To finalize the installation, see [[Setup]].

== External links ==

* [http://cctbx.sourceforge.net/current_cvs/installation.html#manually-building-from-sources-under-unix Manually building from sources under Unix]

Ha14 installation

2014-02-07T07:16:40Z

Hattne: Used mbox for the warning.

IMPORTANT: installing ''cctbx.xfel'' is not necessary for most users, as ''cctbx.xfel'' is installed at SLAC already and is generally accessible. General users should instead go to [[setup]] to configure their unix account at SLAC to run ''cctbx.xfel.''

This document aims to be a step-by-step installation guide for ''cctbx.xfel''. Note that in this text <code>$</code> denotes the prompt of any unprivileged user on the system. Certain shells may be configured to use <code>%</code> or <code>></code> for that purpose. The user prompt is distinct from <code>#</code>, which denotes the prompt of the superuser. The prompt should not be included in the command typed to the shell. This installation can be performed by any regular user—no superuser privileges are required.

== Prerequisite: install and set up the PSDM suite ==

See [[Set up PSDM software]]. It is assumed that <code>ana_env.sh</code> or <code>ana_env.csh</code> has been sourced, a release directory has been configured in <code>~/myrelease</code> or <code>~/analisys-rel</code>, and that an empty analysis package has been set up in <code>~/myrelease/my_ana_pkg</code> or <code>~/analisys-rel/my_ana_pkg</code>



== Download and extract ''cctbx'' and ''LABELIT'' source bundles ==





Note that developers should follow the instructions in [[Installation for developers]] instead. This step has to be performed on a host with Internet access. Not all hosts at SLAC have that, but the members of e.g. psexport pool do. In what follows, the absolute path <code>/path/to/cctbx/directory</code> should be replaced with something more appropriate. The directory should accessible everywhere ''cctbx.xfel'' is to be run.
$ mkdir -p /path/to/cctbx/directory
$ cd /path/to/cctbx/directory
$ wget http://cci.lbl.gov/cctbx_build/results/2013_08_13_0005/cctbx_bundle.tar.gz
$ wget http://cci.lbl.gov/~hattne/labelit/labelit_bundle_20130814.tar.gz

{{ mbox
| type = speedy
| text = New bundles are automatically prepared nightly. It is, however, not advisable to automatically update from the latest bundle. Please check with the program authors before upgrading a critical installation. Also note that the labelit_bundle cannot currently be downloaded.
}}

Extract the sources:
$ mkdir cctbx_project
$ tar -xpvzf cctbx_bundle.tar.gz -C cctbx_project --strip-components=1
$ tar -xpvzf labelit_bundle_20130814.tar.gz


Add the <code>cxi_user</code> directory and its <code>__init__.py</code> module. This defines the default integration algorithm used by ''cctbx.xfel''.
$ mkdir cxi_user
$ cat > cxi_user/__init__.py << EOF
from xfel.mono_simulation.mono_treatment import post_outlier_rejection
from xfel.mono_simulation.mono_treatment import pre_get_predictions
EOF

== Configure the source tree ==
This next two steps require access to a C++ compiler and a Python interpreter. At SLAC, the members of the psexport pool do not have any compilers installed, but the interactive nodes, reachable via psanacs, do. The Python interpreter used to configure the source tree must be the same interpreter the PSDM suite uses. That interpreter can be located using find, e.g.
$ find $SIT_ROOT/sw/external/python -perm /0111 -type f -wholename "*/$SIT_ARCH/*/python" 2> /dev/null
At SLAC this interpreter is located somewhere under <code>/reg/g/psdm/sw/external/python</code>. Then create and initialize the build directory using the appropriate Python interpreter. Developers may want to use slightly different configuration options, see [[Installation for developers]].
$ mkdir -p /path/to/cctbx_build
$ cd /path/to/cctbx_build
$ python /path/to/cctbx/directory/cctbx_project/libtbx/configure.py xfel
where <code>python</code> should be substituted with output of the previous find command. The name and location of the build directory is arbitrary; <code>/path/to/cctbx/directory/cctbx_build</code> is a common choice.

Now, initialise the running shell using the newly created configuration files. bash-users should do
$ . setpaths.sh
while csh-users should instead run
% source setpaths.csh

== Compile the sources ==
If building at SLAC, log into one of the psanacs nodes. Then compile:
$ make
On SLAC's interactive nodes, this takes just over 6 minutes. To finalize the installation, see [[Setup]].

== External links ==

* [http://cctbx.sourceforge.net/current_cvs/installation.html#manually-building-from-sources-under-unix Manually building from sources under Unix]

File:Ambox speedy deletion.png

2014-02-07T07:12:43Z

Hattne: http://commons.wikimedia.org/wiki/File:Ambox_speedy_deletion.png

http://commons.wikimedia.org/wiki/File:Ambox_speedy_deletion.png

File:Ambox notice.png

2014-02-07T07:10:49Z

Hattne: http://en.wikipedia.org/wiki/File:Ambox_notice.png

http://en.wikipedia.org/wiki/File:Ambox_notice.png

File:Imbox notice.png

2014-02-07T02:12:32Z

Hattne: http://commons.wikimedia.org/wiki/File:Imbox_notice.png

http://commons.wikimedia.org/wiki/File:Imbox_notice.png

File:Lua-logo-nolabel.svg

2014-02-07T02:10:39Z

Hattne: http://commons.wikimedia.org/wiki/File:Lua-logo-nolabel.svg

http://commons.wikimedia.org/wiki/File:Lua-logo-nolabel.svg

File:Edit-copy green.svg

2014-02-07T01:44:07Z

Hattne: http://commons.wikimedia.org/wiki/File:Edit-copy_green.svg

http://commons.wikimedia.org/wiki/File:Edit-copy_green.svg

File:Imbox content.png

2014-02-06T21:46:34Z

Hattne: http://commons.wikimedia.org/wiki/File:Imbox_content.png

http://commons.wikimedia.org/wiki/File:Imbox_content.png

File:Template-info.png

2014-02-06T20:48:40Z

Hattne: From http://commons.wikimedia.org/wiki/File:Template-info.png

From http://commons.wikimedia.org/wiki/File:Template-info.png

Set up ssh-agent

2014-02-05T21:41:20Z

Hattne: Nitpicks.

Checking out or updating the ''PHENIX'' sources will establish many ssh-connections to several servers hosting the source repositories. While not necessary for a successful installation, having an ssh-agent running on the machine used to access the repositories will save much error-prone password-typing. The idea is to provide the remote server with the public half of a cryptographic key pair, which enables it to encrypt a randomly generated challenge and send it to the ssh-agent. If the agent has acquired the corresponding private key it can respond with the decrypted challenge, thereby proving to the remote server that it has access to the decrypted private key. The ssh-agent has authenticated with the server without any user intervention.

== Creating a cryptographic key pair ==

The first step is to generate a complementary pair of cryptographic keys. This step can be skipped if a key pair already exists—it is perfectly safe to use the same key pair to authenticate with any number of remote systems. One of the keys is public and the other is private. The public component of the key pair can be freely distributed without compromising security; indeed, any system one wishes to authenticate with will need access to the public key. The private component must be kept secret. In addition, the private key can be encrypted using a passphrase without which it cannot be used. It is possible to generate a key pair without a passphrase, but then anybody who gains access to the key also acquires access to all systems the key unlocks.

In the example below, an RSA key pair is generated in the default location, /home/user/.ssh/id_rsa and /home/user/.ssh/id_rsa.pub. The passphrase should be complex—when using ssh-agent one will not need to type it very often.
$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/user/.ssh/id_rsa): [PRESS RETURN]
Enter passphrase (empty for no passphrase): [ENTER A COMPLEX PASSPHRASE AND PRESS RETURN]
Enter same passphrase again: [REPEAT COMPLEX PASSPHRASE AND PRESS RETURN]
Your identification has been saved in /home/user/.ssh/id_rsa.
Your public key has been saved in /home/user/.ssh/id_rsa.pub.
The key fingerprint is:
ba:f2:7a:78:aa:b0:33:1d:53:de:63:01:62:15:d6:c9 user@host
Then, start the ssh-agent, and add the decrypted private key of the newly generated key pair to the ssh-agent's cache.
$ eval `ssh-agent`
$ ssh-add
Enter passphrase for /home/user/.ssh/id_rsa: [ENTER COMPLEX PASSPHRASE AND PRESS RETURN]
Identity added: /home/user/.ssh/id_rsa (/home/user/.ssh/id_rsa)
New ssh-processes will now act as clients to the ssh-agent, and attempt to use the keys it holds to authenticate with the remote system.

== Distribute the public key to the remote system ==

The remote system needs access to the public component of the key pair to generate the authenticating challenge for the ssh-agent. This means that the public component of the key pair has to be present in <code>~/.ssh/authorized_keys</code> on the remote system.
$ cat ~/.ssh/id_rsa.pub | ssh user@remote.host "cat - >> ~/.ssh/authorized_keys"
This will append the public key of the recently created key pair to <code>~/.ssh/authorized_keys</code> on the remote system. It is now possible to access remote.host without typing a password on the command line.

=== Distribute the public key to SourceForge ===

The servers at SourceForge can be made aware of the public key, too. To do so, access the account from the web interface and navigate to [http://sourceforge.net/account/services SSH Settings], and choose [https://sourceforge.net/account/ssh Edit SSH Keys for Shell/CVS]. The public key can now be pasted into the text box. Note that it may take several minutes for the change to take effect.

== Using ''keychain'' ==

An ssh-agent started from command line only affects ssh-connections started from the same shell, and separate ssh-agents would be required for each new login shell. The private key would have to be decrypted from every login shell separately. Furthermore, ssh-agents are not necessarily terminated when the login shell exits, potentially cluttering the system with orphaned ssh-agents. ''keychain'' is a shell script that was written to avoid this situation: it probes the system for any running ssh-agents, and attempts to use a running agent instead of starting a new daemon. If no ssh-agents are running, ''keychain'' starts a new process and uses ssh-add to add the specified private keys.

At CCI and SLAC ''keychain'' is installed at <code>~hattne/gentoo/usr/bin/keychain</code>. At NERSC it is at <code>/global/project/projectdirs/lcls/gentoo/usr/bin/keychain</code>. It is recommended to run ''keychain'' from the shell configuration files for interactive, login shells. To do so, bash-users can add the following lines to their <code>~/.bash_profile</code>:
test -x /path/to/keychain && \
eval `/path/to/keychain --agents ssh --eval id_rsa --inherit any-once --stop others`
csh-users would modify <code>~/.login</code> instead.

== Using agent forwarding ==

It is not necessary to start ssh-agents on every system logged in to. If an ssh-agent is running on the host used to access a remote system, a connection to the agent can be forwarded using the ssh’s <code>-A</code> option. Any ssh-processes on the remote system will now query the ssh-agent on the host logged in from. ''keychain'' will recognize that agent-forwarding is in effect and will not start a new agent on the remote system.

== External links ==

* [http://www.openbsd.org/cgi-bin/man.cgi?query=ssh-agent OpenBSD manual pages: ssh-agent]
* [http://www.openbsd.org/cgi-bin/man.cgi?query=ssh-keygen OpenBSD manual pages: ssh-keygen]
* [http://www.openbsd.org/cgi-bin/man.cgi?query=ssh-add OpenBSD manual pages: ssh-add]
* [https://sourceforge.net/apps/trac/sourceforge/wiki/SSH%20keys SSH keys - SourceForge]
* [http://www.funtoo.org/Keychain Keychain—Funtoo]

Ha14 installation

2014-02-05T21:37:03Z

Hattne: Point to finalizing the installation using instructions in Setup.

IMPORTANT: installing ''cctbx.xfel'' is not necessary for most users, as ''cctbx.xfel'' is installed at SLAC already and is generally accessible. General users should instead go to [[setup]] to configure their unix account at SLAC to run ''cctbx.xfel.''

This document aims to be a step-by-step installation guide for ''cctbx.xfel''. Note that in this text <code>$</code> denotes the prompt of any unprivileged user on the system. Certain shells may be configured to use <code>%</code> or <code>></code> for that purpose. The user prompt is distinct from <code>#</code>, which denotes the prompt of the superuser. The prompt should not be included in the command typed to the shell. This installation can be performed by any regular user—no superuser privileges are required.

== Prerequisite: install and set up the PSDM suite ==

See [[Set up PSDM software]]. It is assumed that <code>ana_env.sh</code> or <code>ana_env.csh</code> has been sourced, a release directory has been configured in <code>~/myrelease</code> or <code>~/analisys-rel</code>, and that an empty analysis package has been set up in <code>~/myrelease/my_ana_pkg</code> or <code>~/analisys-rel/my_ana_pkg</code>



== Download and extract ''cctbx'' and ''LABELIT'' source bundles ==





Note that developers should follow the instructions in [[Installation for developers]] instead. This step has to be performed on a host with Internet access. Not all hosts at SLAC have that, but the members of e.g. psexport pool do. In what follows, the absolute path <code>/path/to/cctbx/directory</code> should be replaced with something more appropriate. The directory should accessible everywhere ''cctbx.xfel'' is to be run.
$ mkdir -p /path/to/cctbx/directory
$ cd /path/to/cctbx/directory
$ wget http://cci.lbl.gov/cctbx_build/results/2013_08_13_0005/cctbx_bundle.tar.gz
$ wget http://cci.lbl.gov/~hattne/labelit/labelit_bundle_20130814.tar.gz
Note: new bundles are automatically prepared nightly. It is, however, not advisable to automatically update from the latest bundle. Please check with the program authors before upgrading a critical installation. 

Extract the sources:
$ mkdir cctbx_project
$ tar -xpvzf cctbx_bundle.tar.gz -C cctbx_project --strip-components=1
$ tar -xpvzf labelit_bundle_20130814.tar.gz


Add the <code>cxi_user</code> directory and its <code>__init__.py</code> module. This defines the default integration algorithm used by ''cctbx.xfel''.
$ mkdir cxi_user
$ cat > cxi_user/__init__.py << EOF
from xfel.mono_simulation.mono_treatment import post_outlier_rejection
from xfel.mono_simulation.mono_treatment import pre_get_predictions
EOF

== Configure the source tree ==
This next two steps require access to a C++ compiler and a Python interpreter. At SLAC, the members of the psexport pool do not have any compilers installed, but the interactive nodes, reachable via psanacs, do. The Python interpreter used to configure the source tree must be the same interpreter the PSDM suite uses. That interpreter can be located using find, e.g.
$ find $SIT_ROOT/sw/external/python -perm /0111 -type f -wholename "*/$SIT_ARCH/*/python" 2> /dev/null
At SLAC this interpreter is located somewhere under <code>/reg/g/psdm/sw/external/python</code>. Then create and initialize the build directory using the appropriate Python interpreter. Developers may want to use slightly different configuration options, see [[Installation for developers]].
$ mkdir -p /path/to/cctbx_build
$ cd /path/to/cctbx_build
$ python /path/to/cctbx/directory/cctbx_project/libtbx/configure.py xfel
where <code>python</code> should be substituted with output of the previous find command. The name and location of the build directory is arbitrary; <code>/path/to/cctbx/directory/cctbx_build</code> is a common choice.

Now, initialise the running shell using the newly created configuration files. bash-users should do
$ . setpaths.sh
while csh-users should instead run
% source setpaths.csh

== Compile the sources ==
If building at SLAC, log into one of the psanacs nodes. Then compile:
$ make
On SLAC's interactive nodes, this takes just over 6 minutes. To finalize the installation, see [[Setup]].

== External links ==

* [http://cctbx.sourceforge.net/current_cvs/installation.html#manually-building-from-sources-under-unix Manually building from sources under Unix]

Set up PSDM software

2014-02-05T21:20:59Z

Hattne: Comments, thanks to mamin@.

== Prerequisites ==



The PSDM software distribution must be installed. PSDM provides installation and maintenance instructions at [https://confluence.slac.stanford.edu/display/PCDS/PSDM+software+distribution PSDM software distribution]. The distribution can be installed anywhere on the filesystem; it is generally possible to install the PSDM software distribution without superuser privileges.

The installation root directory affects the absolute paths used in these instructions and differs from site to site. Some common installation root directories are listed below.
{| class="wikitable sortable"
|-
! Site
!class="unsortable"| Installation root directory
!class="unsortable"| Notes
|-
| BNL
| <code>/usr/local/crys/psdm</code>
|
|-
| CCI, LBNL
| <code>/net/viper/raid1/psdm</code>
| Only available on viper.lbl.gov
|-
| SLAC
| <code>/reg/g/psdm</code>
|
|}

== Create a test release and an analysis package ==

In what follows it is assumed that the PSDM software distribution is installed at <code>/reg/g/psdm</code>. Substitute <code>/reg/g/psdm</code> with the appropriate path in the instructions below. These steps only need to be performed once.

A ''test release'', which refers to a particular release of the LCLS packages, is represented by a directory in the file system, often called <code>analysis-rel</code> or <code>myrelease</code>. An analysis package within the test release in turn refers to the files implementing the analysis modules of the package. Details about installing of a test release and analysis packages are covered in [https://confluence.slac.stanford.edu/display/PCDS/Analysis+Workbook.+Quick+Tour Analysis Workbook. Quick Tour]. In a bash shell, the environment is prepared for PSDM using
$ . /reg/g/psdm/etc/ana_env.sh
For csh, the corresponding command is
% source /reg/g/psdm/etc/ana_env.csh
Sourcing <code>ana_env.sh</code> or <code>ana_env.csh</code> only modifies the environment of the current shell, and would have to be repeated every time a new shell is started. To make the changes persistent, add
test -r /reg/g/psdm/etc/ana_env.sh && . /reg/g/psdm/etc/ana_env.sh
to <code>~/.bashrc</code>, or
test -r /reg/g/psdm/etc/ana_env.csh && source /reg/g/psdm/etc/ana_env.csh
to <code>~/.cshrc</code> as appropriate.

This ensures the environment is properly prepared for non-interactive shells, e.g. the shells that are used to run jobs submitted to the cluster. To also be able to use the PSDM software from interactive shells, it is recommended that bash-users put
test -r "${HOME}/.bashrc" && . "${HOME}/.bashrc"
near the top of <code>~/.bash_profile</code>. There is no need to edit any additional files for csh-users, because csh always reads <code>~/.cshrc</code>.

A test released based on the most current LCLS software is then set up in the current working directory using
$ newrel ana-current myrelease
$ cd myrelease
$ sit_setup
$ newpkg my_ana_pkg
Note that the names for the test release and the analysis package, myrelease and my_ana_pkg, as well as the location of the myrelease directory are common, but nevertheless arbitrary, choices.

== Keeping up to date with PSDM releases==



Since the ''cctbx.xfel'' modules do not critically depend on any particular version of the PSDM software, it is generally possible to closely track the latest version of the PSDM distribution. To upgrade to the current PSDM release, do
$ cd myrelease
$ relupgrade ana-current
If <code>relinfo</code> does not output <code>ana-current</code> after that, one can edit <code>~/myrelease/.sit_release</code> manually to that effect. Then rebuild
$ scons -c
$ scons

== External links ==

* More detailed manual: [https://confluence.slac.stanford.edu/display/PCDS/Pyana+User+Manual Pyana User Manual]
* [https://confluence.slac.stanford.edu/display/PCDS/Analysis+Workbook.+Packages+and+Releases Analysis Workbook. Packages and Releases]

Ha14 installation

2014-02-05T20:37:28Z

Hattne: Better find command

IMPORTANT: installing ''cctbx.xfel'' is not necessary for most users, as ''cctbx.xfel'' is installed at SLAC already and is generally accessible. General users should instead go to [[setup]] to configure their unix account at SLAC to run ''cctbx.xfel.''

This document aims to be a step-by-step installation guide for ''cctbx.xfel''. Note that in this text <code>$</code> denotes the prompt of any unprivileged user on the system. Certain shells may be configured to use <code>%</code> or <code>></code> for that purpose. The user prompt is distinct from <code>#</code>, which denotes the prompt of the superuser. The prompt should not be included in the command typed to the shell. This installation can be performed by any regular user—no superuser privileges are required.

== Prerequisite: install and set up the PSDM suite ==

See [[Set up PSDM software]]. It is assumed that <code>ana_env.sh</code> or <code>ana_env.csh</code> has been sourced, a release directory has been configured in <code>~/myrelease</code> or <code>~/analisys-rel</code>, and that an empty analysis package has been set up in <code>~/myrelease/my_ana_pkg</code> or <code>~/analisys-rel/my_ana_pkg</code>



== Download and extract ''cctbx'' and ''LABELIT'' source bundles ==





Note that developers should follow the instructions in [[Installation for developers]] instead. This step has to be performed on a host with Internet access. Not all hosts at SLAC have that, but the members of e.g. psexport pool do. In what follows, the absolute path <code>/path/to/cctbx/directory</code> should be replaced with something more appropriate. The directory should accessible everywhere ''cctbx.xfel'' is to be run.
$ mkdir -p /path/to/cctbx/directory
$ cd /path/to/cctbx/directory
$ wget http://cci.lbl.gov/cctbx_build/results/2013_08_13_0005/cctbx_bundle.tar.gz
$ wget http://cci.lbl.gov/~hattne/labelit/labelit_bundle_20130814.tar.gz
Note: new bundles are automatically prepared nightly. It is, however, not advisable to automatically update from the latest bundle. Please check with the program authors before upgrading a critical installation. 

Extract the sources:
$ mkdir cctbx_project
$ tar -xpvzf cctbx_bundle.tar.gz -C cctbx_project --strip-components=1
$ tar -xpvzf labelit_bundle_20130814.tar.gz


Add the <code>cxi_user</code> directory and its <code>__init__.py</code> module. This defines the default integration algorithm used by ''cctbx.xfel''.
$ mkdir cxi_user
$ cat > cxi_user/__init__.py << EOF
from xfel.mono_simulation.mono_treatment import post_outlier_rejection
from xfel.mono_simulation.mono_treatment import pre_get_predictions
EOF

== Configure the source tree ==
This next two steps require access to a C++ compiler and a Python interpreter. At SLAC, the members of the psexport pool do not have any compilers installed, but the interactive nodes, reachable via psanacs, do. The Python interpreter used to configure the source tree must be the same interpreter the PSDM suite uses. That interpreter can be located using find, e.g.
$ find $SIT_ROOT/sw/external/python -perm /0111 -type f -wholename "*/$SIT_ARCH/*/python" 2> /dev/null
At SLAC this interpreter is located somewhere under <code>/reg/g/psdm/sw/external/python</code>. Then create and initialize the build directory using the appropriate Python interpreter. Developers may want to use slightly different configuration options, see [[Installation for developers]].
$ mkdir -p /path/to/cctbx_build
$ cd /path/to/cctbx_build
$ python /path/to/cctbx/directory/cctbx_project/libtbx/configure.py xfel
where <code>python</code> should be substituted with output of the previous find command. The name and location of the build directory is arbitrary; <code>/path/to/cctbx/directory/cctbx_build</code> is a common choice.

Now, initialise the running shell using the newly created configuration files. bash-users should do
$ . setpaths.sh
while csh-users should instead run
% source setpaths.csh

== Compile the sources ==
If building at SLAC, log into one of the psanacs nodes. Then compile:
$ make
On SLAC's interactive nodes, this takes just over 6 minutes.

== External links ==

* [http://cctbx.sourceforge.net/current_cvs/installation.html#manually-building-from-sources-under-unix Manually building from sources under Unix]

Ha14 installation

2014-02-05T20:20:10Z

Hattne: Fix broken find command

IMPORTANT: installing ''cctbx.xfel'' is not necessary for most users, as ''cctbx.xfel'' is installed at SLAC already and is generally accessible. General users should instead go to [[setup]] to configure their unix account at SLAC to run ''cctbx.xfel.''

This document aims to be a step-by-step installation guide for ''cctbx.xfel''. Note that in this text <code>$</code> denotes the prompt of any unprivileged user on the system. Certain shells may be configured to use <code>%</code> or <code>></code> for that purpose. The user prompt is distinct from <code>#</code>, which denotes the prompt of the superuser. The prompt should not be included in the command typed to the shell. This installation can be performed by any regular user—no superuser privileges are required.

== Prerequisite: install and set up the PSDM suite ==

See [[Set up PSDM software]]. It is assumed that <code>ana_env.sh</code> or <code>ana_env.csh</code> has been sourced, a release directory has been configured in <code>~/myrelease</code> or <code>~/analisys-rel</code>, and that an empty analysis package has been set up in <code>~/myrelease/my_ana_pkg</code> or <code>~/analisys-rel/my_ana_pkg</code>



== Download and extract ''cctbx'' and ''LABELIT'' source bundles ==





Note that developers should follow the instructions in [[Installation for developers]] instead. This step has to be performed on a host with Internet access. Not all hosts at SLAC have that, but the members of e.g. psexport pool do. In what follows, the absolute path <code>/path/to/cctbx/directory</code> should be replaced with something more appropriate. The directory should accessible everywhere ''cctbx.xfel'' is to be run.
$ mkdir -p /path/to/cctbx/directory
$ cd /path/to/cctbx/directory
$ wget http://cci.lbl.gov/cctbx_build/results/2013_08_13_0005/cctbx_bundle.tar.gz
$ wget http://cci.lbl.gov/~hattne/labelit/labelit_bundle_20130814.tar.gz
Note: new bundles are automatically prepared nightly. It is, however, not advisable to automatically update from the latest bundle. Please check with the program authors before upgrading a critical installation. 

Extract the sources:
$ mkdir cctbx_project
$ tar -xpvzf cctbx_bundle.tar.gz -C cctbx_project --strip-components=1
$ tar -xpvzf labelit_bundle_20130814.tar.gz


Add the <code>cxi_user</code> directory and its <code>__init__.py</code> module. This defines the default integration algorithm used by ''cctbx.xfel''.
$ mkdir cxi_user
$ cat > cxi_user/__init__.py << EOF
from xfel.mono_simulation.mono_treatment import post_outlier_rejection
from xfel.mono_simulation.mono_treatment import pre_get_predictions
EOF

== Configure the source tree ==
This next two steps require access to a C++ compiler and a Python interpreter. At SLAC, the members of the psexport pool do not have any compilers installed, but the interactive nodes, reachable via psanacs, do. The Python interpreter used to configure the source tree must be the same interpreter the PSDM suite uses. That interpreter can be located using find, e.g.
$ find $SIT_ROOT/sw/external/python -perm /0111 -name python -type f 2> /dev/null | grep $SIT_ARCH
At SLAC this interpreter is located somewhere under <code>/reg/g/psdm/sw/external/python</code>. Then create and initialize the build directory using the appropriate Python interpreter. Developers may want to use slightly different configuration options, see [[Installation for developers]].
$ mkdir -p /path/to/cctbx_build
$ cd /path/to/cctbx_build
$ python /path/to/cctbx/directory/cctbx_project/libtbx/configure.py xfel
where <code>python</code> should be substituted with output of the previous find command. The name and location of the build directory is arbitrary; <code>/path/to/cctbx/directory/cctbx_build</code> is a common choice.

Now, initialise the running shell using the newly created configuration files. bash-users should do
$ . setpaths.sh
while csh-users should instead run
% source setpaths.csh

== Compile the sources ==
If building at SLAC, log into one of the psanacs nodes. Then compile:
$ make
On SLAC's interactive nodes, this takes just over 6 minutes.

== External links ==

* [http://cctbx.sourceforge.net/current_cvs/installation.html#manually-building-from-sources-under-unix Manually building from sources under Unix]

Ha14 installation

2014-02-05T20:02:18Z

Hattne: Consistency with developer instructions.

IMPORTANT: installing ''cctbx.xfel'' is not necessary for most users, as ''cctbx.xfel'' is installed at SLAC already and is generally accessible. General users should instead go to [[setup]] to configure their unix account at SLAC to run ''cctbx.xfel.''

This document aims to be a step-by-step installation guide for ''cctbx.xfel''. Note that in this text <code>$</code> denotes the prompt of any unprivileged user on the system. Certain shells may be configured to use <code>%</code> or <code>></code> for that purpose. The user prompt is distinct from <code>#</code>, which denotes the prompt of the superuser. The prompt should not be included in the command typed to the shell. This installation can be performed by any regular user—no superuser privileges are required.

== Prerequisite: install and set up the PSDM suite ==

See [[Set up PSDM software]]. It is assumed that <code>ana_env.sh</code> or <code>ana_env.csh</code> has been sourced, a release directory has been configured in <code>~/myrelease</code> or <code>~/analisys-rel</code>, and that an empty analysis package has been set up in <code>~/myrelease/my_ana_pkg</code> or <code>~/analisys-rel/my_ana_pkg</code>



== Download and extract ''cctbx'' and ''LABELIT'' source bundles ==





Note that developers should follow the instructions in [[Installation for developers]] instead. This step has to be performed on a host with Internet access. Not all hosts at SLAC have that, but the members of e.g. psexport pool do. In what follows, the absolute path <code>/path/to/cctbx/directory</code> should be replaced with something more appropriate. The directory should accessible everywhere ''cctbx.xfel'' is to be run.
$ mkdir -p /path/to/cctbx/directory
$ cd /path/to/cctbx/directory
$ wget http://cci.lbl.gov/cctbx_build/results/2013_08_13_0005/cctbx_bundle.tar.gz
$ wget http://cci.lbl.gov/~hattne/labelit/labelit_bundle_20130814.tar.gz
Note: new bundles are automatically prepared nightly. It is, however, not advisable to automatically update from the latest bundle. Please check with the program authors before upgrading a critical installation. 

Extract the sources:
$ mkdir cctbx_project
$ tar -xpvzf cctbx_bundle.tar.gz -C cctbx_project --strip-components=1
$ tar -xpvzf labelit_bundle_20130814.tar.gz


Add the <code>cxi_user</code> directory and its <code>__init__.py</code> module. This defines the default integration algorithm used by ''cctbx.xfel''.
$ mkdir cxi_user
$ cat > cxi_user/__init__.py << EOF
from xfel.mono_simulation.mono_treatment import post_outlier_rejection
from xfel.mono_simulation.mono_treatment import pre_get_predictions
EOF

== Configure the source tree ==
This next two steps require access to a C++ compiler and a Python interpreter. At SLAC, the members of the psexport pool do not have any compilers installed, but the interactive nodes, reachable via psanacs, do. The Python interpreter used to configure the source tree must be the same interpreter the PSDM suite uses. That interpreter can be located using find, e.g.
$ find $SIT_ROOT -executable -name python -type f
At SLAC this interpreter is located somewhere under <code>/reg/g/psdm/sw/external/python</code>. Then create and initialize the build directory using the appropriate Python interpreter. Developers may want to use slightly different configuration options, see [[Installation for developers]].
$ mkdir -p /path/to/cctbx_build
$ cd /path/to/cctbx_build
$ python /path/to/cctbx/directory/cctbx_project/libtbx/configure.py xfel
where <code>python</code> should be substituted with output of the previous find command. The name and location of the build directory is arbitrary; <code>/path/to/cctbx/directory/cctbx_build</code> is a common choice.

Now, initialise the running shell using the newly created configuration files. bash-users should do
$ . setpaths.sh
while csh-users should instead run
% source setpaths.csh

== Compile the sources ==
$ make
On SLAC's interactive nodes, this takes just over 6 minutes.

== External links ==

* [http://cctbx.sourceforge.net/current_cvs/installation.html#manually-building-from-sources-under-unix Manually building from sources under Unix]

Ha14 installation

2014-02-05T19:33:17Z

Hattne: Consistency with developer instructions.

IMPORTANT: installing ''cctbx.xfel'' is not necessary for most users, as ''cctbx.xfel'' is installed at SLAC already and is generally accessible. General users should instead go to [[setup]] to configure their unix account at SLAC to run ''cctbx.xfel.''

This document aims to be a step-by-step installation guide for ''cctbx.xfel''. Note that in this text <code>$</code> denotes the prompt of any unprivileged user on the system. Certain shells may be configured to use <code>%</code> or <code>></code> for that purpose. The user prompt is distinct from <code>#</code>, which denotes the prompt of the superuser. The prompt should not be included in the command typed to the shell. This installation can be performed by any regular user—no superuser privileges are required.

== Prerequisite: install and set up the PSDM suite ==

See [[Set up PSDM software]]. It is assumed that <code>ana_env.sh</code> or <code>ana_env.csh</code> has been sourced, a release directory has been configured in <code>~/myrelease</code> or <code>~/analisys-rel</code>, and that an empty analysis package has been set up in <code>~/myrelease/my_ana_pkg</code> or <code>~/analisys-rel/my_ana_pkg</code>



== Download and extract ''cctbx'' and ''LABELIT'' source bundles ==





Note that developers should follow the instructions in [[Installation for developers]] instead. This step has to be performed on a host with Internet access. Not all hosts at SLAC have that, but the members of e.g. psexport pool do. In what follows, the absolute path <code>/path/to/cctbx/directory</code> should be replaced with something more appropriate. The directory should accessible everywhere ''cctbx.xfel'' is to be run.
$ mkdir -p /path/to/cctbx/directory
$ cd /path/to/cctbx/directory
$ wget http://cci.lbl.gov/cctbx_build/results/2013_08_13_0005/cctbx_bundle.tar.gz
$ wget http://cci.lbl.gov/~hattne/labelit/labelit_bundle_20130814.tar.gz
Note: new bundles are automatically prepared nightly. It is, however, not advisable to automatically update from the latest bundle. Please check with the program authors before upgrading a critical installation. 

Extract the sources:
$ mkdir cctbx_project
$ tar -xpvzf cctbx_bundle.tar.gz -C cctbx_project --strip-components=1
$ tar -xpvzf labelit_bundle_20130814.tar.gz


Add the <code>cxi_user</code> directory and its <code>__init__.py</code> module. This defines the default integration algorithm used by ''cctbx.xfel''.
$ mkdir cxi_user
$ cat > cxi_user/__init__.py << EOF
from xfel.mono_simulation.mono_treatment import post_outlier_rejection
from xfel.mono_simulation.mono_treatment import pre_get_predictions
EOF

== Configure the source tree ==
This next two steps require access to a C++ compiler and a Python interpreter, any minor version between 2.5 and 2.7. At SLAC, the members of the psexport pool do not have any compilers installed, but the interactive nodes, reachable via psanacs, do. The Python interpreter used to configure the source tree must be the same interpreter the PSDM suite uses. That interpreter can be located using find, e.g.
$ find $SIT_ROOT -executable -name python -type f
At SLAC this interpreter is located somewhere under <code>/reg/g/psdm/sw/external/python</code>. Then create and initialize the build directory and compile the sources using the appropriate Python interpreter. Developers may want to use slightly different configuration options, see [[Installation for developers]].
$ mkdir cctbx_build
$ cd cctbx_build
$ python ../cctbx_project/libtbx/configure.py xfel
where <code>python</code> should be substituted with output of the previous find command.

Now, initialise the running shell using the newly created configuration files. bash-users should do
$ . setpaths.sh
while csh-users should instead run
% source setpaths.csh

== Compile the sources ==
$ make
On SLAC's interactive nodes, this takes just over 6 minutes.



== External links ==

* [http://cctbx.sourceforge.net/current_cvs/installation.html#manually-building-from-sources-under-unix Manually building from sources under Unix]

Setup

2014-02-05T18:52:05Z

Hattne: Moved scratch-area setup to tutorial.

== Prerequisities ==

''cctbx.xfel'' must be installed, which in turn requires the PSDM software distribution to be set up. See [[Set up PSDM software]] for the latter and [[Installation]] for the former. ''cctbx.xfel'' can be installed anywhere on the filesystem where write access is granted.

The installation root directory affects the absolute paths used in these instructions and differs from site to site. Some common installation root directories are listed below.
{| class="wikitable sortable"
|-
! Site
!class="unsortable"| Installation root directory
!class="unsortable"| Notes
|-
| BNL
| <code>/usr/local/crys_test/cctbx</code>
|
|-
| CCI, LBNL
|
| No central installation available
|-
| SLAC
| <code>/reg/g/cctbx/sources</code>
|
|}
Developers will generally have ''cctbx.xfel'' installed in their home directory.

== ''cctbx.xfel'' setup ==

To start using ''cctbx.xfel'', the environment has to be configured. The details depend on the ''cctbx.xfel'' installation root directory and the user's shell.
In what follows it is assumed that ''cctbx.xfel'' is installed at <code>/reg/g/cctbx/build</code>. Substitute <code>/reg/g/cctbx/build</code> with the appropriate path in the instructions below. These steps need only be performed once.

In a bash shell, the environment is prepared for ''cctbx.xfel'' using
$ . /reg/g/cctbx/build/setpaths.sh
For csh, the corresponding command is
% source /reg/g/cctbx/build/setpaths.csh
Sourcing <code>setpaths.sh</code> or <code>ana_env.csh</code> only modifies the environment of the current shell, and would have to be repeated every time a new shell is started. To make the changes persistent, add
test -r "/reg/g/cctbx/build/setpaths.sh" && \
. "/reg/g/cctbx/build/setpaths.sh"
to <code>~/.bashrc</code>, or
test -r "/reg/g/cctbx/build/setpaths.csh" && \
source "/reg/g/cctbx/build/setpaths.csh"
to <code>~/.cshrc as appropriate</code>.

== Linking ''cctbx.xfel'' to ''pyana'' ==

Run these commands to associate ''pyana'' and ''cctbx.xfel'':
$ cd ~/myrelease
$ sit_setup
$ cd ~/myrelease/my_ana_pkg
$ ln -fns /reg/g/cctbx/sources/cctbx_project/xfel/cxi/cspad_ana src
$ cd ..
$ ln -s /reg/g/cctbx/sources/cctbx_project/xfel/cxi/lsf.sh .
$ scons
The last step compiles the analysis modules just linked in.



== Run each time on logging on to SLAC ==

Whenever you log on to a fresh shell at SLAC, before you can use any portion of ''cctbx.xfel'' that uses ''pyana'', you must run these commands:
$ cd ~/myrelease
$ sit_setup

== Test the installation ==
At SLAC, test that it is working by e.g.
$ cd ~/myrelease
$ cxi.pyana -c /reg/g/cctbx/tutorials/setup/test.cfg /reg/d/ana11/cxi/data/Mar2013calib/xtc/e236-r0004-s00-c00.xtc
This should start a viewer, displaying dark-subtracted averages from an XTC stream. Press the green run button to see the images. This particular stream has a series of misses at its beginning; hits will appear presently.

More specifically, this command starts ''pyana'' with a configuration file (<code>-c</code>) and an XTC file. For more information, see the [[Overview]] and other tutorial pages.