Overview

Data collection and processing using LCLS involves recording data from an X-ray detector (Jungfrau, Rayonix, CSPAD, etc.) in XTC format, reading the data using DIALS libraries that wrap the LCLS-provided library psana, processing on either the SLAC Shared Science Data Facility (S3DF) or NERSC, and coordinating the processing using cctbx.xfel itself. Since analysis proceeds directly from the raw data, no intermediate conversion is necessary, and it can be done while an experiment is running.

The X-ray detector

Example CSPAD image

The LCLS at full capacity operates at 120 Hz. The incident photon packets are delivered in ≈40 femtosecond wide pulses, each containing ≈10¹² photons. This high repetition rate and compact beam delivery time necessitated the construction of a new detector Hart, P The Cornell-SLAC Pixel Array Detector at LCLS. SLAC Scientific Documents (2012)., where the work of reading out and streaming recorded data at these high speeds is accomplished through the use of 64 sensors, arranged in a quadrangular pattern around a central hole (in the place of a beam stop). Each of the 4 quadrants, containing 16 of the sensors, was adjustable on rails radially away from the central hole to adjust the size of this hole.

The CSPAD was replaced with two detectors: the Rayonix MX340-XFEL (recently retired), and the Jungfrau 16M (in current use).

Indexing, predicting spot locations using a crystal orientation matrix, and integrating reflection intensities requires precise knowledge of the location of the sensors in these detectors in three-dimensional space. For this reason, a portion of this wiki describes the calibration and refinement of the tile metrology.

psana

The LCLS data acquisition systems stream the terabytes of diffraction data collected from the X-ray detector to container files in XTC format. XTC is a linear, sequential-access file format, where individual images can be recorded rapidly by the file system as they are collected. The programmatic interface to interact with these files at LCLS is psana, a C++/Python-based interface.

cctbx.xfel wraps access to XTC using small, custom text files called locators, bypassing the need for users to work with psana directly. For example, the DIALS image viewer can be used to view images using a command like this:

 dials.image_viewer run100.loc load_models=False

Where run100.loc looks like this

 experiment=mfx0000000
 run=100
 detector=rayonix

And the parameter load_models=False makes reading and display the data faster. Further details about the detector or experiment can be provided in the locator.

Note that the cctbx.xfel GUI provides convenient ways to specify the parameters need to the user needs not create these locator files directly.

psana itself is designed with computational parallelization in mind. As each image is independent, processing of each image can be done by separate computer cores. MPI is required, which is configured automatically in LCLS/psana/cctbx builds.

More information about psana: [1]

S3DF

SLAC provides access to the SLAC Shared Science Data Facility (S3DF). There are several pools of computers available:

• s3dflogin: login nodes, not used for processing or computation. Reach these using ssh -YAC <username>@s3dflogin.slac.stanford.edu
• psana: not to be confused with the psana library, these are used for small computation/compilation and for running graphical programs.
• s3dfnx: NoMachine gateway, use the full address s3dfnx.slac.stanford.edu with NoMachine for faster graphical displays.
• Queues: generally users should use the milano queue for routine processing.

Queing is done using slurm. Of note are these commands:

• sbatch: used to submit jobs to the queuing system
• squeue -u <username>: used to list the jobs being run by the current user
• scancel: used to stop a job that is running

Phil

cctbx.xfel provides a GUI but the programs it wraps are driven using Python-based hierarchical interchange language (phil) files, the same format that drives DIALS and PHENIX (though PHENIX calls them .eff files). The format is intuitive and allows easy specification of per-processing run parameters.

cctbx.xfel collects processing parameters using its GUI, then when submitting jobs it writes these configuration settings to a phil file. These settings will include things such as thresholds for determining hits (number of spots on an image, spot brightness cutoff, etc.), unit cell targets for indexing, resolution cutoffs, and so forth.

Technical information regarding phil: [2]

Specific phil files used in this tutorial: phil

cctbx

The computational crystallographic toolbox is a foundational set of python and C++ modules that allow abstraction of the crystallographic experiment. Under continual development, the toolbox provides interfaces for working with crystal models, reflection data, and much more.

Introduction: [3] Homepage: [4]