Merging

The result of Indexing and integration is a set of Python pickle files, each of which essentially contains a table of Miller indices of the observed reflections, their integrated intensities, and estimated errors. In the general case, these files reflect the measurements from single shots, each exposing different crystals with a unique pulse of X-rays. Merging refers to the procedure applied to unite all these observations into a single data set. During merging, a distinct multiplicative factor, which accounts for the variance in pulse intensity and crystal size, is applied to the observations from a single shot to bring all the observations onto a common scale. The intensities for individual reflections are then summed, and their errors are propagated in quadrature. The result of merging is an mtz file suited for further processing, e.g. molecular replacement.

Merging a set of integration files

In cctbx.xfel the per-image scale factors are determined using a scaling reference. This scaling reference is expected to be a previously solved, isomorphous data set. The scale factor is determined by a least-squares fit of the observations to the reference intensities, after applying corrections for polarization<ref>Kahn, R, et al. Macromolecular Crystallography with Synchrotron Radiation: Photographic Data Collection and Polarization Correction. J Appl Cryst 15, 330–337 (1982).</ref>, and a significance filter, which limits the resolution of each diffraction pattern based on the signal-to-noise ratio. Images not conforming to the symmetry of the scaling reference are rejected as outliers, as are images that correlate poorly with the scaling reference and images whose unit cell lies deviates too far from that of the scaling reference.

cxi.merge may take several passes over the integrated images. In order to speed up processing, cxi.merge will write the scaled data to a back end database during each pass. Currently three database back ends are implemented.

• FS is the simplest back end. It stores the scaled intensities, their Miller indices, and information about the shots they were observed during in three flat files on the file system.

• The MySQL back end stores data in a MySQL database. The database must be set up beforehand, and credentials to access it must be supplied in the parameters passed to cxi.merge.

• The SQLite back end uses a simple SQLite database, which is written to a single file on the file system. It is easier to use than the MySQL back end and more efficient than the FS backend. Regrettably, the SQLite back end does not appear to work on the Lustre file system.

Compared to indexing and integration, merging is a relatively quick procedure. However, particularly for large datasets, it may significantly strain computational resources. Therefore, it is recommended to merge data on the SLAC's interactive nodes.

$ ssh psanacs.slac.stanford.edu
$ cd myrelease

In cctbx.xfel images are merged using the cxi.merge command.

$ cxi.merge Ls04-lysozyme-merge.phil

Here, Ls04-lysozyme-merge.phil is a phil-file with the parameters to control the merging procedure. In this tutorial only a subset of the available options are defined.

backend

Back end database; FS for flat-file ASCII data storage, MySQL and SQLite for the respective proper database back ends.

d_min

Limiting resolution for scaling and merging

data

Directory containing integrated data in pickle format. Repeat to specify additional directories.

merge_anomalous

True to merge anomalous contributors (i.e. Bijvoet mates), False to preserve them

min_corr

Correlation cutoff for rejecting individual frames

model

The scaling reference, PDB filename containing atomic coordinates and isomorphous CRYST1 record

nproc

Specifies the number of scaling processes cxi.merge may have running at any one time

output.prefix

Prefix for all output file names

rawdata.sdfac_auto

True to apply SDFAC correction to each image, assuming negative intensities are normally distributed noise

rescale_with_average_cell

Rescale the images a second time, requiring images to conform to the average unit cell. If set to True, set_average_unit_cell must also be set to True.

set_average_unit_cell

If True set the unit cell of the merged data to the average of the merged images, otherwise use the unit cell of the scaling reference

XXX mention output here, table, rejected images summary

The name of the output file is output.prefix.mtz.

Calculating the CC_1/2 statistic

cxi.xmerge retrieves the scaled, unmerged intensities from the database back end, and calculates the CC_1/2<ref>Karplus, P. A. & Diederichs, K. Linking Crystallographic Model and Data Quality. Science 336, 1030–1033 (2012).</ref> and CC_iso statistics. CC_1/2 is defined as Pearson's correlation coefficient between two sets, such that for each unique reflection the average intensities of two randomly chosen halves of its independent observations are assigned to different set. CC_iso is the correlation coefficient between the merged data and the isomorphous scaling reference. Both statistics are computed in each resolution bin, as well as for the full set of reflections.

Once the database has been populated using cxi.merge, cxi.xmerge can be run, using the parameters defined in a phil-file, Ls-04-lysozyme-xmerge.phil.

$ cxi.xmerge Ls04-lysozyme-xmerge.phil

The options used in this tutorial not already described in Merging a set of integration files

scaling.mtz_column_F

Column name in the reference structure mtz-file with structure factors

scaling.mtz_file

mtz-file with reference structure factors, must have data type F

scaling.log_cutoff

Intensities less than e^{scaling.log_cutoff} will not be included in the calculation

References

<references/>