2017 cxi merge tutorial

cxi.merge for the 2017 Berkeley Lab tutorial

This is an updated, worked example of data merging using cxi.merge. Previous documentation sets are here and here.

In this example, we are given integrated still-shot data collected by Danny Axford at Diamond, for P6 myoglobin, PDB code 5M3S.

• /net/dials/raid1/aaron/zurich0038/jr_006_batches/split_reintegrated/extracted # cctbx-style integration pickles
• /net/dials/raid1/aaron/zurich0038/jr_006_batches/sig_filter/split_reintegrated/extracted # same data, with per-image resolution cutoff during integration

Unix ls reveals 5031 *.pickle files in each directory.

Immediately there is a problem:

$ cxi.print_pickle /net/dials/raid1/aaron/zurich0038/jr_006_batches/sig_filter/split_reintegrated/extracted/*.pickle

...fails on image 0059 with a traceback; it looks like the file is corrupted.

So focus on the data without integration resolution cutoff:

$ cxi.print_pickle /net/dials/raid1/aaron/zurich0038/jr_006_batches/split_reintegrated/extracted/*.pickle

Some conclusions with the aid of grep:

• all integration pickles have space group P6 (good)
• distance and beam center is fixed throughout the integrated dataset
• Unit cells are variable but do seem to cluster around 91.4 91.4 45.9 90 90 120

phenix.fetch_pdb --mtz 5m3s

Merge command file:

#!/bin/csh -f

set effective_params = "d_min=DMIN \
data=/net/dials/raid1/aaron/zurich0038/jr_006_batches/split_reintegrated/extracted/*.pickle \
output.n_bins=10 \
pixel_size=0.172 \
backend=FS \
nproc=1 \
model=5m3s.pdb \
merge_anomalous=True \
plot_single_index_histograms=False \
scaling.algorithm=mark0 \
raw_data.sdfac_auto=False \
scaling.mtz_file=5m3s.mtz \
scaling.show_plots=False \
scaling.log_cutoff=None \
scaling.mtz_column_F=i-obs \
scaling.report_ML=True \
set_average_unit_cell=True \
rescale_with_average_cell=False \
significance_filter.apply=True \
significance_filter.min_ct=30 \
significance_filter.sigma=0.2 \
include_negatives=NEG \
postrefinement.enable=True \
postrefinement.algorithm=rs \
output.prefix=TAG"
set tag = p6m
set dmin = 2.5
set neg = True
set eff = `echo $effective_params|sed -e "s,FS,Flex,g"|sed -e "s,DMIN,$dmin,g"|sed -e "s,NEG,$neg,g"|sed -e "s,TAG,$tag,g"`

cxi.merge ${eff}
exit
cxi.xmerge ${eff}

Initial trial nproc=1 just to see if it runs. Had to fix PDB reference.

Scale-up trial nproc=60, no postrefinement. set the MTZ flag = jobs

 4493 of 5031 integration files were accepted
 0 rejected due to wrong Bravais group
 11 rejected for unit cell outliers
 22 rejected for low signal
 505 rejected due to up-front poor correlation under min_corr parameter
 0 rejected for file errors or no reindex matrix

Usage: 5m3s.mtz does not contain any observations labelled [fobs, imean, i-obs]. Please set scaling.mtz_column_F to one of [iobs].

 File "/net/viper/raid1/sauter/proj-e/modules/cctbx_project/xfel/cxi/util.py", line 13, in is_odd_numbered
   return int(os.path.basename(file_name).split(allowable)[0][-1])%2==1

ValueError: invalid literal for int() with base 10: 'd'

Something is wrong in the ability to determine even/odd numbered-ness.