http://viper.lbl.gov:8080/cctbx.xfel/api.php?action=feedcontributions&feedformat=atom&user=Aaroncctbx_xfel - User contributions [en]2026-09-04T20:10:11ZUser contributionsMediaWiki 1.39.15http://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Cctbx.xfel.merge&diff=1790Cctbx.xfel.merge2026-08-21T23:44:27Z<p>Aaron: </p>
<hr />
<div>{{DISPLAYTITLE:''cctbx.xfel.merge''}}<br />
<br />
'''''cctbx.xfel.merge''''' is the scaling, post-refinement and merging program of cctbx.xfel. It takes<br />
the per-image integrated results produced by ''[[dials.stills_process]]'' — one crystal per still shot —<br />
and reduces the whole ensemble to a single set of merged structure-factor intensities (an MTZ file)<br />
suitable for structure solution and refinement, together with the crystallographic quality<br />
statistics (completeness, multiplicity, ⟨I/σ⟩, CC½, CC<sub>iso</sub>, R-factors, etc.).<br />
<br />
Because serial datasets routinely contain tens of thousands to millions of still shots,<br />
<code>cctbx.xfel.merge</code> is built from the ground up to run under MPI across many nodes. It is<br />
almost always launched with <code>mpirun</code>/<code>srun</code>. Note that MPI uses the term 'ranks'<br />
to distinguish independent processes that are running across one or more nodes.<br />
<br />
= Overview =<br />
<br />
<code>cctbx.xfel.merge</code> is organized as a sequence of '''workers'''. Each worker performs one<br />
task — reading data, applying a correction, filtering, scaling, post-refining, computing a statistic,<br />
or merging — and passes its (possibly modified) experiments and reflections on to the next worker.<br />
The list of workers to run is the "step list", and it is fully configurable: you can add, remove or<br />
reorder steps with <code>dispatch.step_list</code>. This is what makes the program flexible enough to<br />
scale-and-merge in one pass, to scale subsets separately and merge later, or to run a reference-free<br />
bootstrap.<br />
<br />
The '''default''' step list is:<br />
<br />
{| class="wikitable"<br />
! # !! Worker !! Purpose<br />
|-<br />
| 1 || <code>input</code> || Read integrated experiments (<code>.expt</code>) and reflections (<code>.refl</code>) from disk in DIALS format.<br />
|-<br />
| 2 || <code>balance</code> || Redistribute the input load evenly across MPI ranks (needed when there are fewer or more input files than ranks).<br />
|-<br />
| 3 || <code>model_scaling</code> || Build the full Miller list, model intensities from the reference, and the resolution binner used for scaling and post-refinement.<br />
|-<br />
| 4 || <code>modify</code> || Apply per-reflection modifications, e.g. polarization correction (and optionally re-indexing to resolve indexing ambiguity — see <code>modify.cosym</code>/<code>reindex_to_reference</code>).<br />
|-<br />
| 5 || <code>filter</code> || Reject whole experiments (or reflections) by resolution, unit cell, correlation to the reference, energy, etc.<br />
|-<br />
| 6 || <code>scale</code> || Scale each lattice to the reference dataset (<code>mark0</code>) or simply average (<code>mark1</code>).<br />
|-<br />
| 7 || <code>postrefine</code> || Post-refine per-image crystal parameters and partialities (Sauter 2015).<br />
|-<br />
| 8 || <code>statistics_unitcell</code> || Determine and store the average unit cell.<br />
|-<br />
| 9 || <code>statistics_beam</code> || Determine and store the average wavelength.<br />
|-<br />
| 10 || <code>model_statistics</code> || Rebuild the Miller list / binner for statistics (may use the average unit cell).<br />
|-<br />
| 11 || <code>statistics_resolution</code> || Per-crystal resolution statistics.<br />
|-<br />
| 12 || <code>group</code> || Redistribute reflections across ranks so that every observation of a given HKL is gathered on the same rank prior to merging.<br />
|-<br />
| 13 || <code>errors_merge</code> || Correct the merged error estimates with a per-HKL error model (e.g. Ev11, Brewster 2019b, or MM24 (the default), Mittan-Moreau (2025)).<br />
|-<br />
| 14 || <code>statistics_intensity</code> || Intensity statistics by resolution.<br />
|-<br />
| 15 || <code>merge</code> || Merge symmetry-equivalent intensities, gather all HKLs at rank 0, and write "odd", "even" and "all" MTZ files.<br />
|-<br />
| 16 || <code>statistics_intensity_cxi</code> || Correlation and R-factor statistics (ported from cxi.merge): the "Table of Scaling Results" — CC½ (half-dataset correlation, labelled "CC int"), CC<sub>iso</sub>, R<sub>int</sub>, R<sub>split</sub>, R<sub>iso</sub>.<br />
|}<br />
<br />
Additional, non-default workers are available and can be<br />
inserted into the step list, including <code>tdata</code> (write out unit cells for clustering),<br />
<code>export</code> (batch MTZ for ''Careless''), <code>publish</code> (upload merged files to a<br />
Google Drive folder), <code>monitor</code> (resource-usage logging), <code>integrate</code><br />
(on-the-fly integration), <code>prepare</code> (SpReAD energy binning), and<br />
<code>lunus</code> (diffuse-scattering deck). A <code>deltaccint</code> worker (ΔCC½ lattice<br />
filtering) can be added after <code>group</code>. Custom workers can be supplied through the<br />
<code>XFEL_CUSTOM_WORKER_PATH</code> environment variable.<br />
<br />
= Basic usage =<br />
<br />
<code>cctbx.xfel.merge</code> is run under MPI:<br />
<br />
<pre><br />
mpirun cctbx.xfel.merge merge.phil<br />
</pre><br />
<br />
A typical <code>merge.phil</code>:<br />
<br />
<pre><br />
input.path=<path to data><br />
input.experiments_suffix=.expt<br />
input.reflections_suffix=.refl<br />
<br />
filter.algorithm=unit_cell<br />
filter.unit_cell.value.relative_length_tolerance=0.03<br />
filter.outlier.min_corr=0.1<br />
<br />
select.algorithm=significance_filter<br />
select.significance_filter.sigma=0.1<br />
select.significance_filter.min_ct=200<br />
select.significance_filter.max_ct=300<br />
<br />
scaling.model=<model.pdb or model.mtz><br />
scaling.resolution_scalar=0.96<br />
<br />
merging.d_min=1.8<br />
merging.error.model=mm24<br />
merging.merge_anomalous=True<br />
<br />
postrefinement.enable=True<br />
statistics.n_bins=20<br />
<br />
output.output_dir=<output folder><br />
output.prefix=<prefix for output files><br />
</pre><br />
<br />
Explore the parameters with:<br />
* <code>cctbx.xfel.merge -c</code> — show configurable parameters<br />
* <code>cctbx.xfel.merge -c -e 10 -a 2</code> — show all parameters (expert level 10) with full help text<br />
<br />
= Two-stage scaling and merging =<br />
<br />
For large, growing datasets (the mode used by the [[cctbx.xfel GUI]]), it is more efficient to scale<br />
and post-refine each subset of data '''once''', then merge repeatedly as more data arrive. This is<br />
done by running the program twice with different step lists.<br />
<br />
'''Stage 1 — scale each subset''' (stops after per-image work and writes intermediate DIALS files):<br />
<br />
<pre><br />
dispatch.step_list=input balance model_scaling modify filter scale postrefine \<br />
statistics_unitcell statistics_beam model_statistics statistics_resolution<br />
output.save_experiments_and_reflections=True<br />
</pre><br />
<br />
<code>output.save_experiments_and_reflections=True</code> dumps the scaled/post-refined data as<br />
experiment lists and reflection tables that the merge stage can read back in.<br />
<br />
'''Stage 2 — merge the scaled subsets''' (add an <code>input.path</code> for each finished subset):<br />
<br />
<pre><br />
dispatch.step_list=input model_scaling statistics_unitcell statistics_beam \<br />
model_statistics statistics_resolution group errors_merge statistics_intensity \<br />
merge statistics_intensity_cxi<br />
</pre><br />
<br />
= Merging without a reference (bootstrap / mark1) =<br />
<br />
If no scaling/merging reference is available, use a bootstrap procedure. First run the<br />
<code>mark1</code> algorithm, which simply averages the data (all per-image scale factors = 1, no<br />
post-refinement), specifying the target cell and symmetry explicitly:<br />
<br />
<pre><br />
scaling.algorithm=mark1<br />
scaling.unit_cell=<unit cell><br />
scaling.space_group=<space group><br />
merging.error.model=errors_from_sample_residuals<br />
</pre><br />
<br />
The resulting MTZ is then used as the reference (<code>scaling.model=</code>) for subsequent regular<br />
merges. Iterate a few cycles; CC½ typically improves and converges over the first few rounds.<br />
<br />
= Important parameters =<br />
<br />
Parameters are given by their full PHIL path with defaults in parentheses. The program exposes many<br />
more parameters (see <code>-c -e 10 -a 2</code>); those below are the ones most commonly set.<br />
<br />
== Input ==<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>input.path</code> || None || Path(s) to the integrated data — a folder, several folders, or individual files; wildcards allowed; may be repeated. Matching <code>.expt</code> and <code>.refl</code> files must both be present.<br />
|-<br />
| <code>input.experiments_suffix</code> || <code>_integrated.expt</code> || Only load experiment files ending with this suffix.<br />
|-<br />
| <code>input.reflections_suffix</code> || <code>_integrated.refl</code> || Only load reflection files ending with this suffix.<br />
|-<br />
| <code>input.parallel_file_load.method</code> || uniform || <code>uniform</code> spreads input files evenly over all ranks; <code>node_memory</code> distributes so per-node memory limits are respected.<br />
|-<br />
| <code>input.alist.file</code> || None || Path to a text file of experiment tags or filenames; combined with <code>input.alist.op=keep|reject</code> to merge only (or exclude) a specified subset.<br />
|-<br />
| <code>input.persistent_refl_cols</code> || None || Reflection-table columns to preserve through the pruning steps (and into saved tables).<br />
|}<br />
<br />
== Filtering whole experiments (<code>filter</code>) ==<br />
<br />
The <code>filter</code> section accepts or rejects '''entire experiments'''.<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>filter.algorithm</code> || (none) || One or more of <code>n_obs</code>, <code>resolution</code>, <code>unit_cell</code>, <code>energy</code>, <code>bootstrap</code>, <code>experiment_prefix</code>. Nothing is filtered unless listed.<br />
|-<br />
| <code>filter.unit_cell.algorithm</code> || value || How to restrict unit cells: <code>value</code> (compare to a target), <code>range</code>, or <code>cluster</code> (use a precomputed covariance clustering).<br />
|-<br />
| <code>filter.unit_cell.value.relative_length_tolerance</code> || 0.1 || Fractional deviation of cell edge lengths allowed vs. the target cell. Often tightened (e.g. 0.03).<br />
|-<br />
| <code>filter.unit_cell.value.absolute_angle_tolerance</code> || 2.0 || Allowed cell-angle deviation (degrees) vs. the target.<br />
|-<br />
| <code>filter.unit_cell.value.target_unit_cell</code> / <code>target_space_group</code> || Auto || Target cell/symmetry; <code>Auto</code> uses the scaling model.<br />
|-<br />
| <code>filter.outlier.min_corr</code> || 0.1 || Minimum correlation of an image's intensities to the reference model; images below this are rejected. Only applied when a scaling model is present. Use <code>-1</code> to accept all images regardless of correlation.<br />
|-<br />
| <code>filter.resolution.d_min</code> || None || Reject an experiment unless some of its reflections extend beyond this resolution.<br />
|}<br />
<br />
== Selecting individual reflections (<code>select</code>) ==<br />
<br />
The <code>select</code> section accepts or rejects '''individual reflections''' (contrast with<br />
<code>filter</code>, which acts on whole experiments).<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>select.algorithm</code> || (none) || One or more of <code>significance_filter</code>, <code>panel</code>, <code>cspad_sensor</code>, <code>isolation_forest</code>, <code>target_energy</code>.<br />
|-<br />
| <code>select.significance_filter.sigma</code> || 0.5 || Trim each lattice at the resolution where the binned ⟨I/σ⟩ drops below this value. Often reduced to 0.1.<br />
|-<br />
| <code>select.significance_filter.min_ct</code> / <code>max_ct</code> || 10 / 50 || Target minimum/maximum mean reflections per resolution bin when choosing the number of bins. For large unit cells these are often raised (e.g. 200/300); for small cells, use the defaults.<br />
|}<br />
<br />
== Modifying intensities (<code>modify</code>) ==<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>modify.algorithm</code> || polarization || Per-reflection corrections to apply (default applies the polarization correction).<br />
|-<br />
| <code>modify.cosym</code> || — || Resolve an indexing ambiguity across the ensemble using the Gildea/Winter (Brehm/Diederichs) cosym method, optionally anchored to <code>scaling.model</code>.<br />
|-<br />
| <code>modify.reindex_to_reference</code> / <code>reindex_to_abc</code> || — || Alternative re-indexing operators (to a reference model, or an explicit operator).<br />
|}<br />
<br />
== Scaling (<code>scaling</code>) ==<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>scaling.model</code> || None || Reference dataset: a PDB file (uses the CRYST1 cell and computes model intensities, including bulk solvent) or an MTZ from a previous cycle. Required for <code>mark0</code> scaling.<br />
|-<br />
| <code>scaling.algorithm</code> || mark0 || <code>mark0</code>: per-image scaling against the reference. <code>mark1</code>: no scaling, just Monte-Carlo averaging (all scale factors = 1) — used for reference-free bootstrap.<br />
|-<br />
| <code>scaling.unit_cell</code> / <code>scaling.space_group</code> || None || Cell/symmetry to use when no model is provided (i.e. <code>mark1</code>).<br />
|-<br />
| <code>scaling.resolution_scalar</code> || 0.969 || <code>merging.d_min</code> is multiplied by this when deciding which reflections fall within the resolution limit, generating a few extra reference reflections to account for per-shot cell variation (e.g. set 0.96 for a few more).<br />
|-<br />
| <code>scaling.mtz.mtz_column_F</code> || fobs || Column name in the reference MTZ containing the reference structure factors (intensities or amplitudes).<br />
|}<br />
<br />
== Post-refinement (<code>postrefinement</code>) ==<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>postrefinement.enable</code> || True || Enable post-refinement of per-image parameters/partialities (Sauter 2015).<br />
|-<br />
| <code>postrefinement.algorithm</code> || rs || Post-refinement protocol: <code>rs</code>, <code>rs2</code>, <code>rs_hybrid</code>, or <code>eta_deff</code>.<br />
|}<br />
<br />
== Merging and error model (<code>merging</code>) ==<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>merging.d_min</code> || None || High-resolution limit for scaling and merging.<br />
|-<br />
| <code>merging.d_max</code> || None || Low-resolution limit (currently affects only CCiso).<br />
|-<br />
| <code>merging.merge_anomalous</code> || False || <code>True</code> merges Bijvoet mates together (higher multiplicity); <code>False</code> keeps them separate to preserve the anomalous signal.<br />
|-<br />
| <code>merging.minimum_multiplicity</code> || 2 || Do not produce a merged structure factor for HKLs measured fewer than this many times.<br />
|-<br />
| <code>merging.error.model</code> || mm24 || Merged-sigma error model: <code>mm24</code> (Mittan-Moreau 2025, current default), <code>ev11</code> (Evans 2011 / sdfac-refine), <code>ha14</code>, or <code>errors_from_sample_residuals</code> (no refined error terms — used with <code>mark1</code> bootstrap).<br />
|-<br />
| <code>merging.set_average_unit_cell</code> || True || Output MTZ adopts the average unit cell of the data rather than the reference model's cell.<br />
|}<br />
<br />
== Statistics (<code>statistics</code>) ==<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>statistics.n_bins</code> || 10 || Number of resolution bins in the statistics tables. Often raised to 20.<br />
|-<br />
| <code>statistics.cciso.mtz_file</code> || None || Isomorphous reference structure factors for R<sub>iso</sub> / CC<sub>iso</sub>.<br />
|-<br />
| <code>statistics.report_ML</code> || True || Report per-frame max-likelihood attributes (domain size, mosaicity).<br />
|}<br />
<br />
== Output (<code>output</code>) ==<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>output.output_dir</code> || <code>.</code> || Directory for output files (created if it does not exist).<br />
|-<br />
| <code>output.prefix</code> || iobs || Prefix prepended to all output file names.<br />
|-<br />
| <code>output.save_experiments_and_reflections</code> || False || Dump the final experiments/reflections from the last worker as DIALS files — used for the two-stage scale-then-merge workflow.<br />
|-<br />
| <code>output.do_timing</code> || False || Log elapsed time per execution step.<br />
|-<br />
| <code>output.expanded_bookkeeping</code> || False || Store extra provenance columns (input file/reflection indices, odd/even flag, original id) in the saved reflection tables.<br />
|}<br />
<br />
== Multiprocessing (<code>mp</code>) ==<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>mp.method</code> || mpi || Multiprocessing method (only <code>mpi</code> is currently supported).<br />
|-<br />
| <code>mp.debug.cProfile</code> || False || Enable code profiling; a <code>.prof</code> file is written per rank.<br />
|}<br />
<br />
= Output files =<br />
<br />
The <code>merge</code> worker gathers all HKLs at rank 0 and writes three MTZ files, prefixed with<br />
<code>output.prefix</code> (default <code>iobs</code>):<br />
<br />
* <code>&lt;prefix&gt;_all.mtz</code> — the merged dataset using all observations (the main result for structure solution)<br />
* <code>&lt;prefix&gt;_odd.mtz</code> and <code>&lt;prefix&gt;_even.mtz</code> — the two half-datasets used for CC½ and related half-dataset statistics<br />
<br />
A main log records the per-step progress and the merging statistics tables by resolution bin:<br />
completeness, multiplicity and ⟨I/σ⟩, plus the "Table of Scaling Results" reporting CC½ (the<br />
half-dataset correlation, labelled "CC int"), CC<sub>iso</sub>, R<sub>int</sub>, R<sub>split</sub><br />
and R<sub>iso</sub>. The merged MTZ can optionally be<br />
uploaded to a shared [https://github.com/cctbx/cctbx_project/tree/master/xfel/merging/application/publish Google Drive folder] by adding the <code>publish</code> worker.<br />
<br />
= References =<br />
<br />
* Sauter NK (2015). ''J Synchrotron Radiat'' '''22''', 239–48. (post-refinement)<br />
* Uervirojnangkoorn M, ''et al.'' (2015). ''eLife'' 10.7554/eLife.05421. (PRIME / bootstrap)<br />
* Brewster AS, ''et al.'' (2018). ''Acta Cryst.'' D '''74''', 877–894.<br />
* Brewster AS, ''et al.'' (2019a). ''Computational Crystallography Newsletter'' '''10''', 22–39. (XFEL GUI)<br />
* Brewster AS, ''et al.'' (2019b). ''Acta Cryst.'' D '''75''', 959–968. (error models)<br />
* Mittan-Moreau (2025). (mm24 error model)<br />
<br />
= See also =<br />
<br />
* [[dials.stills_process]] — produces the integrated data that <code>cctbx.xfel.merge</code> consumes<br />
* [[cctbx.xfel GUI]] — orchestrates separate scaling and repeated merging as data are collected<br />
* The program README at <code>xfel/merging/README.md</code> in cctbx_project</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Main_Page&diff=1789Main Page2026-08-21T23:42:11Z<p>Aaron: /* cctbx.xfel resources */</p>
<hr />
<div>News! See our [[https://doi.org/10.1101/2025.05.04.652045 new preprint]]! This paper shows current best practices using ''cctbx.xfel'' with ''DIALS'' for serial crystallographic data.<br />
<br />
= Open-source tools for serial crystallographic data processing =<br />
<br />
''cctbx.xfel'' is a suite of software tools designed to process diffraction data from serial femtosecond crystallography (SFX) measurements at an X-ray free-electron laser (XFEL) or a synchrotron. Built on the Computational Crystallographic Toolbox ([https://cctbx.github.io/ ''cctbx'']), the same toolbox on which [http://www.phenix-online.org ''PHENIX''], and [http://dials.github.io DIALS] are built, it enables the user to solve difficult problems relating to processing serial crystallographic data. The programs and modules provided by ''cctbx.xfel'' can reduce a large set of still diffraction images recorded at Stanford’s Linac Coherent Light Source ([http://lcls.slac.stanford.edu LCLS]), [http://sacla.xfel.jp SACLA], the [https://www.xfel.eu/ European XFEL], or a synchrotron, to a single MTZ file containing merged reflection intensities suitable for structure solution.<br />
<br />
== ''cctbx.xfel'' resources ==<br />
<br />
The current best resource is our pre-print, [https://doi.org/10.1101/2025.05.04.652045 ''cctbx.xfel: a suite for processing serial crystallographic data.'']. This paper describes the algorithms and ideas behind the implementation and execution of the two main programs of ''cctbx.xfel'':<br />
<br />
* ''[[Dials.stills_process|dials.stills_process]]'': this program runs through all the first steps of processing data using dials: import (reading image headers), spot finding, indexing, geometry refinement, and integration. Serial images are independent from each other, so this program is optimized for massively parallel systems, but can be adapted to a variety of single and multi-processing modes.<br />
* ''[[Cctbx.xfel.merge|cctbx.xfel.merge]]'': this program runs through the remaining steps of processing: scaling and merging to produce a full dataset (.mtz file). For documentation, see our [https://github.com/cctbx/cctbx_project/tree/master/xfel/merging Github project]<br />
<br />
Additional resources:<br />
<br />
* Our [http://cci.lbl.gov/publications/download/CCN_2019_p22_Brewster.pdf 2019 newsletter article] describes how to use the ''cctbx.xfel'' GUI, an interactive program for handling large XFEL datasets.<br />
<br />
* [[Overview]] to the system architecture at LCLS and real-time progress monitoring of data processing<br />
* Installation: there are two ways to get ''cctbx.xfel'':<br />
** ''cctbx.xfel'' is installed for general use at SLAC. To start using an existing installation, [[Using the pre-built cctbx at LCLS|simply source the current build]]. <br />
** [https://dials.github.io/installation.html Download] a binary bundle of DIALS. Use this option if you will not be processing LCLS data because, for example, if you have serial crystallographic data from a synchrotron.<br />
** Build a psana/cctbx.xfel build [https://github.com/cctbx/cctbx_project/tree/master/xfel/conda_envs from scratch]. Use this option if you want to process LCLS data outside of LCLS or want your own build.<br />
* [[Tutorials]] on pre-processing, data reduction, and merging. <br />
<br />
Other related information:<br />
<br />
* [[Serial XFEL Crystallography References]]<br />
<br />
This project is under active development. For any assistance, please contact the authors.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Dials.stills_process&diff=1788Dials.stills process2026-08-21T23:40:33Z<p>Aaron: </p>
<hr />
<div>{{DISPLAYTITLE:''dials.stills_process''}}<br />
<br />
'''''dials.stills_process''''' is a stand-alone program for processing still diffraction images and is<br />
the core data-reduction program of ''cctbx.xfel''. It takes a set of<br />
still diffraction images — typically serial femtosecond crystallography (SFX) snapshots from an<br />
XFEL (LCLS, SACLA, EuXFEL) or still shots from a synchrotron — and analyzes each one, independently,<br />
through the full DIALS pipeline: '''import → spot-finding → indexing → refinement → integration'''.<br />
Its output (integrated intensities together with the per-image experimental models) is the input to<br />
the merging program ''[[cctbx.xfel.merge]]''.<br />
<br />
Because every image is a separate still exposure of a different crystal (rather than a wedge of<br />
a rotation series), each image is treated as its own experiment with its own crystal model. This<br />
"one crystal per shot" model is what distinguishes stills processing from conventional rotation data<br />
processing (e.g. <code>dials.import</code>, <code>dials.find_spots</code>, <code>dials.index</code>, etc.), <br />
and it drives most of the stills-specific defaults described below.<br />
<br />
= Overview =<br />
<br />
For each image, <code>dials.stills_process</code> runs the following stages in order. Any stage can<br />
be turned off with the <code>dispatch</code> parameters; if a stage is disabled, processing stops<br />
after the last enabled stage.<br />
<br />
# '''Import''' — read the image, apply reference geometry / gain / mask overrides, and build an experiment.<br />
# '''Spot-finding''' — locate strong pixels and build strong reflections (<code>dispatch.find_spots</code>).<br />
# '''Hit finding''' — reject images with too few (or optionally too many) strong spots, so that only "hits" are indexed (<code>dispatch.hit_finder</code>).<br />
# '''Indexing''' — determine the crystal orientation and (optionally constrained) unit cell (<code>dispatch.index</code>).<br />
# '''Refinement''' — refine the crystal model and re-estimate the mosaic/domain-size parameters. Detector and beam are held fixed by default (<code>dispatch.refine</code>).<br />
# '''Integration''' — predict reflections and measure their intensities using the stills integrator (<code>dispatch.integrate</code>), with an optional significance filter.<br />
<br />
The program applies a set of stills-specific defaults on top of the standard DIALS parameters,<br />
including: indexing method <code>fft1d</code>; the stills integrator with profile fitting turned off<br />
(summation integration with a 2-D plane background); the stills refinement weighting strategy with<br />
the detector and beam models fixed; and a re-estimation of per-crystal mosaic parameters<br />
(domain size and mosaic angle) after refinement. These are sensible starting points for serial data<br />
and normally do not need to be changed.<br />
<br />
Failures on individual images are, by default, caught and logged so that processing continues to the<br />
next image (<code>dispatch.squash_errors=True</code>) — essential when reducing tens of thousands of<br />
shots of which only a fraction are hits.<br />
<br />
= Basic usage =<br />
<br />
<pre><br />
# Process a directory of CBF stills, constraining the target cell/symmetry,<br />
# using 32 parallel processes on one node<br />
dials.stills_process /path/to/images/*.cbf process.phil mp.nproc=32<br />
</pre><br />
<br />
A typical <code>process.phil</code> for XFEL data looks like:<br />
<br />
<pre><br />
spotfinder.filter.min_spot_size=2<br />
spotfinder.threshold.dispersion.gain=1.0<br />
spotfinder.threshold.dispersion.global_threshold=100<br />
<br />
indexing.known_symmetry.space_group=P43212<br />
indexing.known_symmetry.unit_cell=78.9,78.9,38.1,90,90,90<br />
indexing.method=fft1d<br />
<br />
input.reference_geometry=refined.expt # from a previous metrology refinement<br />
<br />
output.output_dir=out<br />
mp.nproc=32<br />
</pre><br />
<br />
Useful command-line helpers:<br />
* <code>dials.stills_process -h</code> — help and usage<br />
* <code>dials.stills_process -c -e 2</code> — show the configurable parameters (expert level 2)<br />
* <code>dials.stills_process -c -a2</code> — include attributes (help text) for each parameter<br />
* <code>dials.stills_process ... input.show_image_tags=True</code> — list the image tags in a dataset without processing (useful for selecting subsets with <code>input.image_tag</code>)<br />
<br />
= Important parameters =<br />
<br />
The program exposes the entire DIALS parameter set (spot-finding, indexing, refinement, integration,<br />
profile modelling, prediction). The parameters below are the ones most commonly adjusted for serial<br />
still data. Parameters are given by their full PHIL path with their default value.<br />
<br />
== Input ==<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>input.reference_geometry</code> || None || Path to an <code>.expt</code> file (or image) with a single detector model. Data are processed using ''this'' geometry instead of the geometry in the image headers. This is how a refined detector metrology from <code>cctbx.xfel</code> ensemble refinement is applied — almost always set for XFEL data.<br />
|-<br />
| <code>input.sync_reference_geom</code> || True || When a reference geometry is supplied, copy its refined local frames onto the image's detector hierarchy node-by-node (the hierarchies must match in shape). If False, the reference detector replaces the image detector wholesale.<br />
|-<br />
| <code>input.file_list</code> || None || Path to a text file listing image paths, one per line (alternative to listing files on the command line).<br />
|-<br />
| <code>input.glob</code> || None || Wildcard path(s) for large multi-file datasets (e.g. <code>*.cbf</code>). May be given multiple times.<br />
|-<br />
| <code>input.image_tag</code> || None || Process only images with the given tag(s). For single-image files the tag is the filename; for multi-image files (HDF5) it is <code>filename_imagenumber</code> (image number is 5 digits with leading zeroes). Use <code>input.show_image_tags=True</code> to list them.<br />
|-<br />
| <code>input.max_images</code> || None || Limit the total number of images processed.<br />
|}<br />
<br />
== Dispatch — controlling the pipeline ==<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>dispatch.find_spots</code> || True || Do spot-finding. Required for indexing/integration.<br />
|-<br />
| <code>dispatch.index</code> || True || Attempt to index. Requires <code>find_spots=True</code>.<br />
|-<br />
| <code>dispatch.integrate</code> || True || Integrate indexed images. Ignored if indexing/spot-finding is off.<br />
|-<br />
| <code>dispatch.refine</code> || False || Run an explicit model-refinement step after indexing (the stills indexer already refines; this is an additional pass, rarely needed).<br />
|-<br />
| <code>dispatch.squash_errors</code> || True || If an image fails, log it and continue to the next image. Set False to halt and show the full error when debugging.<br />
|-<br />
| <code>dispatch.hit_finder.enable</code> || True || Enable hit-finding. If disabled, every image is passed to indexing.<br />
|-<br />
| <code>dispatch.hit_finder.minimum_number_of_reflections</code> || 16 || Images with fewer strong spots than this are discarded as non-hits before indexing.<br />
|-<br />
| <code>dispatch.hit_finder.maximum_number_of_reflections</code> || None || If set, images with more strong spots than this are discarded (e.g. ice/junk).<br />
|-<br />
| <code>dispatch.process_percent</code> || None || Process only this percentage of events (evenly sampled). Useful for a quick look at a large dataset.<br />
|-<br />
| <code>dispatch.coset</code> || False || During integration, also integrate a sublattice coset as a negative control for regions with no Bragg diffraction (see <code>integration.coset.transformation</code>).<br />
|-<br />
| <code>dispatch.pre_import</code> || False || Import all data before processing. Only needed when processing multiple ''multi-image'' files at once (not a recommended workflow). Single-file datasets (e.g. HDF5) are always pre-imported automatically.<br />
|}<br />
<br />
== Spot-finding ==<br />
<br />
Spot-finding uses the standard DIALS dispersion thresholding. The most impactful parameters for<br />
serial data:<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>spotfinder.filter.min_spot_size</code> || Auto || Minimum connected pixels for a spot. Auto = 3 for pixel-array detectors (SENSOR_PAD), 6 otherwise. Often lowered to 2 for sparse serial patterns.<br />
|-<br />
| <code>spotfinder.threshold.dispersion.gain</code> || None || Detector gain used in thresholding. Set this ''or'' the detector-model gain, not both (see gain note below).<br />
|-<br />
| <code>spotfinder.threshold.dispersion.sigma_strong</code> || 3 || Number of sigma above background for a pixel to be "strong".<br />
|-<br />
| <code>spotfinder.threshold.dispersion.sigma_background</code> || 6 || Number of sigma for the background/dispersion test.<br />
|-<br />
| <code>spotfinder.threshold.dispersion.global_threshold</code> || 0 || Absolute pixel intensity floor; pixels below this are never considered strong. Raising it (e.g. 100) suppresses noise on many detectors.<br />
|-<br />
| <code>spotfinder.threshold.dispersion.kernel_size</code> || 3 3 || Local-area kernel (pixels) for the dispersion calculation.<br />
|}<br />
<br />
'''Gain note:''' the detector gain should be set on the detector model ''or'' in the processing<br />
parameters, never both. If a supplied gain (<code>spotfinder.threshold.dispersion.gain</code> or<br />
<code>integration.summation.detector_gain</code>) conflicts with a non-unity gain already on the<br />
detector model, the program raises an error. Override with<br />
<code>input.ignore_gain_mismatch=True</code> only if you understand the consequences.<br />
<br />
== Indexing ==<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>indexing.known_symmetry.space_group</code> || None || Target space group. Strongly recommended for serial data — constrains indexing and improves the hit/index rate.<br />
|-<br />
| <code>indexing.known_symmetry.unit_cell</code> || None || Target unit cell (a,b,c,α,β,γ). Used together with the space group as the indexing target.<br />
|-<br />
| <code>indexing.method</code> || fft1d || Basis-vector search method (<code>dials.stills_process</code> defaults to <code>fft1d</code>; <code>fft3d</code> and <code>real_space_grid_search</code> are also available).<br />
|-<br />
| <code>indexing.stills.method_list</code> || None || Ordered list of indexing methods to try in turn; if one fails, the next is attempted (e.g. <code>fft1d real_space_grid_search</code>).<br />
|-<br />
| <code>indexing.stills.known_orientations</code> || None || Paths to previously-processed experiment lists. If given, images are ''not'' re-indexed; the stored crystal orientations are reused (e.g. to re-integrate with new parameters). Supports wildcards; may be repeated.<br />
|-<br />
| <code>indexing.stills.require_known_orientation</code> || False || When known orientations are supplied but an image is not found among them, controls whether to skip it (True) or fall back to indexing from scratch (False).<br />
|-<br />
| <code>indexing.stills.reflection_subsampling.enable</code> || False || Retry indexing on random subsets of the strong spots (from <code>step_start</code>% down to <code>step_stop</code>% in <code>step_size</code>% steps). Can rescue images that fail to index on the full spot list. Especially valuble for multi-lattice images (multiple crystals in the beam) but does increase execution time!<br />
|}<br />
<br />
== Integration ==<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>profile.gaussian_rs.parameters.sigma_b_cutoff</code> || 0.1 || Maximum beam divergence (sigma_b, degrees) before an indexed crystal is rejected prior to integration. Guards against over-broad, poorly-modelled lattices.<br />
|-<br />
| <code>significance_filter.enable</code> || False || Apply the stills significance filter, which trims each crystal to the resolution at which reflections remain statistically significant (see <code>significance_filter.isigi_cutoff</code>, default 1.0). Defaults to false here because it is typically applied in ''[[cctbx.xfel.merge]]''.<br />
|-<br />
| <code>integration.integration_only_overrides.trusted_range</code> || None || Override the panel trusted range [min, max] during integration only (e.g. to include pixels the header marks as untrusted).<br />
|-<br />
| <code>integration.coset.transformation</code> || 6 || Which modulus-2 sublattice transformation defines the coset negative control when <code>dispatch.coset=True</code> (0–6; 6 = body centering). See Sauter & Zwart, ''Acta Cryst.'' D (2009) '''65''':553.<br />
|}<br />
<br />
== Output ==<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>output.output_dir</code> || <code>.</code> || Directory for output files.<br />
|-<br />
| <code>output.composite_output</code> || True || If True, each process writes ''one'' concatenated set of <code>.expt</code>/<code>.refl</code> files covering all its successful images. If False, a separate pair of files is written per image (generates a very large number of files). Composite output is strongly preferred for large serial datasets.<br />
|-<br />
| <code>output.logging_dir</code> || None || Directory for log files. Under MPI, per-rank logs (<code>log_rankNNNN.out</code>, etc.) are written here.<br />
|-<br />
| <code>output.logging_option</code> || normal || <code>normal</code> (all logging), <code>suppressed</code> (silences verbose DIALS refine/index output), or <code>disabled</code> (almost no logging).<br />
|-<br />
| <code>output.*_filename</code> || (templates) || Output filename templates for each stage: <code>strong</code>, <code>indexed</code>, <code>refined_experiments</code>, <code>integrated</code>, <code>integrated_experiments</code>, <code>coset</code>. A <code>%s</code> in the template is replaced with the process number for composite mode or the per-image tag for non-composite mode.<br />
|-<br />
| <code>output.integration_pickle</code> || None || Write legacy <code>cxi.merge</code>/PRIME integration pickle files (e.g. <code>int-%d-%s.pickle</code>). Only needed for legacy merging workflows.<br />
|}<br />
<br />
== Multiprocessing ==<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>mp.method</code> || multiprocessing || Parallelization backend: <code>multiprocessing</code> (single node), or <code>mpi</code> for multi-node jobs (also <code>sge</code>, <code>lsf</code>, <code>pbs</code>).<br />
|-<br />
| <code>mp.nproc</code> || 1 || Number of processes (for the <code>multiprocessing</code> method). Set to the number of cores per node.<br />
|-<br />
| <code>mp.composite_stride</code> || None || For MPI composite output, how many ranks aggregate their results into one set of output files. Ideally set to the number of processes per node so that one composite file set is written per node rather than per rank.<br />
|}<br />
<br />
Under MPI with more than two ranks, the program runs a client/server model: rank 0 (or rank 2 for<br />
psana2 <code>.loc</code> inputs) hands out images to worker ranks on demand, giving good load<br />
balancing across the many fast-failing non-hits typical of serial data.<br />
<br />
= Output files =<br />
<br />
With the default composite output, each process (or each aggregating rank under MPI) writes<br />
concatenated files such as:<br />
<br />
* <code>&lt;tag&gt;_strong.refl</code> — strong spots from spot-finding<br />
* <code>&lt;tag&gt;_indexed.refl</code> and <code>&lt;tag&gt;_refined.expt</code> — indexed reflections and their crystal models<br />
* <code>&lt;tag&gt;_integrated.refl</code> and <code>&lt;tag&gt;_integrated.expt</code> — integrated intensities and final models (the input to ''[[cctbx.xfel.merge]]'')<br />
<br />
Per-image progress and status codes are recorded in <code>debug/debug_&lt;rank&gt;.txt</code>, and<br />
overall logging goes to <code>dials.process.log</code> (or per-rank logs under MPI). These integrated<br />
<code>.expt</code>/<code>.refl</code> pairs are then scaled and merged with<br />
''[[cctbx.xfel.merge]]''.<br />
<br />
= See also =<br />
<br />
* ''[[cctbx.xfel.merge]]'' — scaling, post-refinement and merging of the integrated data<br />
* The [http://dials.github.io DIALS documentation] for the underlying spot-finding, indexing, refinement and integration algorithms</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Dials.stills_process&diff=1787Dials.stills process2026-08-21T23:39:39Z<p>Aaron: </p>
<hr />
<div>{{DISPLAYTITLE:dials.stills_process}}<br />
<br />
'''''dials.stills_process''''' is a stand-alone program for processing still diffraction images and is<br />
the core data-reduction program of ''cctbx.xfel''. It takes a set of<br />
still diffraction images — typically serial femtosecond crystallography (SFX) snapshots from an<br />
XFEL (LCLS, SACLA, EuXFEL) or still shots from a synchrotron — and analyzes each one, independently,<br />
through the full DIALS pipeline: '''import → spot-finding → indexing → refinement → integration'''.<br />
Its output (integrated intensities together with the per-image experimental models) is the input to<br />
the merging program ''[[cctbx.xfel.merge]]''.<br />
<br />
Because every image is a separate still exposure of a different crystal (rather than a wedge of<br />
a rotation series), each image is treated as its own experiment with its own crystal model. This<br />
"one crystal per shot" model is what distinguishes stills processing from conventional rotation data<br />
processing (e.g. <code>dials.import</code>, <code>dials.find_spots</code>, <code>dials.index</code>, etc.), <br />
and it drives most of the stills-specific defaults described below.<br />
<br />
= Overview =<br />
<br />
For each image, <code>dials.stills_process</code> runs the following stages in order. Any stage can<br />
be turned off with the <code>dispatch</code> parameters; if a stage is disabled, processing stops<br />
after the last enabled stage.<br />
<br />
# '''Import''' — read the image, apply reference geometry / gain / mask overrides, and build an experiment.<br />
# '''Spot-finding''' — locate strong pixels and build strong reflections (<code>dispatch.find_spots</code>).<br />
# '''Hit finding''' — reject images with too few (or optionally too many) strong spots, so that only "hits" are indexed (<code>dispatch.hit_finder</code>).<br />
# '''Indexing''' — determine the crystal orientation and (optionally constrained) unit cell (<code>dispatch.index</code>).<br />
# '''Refinement''' — refine the crystal model and re-estimate the mosaic/domain-size parameters. Detector and beam are held fixed by default (<code>dispatch.refine</code>).<br />
# '''Integration''' — predict reflections and measure their intensities using the stills integrator (<code>dispatch.integrate</code>), with an optional significance filter.<br />
<br />
The program applies a set of stills-specific defaults on top of the standard DIALS parameters,<br />
including: indexing method <code>fft1d</code>; the stills integrator with profile fitting turned off<br />
(summation integration with a 2-D plane background); the stills refinement weighting strategy with<br />
the detector and beam models fixed; and a re-estimation of per-crystal mosaic parameters<br />
(domain size and mosaic angle) after refinement. These are sensible starting points for serial data<br />
and normally do not need to be changed.<br />
<br />
Failures on individual images are, by default, caught and logged so that processing continues to the<br />
next image (<code>dispatch.squash_errors=True</code>) — essential when reducing tens of thousands of<br />
shots of which only a fraction are hits.<br />
<br />
= Basic usage =<br />
<br />
<pre><br />
# Process a directory of CBF stills, constraining the target cell/symmetry,<br />
# using 32 parallel processes on one node<br />
dials.stills_process /path/to/images/*.cbf process.phil mp.nproc=32<br />
</pre><br />
<br />
A typical <code>process.phil</code> for XFEL data looks like:<br />
<br />
<pre><br />
spotfinder.filter.min_spot_size=2<br />
spotfinder.threshold.dispersion.gain=1.0<br />
spotfinder.threshold.dispersion.global_threshold=100<br />
<br />
indexing.known_symmetry.space_group=P43212<br />
indexing.known_symmetry.unit_cell=78.9,78.9,38.1,90,90,90<br />
indexing.method=fft1d<br />
<br />
input.reference_geometry=refined.expt # from a previous metrology refinement<br />
<br />
output.output_dir=out<br />
mp.nproc=32<br />
</pre><br />
<br />
Useful command-line helpers:<br />
* <code>dials.stills_process -h</code> — help and usage<br />
* <code>dials.stills_process -c -e 2</code> — show the configurable parameters (expert level 2)<br />
* <code>dials.stills_process -c -a2</code> — include attributes (help text) for each parameter<br />
* <code>dials.stills_process ... input.show_image_tags=True</code> — list the image tags in a dataset without processing (useful for selecting subsets with <code>input.image_tag</code>)<br />
<br />
= Important parameters =<br />
<br />
The program exposes the entire DIALS parameter set (spot-finding, indexing, refinement, integration,<br />
profile modelling, prediction). The parameters below are the ones most commonly adjusted for serial<br />
still data. Parameters are given by their full PHIL path with their default value.<br />
<br />
== Input ==<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>input.reference_geometry</code> || None || Path to an <code>.expt</code> file (or image) with a single detector model. Data are processed using ''this'' geometry instead of the geometry in the image headers. This is how a refined detector metrology from <code>cctbx.xfel</code> ensemble refinement is applied — almost always set for XFEL data.<br />
|-<br />
| <code>input.sync_reference_geom</code> || True || When a reference geometry is supplied, copy its refined local frames onto the image's detector hierarchy node-by-node (the hierarchies must match in shape). If False, the reference detector replaces the image detector wholesale.<br />
|-<br />
| <code>input.file_list</code> || None || Path to a text file listing image paths, one per line (alternative to listing files on the command line).<br />
|-<br />
| <code>input.glob</code> || None || Wildcard path(s) for large multi-file datasets (e.g. <code>*.cbf</code>). May be given multiple times.<br />
|-<br />
| <code>input.image_tag</code> || None || Process only images with the given tag(s). For single-image files the tag is the filename; for multi-image files (HDF5) it is <code>filename_imagenumber</code> (image number is 5 digits with leading zeroes). Use <code>input.show_image_tags=True</code> to list them.<br />
|-<br />
| <code>input.max_images</code> || None || Limit the total number of images processed.<br />
|}<br />
<br />
== Dispatch — controlling the pipeline ==<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>dispatch.find_spots</code> || True || Do spot-finding. Required for indexing/integration.<br />
|-<br />
| <code>dispatch.index</code> || True || Attempt to index. Requires <code>find_spots=True</code>.<br />
|-<br />
| <code>dispatch.integrate</code> || True || Integrate indexed images. Ignored if indexing/spot-finding is off.<br />
|-<br />
| <code>dispatch.refine</code> || False || Run an explicit model-refinement step after indexing (the stills indexer already refines; this is an additional pass, rarely needed).<br />
|-<br />
| <code>dispatch.squash_errors</code> || True || If an image fails, log it and continue to the next image. Set False to halt and show the full error when debugging.<br />
|-<br />
| <code>dispatch.hit_finder.enable</code> || True || Enable hit-finding. If disabled, every image is passed to indexing.<br />
|-<br />
| <code>dispatch.hit_finder.minimum_number_of_reflections</code> || 16 || Images with fewer strong spots than this are discarded as non-hits before indexing.<br />
|-<br />
| <code>dispatch.hit_finder.maximum_number_of_reflections</code> || None || If set, images with more strong spots than this are discarded (e.g. ice/junk).<br />
|-<br />
| <code>dispatch.process_percent</code> || None || Process only this percentage of events (evenly sampled). Useful for a quick look at a large dataset.<br />
|-<br />
| <code>dispatch.coset</code> || False || During integration, also integrate a sublattice coset as a negative control for regions with no Bragg diffraction (see <code>integration.coset.transformation</code>).<br />
|-<br />
| <code>dispatch.pre_import</code> || False || Import all data before processing. Only needed when processing multiple ''multi-image'' files at once (not a recommended workflow). Single-file datasets (e.g. HDF5) are always pre-imported automatically.<br />
|}<br />
<br />
== Spot-finding ==<br />
<br />
Spot-finding uses the standard DIALS dispersion thresholding. The most impactful parameters for<br />
serial data:<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>spotfinder.filter.min_spot_size</code> || Auto || Minimum connected pixels for a spot. Auto = 3 for pixel-array detectors (SENSOR_PAD), 6 otherwise. Often lowered to 2 for sparse serial patterns.<br />
|-<br />
| <code>spotfinder.threshold.dispersion.gain</code> || None || Detector gain used in thresholding. Set this ''or'' the detector-model gain, not both (see gain note below).<br />
|-<br />
| <code>spotfinder.threshold.dispersion.sigma_strong</code> || 3 || Number of sigma above background for a pixel to be "strong".<br />
|-<br />
| <code>spotfinder.threshold.dispersion.sigma_background</code> || 6 || Number of sigma for the background/dispersion test.<br />
|-<br />
| <code>spotfinder.threshold.dispersion.global_threshold</code> || 0 || Absolute pixel intensity floor; pixels below this are never considered strong. Raising it (e.g. 100) suppresses noise on many detectors.<br />
|-<br />
| <code>spotfinder.threshold.dispersion.kernel_size</code> || 3 3 || Local-area kernel (pixels) for the dispersion calculation.<br />
|}<br />
<br />
'''Gain note:''' the detector gain should be set on the detector model ''or'' in the processing<br />
parameters, never both. If a supplied gain (<code>spotfinder.threshold.dispersion.gain</code> or<br />
<code>integration.summation.detector_gain</code>) conflicts with a non-unity gain already on the<br />
detector model, the program raises an error. Override with<br />
<code>input.ignore_gain_mismatch=True</code> only if you understand the consequences.<br />
<br />
== Indexing ==<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>indexing.known_symmetry.space_group</code> || None || Target space group. Strongly recommended for serial data — constrains indexing and improves the hit/index rate.<br />
|-<br />
| <code>indexing.known_symmetry.unit_cell</code> || None || Target unit cell (a,b,c,α,β,γ). Used together with the space group as the indexing target.<br />
|-<br />
| <code>indexing.method</code> || fft1d || Basis-vector search method (<code>dials.stills_process</code> defaults to <code>fft1d</code>; <code>fft3d</code> and <code>real_space_grid_search</code> are also available).<br />
|-<br />
| <code>indexing.stills.method_list</code> || None || Ordered list of indexing methods to try in turn; if one fails, the next is attempted (e.g. <code>fft1d real_space_grid_search</code>).<br />
|-<br />
| <code>indexing.stills.known_orientations</code> || None || Paths to previously-processed experiment lists. If given, images are ''not'' re-indexed; the stored crystal orientations are reused (e.g. to re-integrate with new parameters). Supports wildcards; may be repeated.<br />
|-<br />
| <code>indexing.stills.require_known_orientation</code> || False || When known orientations are supplied but an image is not found among them, controls whether to skip it (True) or fall back to indexing from scratch (False).<br />
|-<br />
| <code>indexing.stills.reflection_subsampling.enable</code> || False || Retry indexing on random subsets of the strong spots (from <code>step_start</code>% down to <code>step_stop</code>% in <code>step_size</code>% steps). Can rescue images that fail to index on the full spot list. Especially valuble for multi-lattice images (multiple crystals in the beam) but does increase execution time!<br />
|}<br />
<br />
== Integration ==<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>profile.gaussian_rs.parameters.sigma_b_cutoff</code> || 0.1 || Maximum beam divergence (sigma_b, degrees) before an indexed crystal is rejected prior to integration. Guards against over-broad, poorly-modelled lattices.<br />
|-<br />
| <code>significance_filter.enable</code> || False || Apply the stills significance filter, which trims each crystal to the resolution at which reflections remain statistically significant (see <code>significance_filter.isigi_cutoff</code>, default 1.0). Defaults to false here because it is typically applied in ''[[cctbx.xfel.merge]]''.<br />
|-<br />
| <code>integration.integration_only_overrides.trusted_range</code> || None || Override the panel trusted range [min, max] during integration only (e.g. to include pixels the header marks as untrusted).<br />
|-<br />
| <code>integration.coset.transformation</code> || 6 || Which modulus-2 sublattice transformation defines the coset negative control when <code>dispatch.coset=True</code> (0–6; 6 = body centering). See Sauter & Zwart, ''Acta Cryst.'' D (2009) '''65''':553.<br />
|}<br />
<br />
== Output ==<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>output.output_dir</code> || <code>.</code> || Directory for output files.<br />
|-<br />
| <code>output.composite_output</code> || True || If True, each process writes ''one'' concatenated set of <code>.expt</code>/<code>.refl</code> files covering all its successful images. If False, a separate pair of files is written per image (generates a very large number of files). Composite output is strongly preferred for large serial datasets.<br />
|-<br />
| <code>output.logging_dir</code> || None || Directory for log files. Under MPI, per-rank logs (<code>log_rankNNNN.out</code>, etc.) are written here.<br />
|-<br />
| <code>output.logging_option</code> || normal || <code>normal</code> (all logging), <code>suppressed</code> (silences verbose DIALS refine/index output), or <code>disabled</code> (almost no logging).<br />
|-<br />
| <code>output.*_filename</code> || (templates) || Output filename templates for each stage: <code>strong</code>, <code>indexed</code>, <code>refined_experiments</code>, <code>integrated</code>, <code>integrated_experiments</code>, <code>coset</code>. A <code>%s</code> in the template is replaced with the process number for composite mode or the per-image tag for non-composite mode.<br />
|-<br />
| <code>output.integration_pickle</code> || None || Write legacy <code>cxi.merge</code>/PRIME integration pickle files (e.g. <code>int-%d-%s.pickle</code>). Only needed for legacy merging workflows.<br />
|}<br />
<br />
== Multiprocessing ==<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>mp.method</code> || multiprocessing || Parallelization backend: <code>multiprocessing</code> (single node), or <code>mpi</code> for multi-node jobs (also <code>sge</code>, <code>lsf</code>, <code>pbs</code>).<br />
|-<br />
| <code>mp.nproc</code> || 1 || Number of processes (for the <code>multiprocessing</code> method). Set to the number of cores per node.<br />
|-<br />
| <code>mp.composite_stride</code> || None || For MPI composite output, how many ranks aggregate their results into one set of output files. Ideally set to the number of processes per node so that one composite file set is written per node rather than per rank.<br />
|}<br />
<br />
Under MPI with more than two ranks, the program runs a client/server model: rank 0 (or rank 2 for<br />
psana2 <code>.loc</code> inputs) hands out images to worker ranks on demand, giving good load<br />
balancing across the many fast-failing non-hits typical of serial data.<br />
<br />
= Output files =<br />
<br />
With the default composite output, each process (or each aggregating rank under MPI) writes<br />
concatenated files such as:<br />
<br />
* <code>&lt;tag&gt;_strong.refl</code> — strong spots from spot-finding<br />
* <code>&lt;tag&gt;_indexed.refl</code> and <code>&lt;tag&gt;_refined.expt</code> — indexed reflections and their crystal models<br />
* <code>&lt;tag&gt;_integrated.refl</code> and <code>&lt;tag&gt;_integrated.expt</code> — integrated intensities and final models (the input to ''[[cctbx.xfel.merge]]'')<br />
<br />
Per-image progress and status codes are recorded in <code>debug/debug_&lt;rank&gt;.txt</code>, and<br />
overall logging goes to <code>dials.process.log</code> (or per-rank logs under MPI). These integrated<br />
<code>.expt</code>/<code>.refl</code> pairs are then scaled and merged with<br />
''[[cctbx.xfel.merge]]''.<br />
<br />
= See also =<br />
<br />
* ''[[cctbx.xfel.merge]]'' — scaling, post-refinement and merging of the integrated data<br />
* The [http://dials.github.io DIALS documentation] for the underlying spot-finding, indexing, refinement and integration algorithms</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Main_Page&diff=1786Main Page2026-08-21T23:32:10Z<p>Aaron: /* cctbx.xfel resources */</p>
<hr />
<div>News! See our [[https://doi.org/10.1101/2025.05.04.652045 new preprint]]! This paper shows current best practices using ''cctbx.xfel'' with ''DIALS'' for serial crystallographic data.<br />
<br />
= Open-source tools for serial crystallographic data processing =<br />
<br />
''cctbx.xfel'' is a suite of software tools designed to process diffraction data from serial femtosecond crystallography (SFX) measurements at an X-ray free-electron laser (XFEL) or a synchrotron. Built on the Computational Crystallographic Toolbox ([https://cctbx.github.io/ ''cctbx'']), the same toolbox on which [http://www.phenix-online.org ''PHENIX''], and [http://dials.github.io DIALS] are built, it enables the user to solve difficult problems relating to processing serial crystallographic data. The programs and modules provided by ''cctbx.xfel'' can reduce a large set of still diffraction images recorded at Stanford’s Linac Coherent Light Source ([http://lcls.slac.stanford.edu LCLS]), [http://sacla.xfel.jp SACLA], the [https://www.xfel.eu/ European XFEL], or a synchrotron, to a single MTZ file containing merged reflection intensities suitable for structure solution.<br />
<br />
== ''cctbx.xfel'' resources ==<br />
<br />
The current best resource is our pre-print, [https://doi.org/10.1101/2025.05.04.652045 ''cctbx.xfel: a suite for processing serial crystallographic data.'']. This paper describes the algorithms and ideas behind the implementation and execution of the two main programs of ''cctbx.xfel'':<br />
<br />
* ''[[Dials.stills_process]]'': this program runs through all the first steps of processing data using dials: import (reading image headers), spot finding, indexing, geometry refinement, and integration. Serial images are independent from each other, so this program is optimized for massively parallel systems, but can be adapted to a variety of single and multi-processing modes.<br />
* ''[[Cctbx.xfel.merge]]'': this program runs through the remaining steps of processing: scaling and merging to produce a full dataset (.mtz file). For documentation, see our [https://github.com/cctbx/cctbx_project/tree/master/xfel/merging Github project]<br />
<br />
Additional resources:<br />
<br />
* Our [http://cci.lbl.gov/publications/download/CCN_2019_p22_Brewster.pdf 2019 newsletter article] describes how to use the ''cctbx.xfel'' GUI, an interactive program for handling large XFEL datasets.<br />
<br />
* [[Overview]] to the system architecture at LCLS and real-time progress monitoring of data processing<br />
* Installation: there are two ways to get ''cctbx.xfel'':<br />
** ''cctbx.xfel'' is installed for general use at SLAC. To start using an existing installation, [[Using the pre-built cctbx at LCLS|simply source the current build]]. <br />
** [https://dials.github.io/installation.html Download] a binary bundle of DIALS. Use this option if you will not be processing LCLS data because, for example, if you have serial crystallographic data from a synchrotron.<br />
** Build a psana/cctbx.xfel build [https://github.com/cctbx/cctbx_project/tree/master/xfel/conda_envs from scratch]. Use this option if you want to process LCLS data outside of LCLS or want your own build.<br />
* [[Tutorials]] on pre-processing, data reduction, and merging. <br />
<br />
Other related information:<br />
<br />
* [[Serial XFEL Crystallography References]]<br />
<br />
This project is under active development. For any assistance, please contact the authors.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Cctbx.xfel.merge&diff=1785Cctbx.xfel.merge2026-08-21T23:26:17Z<p>Aaron: /* Basic usage */</p>
<hr />
<div>= ''cctbx.xfel.merge'' =<br />
<br />
'''''cctbx.xfel.merge''''' is the scaling, post-refinement and merging program of cctbx.xfel. It takes<br />
the per-image integrated results produced by ''[[dials.stills_process]]'' — one crystal per still shot —<br />
and reduces the whole ensemble to a single set of merged structure-factor intensities (an MTZ file)<br />
suitable for structure solution and refinement, together with the crystallographic quality<br />
statistics (completeness, multiplicity, ⟨I/σ⟩, CC½, CC<sub>iso</sub>, R-factors, etc.).<br />
<br />
Because serial datasets routinely contain tens of thousands to millions of still shots,<br />
<code>cctbx.xfel.merge</code> is built from the ground up to run under MPI across many nodes. It is<br />
almost always launched with <code>mpirun</code>/<code>srun</code>. Note that MPI uses the term 'ranks'<br />
to distinguish independent processes that are running across one or more nodes.<br />
<br />
== Overview ==<br />
<br />
<code>cctbx.xfel.merge</code> is organized as a sequence of '''workers'''. Each worker performs one<br />
task — reading data, applying a correction, filtering, scaling, post-refining, computing a statistic,<br />
or merging — and passes its (possibly modified) experiments and reflections on to the next worker.<br />
The list of workers to run is the "step list", and it is fully configurable: you can add, remove or<br />
reorder steps with <code>dispatch.step_list</code>. This is what makes the program flexible enough to<br />
scale-and-merge in one pass, to scale subsets separately and merge later, or to run a reference-free<br />
bootstrap.<br />
<br />
The '''default''' step list is:<br />
<br />
{| class="wikitable"<br />
! # !! Worker !! Purpose<br />
|-<br />
| 1 || <code>input</code> || Read integrated experiments (<code>.expt</code>) and reflections (<code>.refl</code>) from disk in DIALS format.<br />
|-<br />
| 2 || <code>balance</code> || Redistribute the input load evenly across MPI ranks (needed when there are fewer or more input files than ranks).<br />
|-<br />
| 3 || <code>model_scaling</code> || Build the full Miller list, model intensities from the reference, and the resolution binner used for scaling and post-refinement.<br />
|-<br />
| 4 || <code>modify</code> || Apply per-reflection modifications, e.g. polarization correction (and optionally re-indexing to resolve indexing ambiguity — see <code>modify.cosym</code>/<code>reindex_to_reference</code>).<br />
|-<br />
| 5 || <code>filter</code> || Reject whole experiments (or reflections) by resolution, unit cell, correlation to the reference, energy, etc.<br />
|-<br />
| 6 || <code>scale</code> || Scale each lattice to the reference dataset (<code>mark0</code>) or simply average (<code>mark1</code>).<br />
|-<br />
| 7 || <code>postrefine</code> || Post-refine per-image crystal parameters and partialities (Sauter 2015).<br />
|-<br />
| 8 || <code>statistics_unitcell</code> || Determine and store the average unit cell.<br />
|-<br />
| 9 || <code>statistics_beam</code> || Determine and store the average wavelength.<br />
|-<br />
| 10 || <code>model_statistics</code> || Rebuild the Miller list / binner for statistics (may use the average unit cell).<br />
|-<br />
| 11 || <code>statistics_resolution</code> || Per-crystal resolution statistics.<br />
|-<br />
| 12 || <code>group</code> || Redistribute reflections across ranks so that every observation of a given HKL is gathered on the same rank prior to merging.<br />
|-<br />
| 13 || <code>errors_merge</code> || Correct the merged error estimates with a per-HKL error model (e.g. Ev11, Brewster 2019b, or MM24 (the default), Mittan-Moreau (2025)).<br />
|-<br />
| 14 || <code>statistics_intensity</code> || Intensity statistics by resolution.<br />
|-<br />
| 15 || <code>merge</code> || Merge symmetry-equivalent intensities, gather all HKLs at rank 0, and write "odd", "even" and "all" MTZ files.<br />
|-<br />
| 16 || <code>statistics_intensity_cxi</code> || Correlation and R-factor statistics (ported from cxi.merge): the "Table of Scaling Results" — CC½ (half-dataset correlation, labelled "CC int"), CC<sub>iso</sub>, R<sub>int</sub>, R<sub>split</sub>, R<sub>iso</sub>.<br />
|}<br />
<br />
Additional, non-default workers are available and can be<br />
inserted into the step list, including <code>tdata</code> (write out unit cells for clustering),<br />
<code>export</code> (batch MTZ for ''Careless''), <code>publish</code> (upload merged files to a<br />
Google Drive folder), <code>monitor</code> (resource-usage logging), <code>integrate</code><br />
(on-the-fly integration), <code>prepare</code> (SpReAD energy binning), and<br />
<code>lunus</code> (diffuse-scattering deck). A <code>deltaccint</code> worker (ΔCC½ lattice<br />
filtering) can be added after <code>group</code>. Custom workers can be supplied through the<br />
<code>XFEL_CUSTOM_WORKER_PATH</code> environment variable.<br />
<br />
== Basic usage ==<br />
<br />
<code>cctbx.xfel.merge</code> is run under MPI:<br />
<br />
<pre><br />
mpirun cctbx.xfel.merge merge.phil<br />
</pre><br />
<br />
A typical <code>merge.phil</code>:<br />
<br />
<pre><br />
input.path=<path to data><br />
input.experiments_suffix=.expt<br />
input.reflections_suffix=.refl<br />
<br />
filter.algorithm=unit_cell<br />
filter.unit_cell.value.relative_length_tolerance=0.03<br />
filter.outlier.min_corr=0.1<br />
<br />
select.algorithm=significance_filter<br />
select.significance_filter.sigma=0.1<br />
select.significance_filter.min_ct=200<br />
select.significance_filter.max_ct=300<br />
<br />
scaling.model=<model.pdb or model.mtz><br />
scaling.resolution_scalar=0.96<br />
<br />
merging.d_min=1.8<br />
merging.error.model=mm24<br />
merging.merge_anomalous=True<br />
<br />
postrefinement.enable=True<br />
statistics.n_bins=20<br />
<br />
output.output_dir=<output folder><br />
output.prefix=<prefix for output files><br />
</pre><br />
<br />
Explore the parameters with:<br />
* <code>cctbx.xfel.merge -c</code> — show configurable parameters<br />
* <code>cctbx.xfel.merge -c -e 10 -a 2</code> — show all parameters (expert level 10) with full help text<br />
<br />
== Two-stage scaling and merging ==<br />
<br />
For large, growing datasets (the mode used by the [[cctbx.xfel GUI]]), it is more efficient to scale<br />
and post-refine each subset of data '''once''', then merge repeatedly as more data arrive. This is<br />
done by running the program twice with different step lists.<br />
<br />
'''Stage 1 — scale each subset''' (stops after per-image work and writes intermediate DIALS files):<br />
<br />
<pre><br />
dispatch.step_list=input balance model_scaling modify filter scale postrefine \<br />
statistics_unitcell statistics_beam model_statistics statistics_resolution<br />
output.save_experiments_and_reflections=True<br />
</pre><br />
<br />
<code>output.save_experiments_and_reflections=True</code> dumps the scaled/post-refined data as<br />
experiment lists and reflection tables that the merge stage can read back in.<br />
<br />
'''Stage 2 — merge the scaled subsets''' (add an <code>input.path</code> for each finished subset):<br />
<br />
<pre><br />
dispatch.step_list=input model_scaling statistics_unitcell statistics_beam \<br />
model_statistics statistics_resolution group errors_merge statistics_intensity \<br />
merge statistics_intensity_cxi<br />
</pre><br />
<br />
== Merging without a reference (bootstrap / mark1) ==<br />
<br />
If no scaling/merging reference is available, use a bootstrap procedure. First run the<br />
<code>mark1</code> algorithm, which simply averages the data (all per-image scale factors = 1, no<br />
post-refinement), specifying the target cell and symmetry explicitly:<br />
<br />
<pre><br />
scaling.algorithm=mark1<br />
scaling.unit_cell=<unit cell><br />
scaling.space_group=<space group><br />
merging.error.model=errors_from_sample_residuals<br />
</pre><br />
<br />
The resulting MTZ is then used as the reference (<code>scaling.model=</code>) for subsequent regular<br />
merges. Iterate a few cycles; CC½ typically improves and converges over the first few rounds.<br />
<br />
== Important parameters ==<br />
<br />
Parameters are given by their full PHIL path with defaults in parentheses. The program exposes many<br />
more parameters (see <code>-c -e 10 -a 2</code>); those below are the ones most commonly set.<br />
<br />
=== Input ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>input.path</code> || None || Path(s) to the integrated data — a folder, several folders, or individual files; wildcards allowed; may be repeated. Matching <code>.expt</code> and <code>.refl</code> files must both be present.<br />
|-<br />
| <code>input.experiments_suffix</code> || <code>_integrated.expt</code> || Only load experiment files ending with this suffix.<br />
|-<br />
| <code>input.reflections_suffix</code> || <code>_integrated.refl</code> || Only load reflection files ending with this suffix.<br />
|-<br />
| <code>input.parallel_file_load.method</code> || uniform || <code>uniform</code> spreads input files evenly over all ranks; <code>node_memory</code> distributes so per-node memory limits are respected.<br />
|-<br />
| <code>input.alist.file</code> || None || Path to a text file of experiment tags or filenames; combined with <code>input.alist.op=keep|reject</code> to merge only (or exclude) a specified subset.<br />
|-<br />
| <code>input.persistent_refl_cols</code> || None || Reflection-table columns to preserve through the pruning steps (and into saved tables).<br />
|}<br />
<br />
=== Filtering whole experiments (<code>filter</code>) ===<br />
<br />
The <code>filter</code> section accepts or rejects '''entire experiments'''.<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>filter.algorithm</code> || (none) || One or more of <code>n_obs</code>, <code>resolution</code>, <code>unit_cell</code>, <code>energy</code>, <code>bootstrap</code>, <code>experiment_prefix</code>. Nothing is filtered unless listed.<br />
|-<br />
| <code>filter.unit_cell.algorithm</code> || value || How to restrict unit cells: <code>value</code> (compare to a target), <code>range</code>, or <code>cluster</code> (use a precomputed covariance clustering).<br />
|-<br />
| <code>filter.unit_cell.value.relative_length_tolerance</code> || 0.1 || Fractional deviation of cell edge lengths allowed vs. the target cell. Often tightened (e.g. 0.03).<br />
|-<br />
| <code>filter.unit_cell.value.absolute_angle_tolerance</code> || 2.0 || Allowed cell-angle deviation (degrees) vs. the target.<br />
|-<br />
| <code>filter.unit_cell.value.target_unit_cell</code> / <code>target_space_group</code> || Auto || Target cell/symmetry; <code>Auto</code> uses the scaling model.<br />
|-<br />
| <code>filter.outlier.min_corr</code> || 0.1 || Minimum correlation of an image's intensities to the reference model; images below this are rejected. Only applied when a scaling model is present. Use <code>-1</code> to accept all images regardless of correlation.<br />
|-<br />
| <code>filter.resolution.d_min</code> || None || Reject an experiment unless some of its reflections extend beyond this resolution.<br />
|}<br />
<br />
=== Selecting individual reflections (<code>select</code>) ===<br />
<br />
The <code>select</code> section accepts or rejects '''individual reflections''' (contrast with<br />
<code>filter</code>, which acts on whole experiments).<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>select.algorithm</code> || (none) || One or more of <code>significance_filter</code>, <code>panel</code>, <code>cspad_sensor</code>, <code>isolation_forest</code>, <code>target_energy</code>.<br />
|-<br />
| <code>select.significance_filter.sigma</code> || 0.5 || Trim each lattice at the resolution where the binned ⟨I/σ⟩ drops below this value. Often reduced to 0.1.<br />
|-<br />
| <code>select.significance_filter.min_ct</code> / <code>max_ct</code> || 10 / 50 || Target minimum/maximum mean reflections per resolution bin when choosing the number of bins. For large unit cells these are often raised (e.g. 200/300); for small cells, use the defaults.<br />
|}<br />
<br />
=== Modifying intensities (<code>modify</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>modify.algorithm</code> || polarization || Per-reflection corrections to apply (default applies the polarization correction).<br />
|-<br />
| <code>modify.cosym</code> || — || Resolve an indexing ambiguity across the ensemble using the Gildea/Winter (Brehm/Diederichs) cosym method, optionally anchored to <code>scaling.model</code>.<br />
|-<br />
| <code>modify.reindex_to_reference</code> / <code>reindex_to_abc</code> || — || Alternative re-indexing operators (to a reference model, or an explicit operator).<br />
|}<br />
<br />
=== Scaling (<code>scaling</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>scaling.model</code> || None || Reference dataset: a PDB file (uses the CRYST1 cell and computes model intensities, including bulk solvent) or an MTZ from a previous cycle. Required for <code>mark0</code> scaling.<br />
|-<br />
| <code>scaling.algorithm</code> || mark0 || <code>mark0</code>: per-image scaling against the reference. <code>mark1</code>: no scaling, just Monte-Carlo averaging (all scale factors = 1) — used for reference-free bootstrap.<br />
|-<br />
| <code>scaling.unit_cell</code> / <code>scaling.space_group</code> || None || Cell/symmetry to use when no model is provided (i.e. <code>mark1</code>).<br />
|-<br />
| <code>scaling.resolution_scalar</code> || 0.969 || <code>merging.d_min</code> is multiplied by this when deciding which reflections fall within the resolution limit, generating a few extra reference reflections to account for per-shot cell variation (e.g. set 0.96 for a few more).<br />
|-<br />
| <code>scaling.mtz.mtz_column_F</code> || fobs || Column name in the reference MTZ containing the reference structure factors (intensities or amplitudes).<br />
|}<br />
<br />
=== Post-refinement (<code>postrefinement</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>postrefinement.enable</code> || True || Enable post-refinement of per-image parameters/partialities (Sauter 2015).<br />
|-<br />
| <code>postrefinement.algorithm</code> || rs || Post-refinement protocol: <code>rs</code>, <code>rs2</code>, <code>rs_hybrid</code>, or <code>eta_deff</code>.<br />
|}<br />
<br />
=== Merging and error model (<code>merging</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>merging.d_min</code> || None || High-resolution limit for scaling and merging.<br />
|-<br />
| <code>merging.d_max</code> || None || Low-resolution limit (currently affects only CCiso).<br />
|-<br />
| <code>merging.merge_anomalous</code> || False || <code>True</code> merges Bijvoet mates together (higher multiplicity); <code>False</code> keeps them separate to preserve the anomalous signal.<br />
|-<br />
| <code>merging.minimum_multiplicity</code> || 2 || Do not produce a merged structure factor for HKLs measured fewer than this many times.<br />
|-<br />
| <code>merging.error.model</code> || mm24 || Merged-sigma error model: <code>mm24</code> (Mittan-Moreau 2025, current default), <code>ev11</code> (Evans 2011 / sdfac-refine), <code>ha14</code>, or <code>errors_from_sample_residuals</code> (no refined error terms — used with <code>mark1</code> bootstrap).<br />
|-<br />
| <code>merging.set_average_unit_cell</code> || True || Output MTZ adopts the average unit cell of the data rather than the reference model's cell.<br />
|}<br />
<br />
=== Statistics (<code>statistics</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>statistics.n_bins</code> || 10 || Number of resolution bins in the statistics tables. Often raised to 20.<br />
|-<br />
| <code>statistics.cciso.mtz_file</code> || None || Isomorphous reference structure factors for R<sub>iso</sub> / CC<sub>iso</sub>.<br />
|-<br />
| <code>statistics.report_ML</code> || True || Report per-frame max-likelihood attributes (domain size, mosaicity).<br />
|}<br />
<br />
=== Output (<code>output</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>output.output_dir</code> || <code>.</code> || Directory for output files (created if it does not exist).<br />
|-<br />
| <code>output.prefix</code> || iobs || Prefix prepended to all output file names.<br />
|-<br />
| <code>output.save_experiments_and_reflections</code> || False || Dump the final experiments/reflections from the last worker as DIALS files — used for the two-stage scale-then-merge workflow.<br />
|-<br />
| <code>output.do_timing</code> || False || Log elapsed time per execution step.<br />
|-<br />
| <code>output.expanded_bookkeeping</code> || False || Store extra provenance columns (input file/reflection indices, odd/even flag, original id) in the saved reflection tables.<br />
|}<br />
<br />
=== Multiprocessing (<code>mp</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>mp.method</code> || mpi || Multiprocessing method (only <code>mpi</code> is currently supported).<br />
|-<br />
| <code>mp.debug.cProfile</code> || False || Enable code profiling; a <code>.prof</code> file is written per rank.<br />
|}<br />
<br />
== Output files ==<br />
<br />
The <code>merge</code> worker gathers all HKLs at rank 0 and writes three MTZ files, prefixed with<br />
<code>output.prefix</code> (default <code>iobs</code>):<br />
<br />
* <code>&lt;prefix&gt;_all.mtz</code> — the merged dataset using all observations (the main result for structure solution)<br />
* <code>&lt;prefix&gt;_odd.mtz</code> and <code>&lt;prefix&gt;_even.mtz</code> — the two half-datasets used for CC½ and related half-dataset statistics<br />
<br />
A main log records the per-step progress and the merging statistics tables by resolution bin:<br />
completeness, multiplicity and ⟨I/σ⟩, plus the "Table of Scaling Results" reporting CC½ (the<br />
half-dataset correlation, labelled "CC int"), CC<sub>iso</sub>, R<sub>int</sub>, R<sub>split</sub><br />
and R<sub>iso</sub>. The merged MTZ can optionally be<br />
uploaded to a shared [https://github.com/cctbx/cctbx_project/tree/master/xfel/merging/application/publish Google Drive folder] by adding the <code>publish</code> worker.<br />
<br />
== References ==<br />
<br />
* Sauter NK (2015). ''J Synchrotron Radiat'' '''22''', 239–48. (post-refinement)<br />
* Uervirojnangkoorn M, ''et al.'' (2015). ''eLife'' 10.7554/eLife.05421. (PRIME / bootstrap)<br />
* Brewster AS, ''et al.'' (2018). ''Acta Cryst.'' D '''74''', 877–894.<br />
* Brewster AS, ''et al.'' (2019a). ''Computational Crystallography Newsletter'' '''10''', 22–39. (XFEL GUI)<br />
* Brewster AS, ''et al.'' (2019b). ''Acta Cryst.'' D '''75''', 959–968. (error models)<br />
* Mittan-Moreau (2025). (mm24 error model)<br />
<br />
== See also ==<br />
<br />
* [[dials.stills_process]] — produces the integrated data that <code>cctbx.xfel.merge</code> consumes<br />
* [[cctbx.xfel GUI]] — orchestrates separate scaling and repeated merging as data are collected<br />
* The program README at <code>xfel/merging/README.md</code> in cctbx_project</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Cctbx.xfel.merge&diff=1784Cctbx.xfel.merge2026-08-21T23:25:39Z<p>Aaron: /* Statistics (statistics) */</p>
<hr />
<div>= ''cctbx.xfel.merge'' =<br />
<br />
'''''cctbx.xfel.merge''''' is the scaling, post-refinement and merging program of cctbx.xfel. It takes<br />
the per-image integrated results produced by ''[[dials.stills_process]]'' — one crystal per still shot —<br />
and reduces the whole ensemble to a single set of merged structure-factor intensities (an MTZ file)<br />
suitable for structure solution and refinement, together with the crystallographic quality<br />
statistics (completeness, multiplicity, ⟨I/σ⟩, CC½, CC<sub>iso</sub>, R-factors, etc.).<br />
<br />
Because serial datasets routinely contain tens of thousands to millions of still shots,<br />
<code>cctbx.xfel.merge</code> is built from the ground up to run under MPI across many nodes. It is<br />
almost always launched with <code>mpirun</code>/<code>srun</code>. Note that MPI uses the term 'ranks'<br />
to distinguish independent processes that are running across one or more nodes.<br />
<br />
== Overview ==<br />
<br />
<code>cctbx.xfel.merge</code> is organized as a sequence of '''workers'''. Each worker performs one<br />
task — reading data, applying a correction, filtering, scaling, post-refining, computing a statistic,<br />
or merging — and passes its (possibly modified) experiments and reflections on to the next worker.<br />
The list of workers to run is the "step list", and it is fully configurable: you can add, remove or<br />
reorder steps with <code>dispatch.step_list</code>. This is what makes the program flexible enough to<br />
scale-and-merge in one pass, to scale subsets separately and merge later, or to run a reference-free<br />
bootstrap.<br />
<br />
The '''default''' step list is:<br />
<br />
{| class="wikitable"<br />
! # !! Worker !! Purpose<br />
|-<br />
| 1 || <code>input</code> || Read integrated experiments (<code>.expt</code>) and reflections (<code>.refl</code>) from disk in DIALS format.<br />
|-<br />
| 2 || <code>balance</code> || Redistribute the input load evenly across MPI ranks (needed when there are fewer or more input files than ranks).<br />
|-<br />
| 3 || <code>model_scaling</code> || Build the full Miller list, model intensities from the reference, and the resolution binner used for scaling and post-refinement.<br />
|-<br />
| 4 || <code>modify</code> || Apply per-reflection modifications, e.g. polarization correction (and optionally re-indexing to resolve indexing ambiguity — see <code>modify.cosym</code>/<code>reindex_to_reference</code>).<br />
|-<br />
| 5 || <code>filter</code> || Reject whole experiments (or reflections) by resolution, unit cell, correlation to the reference, energy, etc.<br />
|-<br />
| 6 || <code>scale</code> || Scale each lattice to the reference dataset (<code>mark0</code>) or simply average (<code>mark1</code>).<br />
|-<br />
| 7 || <code>postrefine</code> || Post-refine per-image crystal parameters and partialities (Sauter 2015).<br />
|-<br />
| 8 || <code>statistics_unitcell</code> || Determine and store the average unit cell.<br />
|-<br />
| 9 || <code>statistics_beam</code> || Determine and store the average wavelength.<br />
|-<br />
| 10 || <code>model_statistics</code> || Rebuild the Miller list / binner for statistics (may use the average unit cell).<br />
|-<br />
| 11 || <code>statistics_resolution</code> || Per-crystal resolution statistics.<br />
|-<br />
| 12 || <code>group</code> || Redistribute reflections across ranks so that every observation of a given HKL is gathered on the same rank prior to merging.<br />
|-<br />
| 13 || <code>errors_merge</code> || Correct the merged error estimates with a per-HKL error model (e.g. Ev11, Brewster 2019b, or MM24 (the default), Mittan-Moreau (2025)).<br />
|-<br />
| 14 || <code>statistics_intensity</code> || Intensity statistics by resolution.<br />
|-<br />
| 15 || <code>merge</code> || Merge symmetry-equivalent intensities, gather all HKLs at rank 0, and write "odd", "even" and "all" MTZ files.<br />
|-<br />
| 16 || <code>statistics_intensity_cxi</code> || Correlation and R-factor statistics (ported from cxi.merge): the "Table of Scaling Results" — CC½ (half-dataset correlation, labelled "CC int"), CC<sub>iso</sub>, R<sub>int</sub>, R<sub>split</sub>, R<sub>iso</sub>.<br />
|}<br />
<br />
Additional, non-default workers are available and can be<br />
inserted into the step list, including <code>tdata</code> (write out unit cells for clustering),<br />
<code>export</code> (batch MTZ for ''Careless''), <code>publish</code> (upload merged files to a<br />
Google Drive folder), <code>monitor</code> (resource-usage logging), <code>integrate</code><br />
(on-the-fly integration), <code>prepare</code> (SpReAD energy binning), and<br />
<code>lunus</code> (diffuse-scattering deck). A <code>deltaccint</code> worker (ΔCC½ lattice<br />
filtering) can be added after <code>group</code>. Custom workers can be supplied through the<br />
<code>XFEL_CUSTOM_WORKER_PATH</code> environment variable.<br />
<br />
== Basic usage ==<br />
<br />
<code>cctbx.xfel.merge</code> is run under MPI:<br />
<br />
<pre><br />
mpirun cctbx.xfel.merge merge.phil<br />
</pre><br />
<br />
A typical <code>merge.phil</code>:<br />
<br />
<pre><br />
input.path=<path to data><br />
input.experiments_suffix=.expt<br />
input.reflections_suffix=.refl<br />
<br />
filter.algorithm=unit_cell<br />
filter.unit_cell.value.relative_length_tolerance=0.03<br />
filter.outlier.min_corr=0.1<br />
<br />
select.algorithm=significance_filter<br />
select.significance_filter.sigma=0.1<br />
select.significance_filter.min_ct=200<br />
select.significance_filter.max_ct=300<br />
<br />
scaling.model=<model.pdb or model.mtz><br />
scaling.resolution_scalar=0.96<br />
<br />
merging.d_min=1.8<br />
merging.error.model=mm24<br />
merging.merge_anomalous=True<br />
<br />
postrefinement.enable=True<br />
statistics.n_bins=20<br />
<br />
output.do_timing=True<br />
output.output_dir=<output folder><br />
output.prefix=<prefix for output files><br />
</pre><br />
<br />
Explore the parameters with:<br />
* <code>cctbx.xfel.merge -c</code> — show configurable parameters<br />
* <code>cctbx.xfel.merge -c -e 10 -a 2</code> — show all parameters (expert level 10) with full help text<br />
<br />
== Two-stage scaling and merging ==<br />
<br />
For large, growing datasets (the mode used by the [[cctbx.xfel GUI]]), it is more efficient to scale<br />
and post-refine each subset of data '''once''', then merge repeatedly as more data arrive. This is<br />
done by running the program twice with different step lists.<br />
<br />
'''Stage 1 — scale each subset''' (stops after per-image work and writes intermediate DIALS files):<br />
<br />
<pre><br />
dispatch.step_list=input balance model_scaling modify filter scale postrefine \<br />
statistics_unitcell statistics_beam model_statistics statistics_resolution<br />
output.save_experiments_and_reflections=True<br />
</pre><br />
<br />
<code>output.save_experiments_and_reflections=True</code> dumps the scaled/post-refined data as<br />
experiment lists and reflection tables that the merge stage can read back in.<br />
<br />
'''Stage 2 — merge the scaled subsets''' (add an <code>input.path</code> for each finished subset):<br />
<br />
<pre><br />
dispatch.step_list=input model_scaling statistics_unitcell statistics_beam \<br />
model_statistics statistics_resolution group errors_merge statistics_intensity \<br />
merge statistics_intensity_cxi<br />
</pre><br />
<br />
== Merging without a reference (bootstrap / mark1) ==<br />
<br />
If no scaling/merging reference is available, use a bootstrap procedure. First run the<br />
<code>mark1</code> algorithm, which simply averages the data (all per-image scale factors = 1, no<br />
post-refinement), specifying the target cell and symmetry explicitly:<br />
<br />
<pre><br />
scaling.algorithm=mark1<br />
scaling.unit_cell=<unit cell><br />
scaling.space_group=<space group><br />
merging.error.model=errors_from_sample_residuals<br />
</pre><br />
<br />
The resulting MTZ is then used as the reference (<code>scaling.model=</code>) for subsequent regular<br />
merges. Iterate a few cycles; CC½ typically improves and converges over the first few rounds.<br />
<br />
== Important parameters ==<br />
<br />
Parameters are given by their full PHIL path with defaults in parentheses. The program exposes many<br />
more parameters (see <code>-c -e 10 -a 2</code>); those below are the ones most commonly set.<br />
<br />
=== Input ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>input.path</code> || None || Path(s) to the integrated data — a folder, several folders, or individual files; wildcards allowed; may be repeated. Matching <code>.expt</code> and <code>.refl</code> files must both be present.<br />
|-<br />
| <code>input.experiments_suffix</code> || <code>_integrated.expt</code> || Only load experiment files ending with this suffix.<br />
|-<br />
| <code>input.reflections_suffix</code> || <code>_integrated.refl</code> || Only load reflection files ending with this suffix.<br />
|-<br />
| <code>input.parallel_file_load.method</code> || uniform || <code>uniform</code> spreads input files evenly over all ranks; <code>node_memory</code> distributes so per-node memory limits are respected.<br />
|-<br />
| <code>input.alist.file</code> || None || Path to a text file of experiment tags or filenames; combined with <code>input.alist.op=keep|reject</code> to merge only (or exclude) a specified subset.<br />
|-<br />
| <code>input.persistent_refl_cols</code> || None || Reflection-table columns to preserve through the pruning steps (and into saved tables).<br />
|}<br />
<br />
=== Filtering whole experiments (<code>filter</code>) ===<br />
<br />
The <code>filter</code> section accepts or rejects '''entire experiments'''.<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>filter.algorithm</code> || (none) || One or more of <code>n_obs</code>, <code>resolution</code>, <code>unit_cell</code>, <code>energy</code>, <code>bootstrap</code>, <code>experiment_prefix</code>. Nothing is filtered unless listed.<br />
|-<br />
| <code>filter.unit_cell.algorithm</code> || value || How to restrict unit cells: <code>value</code> (compare to a target), <code>range</code>, or <code>cluster</code> (use a precomputed covariance clustering).<br />
|-<br />
| <code>filter.unit_cell.value.relative_length_tolerance</code> || 0.1 || Fractional deviation of cell edge lengths allowed vs. the target cell. Often tightened (e.g. 0.03).<br />
|-<br />
| <code>filter.unit_cell.value.absolute_angle_tolerance</code> || 2.0 || Allowed cell-angle deviation (degrees) vs. the target.<br />
|-<br />
| <code>filter.unit_cell.value.target_unit_cell</code> / <code>target_space_group</code> || Auto || Target cell/symmetry; <code>Auto</code> uses the scaling model.<br />
|-<br />
| <code>filter.outlier.min_corr</code> || 0.1 || Minimum correlation of an image's intensities to the reference model; images below this are rejected. Only applied when a scaling model is present. Use <code>-1</code> to accept all images regardless of correlation.<br />
|-<br />
| <code>filter.resolution.d_min</code> || None || Reject an experiment unless some of its reflections extend beyond this resolution.<br />
|}<br />
<br />
=== Selecting individual reflections (<code>select</code>) ===<br />
<br />
The <code>select</code> section accepts or rejects '''individual reflections''' (contrast with<br />
<code>filter</code>, which acts on whole experiments).<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>select.algorithm</code> || (none) || One or more of <code>significance_filter</code>, <code>panel</code>, <code>cspad_sensor</code>, <code>isolation_forest</code>, <code>target_energy</code>.<br />
|-<br />
| <code>select.significance_filter.sigma</code> || 0.5 || Trim each lattice at the resolution where the binned ⟨I/σ⟩ drops below this value. Often reduced to 0.1.<br />
|-<br />
| <code>select.significance_filter.min_ct</code> / <code>max_ct</code> || 10 / 50 || Target minimum/maximum mean reflections per resolution bin when choosing the number of bins. For large unit cells these are often raised (e.g. 200/300); for small cells, use the defaults.<br />
|}<br />
<br />
=== Modifying intensities (<code>modify</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>modify.algorithm</code> || polarization || Per-reflection corrections to apply (default applies the polarization correction).<br />
|-<br />
| <code>modify.cosym</code> || — || Resolve an indexing ambiguity across the ensemble using the Gildea/Winter (Brehm/Diederichs) cosym method, optionally anchored to <code>scaling.model</code>.<br />
|-<br />
| <code>modify.reindex_to_reference</code> / <code>reindex_to_abc</code> || — || Alternative re-indexing operators (to a reference model, or an explicit operator).<br />
|}<br />
<br />
=== Scaling (<code>scaling</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>scaling.model</code> || None || Reference dataset: a PDB file (uses the CRYST1 cell and computes model intensities, including bulk solvent) or an MTZ from a previous cycle. Required for <code>mark0</code> scaling.<br />
|-<br />
| <code>scaling.algorithm</code> || mark0 || <code>mark0</code>: per-image scaling against the reference. <code>mark1</code>: no scaling, just Monte-Carlo averaging (all scale factors = 1) — used for reference-free bootstrap.<br />
|-<br />
| <code>scaling.unit_cell</code> / <code>scaling.space_group</code> || None || Cell/symmetry to use when no model is provided (i.e. <code>mark1</code>).<br />
|-<br />
| <code>scaling.resolution_scalar</code> || 0.969 || <code>merging.d_min</code> is multiplied by this when deciding which reflections fall within the resolution limit, generating a few extra reference reflections to account for per-shot cell variation (e.g. set 0.96 for a few more).<br />
|-<br />
| <code>scaling.mtz.mtz_column_F</code> || fobs || Column name in the reference MTZ containing the reference structure factors (intensities or amplitudes).<br />
|}<br />
<br />
=== Post-refinement (<code>postrefinement</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>postrefinement.enable</code> || True || Enable post-refinement of per-image parameters/partialities (Sauter 2015).<br />
|-<br />
| <code>postrefinement.algorithm</code> || rs || Post-refinement protocol: <code>rs</code>, <code>rs2</code>, <code>rs_hybrid</code>, or <code>eta_deff</code>.<br />
|}<br />
<br />
=== Merging and error model (<code>merging</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>merging.d_min</code> || None || High-resolution limit for scaling and merging.<br />
|-<br />
| <code>merging.d_max</code> || None || Low-resolution limit (currently affects only CCiso).<br />
|-<br />
| <code>merging.merge_anomalous</code> || False || <code>True</code> merges Bijvoet mates together (higher multiplicity); <code>False</code> keeps them separate to preserve the anomalous signal.<br />
|-<br />
| <code>merging.minimum_multiplicity</code> || 2 || Do not produce a merged structure factor for HKLs measured fewer than this many times.<br />
|-<br />
| <code>merging.error.model</code> || mm24 || Merged-sigma error model: <code>mm24</code> (Mittan-Moreau 2025, current default), <code>ev11</code> (Evans 2011 / sdfac-refine), <code>ha14</code>, or <code>errors_from_sample_residuals</code> (no refined error terms — used with <code>mark1</code> bootstrap).<br />
|-<br />
| <code>merging.set_average_unit_cell</code> || True || Output MTZ adopts the average unit cell of the data rather than the reference model's cell.<br />
|}<br />
<br />
=== Statistics (<code>statistics</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>statistics.n_bins</code> || 10 || Number of resolution bins in the statistics tables. Often raised to 20.<br />
|-<br />
| <code>statistics.cciso.mtz_file</code> || None || Isomorphous reference structure factors for R<sub>iso</sub> / CC<sub>iso</sub>.<br />
|-<br />
| <code>statistics.report_ML</code> || True || Report per-frame max-likelihood attributes (domain size, mosaicity).<br />
|}<br />
<br />
=== Output (<code>output</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>output.output_dir</code> || <code>.</code> || Directory for output files (created if it does not exist).<br />
|-<br />
| <code>output.prefix</code> || iobs || Prefix prepended to all output file names.<br />
|-<br />
| <code>output.save_experiments_and_reflections</code> || False || Dump the final experiments/reflections from the last worker as DIALS files — used for the two-stage scale-then-merge workflow.<br />
|-<br />
| <code>output.do_timing</code> || False || Log elapsed time per execution step.<br />
|-<br />
| <code>output.expanded_bookkeeping</code> || False || Store extra provenance columns (input file/reflection indices, odd/even flag, original id) in the saved reflection tables.<br />
|}<br />
<br />
=== Multiprocessing (<code>mp</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>mp.method</code> || mpi || Multiprocessing method (only <code>mpi</code> is currently supported).<br />
|-<br />
| <code>mp.debug.cProfile</code> || False || Enable code profiling; a <code>.prof</code> file is written per rank.<br />
|}<br />
<br />
== Output files ==<br />
<br />
The <code>merge</code> worker gathers all HKLs at rank 0 and writes three MTZ files, prefixed with<br />
<code>output.prefix</code> (default <code>iobs</code>):<br />
<br />
* <code>&lt;prefix&gt;_all.mtz</code> — the merged dataset using all observations (the main result for structure solution)<br />
* <code>&lt;prefix&gt;_odd.mtz</code> and <code>&lt;prefix&gt;_even.mtz</code> — the two half-datasets used for CC½ and related half-dataset statistics<br />
<br />
A main log records the per-step progress and the merging statistics tables by resolution bin:<br />
completeness, multiplicity and ⟨I/σ⟩, plus the "Table of Scaling Results" reporting CC½ (the<br />
half-dataset correlation, labelled "CC int"), CC<sub>iso</sub>, R<sub>int</sub>, R<sub>split</sub><br />
and R<sub>iso</sub>. The merged MTZ can optionally be<br />
uploaded to a shared [https://github.com/cctbx/cctbx_project/tree/master/xfel/merging/application/publish Google Drive folder] by adding the <code>publish</code> worker.<br />
<br />
== References ==<br />
<br />
* Sauter NK (2015). ''J Synchrotron Radiat'' '''22''', 239–48. (post-refinement)<br />
* Uervirojnangkoorn M, ''et al.'' (2015). ''eLife'' 10.7554/eLife.05421. (PRIME / bootstrap)<br />
* Brewster AS, ''et al.'' (2018). ''Acta Cryst.'' D '''74''', 877–894.<br />
* Brewster AS, ''et al.'' (2019a). ''Computational Crystallography Newsletter'' '''10''', 22–39. (XFEL GUI)<br />
* Brewster AS, ''et al.'' (2019b). ''Acta Cryst.'' D '''75''', 959–968. (error models)<br />
* Mittan-Moreau (2025). (mm24 error model)<br />
<br />
== See also ==<br />
<br />
* [[dials.stills_process]] — produces the integrated data that <code>cctbx.xfel.merge</code> consumes<br />
* [[cctbx.xfel GUI]] — orchestrates separate scaling and repeated merging as data are collected<br />
* The program README at <code>xfel/merging/README.md</code> in cctbx_project</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Cctbx.xfel.merge&diff=1783Cctbx.xfel.merge2026-08-21T23:24:52Z<p>Aaron: /* Merging and error model (merging) */</p>
<hr />
<div>= ''cctbx.xfel.merge'' =<br />
<br />
'''''cctbx.xfel.merge''''' is the scaling, post-refinement and merging program of cctbx.xfel. It takes<br />
the per-image integrated results produced by ''[[dials.stills_process]]'' — one crystal per still shot —<br />
and reduces the whole ensemble to a single set of merged structure-factor intensities (an MTZ file)<br />
suitable for structure solution and refinement, together with the crystallographic quality<br />
statistics (completeness, multiplicity, ⟨I/σ⟩, CC½, CC<sub>iso</sub>, R-factors, etc.).<br />
<br />
Because serial datasets routinely contain tens of thousands to millions of still shots,<br />
<code>cctbx.xfel.merge</code> is built from the ground up to run under MPI across many nodes. It is<br />
almost always launched with <code>mpirun</code>/<code>srun</code>. Note that MPI uses the term 'ranks'<br />
to distinguish independent processes that are running across one or more nodes.<br />
<br />
== Overview ==<br />
<br />
<code>cctbx.xfel.merge</code> is organized as a sequence of '''workers'''. Each worker performs one<br />
task — reading data, applying a correction, filtering, scaling, post-refining, computing a statistic,<br />
or merging — and passes its (possibly modified) experiments and reflections on to the next worker.<br />
The list of workers to run is the "step list", and it is fully configurable: you can add, remove or<br />
reorder steps with <code>dispatch.step_list</code>. This is what makes the program flexible enough to<br />
scale-and-merge in one pass, to scale subsets separately and merge later, or to run a reference-free<br />
bootstrap.<br />
<br />
The '''default''' step list is:<br />
<br />
{| class="wikitable"<br />
! # !! Worker !! Purpose<br />
|-<br />
| 1 || <code>input</code> || Read integrated experiments (<code>.expt</code>) and reflections (<code>.refl</code>) from disk in DIALS format.<br />
|-<br />
| 2 || <code>balance</code> || Redistribute the input load evenly across MPI ranks (needed when there are fewer or more input files than ranks).<br />
|-<br />
| 3 || <code>model_scaling</code> || Build the full Miller list, model intensities from the reference, and the resolution binner used for scaling and post-refinement.<br />
|-<br />
| 4 || <code>modify</code> || Apply per-reflection modifications, e.g. polarization correction (and optionally re-indexing to resolve indexing ambiguity — see <code>modify.cosym</code>/<code>reindex_to_reference</code>).<br />
|-<br />
| 5 || <code>filter</code> || Reject whole experiments (or reflections) by resolution, unit cell, correlation to the reference, energy, etc.<br />
|-<br />
| 6 || <code>scale</code> || Scale each lattice to the reference dataset (<code>mark0</code>) or simply average (<code>mark1</code>).<br />
|-<br />
| 7 || <code>postrefine</code> || Post-refine per-image crystal parameters and partialities (Sauter 2015).<br />
|-<br />
| 8 || <code>statistics_unitcell</code> || Determine and store the average unit cell.<br />
|-<br />
| 9 || <code>statistics_beam</code> || Determine and store the average wavelength.<br />
|-<br />
| 10 || <code>model_statistics</code> || Rebuild the Miller list / binner for statistics (may use the average unit cell).<br />
|-<br />
| 11 || <code>statistics_resolution</code> || Per-crystal resolution statistics.<br />
|-<br />
| 12 || <code>group</code> || Redistribute reflections across ranks so that every observation of a given HKL is gathered on the same rank prior to merging.<br />
|-<br />
| 13 || <code>errors_merge</code> || Correct the merged error estimates with a per-HKL error model (e.g. Ev11, Brewster 2019b, or MM24 (the default), Mittan-Moreau (2025)).<br />
|-<br />
| 14 || <code>statistics_intensity</code> || Intensity statistics by resolution.<br />
|-<br />
| 15 || <code>merge</code> || Merge symmetry-equivalent intensities, gather all HKLs at rank 0, and write "odd", "even" and "all" MTZ files.<br />
|-<br />
| 16 || <code>statistics_intensity_cxi</code> || Correlation and R-factor statistics (ported from cxi.merge): the "Table of Scaling Results" — CC½ (half-dataset correlation, labelled "CC int"), CC<sub>iso</sub>, R<sub>int</sub>, R<sub>split</sub>, R<sub>iso</sub>.<br />
|}<br />
<br />
Additional, non-default workers are available and can be<br />
inserted into the step list, including <code>tdata</code> (write out unit cells for clustering),<br />
<code>export</code> (batch MTZ for ''Careless''), <code>publish</code> (upload merged files to a<br />
Google Drive folder), <code>monitor</code> (resource-usage logging), <code>integrate</code><br />
(on-the-fly integration), <code>prepare</code> (SpReAD energy binning), and<br />
<code>lunus</code> (diffuse-scattering deck). A <code>deltaccint</code> worker (ΔCC½ lattice<br />
filtering) can be added after <code>group</code>. Custom workers can be supplied through the<br />
<code>XFEL_CUSTOM_WORKER_PATH</code> environment variable.<br />
<br />
== Basic usage ==<br />
<br />
<code>cctbx.xfel.merge</code> is run under MPI:<br />
<br />
<pre><br />
mpirun cctbx.xfel.merge merge.phil<br />
</pre><br />
<br />
A typical <code>merge.phil</code>:<br />
<br />
<pre><br />
input.path=<path to data><br />
input.experiments_suffix=.expt<br />
input.reflections_suffix=.refl<br />
<br />
filter.algorithm=unit_cell<br />
filter.unit_cell.value.relative_length_tolerance=0.03<br />
filter.outlier.min_corr=0.1<br />
<br />
select.algorithm=significance_filter<br />
select.significance_filter.sigma=0.1<br />
select.significance_filter.min_ct=200<br />
select.significance_filter.max_ct=300<br />
<br />
scaling.model=<model.pdb or model.mtz><br />
scaling.resolution_scalar=0.96<br />
<br />
merging.d_min=1.8<br />
merging.error.model=mm24<br />
merging.merge_anomalous=True<br />
<br />
postrefinement.enable=True<br />
statistics.n_bins=20<br />
<br />
output.do_timing=True<br />
output.output_dir=<output folder><br />
output.prefix=<prefix for output files><br />
</pre><br />
<br />
Explore the parameters with:<br />
* <code>cctbx.xfel.merge -c</code> — show configurable parameters<br />
* <code>cctbx.xfel.merge -c -e 10 -a 2</code> — show all parameters (expert level 10) with full help text<br />
<br />
== Two-stage scaling and merging ==<br />
<br />
For large, growing datasets (the mode used by the [[cctbx.xfel GUI]]), it is more efficient to scale<br />
and post-refine each subset of data '''once''', then merge repeatedly as more data arrive. This is<br />
done by running the program twice with different step lists.<br />
<br />
'''Stage 1 — scale each subset''' (stops after per-image work and writes intermediate DIALS files):<br />
<br />
<pre><br />
dispatch.step_list=input balance model_scaling modify filter scale postrefine \<br />
statistics_unitcell statistics_beam model_statistics statistics_resolution<br />
output.save_experiments_and_reflections=True<br />
</pre><br />
<br />
<code>output.save_experiments_and_reflections=True</code> dumps the scaled/post-refined data as<br />
experiment lists and reflection tables that the merge stage can read back in.<br />
<br />
'''Stage 2 — merge the scaled subsets''' (add an <code>input.path</code> for each finished subset):<br />
<br />
<pre><br />
dispatch.step_list=input model_scaling statistics_unitcell statistics_beam \<br />
model_statistics statistics_resolution group errors_merge statistics_intensity \<br />
merge statistics_intensity_cxi<br />
</pre><br />
<br />
== Merging without a reference (bootstrap / mark1) ==<br />
<br />
If no scaling/merging reference is available, use a bootstrap procedure. First run the<br />
<code>mark1</code> algorithm, which simply averages the data (all per-image scale factors = 1, no<br />
post-refinement), specifying the target cell and symmetry explicitly:<br />
<br />
<pre><br />
scaling.algorithm=mark1<br />
scaling.unit_cell=<unit cell><br />
scaling.space_group=<space group><br />
merging.error.model=errors_from_sample_residuals<br />
</pre><br />
<br />
The resulting MTZ is then used as the reference (<code>scaling.model=</code>) for subsequent regular<br />
merges. Iterate a few cycles; CC½ typically improves and converges over the first few rounds.<br />
<br />
== Important parameters ==<br />
<br />
Parameters are given by their full PHIL path with defaults in parentheses. The program exposes many<br />
more parameters (see <code>-c -e 10 -a 2</code>); those below are the ones most commonly set.<br />
<br />
=== Input ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>input.path</code> || None || Path(s) to the integrated data — a folder, several folders, or individual files; wildcards allowed; may be repeated. Matching <code>.expt</code> and <code>.refl</code> files must both be present.<br />
|-<br />
| <code>input.experiments_suffix</code> || <code>_integrated.expt</code> || Only load experiment files ending with this suffix.<br />
|-<br />
| <code>input.reflections_suffix</code> || <code>_integrated.refl</code> || Only load reflection files ending with this suffix.<br />
|-<br />
| <code>input.parallel_file_load.method</code> || uniform || <code>uniform</code> spreads input files evenly over all ranks; <code>node_memory</code> distributes so per-node memory limits are respected.<br />
|-<br />
| <code>input.alist.file</code> || None || Path to a text file of experiment tags or filenames; combined with <code>input.alist.op=keep|reject</code> to merge only (or exclude) a specified subset.<br />
|-<br />
| <code>input.persistent_refl_cols</code> || None || Reflection-table columns to preserve through the pruning steps (and into saved tables).<br />
|}<br />
<br />
=== Filtering whole experiments (<code>filter</code>) ===<br />
<br />
The <code>filter</code> section accepts or rejects '''entire experiments'''.<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>filter.algorithm</code> || (none) || One or more of <code>n_obs</code>, <code>resolution</code>, <code>unit_cell</code>, <code>energy</code>, <code>bootstrap</code>, <code>experiment_prefix</code>. Nothing is filtered unless listed.<br />
|-<br />
| <code>filter.unit_cell.algorithm</code> || value || How to restrict unit cells: <code>value</code> (compare to a target), <code>range</code>, or <code>cluster</code> (use a precomputed covariance clustering).<br />
|-<br />
| <code>filter.unit_cell.value.relative_length_tolerance</code> || 0.1 || Fractional deviation of cell edge lengths allowed vs. the target cell. Often tightened (e.g. 0.03).<br />
|-<br />
| <code>filter.unit_cell.value.absolute_angle_tolerance</code> || 2.0 || Allowed cell-angle deviation (degrees) vs. the target.<br />
|-<br />
| <code>filter.unit_cell.value.target_unit_cell</code> / <code>target_space_group</code> || Auto || Target cell/symmetry; <code>Auto</code> uses the scaling model.<br />
|-<br />
| <code>filter.outlier.min_corr</code> || 0.1 || Minimum correlation of an image's intensities to the reference model; images below this are rejected. Only applied when a scaling model is present. Use <code>-1</code> to accept all images regardless of correlation.<br />
|-<br />
| <code>filter.resolution.d_min</code> || None || Reject an experiment unless some of its reflections extend beyond this resolution.<br />
|}<br />
<br />
=== Selecting individual reflections (<code>select</code>) ===<br />
<br />
The <code>select</code> section accepts or rejects '''individual reflections''' (contrast with<br />
<code>filter</code>, which acts on whole experiments).<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>select.algorithm</code> || (none) || One or more of <code>significance_filter</code>, <code>panel</code>, <code>cspad_sensor</code>, <code>isolation_forest</code>, <code>target_energy</code>.<br />
|-<br />
| <code>select.significance_filter.sigma</code> || 0.5 || Trim each lattice at the resolution where the binned ⟨I/σ⟩ drops below this value. Often reduced to 0.1.<br />
|-<br />
| <code>select.significance_filter.min_ct</code> / <code>max_ct</code> || 10 / 50 || Target minimum/maximum mean reflections per resolution bin when choosing the number of bins. For large unit cells these are often raised (e.g. 200/300); for small cells, use the defaults.<br />
|}<br />
<br />
=== Modifying intensities (<code>modify</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>modify.algorithm</code> || polarization || Per-reflection corrections to apply (default applies the polarization correction).<br />
|-<br />
| <code>modify.cosym</code> || — || Resolve an indexing ambiguity across the ensemble using the Gildea/Winter (Brehm/Diederichs) cosym method, optionally anchored to <code>scaling.model</code>.<br />
|-<br />
| <code>modify.reindex_to_reference</code> / <code>reindex_to_abc</code> || — || Alternative re-indexing operators (to a reference model, or an explicit operator).<br />
|}<br />
<br />
=== Scaling (<code>scaling</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>scaling.model</code> || None || Reference dataset: a PDB file (uses the CRYST1 cell and computes model intensities, including bulk solvent) or an MTZ from a previous cycle. Required for <code>mark0</code> scaling.<br />
|-<br />
| <code>scaling.algorithm</code> || mark0 || <code>mark0</code>: per-image scaling against the reference. <code>mark1</code>: no scaling, just Monte-Carlo averaging (all scale factors = 1) — used for reference-free bootstrap.<br />
|-<br />
| <code>scaling.unit_cell</code> / <code>scaling.space_group</code> || None || Cell/symmetry to use when no model is provided (i.e. <code>mark1</code>).<br />
|-<br />
| <code>scaling.resolution_scalar</code> || 0.969 || <code>merging.d_min</code> is multiplied by this when deciding which reflections fall within the resolution limit, generating a few extra reference reflections to account for per-shot cell variation (e.g. set 0.96 for a few more).<br />
|-<br />
| <code>scaling.mtz.mtz_column_F</code> || fobs || Column name in the reference MTZ containing the reference structure factors (intensities or amplitudes).<br />
|}<br />
<br />
=== Post-refinement (<code>postrefinement</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>postrefinement.enable</code> || True || Enable post-refinement of per-image parameters/partialities (Sauter 2015).<br />
|-<br />
| <code>postrefinement.algorithm</code> || rs || Post-refinement protocol: <code>rs</code>, <code>rs2</code>, <code>rs_hybrid</code>, or <code>eta_deff</code>.<br />
|}<br />
<br />
=== Merging and error model (<code>merging</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>merging.d_min</code> || None || High-resolution limit for scaling and merging.<br />
|-<br />
| <code>merging.d_max</code> || None || Low-resolution limit (currently affects only CCiso).<br />
|-<br />
| <code>merging.merge_anomalous</code> || False || <code>True</code> merges Bijvoet mates together (higher multiplicity); <code>False</code> keeps them separate to preserve the anomalous signal.<br />
|-<br />
| <code>merging.minimum_multiplicity</code> || 2 || Do not produce a merged structure factor for HKLs measured fewer than this many times.<br />
|-<br />
| <code>merging.error.model</code> || mm24 || Merged-sigma error model: <code>mm24</code> (Mittan-Moreau 2025, current default), <code>ev11</code> (Evans 2011 / sdfac-refine), <code>ha14</code>, or <code>errors_from_sample_residuals</code> (no refined error terms — used with <code>mark1</code> bootstrap).<br />
|-<br />
| <code>merging.set_average_unit_cell</code> || True || Output MTZ adopts the average unit cell of the data rather than the reference model's cell.<br />
|}<br />
<br />
=== Statistics (<code>statistics</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>statistics.n_bins</code> || 10 || Number of resolution bins in the statistics tables.<br />
|-<br />
| <code>statistics.cciso.mtz_file</code> || None || Isomorphous reference structure factors for R<sub>iso</sub> / CC<sub>iso</sub>.<br />
|-<br />
| <code>statistics.report_ML</code> || True || Report per-frame max-likelihood attributes (domain size, mosaicity).<br />
|}<br />
<br />
=== Output (<code>output</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>output.output_dir</code> || <code>.</code> || Directory for output files (created if it does not exist).<br />
|-<br />
| <code>output.prefix</code> || iobs || Prefix prepended to all output file names.<br />
|-<br />
| <code>output.save_experiments_and_reflections</code> || False || Dump the final experiments/reflections from the last worker as DIALS files — used for the two-stage scale-then-merge workflow.<br />
|-<br />
| <code>output.do_timing</code> || False || Log elapsed time per execution step.<br />
|-<br />
| <code>output.expanded_bookkeeping</code> || False || Store extra provenance columns (input file/reflection indices, odd/even flag, original id) in the saved reflection tables.<br />
|}<br />
<br />
=== Multiprocessing (<code>mp</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>mp.method</code> || mpi || Multiprocessing method (only <code>mpi</code> is currently supported).<br />
|-<br />
| <code>mp.debug.cProfile</code> || False || Enable code profiling; a <code>.prof</code> file is written per rank.<br />
|}<br />
<br />
== Output files ==<br />
<br />
The <code>merge</code> worker gathers all HKLs at rank 0 and writes three MTZ files, prefixed with<br />
<code>output.prefix</code> (default <code>iobs</code>):<br />
<br />
* <code>&lt;prefix&gt;_all.mtz</code> — the merged dataset using all observations (the main result for structure solution)<br />
* <code>&lt;prefix&gt;_odd.mtz</code> and <code>&lt;prefix&gt;_even.mtz</code> — the two half-datasets used for CC½ and related half-dataset statistics<br />
<br />
A main log records the per-step progress and the merging statistics tables by resolution bin:<br />
completeness, multiplicity and ⟨I/σ⟩, plus the "Table of Scaling Results" reporting CC½ (the<br />
half-dataset correlation, labelled "CC int"), CC<sub>iso</sub>, R<sub>int</sub>, R<sub>split</sub><br />
and R<sub>iso</sub>. The merged MTZ can optionally be<br />
uploaded to a shared [https://github.com/cctbx/cctbx_project/tree/master/xfel/merging/application/publish Google Drive folder] by adding the <code>publish</code> worker.<br />
<br />
== References ==<br />
<br />
* Sauter NK (2015). ''J Synchrotron Radiat'' '''22''', 239–48. (post-refinement)<br />
* Uervirojnangkoorn M, ''et al.'' (2015). ''eLife'' 10.7554/eLife.05421. (PRIME / bootstrap)<br />
* Brewster AS, ''et al.'' (2018). ''Acta Cryst.'' D '''74''', 877–894.<br />
* Brewster AS, ''et al.'' (2019a). ''Computational Crystallography Newsletter'' '''10''', 22–39. (XFEL GUI)<br />
* Brewster AS, ''et al.'' (2019b). ''Acta Cryst.'' D '''75''', 959–968. (error models)<br />
* Mittan-Moreau (2025). (mm24 error model)<br />
<br />
== See also ==<br />
<br />
* [[dials.stills_process]] — produces the integrated data that <code>cctbx.xfel.merge</code> consumes<br />
* [[cctbx.xfel GUI]] — orchestrates separate scaling and repeated merging as data are collected<br />
* The program README at <code>xfel/merging/README.md</code> in cctbx_project</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Cctbx.xfel.merge&diff=1782Cctbx.xfel.merge2026-08-21T23:24:39Z<p>Aaron: /* Merging and error model (merging) */</p>
<hr />
<div>= ''cctbx.xfel.merge'' =<br />
<br />
'''''cctbx.xfel.merge''''' is the scaling, post-refinement and merging program of cctbx.xfel. It takes<br />
the per-image integrated results produced by ''[[dials.stills_process]]'' — one crystal per still shot —<br />
and reduces the whole ensemble to a single set of merged structure-factor intensities (an MTZ file)<br />
suitable for structure solution and refinement, together with the crystallographic quality<br />
statistics (completeness, multiplicity, ⟨I/σ⟩, CC½, CC<sub>iso</sub>, R-factors, etc.).<br />
<br />
Because serial datasets routinely contain tens of thousands to millions of still shots,<br />
<code>cctbx.xfel.merge</code> is built from the ground up to run under MPI across many nodes. It is<br />
almost always launched with <code>mpirun</code>/<code>srun</code>. Note that MPI uses the term 'ranks'<br />
to distinguish independent processes that are running across one or more nodes.<br />
<br />
== Overview ==<br />
<br />
<code>cctbx.xfel.merge</code> is organized as a sequence of '''workers'''. Each worker performs one<br />
task — reading data, applying a correction, filtering, scaling, post-refining, computing a statistic,<br />
or merging — and passes its (possibly modified) experiments and reflections on to the next worker.<br />
The list of workers to run is the "step list", and it is fully configurable: you can add, remove or<br />
reorder steps with <code>dispatch.step_list</code>. This is what makes the program flexible enough to<br />
scale-and-merge in one pass, to scale subsets separately and merge later, or to run a reference-free<br />
bootstrap.<br />
<br />
The '''default''' step list is:<br />
<br />
{| class="wikitable"<br />
! # !! Worker !! Purpose<br />
|-<br />
| 1 || <code>input</code> || Read integrated experiments (<code>.expt</code>) and reflections (<code>.refl</code>) from disk in DIALS format.<br />
|-<br />
| 2 || <code>balance</code> || Redistribute the input load evenly across MPI ranks (needed when there are fewer or more input files than ranks).<br />
|-<br />
| 3 || <code>model_scaling</code> || Build the full Miller list, model intensities from the reference, and the resolution binner used for scaling and post-refinement.<br />
|-<br />
| 4 || <code>modify</code> || Apply per-reflection modifications, e.g. polarization correction (and optionally re-indexing to resolve indexing ambiguity — see <code>modify.cosym</code>/<code>reindex_to_reference</code>).<br />
|-<br />
| 5 || <code>filter</code> || Reject whole experiments (or reflections) by resolution, unit cell, correlation to the reference, energy, etc.<br />
|-<br />
| 6 || <code>scale</code> || Scale each lattice to the reference dataset (<code>mark0</code>) or simply average (<code>mark1</code>).<br />
|-<br />
| 7 || <code>postrefine</code> || Post-refine per-image crystal parameters and partialities (Sauter 2015).<br />
|-<br />
| 8 || <code>statistics_unitcell</code> || Determine and store the average unit cell.<br />
|-<br />
| 9 || <code>statistics_beam</code> || Determine and store the average wavelength.<br />
|-<br />
| 10 || <code>model_statistics</code> || Rebuild the Miller list / binner for statistics (may use the average unit cell).<br />
|-<br />
| 11 || <code>statistics_resolution</code> || Per-crystal resolution statistics.<br />
|-<br />
| 12 || <code>group</code> || Redistribute reflections across ranks so that every observation of a given HKL is gathered on the same rank prior to merging.<br />
|-<br />
| 13 || <code>errors_merge</code> || Correct the merged error estimates with a per-HKL error model (e.g. Ev11, Brewster 2019b, or MM24 (the default), Mittan-Moreau (2025)).<br />
|-<br />
| 14 || <code>statistics_intensity</code> || Intensity statistics by resolution.<br />
|-<br />
| 15 || <code>merge</code> || Merge symmetry-equivalent intensities, gather all HKLs at rank 0, and write "odd", "even" and "all" MTZ files.<br />
|-<br />
| 16 || <code>statistics_intensity_cxi</code> || Correlation and R-factor statistics (ported from cxi.merge): the "Table of Scaling Results" — CC½ (half-dataset correlation, labelled "CC int"), CC<sub>iso</sub>, R<sub>int</sub>, R<sub>split</sub>, R<sub>iso</sub>.<br />
|}<br />
<br />
Additional, non-default workers are available and can be<br />
inserted into the step list, including <code>tdata</code> (write out unit cells for clustering),<br />
<code>export</code> (batch MTZ for ''Careless''), <code>publish</code> (upload merged files to a<br />
Google Drive folder), <code>monitor</code> (resource-usage logging), <code>integrate</code><br />
(on-the-fly integration), <code>prepare</code> (SpReAD energy binning), and<br />
<code>lunus</code> (diffuse-scattering deck). A <code>deltaccint</code> worker (ΔCC½ lattice<br />
filtering) can be added after <code>group</code>. Custom workers can be supplied through the<br />
<code>XFEL_CUSTOM_WORKER_PATH</code> environment variable.<br />
<br />
== Basic usage ==<br />
<br />
<code>cctbx.xfel.merge</code> is run under MPI:<br />
<br />
<pre><br />
mpirun cctbx.xfel.merge merge.phil<br />
</pre><br />
<br />
A typical <code>merge.phil</code>:<br />
<br />
<pre><br />
input.path=<path to data><br />
input.experiments_suffix=.expt<br />
input.reflections_suffix=.refl<br />
<br />
filter.algorithm=unit_cell<br />
filter.unit_cell.value.relative_length_tolerance=0.03<br />
filter.outlier.min_corr=0.1<br />
<br />
select.algorithm=significance_filter<br />
select.significance_filter.sigma=0.1<br />
select.significance_filter.min_ct=200<br />
select.significance_filter.max_ct=300<br />
<br />
scaling.model=<model.pdb or model.mtz><br />
scaling.resolution_scalar=0.96<br />
<br />
merging.d_min=1.8<br />
merging.error.model=mm24<br />
merging.merge_anomalous=True<br />
<br />
postrefinement.enable=True<br />
statistics.n_bins=20<br />
<br />
output.do_timing=True<br />
output.output_dir=<output folder><br />
output.prefix=<prefix for output files><br />
</pre><br />
<br />
Explore the parameters with:<br />
* <code>cctbx.xfel.merge -c</code> — show configurable parameters<br />
* <code>cctbx.xfel.merge -c -e 10 -a 2</code> — show all parameters (expert level 10) with full help text<br />
<br />
== Two-stage scaling and merging ==<br />
<br />
For large, growing datasets (the mode used by the [[cctbx.xfel GUI]]), it is more efficient to scale<br />
and post-refine each subset of data '''once''', then merge repeatedly as more data arrive. This is<br />
done by running the program twice with different step lists.<br />
<br />
'''Stage 1 — scale each subset''' (stops after per-image work and writes intermediate DIALS files):<br />
<br />
<pre><br />
dispatch.step_list=input balance model_scaling modify filter scale postrefine \<br />
statistics_unitcell statistics_beam model_statistics statistics_resolution<br />
output.save_experiments_and_reflections=True<br />
</pre><br />
<br />
<code>output.save_experiments_and_reflections=True</code> dumps the scaled/post-refined data as<br />
experiment lists and reflection tables that the merge stage can read back in.<br />
<br />
'''Stage 2 — merge the scaled subsets''' (add an <code>input.path</code> for each finished subset):<br />
<br />
<pre><br />
dispatch.step_list=input model_scaling statistics_unitcell statistics_beam \<br />
model_statistics statistics_resolution group errors_merge statistics_intensity \<br />
merge statistics_intensity_cxi<br />
</pre><br />
<br />
== Merging without a reference (bootstrap / mark1) ==<br />
<br />
If no scaling/merging reference is available, use a bootstrap procedure. First run the<br />
<code>mark1</code> algorithm, which simply averages the data (all per-image scale factors = 1, no<br />
post-refinement), specifying the target cell and symmetry explicitly:<br />
<br />
<pre><br />
scaling.algorithm=mark1<br />
scaling.unit_cell=<unit cell><br />
scaling.space_group=<space group><br />
merging.error.model=errors_from_sample_residuals<br />
</pre><br />
<br />
The resulting MTZ is then used as the reference (<code>scaling.model=</code>) for subsequent regular<br />
merges. Iterate a few cycles; CC½ typically improves and converges over the first few rounds.<br />
<br />
== Important parameters ==<br />
<br />
Parameters are given by their full PHIL path with defaults in parentheses. The program exposes many<br />
more parameters (see <code>-c -e 10 -a 2</code>); those below are the ones most commonly set.<br />
<br />
=== Input ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>input.path</code> || None || Path(s) to the integrated data — a folder, several folders, or individual files; wildcards allowed; may be repeated. Matching <code>.expt</code> and <code>.refl</code> files must both be present.<br />
|-<br />
| <code>input.experiments_suffix</code> || <code>_integrated.expt</code> || Only load experiment files ending with this suffix.<br />
|-<br />
| <code>input.reflections_suffix</code> || <code>_integrated.refl</code> || Only load reflection files ending with this suffix.<br />
|-<br />
| <code>input.parallel_file_load.method</code> || uniform || <code>uniform</code> spreads input files evenly over all ranks; <code>node_memory</code> distributes so per-node memory limits are respected.<br />
|-<br />
| <code>input.alist.file</code> || None || Path to a text file of experiment tags or filenames; combined with <code>input.alist.op=keep|reject</code> to merge only (or exclude) a specified subset.<br />
|-<br />
| <code>input.persistent_refl_cols</code> || None || Reflection-table columns to preserve through the pruning steps (and into saved tables).<br />
|}<br />
<br />
=== Filtering whole experiments (<code>filter</code>) ===<br />
<br />
The <code>filter</code> section accepts or rejects '''entire experiments'''.<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>filter.algorithm</code> || (none) || One or more of <code>n_obs</code>, <code>resolution</code>, <code>unit_cell</code>, <code>energy</code>, <code>bootstrap</code>, <code>experiment_prefix</code>. Nothing is filtered unless listed.<br />
|-<br />
| <code>filter.unit_cell.algorithm</code> || value || How to restrict unit cells: <code>value</code> (compare to a target), <code>range</code>, or <code>cluster</code> (use a precomputed covariance clustering).<br />
|-<br />
| <code>filter.unit_cell.value.relative_length_tolerance</code> || 0.1 || Fractional deviation of cell edge lengths allowed vs. the target cell. Often tightened (e.g. 0.03).<br />
|-<br />
| <code>filter.unit_cell.value.absolute_angle_tolerance</code> || 2.0 || Allowed cell-angle deviation (degrees) vs. the target.<br />
|-<br />
| <code>filter.unit_cell.value.target_unit_cell</code> / <code>target_space_group</code> || Auto || Target cell/symmetry; <code>Auto</code> uses the scaling model.<br />
|-<br />
| <code>filter.outlier.min_corr</code> || 0.1 || Minimum correlation of an image's intensities to the reference model; images below this are rejected. Only applied when a scaling model is present. Use <code>-1</code> to accept all images regardless of correlation.<br />
|-<br />
| <code>filter.resolution.d_min</code> || None || Reject an experiment unless some of its reflections extend beyond this resolution.<br />
|}<br />
<br />
=== Selecting individual reflections (<code>select</code>) ===<br />
<br />
The <code>select</code> section accepts or rejects '''individual reflections''' (contrast with<br />
<code>filter</code>, which acts on whole experiments).<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>select.algorithm</code> || (none) || One or more of <code>significance_filter</code>, <code>panel</code>, <code>cspad_sensor</code>, <code>isolation_forest</code>, <code>target_energy</code>.<br />
|-<br />
| <code>select.significance_filter.sigma</code> || 0.5 || Trim each lattice at the resolution where the binned ⟨I/σ⟩ drops below this value. Often reduced to 0.1.<br />
|-<br />
| <code>select.significance_filter.min_ct</code> / <code>max_ct</code> || 10 / 50 || Target minimum/maximum mean reflections per resolution bin when choosing the number of bins. For large unit cells these are often raised (e.g. 200/300); for small cells, use the defaults.<br />
|}<br />
<br />
=== Modifying intensities (<code>modify</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>modify.algorithm</code> || polarization || Per-reflection corrections to apply (default applies the polarization correction).<br />
|-<br />
| <code>modify.cosym</code> || — || Resolve an indexing ambiguity across the ensemble using the Gildea/Winter (Brehm/Diederichs) cosym method, optionally anchored to <code>scaling.model</code>.<br />
|-<br />
| <code>modify.reindex_to_reference</code> / <code>reindex_to_abc</code> || — || Alternative re-indexing operators (to a reference model, or an explicit operator).<br />
|}<br />
<br />
=== Scaling (<code>scaling</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>scaling.model</code> || None || Reference dataset: a PDB file (uses the CRYST1 cell and computes model intensities, including bulk solvent) or an MTZ from a previous cycle. Required for <code>mark0</code> scaling.<br />
|-<br />
| <code>scaling.algorithm</code> || mark0 || <code>mark0</code>: per-image scaling against the reference. <code>mark1</code>: no scaling, just Monte-Carlo averaging (all scale factors = 1) — used for reference-free bootstrap.<br />
|-<br />
| <code>scaling.unit_cell</code> / <code>scaling.space_group</code> || None || Cell/symmetry to use when no model is provided (i.e. <code>mark1</code>).<br />
|-<br />
| <code>scaling.resolution_scalar</code> || 0.969 || <code>merging.d_min</code> is multiplied by this when deciding which reflections fall within the resolution limit, generating a few extra reference reflections to account for per-shot cell variation (e.g. set 0.96 for a few more).<br />
|-<br />
| <code>scaling.mtz.mtz_column_F</code> || fobs || Column name in the reference MTZ containing the reference structure factors (intensities or amplitudes).<br />
|}<br />
<br />
=== Post-refinement (<code>postrefinement</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>postrefinement.enable</code> || True || Enable post-refinement of per-image parameters/partialities (Sauter 2015).<br />
|-<br />
| <code>postrefinement.algorithm</code> || rs || Post-refinement protocol: <code>rs</code>, <code>rs2</code>, <code>rs_hybrid</code>, or <code>eta_deff</code>.<br />
|}<br />
<br />
=== Merging and error model (<code>merging</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>merging.d_min</code> || None || High-resolution limit for scaling and merging.<br />
|-<br />
| <code>merging.d_max</code> || None || Low-resolution limit (currently affects only CCiso).<br />
|-<br />
| <code>merging.merge_anomalous</code> || False || <code>True</code> merges Bijvoet mates together (higher multiplicity); <code>False</code> keeps them separate to preserve the anomalous signal.<br />
|-<br />
| <code>merging.minimum_multiplicity</code> || 2 || Do not produce a merged structure factor for HKLs measured fewer than this many times.<br />
|-<br />
| <code>merging.error.model</code> || mm24 || Merged-sigma error model: <code>mm24</code> (Mittan-Moreau 2025, current default), <code>ev11</code> (Evans 2011 / sdfac-refine), <code>ha14</code>, or <code>errors_from_sample_residuals</code> (no refined error terms — used with <code>mark1</code> bootstrap).<br />
|-<br />
| <code>merging.set_average_unit_cell</code> || True || Output MTZ adopts the (cluster-derived) average unit cell of the data rather than the reference model's cell.<br />
|}<br />
<br />
=== Statistics (<code>statistics</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>statistics.n_bins</code> || 10 || Number of resolution bins in the statistics tables.<br />
|-<br />
| <code>statistics.cciso.mtz_file</code> || None || Isomorphous reference structure factors for R<sub>iso</sub> / CC<sub>iso</sub>.<br />
|-<br />
| <code>statistics.report_ML</code> || True || Report per-frame max-likelihood attributes (domain size, mosaicity).<br />
|}<br />
<br />
=== Output (<code>output</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>output.output_dir</code> || <code>.</code> || Directory for output files (created if it does not exist).<br />
|-<br />
| <code>output.prefix</code> || iobs || Prefix prepended to all output file names.<br />
|-<br />
| <code>output.save_experiments_and_reflections</code> || False || Dump the final experiments/reflections from the last worker as DIALS files — used for the two-stage scale-then-merge workflow.<br />
|-<br />
| <code>output.do_timing</code> || False || Log elapsed time per execution step.<br />
|-<br />
| <code>output.expanded_bookkeeping</code> || False || Store extra provenance columns (input file/reflection indices, odd/even flag, original id) in the saved reflection tables.<br />
|}<br />
<br />
=== Multiprocessing (<code>mp</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>mp.method</code> || mpi || Multiprocessing method (only <code>mpi</code> is currently supported).<br />
|-<br />
| <code>mp.debug.cProfile</code> || False || Enable code profiling; a <code>.prof</code> file is written per rank.<br />
|}<br />
<br />
== Output files ==<br />
<br />
The <code>merge</code> worker gathers all HKLs at rank 0 and writes three MTZ files, prefixed with<br />
<code>output.prefix</code> (default <code>iobs</code>):<br />
<br />
* <code>&lt;prefix&gt;_all.mtz</code> — the merged dataset using all observations (the main result for structure solution)<br />
* <code>&lt;prefix&gt;_odd.mtz</code> and <code>&lt;prefix&gt;_even.mtz</code> — the two half-datasets used for CC½ and related half-dataset statistics<br />
<br />
A main log records the per-step progress and the merging statistics tables by resolution bin:<br />
completeness, multiplicity and ⟨I/σ⟩, plus the "Table of Scaling Results" reporting CC½ (the<br />
half-dataset correlation, labelled "CC int"), CC<sub>iso</sub>, R<sub>int</sub>, R<sub>split</sub><br />
and R<sub>iso</sub>. The merged MTZ can optionally be<br />
uploaded to a shared [https://github.com/cctbx/cctbx_project/tree/master/xfel/merging/application/publish Google Drive folder] by adding the <code>publish</code> worker.<br />
<br />
== References ==<br />
<br />
* Sauter NK (2015). ''J Synchrotron Radiat'' '''22''', 239–48. (post-refinement)<br />
* Uervirojnangkoorn M, ''et al.'' (2015). ''eLife'' 10.7554/eLife.05421. (PRIME / bootstrap)<br />
* Brewster AS, ''et al.'' (2018). ''Acta Cryst.'' D '''74''', 877–894.<br />
* Brewster AS, ''et al.'' (2019a). ''Computational Crystallography Newsletter'' '''10''', 22–39. (XFEL GUI)<br />
* Brewster AS, ''et al.'' (2019b). ''Acta Cryst.'' D '''75''', 959–968. (error models)<br />
* Mittan-Moreau (2025). (mm24 error model)<br />
<br />
== See also ==<br />
<br />
* [[dials.stills_process]] — produces the integrated data that <code>cctbx.xfel.merge</code> consumes<br />
* [[cctbx.xfel GUI]] — orchestrates separate scaling and repeated merging as data are collected<br />
* The program README at <code>xfel/merging/README.md</code> in cctbx_project</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Cctbx.xfel.merge&diff=1781Cctbx.xfel.merge2026-08-21T23:22:00Z<p>Aaron: /* Selecting individual reflections (select) */</p>
<hr />
<div>= ''cctbx.xfel.merge'' =<br />
<br />
'''''cctbx.xfel.merge''''' is the scaling, post-refinement and merging program of cctbx.xfel. It takes<br />
the per-image integrated results produced by ''[[dials.stills_process]]'' — one crystal per still shot —<br />
and reduces the whole ensemble to a single set of merged structure-factor intensities (an MTZ file)<br />
suitable for structure solution and refinement, together with the crystallographic quality<br />
statistics (completeness, multiplicity, ⟨I/σ⟩, CC½, CC<sub>iso</sub>, R-factors, etc.).<br />
<br />
Because serial datasets routinely contain tens of thousands to millions of still shots,<br />
<code>cctbx.xfel.merge</code> is built from the ground up to run under MPI across many nodes. It is<br />
almost always launched with <code>mpirun</code>/<code>srun</code>. Note that MPI uses the term 'ranks'<br />
to distinguish independent processes that are running across one or more nodes.<br />
<br />
== Overview ==<br />
<br />
<code>cctbx.xfel.merge</code> is organized as a sequence of '''workers'''. Each worker performs one<br />
task — reading data, applying a correction, filtering, scaling, post-refining, computing a statistic,<br />
or merging — and passes its (possibly modified) experiments and reflections on to the next worker.<br />
The list of workers to run is the "step list", and it is fully configurable: you can add, remove or<br />
reorder steps with <code>dispatch.step_list</code>. This is what makes the program flexible enough to<br />
scale-and-merge in one pass, to scale subsets separately and merge later, or to run a reference-free<br />
bootstrap.<br />
<br />
The '''default''' step list is:<br />
<br />
{| class="wikitable"<br />
! # !! Worker !! Purpose<br />
|-<br />
| 1 || <code>input</code> || Read integrated experiments (<code>.expt</code>) and reflections (<code>.refl</code>) from disk in DIALS format.<br />
|-<br />
| 2 || <code>balance</code> || Redistribute the input load evenly across MPI ranks (needed when there are fewer or more input files than ranks).<br />
|-<br />
| 3 || <code>model_scaling</code> || Build the full Miller list, model intensities from the reference, and the resolution binner used for scaling and post-refinement.<br />
|-<br />
| 4 || <code>modify</code> || Apply per-reflection modifications, e.g. polarization correction (and optionally re-indexing to resolve indexing ambiguity — see <code>modify.cosym</code>/<code>reindex_to_reference</code>).<br />
|-<br />
| 5 || <code>filter</code> || Reject whole experiments (or reflections) by resolution, unit cell, correlation to the reference, energy, etc.<br />
|-<br />
| 6 || <code>scale</code> || Scale each lattice to the reference dataset (<code>mark0</code>) or simply average (<code>mark1</code>).<br />
|-<br />
| 7 || <code>postrefine</code> || Post-refine per-image crystal parameters and partialities (Sauter 2015).<br />
|-<br />
| 8 || <code>statistics_unitcell</code> || Determine and store the average unit cell.<br />
|-<br />
| 9 || <code>statistics_beam</code> || Determine and store the average wavelength.<br />
|-<br />
| 10 || <code>model_statistics</code> || Rebuild the Miller list / binner for statistics (may use the average unit cell).<br />
|-<br />
| 11 || <code>statistics_resolution</code> || Per-crystal resolution statistics.<br />
|-<br />
| 12 || <code>group</code> || Redistribute reflections across ranks so that every observation of a given HKL is gathered on the same rank prior to merging.<br />
|-<br />
| 13 || <code>errors_merge</code> || Correct the merged error estimates with a per-HKL error model (e.g. Ev11, Brewster 2019b, or MM24 (the default), Mittan-Moreau (2025)).<br />
|-<br />
| 14 || <code>statistics_intensity</code> || Intensity statistics by resolution.<br />
|-<br />
| 15 || <code>merge</code> || Merge symmetry-equivalent intensities, gather all HKLs at rank 0, and write "odd", "even" and "all" MTZ files.<br />
|-<br />
| 16 || <code>statistics_intensity_cxi</code> || Correlation and R-factor statistics (ported from cxi.merge): the "Table of Scaling Results" — CC½ (half-dataset correlation, labelled "CC int"), CC<sub>iso</sub>, R<sub>int</sub>, R<sub>split</sub>, R<sub>iso</sub>.<br />
|}<br />
<br />
Additional, non-default workers are available and can be<br />
inserted into the step list, including <code>tdata</code> (write out unit cells for clustering),<br />
<code>export</code> (batch MTZ for ''Careless''), <code>publish</code> (upload merged files to a<br />
Google Drive folder), <code>monitor</code> (resource-usage logging), <code>integrate</code><br />
(on-the-fly integration), <code>prepare</code> (SpReAD energy binning), and<br />
<code>lunus</code> (diffuse-scattering deck). A <code>deltaccint</code> worker (ΔCC½ lattice<br />
filtering) can be added after <code>group</code>. Custom workers can be supplied through the<br />
<code>XFEL_CUSTOM_WORKER_PATH</code> environment variable.<br />
<br />
== Basic usage ==<br />
<br />
<code>cctbx.xfel.merge</code> is run under MPI:<br />
<br />
<pre><br />
mpirun cctbx.xfel.merge merge.phil<br />
</pre><br />
<br />
A typical <code>merge.phil</code>:<br />
<br />
<pre><br />
input.path=<path to data><br />
input.experiments_suffix=.expt<br />
input.reflections_suffix=.refl<br />
<br />
filter.algorithm=unit_cell<br />
filter.unit_cell.value.relative_length_tolerance=0.03<br />
filter.outlier.min_corr=0.1<br />
<br />
select.algorithm=significance_filter<br />
select.significance_filter.sigma=0.1<br />
select.significance_filter.min_ct=200<br />
select.significance_filter.max_ct=300<br />
<br />
scaling.model=<model.pdb or model.mtz><br />
scaling.resolution_scalar=0.96<br />
<br />
merging.d_min=1.8<br />
merging.error.model=mm24<br />
merging.merge_anomalous=True<br />
<br />
postrefinement.enable=True<br />
statistics.n_bins=20<br />
<br />
output.do_timing=True<br />
output.output_dir=<output folder><br />
output.prefix=<prefix for output files><br />
</pre><br />
<br />
Explore the parameters with:<br />
* <code>cctbx.xfel.merge -c</code> — show configurable parameters<br />
* <code>cctbx.xfel.merge -c -e 10 -a 2</code> — show all parameters (expert level 10) with full help text<br />
<br />
== Two-stage scaling and merging ==<br />
<br />
For large, growing datasets (the mode used by the [[cctbx.xfel GUI]]), it is more efficient to scale<br />
and post-refine each subset of data '''once''', then merge repeatedly as more data arrive. This is<br />
done by running the program twice with different step lists.<br />
<br />
'''Stage 1 — scale each subset''' (stops after per-image work and writes intermediate DIALS files):<br />
<br />
<pre><br />
dispatch.step_list=input balance model_scaling modify filter scale postrefine \<br />
statistics_unitcell statistics_beam model_statistics statistics_resolution<br />
output.save_experiments_and_reflections=True<br />
</pre><br />
<br />
<code>output.save_experiments_and_reflections=True</code> dumps the scaled/post-refined data as<br />
experiment lists and reflection tables that the merge stage can read back in.<br />
<br />
'''Stage 2 — merge the scaled subsets''' (add an <code>input.path</code> for each finished subset):<br />
<br />
<pre><br />
dispatch.step_list=input model_scaling statistics_unitcell statistics_beam \<br />
model_statistics statistics_resolution group errors_merge statistics_intensity \<br />
merge statistics_intensity_cxi<br />
</pre><br />
<br />
== Merging without a reference (bootstrap / mark1) ==<br />
<br />
If no scaling/merging reference is available, use a bootstrap procedure. First run the<br />
<code>mark1</code> algorithm, which simply averages the data (all per-image scale factors = 1, no<br />
post-refinement), specifying the target cell and symmetry explicitly:<br />
<br />
<pre><br />
scaling.algorithm=mark1<br />
scaling.unit_cell=<unit cell><br />
scaling.space_group=<space group><br />
merging.error.model=errors_from_sample_residuals<br />
</pre><br />
<br />
The resulting MTZ is then used as the reference (<code>scaling.model=</code>) for subsequent regular<br />
merges. Iterate a few cycles; CC½ typically improves and converges over the first few rounds.<br />
<br />
== Important parameters ==<br />
<br />
Parameters are given by their full PHIL path with defaults in parentheses. The program exposes many<br />
more parameters (see <code>-c -e 10 -a 2</code>); those below are the ones most commonly set.<br />
<br />
=== Input ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>input.path</code> || None || Path(s) to the integrated data — a folder, several folders, or individual files; wildcards allowed; may be repeated. Matching <code>.expt</code> and <code>.refl</code> files must both be present.<br />
|-<br />
| <code>input.experiments_suffix</code> || <code>_integrated.expt</code> || Only load experiment files ending with this suffix.<br />
|-<br />
| <code>input.reflections_suffix</code> || <code>_integrated.refl</code> || Only load reflection files ending with this suffix.<br />
|-<br />
| <code>input.parallel_file_load.method</code> || uniform || <code>uniform</code> spreads input files evenly over all ranks; <code>node_memory</code> distributes so per-node memory limits are respected.<br />
|-<br />
| <code>input.alist.file</code> || None || Path to a text file of experiment tags or filenames; combined with <code>input.alist.op=keep|reject</code> to merge only (or exclude) a specified subset.<br />
|-<br />
| <code>input.persistent_refl_cols</code> || None || Reflection-table columns to preserve through the pruning steps (and into saved tables).<br />
|}<br />
<br />
=== Filtering whole experiments (<code>filter</code>) ===<br />
<br />
The <code>filter</code> section accepts or rejects '''entire experiments'''.<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>filter.algorithm</code> || (none) || One or more of <code>n_obs</code>, <code>resolution</code>, <code>unit_cell</code>, <code>energy</code>, <code>bootstrap</code>, <code>experiment_prefix</code>. Nothing is filtered unless listed.<br />
|-<br />
| <code>filter.unit_cell.algorithm</code> || value || How to restrict unit cells: <code>value</code> (compare to a target), <code>range</code>, or <code>cluster</code> (use a precomputed covariance clustering).<br />
|-<br />
| <code>filter.unit_cell.value.relative_length_tolerance</code> || 0.1 || Fractional deviation of cell edge lengths allowed vs. the target cell. Often tightened (e.g. 0.03).<br />
|-<br />
| <code>filter.unit_cell.value.absolute_angle_tolerance</code> || 2.0 || Allowed cell-angle deviation (degrees) vs. the target.<br />
|-<br />
| <code>filter.unit_cell.value.target_unit_cell</code> / <code>target_space_group</code> || Auto || Target cell/symmetry; <code>Auto</code> uses the scaling model.<br />
|-<br />
| <code>filter.outlier.min_corr</code> || 0.1 || Minimum correlation of an image's intensities to the reference model; images below this are rejected. Only applied when a scaling model is present. Use <code>-1</code> to accept all images regardless of correlation.<br />
|-<br />
| <code>filter.resolution.d_min</code> || None || Reject an experiment unless some of its reflections extend beyond this resolution.<br />
|}<br />
<br />
=== Selecting individual reflections (<code>select</code>) ===<br />
<br />
The <code>select</code> section accepts or rejects '''individual reflections''' (contrast with<br />
<code>filter</code>, which acts on whole experiments).<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>select.algorithm</code> || (none) || One or more of <code>significance_filter</code>, <code>panel</code>, <code>cspad_sensor</code>, <code>isolation_forest</code>, <code>target_energy</code>.<br />
|-<br />
| <code>select.significance_filter.sigma</code> || 0.5 || Trim each lattice at the resolution where the binned ⟨I/σ⟩ drops below this value. Often reduced to 0.1.<br />
|-<br />
| <code>select.significance_filter.min_ct</code> / <code>max_ct</code> || 10 / 50 || Target minimum/maximum mean reflections per resolution bin when choosing the number of bins. For large unit cells these are often raised (e.g. 200/300); for small cells, use the defaults.<br />
|}<br />
<br />
=== Modifying intensities (<code>modify</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>modify.algorithm</code> || polarization || Per-reflection corrections to apply (default applies the polarization correction).<br />
|-<br />
| <code>modify.cosym</code> || — || Resolve an indexing ambiguity across the ensemble using the Gildea/Winter (Brehm/Diederichs) cosym method, optionally anchored to <code>scaling.model</code>.<br />
|-<br />
| <code>modify.reindex_to_reference</code> / <code>reindex_to_abc</code> || — || Alternative re-indexing operators (to a reference model, or an explicit operator).<br />
|}<br />
<br />
=== Scaling (<code>scaling</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>scaling.model</code> || None || Reference dataset: a PDB file (uses the CRYST1 cell and computes model intensities, including bulk solvent) or an MTZ from a previous cycle. Required for <code>mark0</code> scaling.<br />
|-<br />
| <code>scaling.algorithm</code> || mark0 || <code>mark0</code>: per-image scaling against the reference. <code>mark1</code>: no scaling, just Monte-Carlo averaging (all scale factors = 1) — used for reference-free bootstrap.<br />
|-<br />
| <code>scaling.unit_cell</code> / <code>scaling.space_group</code> || None || Cell/symmetry to use when no model is provided (i.e. <code>mark1</code>).<br />
|-<br />
| <code>scaling.resolution_scalar</code> || 0.969 || <code>merging.d_min</code> is multiplied by this when deciding which reflections fall within the resolution limit, generating a few extra reference reflections to account for per-shot cell variation (e.g. set 0.96 for a few more).<br />
|-<br />
| <code>scaling.mtz.mtz_column_F</code> || fobs || Column name in the reference MTZ containing the reference structure factors (intensities or amplitudes).<br />
|}<br />
<br />
=== Post-refinement (<code>postrefinement</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>postrefinement.enable</code> || True || Enable post-refinement of per-image parameters/partialities (Sauter 2015).<br />
|-<br />
| <code>postrefinement.algorithm</code> || rs || Post-refinement protocol: <code>rs</code>, <code>rs2</code>, <code>rs_hybrid</code>, or <code>eta_deff</code>.<br />
|}<br />
<br />
=== Merging and error model (<code>merging</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>merging.d_min</code> || None || High-resolution limit for scaling and merging.<br />
|-<br />
| <code>merging.d_max</code> || None || Low-resolution limit (currently affects only CCiso).<br />
|-<br />
| <code>merging.merge_anomalous</code> || False || <code>True</code> merges Bijvoet mates together (higher multiplicity); <code>False</code> keeps them separate to preserve the anomalous signal.<br />
|-<br />
| <code>merging.minimum_multiplicity</code> || 2 || Do not produce a merged structure factor for HKLs measured fewer than this many times.<br />
|-<br />
| <code>merging.error.model</code> || mm24 || Merged-sigma error model: <code>mm24</code> (Mittan-Moreau 2025, current default), <code>ev11</code> (Evans 2011 / sdfac-refine, recommended in the README), <code>ha14</code>, or <code>errors_from_sample_residuals</code> (no refined error terms — used with <code>mark1</code> bootstrap).<br />
|-<br />
| <code>merging.set_average_unit_cell</code> || True || Output MTZ adopts the (cluster-derived) average unit cell of the data rather than the reference model's cell.<br />
|}<br />
<br />
=== Statistics (<code>statistics</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>statistics.n_bins</code> || 10 || Number of resolution bins in the statistics tables.<br />
|-<br />
| <code>statistics.cciso.mtz_file</code> || None || Isomorphous reference structure factors for R<sub>iso</sub> / CC<sub>iso</sub>.<br />
|-<br />
| <code>statistics.report_ML</code> || True || Report per-frame max-likelihood attributes (domain size, mosaicity).<br />
|}<br />
<br />
=== Output (<code>output</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>output.output_dir</code> || <code>.</code> || Directory for output files (created if it does not exist).<br />
|-<br />
| <code>output.prefix</code> || iobs || Prefix prepended to all output file names.<br />
|-<br />
| <code>output.save_experiments_and_reflections</code> || False || Dump the final experiments/reflections from the last worker as DIALS files — used for the two-stage scale-then-merge workflow.<br />
|-<br />
| <code>output.do_timing</code> || False || Log elapsed time per execution step.<br />
|-<br />
| <code>output.expanded_bookkeeping</code> || False || Store extra provenance columns (input file/reflection indices, odd/even flag, original id) in the saved reflection tables.<br />
|}<br />
<br />
=== Multiprocessing (<code>mp</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>mp.method</code> || mpi || Multiprocessing method (only <code>mpi</code> is currently supported).<br />
|-<br />
| <code>mp.debug.cProfile</code> || False || Enable code profiling; a <code>.prof</code> file is written per rank.<br />
|}<br />
<br />
== Output files ==<br />
<br />
The <code>merge</code> worker gathers all HKLs at rank 0 and writes three MTZ files, prefixed with<br />
<code>output.prefix</code> (default <code>iobs</code>):<br />
<br />
* <code>&lt;prefix&gt;_all.mtz</code> — the merged dataset using all observations (the main result for structure solution)<br />
* <code>&lt;prefix&gt;_odd.mtz</code> and <code>&lt;prefix&gt;_even.mtz</code> — the two half-datasets used for CC½ and related half-dataset statistics<br />
<br />
A main log records the per-step progress and the merging statistics tables by resolution bin:<br />
completeness, multiplicity and ⟨I/σ⟩, plus the "Table of Scaling Results" reporting CC½ (the<br />
half-dataset correlation, labelled "CC int"), CC<sub>iso</sub>, R<sub>int</sub>, R<sub>split</sub><br />
and R<sub>iso</sub>. The merged MTZ can optionally be<br />
uploaded to a shared [https://github.com/cctbx/cctbx_project/tree/master/xfel/merging/application/publish Google Drive folder] by adding the <code>publish</code> worker.<br />
<br />
== References ==<br />
<br />
* Sauter NK (2015). ''J Synchrotron Radiat'' '''22''', 239–48. (post-refinement)<br />
* Uervirojnangkoorn M, ''et al.'' (2015). ''eLife'' 10.7554/eLife.05421. (PRIME / bootstrap)<br />
* Brewster AS, ''et al.'' (2018). ''Acta Cryst.'' D '''74''', 877–894.<br />
* Brewster AS, ''et al.'' (2019a). ''Computational Crystallography Newsletter'' '''10''', 22–39. (XFEL GUI)<br />
* Brewster AS, ''et al.'' (2019b). ''Acta Cryst.'' D '''75''', 959–968. (error models)<br />
* Mittan-Moreau (2025). (mm24 error model)<br />
<br />
== See also ==<br />
<br />
* [[dials.stills_process]] — produces the integrated data that <code>cctbx.xfel.merge</code> consumes<br />
* [[cctbx.xfel GUI]] — orchestrates separate scaling and repeated merging as data are collected<br />
* The program README at <code>xfel/merging/README.md</code> in cctbx_project</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Cctbx.xfel.merge&diff=1780Cctbx.xfel.merge2026-08-21T23:18:48Z<p>Aaron: /* Two-stage scaling and merging */</p>
<hr />
<div>= ''cctbx.xfel.merge'' =<br />
<br />
'''''cctbx.xfel.merge''''' is the scaling, post-refinement and merging program of cctbx.xfel. It takes<br />
the per-image integrated results produced by ''[[dials.stills_process]]'' — one crystal per still shot —<br />
and reduces the whole ensemble to a single set of merged structure-factor intensities (an MTZ file)<br />
suitable for structure solution and refinement, together with the crystallographic quality<br />
statistics (completeness, multiplicity, ⟨I/σ⟩, CC½, CC<sub>iso</sub>, R-factors, etc.).<br />
<br />
Because serial datasets routinely contain tens of thousands to millions of still shots,<br />
<code>cctbx.xfel.merge</code> is built from the ground up to run under MPI across many nodes. It is<br />
almost always launched with <code>mpirun</code>/<code>srun</code>. Note that MPI uses the term 'ranks'<br />
to distinguish independent processes that are running across one or more nodes.<br />
<br />
== Overview ==<br />
<br />
<code>cctbx.xfel.merge</code> is organized as a sequence of '''workers'''. Each worker performs one<br />
task — reading data, applying a correction, filtering, scaling, post-refining, computing a statistic,<br />
or merging — and passes its (possibly modified) experiments and reflections on to the next worker.<br />
The list of workers to run is the "step list", and it is fully configurable: you can add, remove or<br />
reorder steps with <code>dispatch.step_list</code>. This is what makes the program flexible enough to<br />
scale-and-merge in one pass, to scale subsets separately and merge later, or to run a reference-free<br />
bootstrap.<br />
<br />
The '''default''' step list is:<br />
<br />
{| class="wikitable"<br />
! # !! Worker !! Purpose<br />
|-<br />
| 1 || <code>input</code> || Read integrated experiments (<code>.expt</code>) and reflections (<code>.refl</code>) from disk in DIALS format.<br />
|-<br />
| 2 || <code>balance</code> || Redistribute the input load evenly across MPI ranks (needed when there are fewer or more input files than ranks).<br />
|-<br />
| 3 || <code>model_scaling</code> || Build the full Miller list, model intensities from the reference, and the resolution binner used for scaling and post-refinement.<br />
|-<br />
| 4 || <code>modify</code> || Apply per-reflection modifications, e.g. polarization correction (and optionally re-indexing to resolve indexing ambiguity — see <code>modify.cosym</code>/<code>reindex_to_reference</code>).<br />
|-<br />
| 5 || <code>filter</code> || Reject whole experiments (or reflections) by resolution, unit cell, correlation to the reference, energy, etc.<br />
|-<br />
| 6 || <code>scale</code> || Scale each lattice to the reference dataset (<code>mark0</code>) or simply average (<code>mark1</code>).<br />
|-<br />
| 7 || <code>postrefine</code> || Post-refine per-image crystal parameters and partialities (Sauter 2015).<br />
|-<br />
| 8 || <code>statistics_unitcell</code> || Determine and store the average unit cell.<br />
|-<br />
| 9 || <code>statistics_beam</code> || Determine and store the average wavelength.<br />
|-<br />
| 10 || <code>model_statistics</code> || Rebuild the Miller list / binner for statistics (may use the average unit cell).<br />
|-<br />
| 11 || <code>statistics_resolution</code> || Per-crystal resolution statistics.<br />
|-<br />
| 12 || <code>group</code> || Redistribute reflections across ranks so that every observation of a given HKL is gathered on the same rank prior to merging.<br />
|-<br />
| 13 || <code>errors_merge</code> || Correct the merged error estimates with a per-HKL error model (e.g. Ev11, Brewster 2019b, or MM24 (the default), Mittan-Moreau (2025)).<br />
|-<br />
| 14 || <code>statistics_intensity</code> || Intensity statistics by resolution.<br />
|-<br />
| 15 || <code>merge</code> || Merge symmetry-equivalent intensities, gather all HKLs at rank 0, and write "odd", "even" and "all" MTZ files.<br />
|-<br />
| 16 || <code>statistics_intensity_cxi</code> || Correlation and R-factor statistics (ported from cxi.merge): the "Table of Scaling Results" — CC½ (half-dataset correlation, labelled "CC int"), CC<sub>iso</sub>, R<sub>int</sub>, R<sub>split</sub>, R<sub>iso</sub>.<br />
|}<br />
<br />
Additional, non-default workers are available and can be<br />
inserted into the step list, including <code>tdata</code> (write out unit cells for clustering),<br />
<code>export</code> (batch MTZ for ''Careless''), <code>publish</code> (upload merged files to a<br />
Google Drive folder), <code>monitor</code> (resource-usage logging), <code>integrate</code><br />
(on-the-fly integration), <code>prepare</code> (SpReAD energy binning), and<br />
<code>lunus</code> (diffuse-scattering deck). A <code>deltaccint</code> worker (ΔCC½ lattice<br />
filtering) can be added after <code>group</code>. Custom workers can be supplied through the<br />
<code>XFEL_CUSTOM_WORKER_PATH</code> environment variable.<br />
<br />
== Basic usage ==<br />
<br />
<code>cctbx.xfel.merge</code> is run under MPI:<br />
<br />
<pre><br />
mpirun cctbx.xfel.merge merge.phil<br />
</pre><br />
<br />
A typical <code>merge.phil</code>:<br />
<br />
<pre><br />
input.path=<path to data><br />
input.experiments_suffix=.expt<br />
input.reflections_suffix=.refl<br />
<br />
filter.algorithm=unit_cell<br />
filter.unit_cell.value.relative_length_tolerance=0.03<br />
filter.outlier.min_corr=0.1<br />
<br />
select.algorithm=significance_filter<br />
select.significance_filter.sigma=0.1<br />
select.significance_filter.min_ct=200<br />
select.significance_filter.max_ct=300<br />
<br />
scaling.model=<model.pdb or model.mtz><br />
scaling.resolution_scalar=0.96<br />
<br />
merging.d_min=1.8<br />
merging.error.model=mm24<br />
merging.merge_anomalous=True<br />
<br />
postrefinement.enable=True<br />
statistics.n_bins=20<br />
<br />
output.do_timing=True<br />
output.output_dir=<output folder><br />
output.prefix=<prefix for output files><br />
</pre><br />
<br />
Explore the parameters with:<br />
* <code>cctbx.xfel.merge -c</code> — show configurable parameters<br />
* <code>cctbx.xfel.merge -c -e 10 -a 2</code> — show all parameters (expert level 10) with full help text<br />
<br />
== Two-stage scaling and merging ==<br />
<br />
For large, growing datasets (the mode used by the [[cctbx.xfel GUI]]), it is more efficient to scale<br />
and post-refine each subset of data '''once''', then merge repeatedly as more data arrive. This is<br />
done by running the program twice with different step lists.<br />
<br />
'''Stage 1 — scale each subset''' (stops after per-image work and writes intermediate DIALS files):<br />
<br />
<pre><br />
dispatch.step_list=input balance model_scaling modify filter scale postrefine \<br />
statistics_unitcell statistics_beam model_statistics statistics_resolution<br />
output.save_experiments_and_reflections=True<br />
</pre><br />
<br />
<code>output.save_experiments_and_reflections=True</code> dumps the scaled/post-refined data as<br />
experiment lists and reflection tables that the merge stage can read back in.<br />
<br />
'''Stage 2 — merge the scaled subsets''' (add an <code>input.path</code> for each finished subset):<br />
<br />
<pre><br />
dispatch.step_list=input model_scaling statistics_unitcell statistics_beam \<br />
model_statistics statistics_resolution group errors_merge statistics_intensity \<br />
merge statistics_intensity_cxi<br />
</pre><br />
<br />
== Merging without a reference (bootstrap / mark1) ==<br />
<br />
If no scaling/merging reference is available, use a bootstrap procedure. First run the<br />
<code>mark1</code> algorithm, which simply averages the data (all per-image scale factors = 1, no<br />
post-refinement), specifying the target cell and symmetry explicitly:<br />
<br />
<pre><br />
scaling.algorithm=mark1<br />
scaling.unit_cell=<unit cell><br />
scaling.space_group=<space group><br />
merging.error.model=errors_from_sample_residuals<br />
</pre><br />
<br />
The resulting MTZ is then used as the reference (<code>scaling.model=</code>) for subsequent regular<br />
merges. Iterate a few cycles; CC½ typically improves and converges over the first few rounds.<br />
<br />
== Important parameters ==<br />
<br />
Parameters are given by their full PHIL path with defaults in parentheses. The program exposes many<br />
more parameters (see <code>-c -e 10 -a 2</code>); those below are the ones most commonly set.<br />
<br />
=== Input ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>input.path</code> || None || Path(s) to the integrated data — a folder, several folders, or individual files; wildcards allowed; may be repeated. Matching <code>.expt</code> and <code>.refl</code> files must both be present.<br />
|-<br />
| <code>input.experiments_suffix</code> || <code>_integrated.expt</code> || Only load experiment files ending with this suffix.<br />
|-<br />
| <code>input.reflections_suffix</code> || <code>_integrated.refl</code> || Only load reflection files ending with this suffix.<br />
|-<br />
| <code>input.parallel_file_load.method</code> || uniform || <code>uniform</code> spreads input files evenly over all ranks; <code>node_memory</code> distributes so per-node memory limits are respected.<br />
|-<br />
| <code>input.alist.file</code> || None || Path to a text file of experiment tags or filenames; combined with <code>input.alist.op=keep|reject</code> to merge only (or exclude) a specified subset.<br />
|-<br />
| <code>input.persistent_refl_cols</code> || None || Reflection-table columns to preserve through the pruning steps (and into saved tables).<br />
|}<br />
<br />
=== Filtering whole experiments (<code>filter</code>) ===<br />
<br />
The <code>filter</code> section accepts or rejects '''entire experiments'''.<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>filter.algorithm</code> || (none) || One or more of <code>n_obs</code>, <code>resolution</code>, <code>unit_cell</code>, <code>energy</code>, <code>bootstrap</code>, <code>experiment_prefix</code>. Nothing is filtered unless listed.<br />
|-<br />
| <code>filter.unit_cell.algorithm</code> || value || How to restrict unit cells: <code>value</code> (compare to a target), <code>range</code>, or <code>cluster</code> (use a precomputed covariance clustering).<br />
|-<br />
| <code>filter.unit_cell.value.relative_length_tolerance</code> || 0.1 || Fractional deviation of cell edge lengths allowed vs. the target cell. Often tightened (e.g. 0.03).<br />
|-<br />
| <code>filter.unit_cell.value.absolute_angle_tolerance</code> || 2.0 || Allowed cell-angle deviation (degrees) vs. the target.<br />
|-<br />
| <code>filter.unit_cell.value.target_unit_cell</code> / <code>target_space_group</code> || Auto || Target cell/symmetry; <code>Auto</code> uses the scaling model.<br />
|-<br />
| <code>filter.outlier.min_corr</code> || 0.1 || Minimum correlation of an image's intensities to the reference model; images below this are rejected. Only applied when a scaling model is present. Use <code>-1</code> to accept all images regardless of correlation.<br />
|-<br />
| <code>filter.resolution.d_min</code> || None || Reject an experiment unless some of its reflections extend beyond this resolution.<br />
|}<br />
<br />
=== Selecting individual reflections (<code>select</code>) ===<br />
<br />
The <code>select</code> section accepts or rejects '''individual reflections''' (contrast with<br />
<code>filter</code>, which acts on whole experiments).<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>select.algorithm</code> || (none) || One or more of <code>significance_filter</code>, <code>panel</code>, <code>cspad_sensor</code>, <code>isolation_forest</code>, <code>target_energy</code>.<br />
|-<br />
| <code>select.significance_filter.sigma</code> || 0.5 || Trim each lattice at the resolution where the binned ⟨I/σ⟩ drops below this value.<br />
|-<br />
| <code>select.significance_filter.min_ct</code> / <code>max_ct</code> || 10 / 50 || Target minimum/maximum mean reflections per resolution bin when choosing the number of bins. For large unit cells these are often raised (e.g. 200/300); for small cells, use the defaults.<br />
|}<br />
<br />
=== Modifying intensities (<code>modify</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>modify.algorithm</code> || polarization || Per-reflection corrections to apply (default applies the polarization correction).<br />
|-<br />
| <code>modify.cosym</code> || — || Resolve an indexing ambiguity across the ensemble using the Gildea/Winter (Brehm/Diederichs) cosym method, optionally anchored to <code>scaling.model</code>.<br />
|-<br />
| <code>modify.reindex_to_reference</code> / <code>reindex_to_abc</code> || — || Alternative re-indexing operators (to a reference model, or an explicit operator).<br />
|}<br />
<br />
=== Scaling (<code>scaling</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>scaling.model</code> || None || Reference dataset: a PDB file (uses the CRYST1 cell and computes model intensities, including bulk solvent) or an MTZ from a previous cycle. Required for <code>mark0</code> scaling.<br />
|-<br />
| <code>scaling.algorithm</code> || mark0 || <code>mark0</code>: per-image scaling against the reference. <code>mark1</code>: no scaling, just Monte-Carlo averaging (all scale factors = 1) — used for reference-free bootstrap.<br />
|-<br />
| <code>scaling.unit_cell</code> / <code>scaling.space_group</code> || None || Cell/symmetry to use when no model is provided (i.e. <code>mark1</code>).<br />
|-<br />
| <code>scaling.resolution_scalar</code> || 0.969 || <code>merging.d_min</code> is multiplied by this when deciding which reflections fall within the resolution limit, generating a few extra reference reflections to account for per-shot cell variation (e.g. set 0.96 for a few more).<br />
|-<br />
| <code>scaling.mtz.mtz_column_F</code> || fobs || Column name in the reference MTZ containing the reference structure factors (intensities or amplitudes).<br />
|}<br />
<br />
=== Post-refinement (<code>postrefinement</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>postrefinement.enable</code> || True || Enable post-refinement of per-image parameters/partialities (Sauter 2015).<br />
|-<br />
| <code>postrefinement.algorithm</code> || rs || Post-refinement protocol: <code>rs</code>, <code>rs2</code>, <code>rs_hybrid</code>, or <code>eta_deff</code>.<br />
|}<br />
<br />
=== Merging and error model (<code>merging</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>merging.d_min</code> || None || High-resolution limit for scaling and merging.<br />
|-<br />
| <code>merging.d_max</code> || None || Low-resolution limit (currently affects only CCiso).<br />
|-<br />
| <code>merging.merge_anomalous</code> || False || <code>True</code> merges Bijvoet mates together (higher multiplicity); <code>False</code> keeps them separate to preserve the anomalous signal.<br />
|-<br />
| <code>merging.minimum_multiplicity</code> || 2 || Do not produce a merged structure factor for HKLs measured fewer than this many times.<br />
|-<br />
| <code>merging.error.model</code> || mm24 || Merged-sigma error model: <code>mm24</code> (Mittan-Moreau 2025, current default), <code>ev11</code> (Evans 2011 / sdfac-refine, recommended in the README), <code>ha14</code>, or <code>errors_from_sample_residuals</code> (no refined error terms — used with <code>mark1</code> bootstrap).<br />
|-<br />
| <code>merging.set_average_unit_cell</code> || True || Output MTZ adopts the (cluster-derived) average unit cell of the data rather than the reference model's cell.<br />
|}<br />
<br />
=== Statistics (<code>statistics</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>statistics.n_bins</code> || 10 || Number of resolution bins in the statistics tables.<br />
|-<br />
| <code>statistics.cciso.mtz_file</code> || None || Isomorphous reference structure factors for R<sub>iso</sub> / CC<sub>iso</sub>.<br />
|-<br />
| <code>statistics.report_ML</code> || True || Report per-frame max-likelihood attributes (domain size, mosaicity).<br />
|}<br />
<br />
=== Output (<code>output</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>output.output_dir</code> || <code>.</code> || Directory for output files (created if it does not exist).<br />
|-<br />
| <code>output.prefix</code> || iobs || Prefix prepended to all output file names.<br />
|-<br />
| <code>output.save_experiments_and_reflections</code> || False || Dump the final experiments/reflections from the last worker as DIALS files — used for the two-stage scale-then-merge workflow.<br />
|-<br />
| <code>output.do_timing</code> || False || Log elapsed time per execution step.<br />
|-<br />
| <code>output.expanded_bookkeeping</code> || False || Store extra provenance columns (input file/reflection indices, odd/even flag, original id) in the saved reflection tables.<br />
|}<br />
<br />
=== Multiprocessing (<code>mp</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>mp.method</code> || mpi || Multiprocessing method (only <code>mpi</code> is currently supported).<br />
|-<br />
| <code>mp.debug.cProfile</code> || False || Enable code profiling; a <code>.prof</code> file is written per rank.<br />
|}<br />
<br />
== Output files ==<br />
<br />
The <code>merge</code> worker gathers all HKLs at rank 0 and writes three MTZ files, prefixed with<br />
<code>output.prefix</code> (default <code>iobs</code>):<br />
<br />
* <code>&lt;prefix&gt;_all.mtz</code> — the merged dataset using all observations (the main result for structure solution)<br />
* <code>&lt;prefix&gt;_odd.mtz</code> and <code>&lt;prefix&gt;_even.mtz</code> — the two half-datasets used for CC½ and related half-dataset statistics<br />
<br />
A main log records the per-step progress and the merging statistics tables by resolution bin:<br />
completeness, multiplicity and ⟨I/σ⟩, plus the "Table of Scaling Results" reporting CC½ (the<br />
half-dataset correlation, labelled "CC int"), CC<sub>iso</sub>, R<sub>int</sub>, R<sub>split</sub><br />
and R<sub>iso</sub>. The merged MTZ can optionally be<br />
uploaded to a shared [https://github.com/cctbx/cctbx_project/tree/master/xfel/merging/application/publish Google Drive folder] by adding the <code>publish</code> worker.<br />
<br />
== References ==<br />
<br />
* Sauter NK (2015). ''J Synchrotron Radiat'' '''22''', 239–48. (post-refinement)<br />
* Uervirojnangkoorn M, ''et al.'' (2015). ''eLife'' 10.7554/eLife.05421. (PRIME / bootstrap)<br />
* Brewster AS, ''et al.'' (2018). ''Acta Cryst.'' D '''74''', 877–894.<br />
* Brewster AS, ''et al.'' (2019a). ''Computational Crystallography Newsletter'' '''10''', 22–39. (XFEL GUI)<br />
* Brewster AS, ''et al.'' (2019b). ''Acta Cryst.'' D '''75''', 959–968. (error models)<br />
* Mittan-Moreau (2025). (mm24 error model)<br />
<br />
== See also ==<br />
<br />
* [[dials.stills_process]] — produces the integrated data that <code>cctbx.xfel.merge</code> consumes<br />
* [[cctbx.xfel GUI]] — orchestrates separate scaling and repeated merging as data are collected<br />
* The program README at <code>xfel/merging/README.md</code> in cctbx_project</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Cctbx.xfel.merge&diff=1779Cctbx.xfel.merge2026-08-21T23:17:17Z<p>Aaron: /* Basic usage */</p>
<hr />
<div>= ''cctbx.xfel.merge'' =<br />
<br />
'''''cctbx.xfel.merge''''' is the scaling, post-refinement and merging program of cctbx.xfel. It takes<br />
the per-image integrated results produced by ''[[dials.stills_process]]'' — one crystal per still shot —<br />
and reduces the whole ensemble to a single set of merged structure-factor intensities (an MTZ file)<br />
suitable for structure solution and refinement, together with the crystallographic quality<br />
statistics (completeness, multiplicity, ⟨I/σ⟩, CC½, CC<sub>iso</sub>, R-factors, etc.).<br />
<br />
Because serial datasets routinely contain tens of thousands to millions of still shots,<br />
<code>cctbx.xfel.merge</code> is built from the ground up to run under MPI across many nodes. It is<br />
almost always launched with <code>mpirun</code>/<code>srun</code>. Note that MPI uses the term 'ranks'<br />
to distinguish independent processes that are running across one or more nodes.<br />
<br />
== Overview ==<br />
<br />
<code>cctbx.xfel.merge</code> is organized as a sequence of '''workers'''. Each worker performs one<br />
task — reading data, applying a correction, filtering, scaling, post-refining, computing a statistic,<br />
or merging — and passes its (possibly modified) experiments and reflections on to the next worker.<br />
The list of workers to run is the "step list", and it is fully configurable: you can add, remove or<br />
reorder steps with <code>dispatch.step_list</code>. This is what makes the program flexible enough to<br />
scale-and-merge in one pass, to scale subsets separately and merge later, or to run a reference-free<br />
bootstrap.<br />
<br />
The '''default''' step list is:<br />
<br />
{| class="wikitable"<br />
! # !! Worker !! Purpose<br />
|-<br />
| 1 || <code>input</code> || Read integrated experiments (<code>.expt</code>) and reflections (<code>.refl</code>) from disk in DIALS format.<br />
|-<br />
| 2 || <code>balance</code> || Redistribute the input load evenly across MPI ranks (needed when there are fewer or more input files than ranks).<br />
|-<br />
| 3 || <code>model_scaling</code> || Build the full Miller list, model intensities from the reference, and the resolution binner used for scaling and post-refinement.<br />
|-<br />
| 4 || <code>modify</code> || Apply per-reflection modifications, e.g. polarization correction (and optionally re-indexing to resolve indexing ambiguity — see <code>modify.cosym</code>/<code>reindex_to_reference</code>).<br />
|-<br />
| 5 || <code>filter</code> || Reject whole experiments (or reflections) by resolution, unit cell, correlation to the reference, energy, etc.<br />
|-<br />
| 6 || <code>scale</code> || Scale each lattice to the reference dataset (<code>mark0</code>) or simply average (<code>mark1</code>).<br />
|-<br />
| 7 || <code>postrefine</code> || Post-refine per-image crystal parameters and partialities (Sauter 2015).<br />
|-<br />
| 8 || <code>statistics_unitcell</code> || Determine and store the average unit cell.<br />
|-<br />
| 9 || <code>statistics_beam</code> || Determine and store the average wavelength.<br />
|-<br />
| 10 || <code>model_statistics</code> || Rebuild the Miller list / binner for statistics (may use the average unit cell).<br />
|-<br />
| 11 || <code>statistics_resolution</code> || Per-crystal resolution statistics.<br />
|-<br />
| 12 || <code>group</code> || Redistribute reflections across ranks so that every observation of a given HKL is gathered on the same rank prior to merging.<br />
|-<br />
| 13 || <code>errors_merge</code> || Correct the merged error estimates with a per-HKL error model (e.g. Ev11, Brewster 2019b, or MM24 (the default), Mittan-Moreau (2025)).<br />
|-<br />
| 14 || <code>statistics_intensity</code> || Intensity statistics by resolution.<br />
|-<br />
| 15 || <code>merge</code> || Merge symmetry-equivalent intensities, gather all HKLs at rank 0, and write "odd", "even" and "all" MTZ files.<br />
|-<br />
| 16 || <code>statistics_intensity_cxi</code> || Correlation and R-factor statistics (ported from cxi.merge): the "Table of Scaling Results" — CC½ (half-dataset correlation, labelled "CC int"), CC<sub>iso</sub>, R<sub>int</sub>, R<sub>split</sub>, R<sub>iso</sub>.<br />
|}<br />
<br />
Additional, non-default workers are available and can be<br />
inserted into the step list, including <code>tdata</code> (write out unit cells for clustering),<br />
<code>export</code> (batch MTZ for ''Careless''), <code>publish</code> (upload merged files to a<br />
Google Drive folder), <code>monitor</code> (resource-usage logging), <code>integrate</code><br />
(on-the-fly integration), <code>prepare</code> (SpReAD energy binning), and<br />
<code>lunus</code> (diffuse-scattering deck). A <code>deltaccint</code> worker (ΔCC½ lattice<br />
filtering) can be added after <code>group</code>. Custom workers can be supplied through the<br />
<code>XFEL_CUSTOM_WORKER_PATH</code> environment variable.<br />
<br />
== Basic usage ==<br />
<br />
<code>cctbx.xfel.merge</code> is run under MPI:<br />
<br />
<pre><br />
mpirun cctbx.xfel.merge merge.phil<br />
</pre><br />
<br />
A typical <code>merge.phil</code>:<br />
<br />
<pre><br />
input.path=<path to data><br />
input.experiments_suffix=.expt<br />
input.reflections_suffix=.refl<br />
<br />
filter.algorithm=unit_cell<br />
filter.unit_cell.value.relative_length_tolerance=0.03<br />
filter.outlier.min_corr=0.1<br />
<br />
select.algorithm=significance_filter<br />
select.significance_filter.sigma=0.1<br />
select.significance_filter.min_ct=200<br />
select.significance_filter.max_ct=300<br />
<br />
scaling.model=<model.pdb or model.mtz><br />
scaling.resolution_scalar=0.96<br />
<br />
merging.d_min=1.8<br />
merging.error.model=mm24<br />
merging.merge_anomalous=True<br />
<br />
postrefinement.enable=True<br />
statistics.n_bins=20<br />
<br />
output.do_timing=True<br />
output.output_dir=<output folder><br />
output.prefix=<prefix for output files><br />
</pre><br />
<br />
Explore the parameters with:<br />
* <code>cctbx.xfel.merge -c</code> — show configurable parameters<br />
* <code>cctbx.xfel.merge -c -e 10 -a 2</code> — show all parameters (expert level 10) with full help text<br />
<br />
== Two-stage scaling and merging ==<br />
<br />
For large, growing datasets (the mode used by the [[cctbx.xfel GUI]]), it is more efficient to scale<br />
and post-refine each subset of data '''once''', then merge repeatedly as more data arrive. This is<br />
done by running the program twice with different step lists.<br />
<br />
'''Stage 1 — scale each subset''' (stops after per-image work and writes intermediate DIALS files):<br />
<br />
<pre><br />
dispatch.step_list=input balance model_scaling modify filter scale postrefine \<br />
statistics_unitcell statistics_beam model_statistics statistics_resolution<br />
input.parallel_file_load.method=uniform<br />
output.save_experiments_and_reflections=True<br />
</pre><br />
<br />
<code>output.save_experiments_and_reflections=True</code> dumps the scaled/post-refined data as<br />
experiment lists and reflection tables that the merge stage can read back in.<br />
<br />
'''Stage 2 — merge the scaled subsets''' (add an <code>input.path</code> for each finished subset):<br />
<br />
<pre><br />
dispatch.step_list=input model_scaling statistics_unitcell statistics_beam \<br />
model_statistics statistics_resolution group errors_merge statistics_intensity \<br />
merge statistics_intensity_cxi<br />
input.parallel_file_load.method=uniform<br />
</pre><br />
<br />
== Merging without a reference (bootstrap / mark1) ==<br />
<br />
If no scaling/merging reference is available, use a bootstrap procedure. First run the<br />
<code>mark1</code> algorithm, which simply averages the data (all per-image scale factors = 1, no<br />
post-refinement), specifying the target cell and symmetry explicitly:<br />
<br />
<pre><br />
scaling.algorithm=mark1<br />
scaling.unit_cell=<unit cell><br />
scaling.space_group=<space group><br />
merging.error.model=errors_from_sample_residuals<br />
</pre><br />
<br />
The resulting MTZ is then used as the reference (<code>scaling.model=</code>) for subsequent regular<br />
merges. Iterate a few cycles; CC½ typically improves and converges over the first few rounds.<br />
<br />
== Important parameters ==<br />
<br />
Parameters are given by their full PHIL path with defaults in parentheses. The program exposes many<br />
more parameters (see <code>-c -e 10 -a 2</code>); those below are the ones most commonly set.<br />
<br />
=== Input ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>input.path</code> || None || Path(s) to the integrated data — a folder, several folders, or individual files; wildcards allowed; may be repeated. Matching <code>.expt</code> and <code>.refl</code> files must both be present.<br />
|-<br />
| <code>input.experiments_suffix</code> || <code>_integrated.expt</code> || Only load experiment files ending with this suffix.<br />
|-<br />
| <code>input.reflections_suffix</code> || <code>_integrated.refl</code> || Only load reflection files ending with this suffix.<br />
|-<br />
| <code>input.parallel_file_load.method</code> || uniform || <code>uniform</code> spreads input files evenly over all ranks; <code>node_memory</code> distributes so per-node memory limits are respected.<br />
|-<br />
| <code>input.alist.file</code> || None || Path to a text file of experiment tags or filenames; combined with <code>input.alist.op=keep|reject</code> to merge only (or exclude) a specified subset.<br />
|-<br />
| <code>input.persistent_refl_cols</code> || None || Reflection-table columns to preserve through the pruning steps (and into saved tables).<br />
|}<br />
<br />
=== Filtering whole experiments (<code>filter</code>) ===<br />
<br />
The <code>filter</code> section accepts or rejects '''entire experiments'''.<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>filter.algorithm</code> || (none) || One or more of <code>n_obs</code>, <code>resolution</code>, <code>unit_cell</code>, <code>energy</code>, <code>bootstrap</code>, <code>experiment_prefix</code>. Nothing is filtered unless listed.<br />
|-<br />
| <code>filter.unit_cell.algorithm</code> || value || How to restrict unit cells: <code>value</code> (compare to a target), <code>range</code>, or <code>cluster</code> (use a precomputed covariance clustering).<br />
|-<br />
| <code>filter.unit_cell.value.relative_length_tolerance</code> || 0.1 || Fractional deviation of cell edge lengths allowed vs. the target cell. Often tightened (e.g. 0.03).<br />
|-<br />
| <code>filter.unit_cell.value.absolute_angle_tolerance</code> || 2.0 || Allowed cell-angle deviation (degrees) vs. the target.<br />
|-<br />
| <code>filter.unit_cell.value.target_unit_cell</code> / <code>target_space_group</code> || Auto || Target cell/symmetry; <code>Auto</code> uses the scaling model.<br />
|-<br />
| <code>filter.outlier.min_corr</code> || 0.1 || Minimum correlation of an image's intensities to the reference model; images below this are rejected. Only applied when a scaling model is present. Use <code>-1</code> to accept all images regardless of correlation.<br />
|-<br />
| <code>filter.resolution.d_min</code> || None || Reject an experiment unless some of its reflections extend beyond this resolution.<br />
|}<br />
<br />
=== Selecting individual reflections (<code>select</code>) ===<br />
<br />
The <code>select</code> section accepts or rejects '''individual reflections''' (contrast with<br />
<code>filter</code>, which acts on whole experiments).<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>select.algorithm</code> || (none) || One or more of <code>significance_filter</code>, <code>panel</code>, <code>cspad_sensor</code>, <code>isolation_forest</code>, <code>target_energy</code>.<br />
|-<br />
| <code>select.significance_filter.sigma</code> || 0.5 || Trim each lattice at the resolution where the binned ⟨I/σ⟩ drops below this value.<br />
|-<br />
| <code>select.significance_filter.min_ct</code> / <code>max_ct</code> || 10 / 50 || Target minimum/maximum mean reflections per resolution bin when choosing the number of bins. For large unit cells these are often raised (e.g. 200/300); for small cells, use the defaults.<br />
|}<br />
<br />
=== Modifying intensities (<code>modify</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>modify.algorithm</code> || polarization || Per-reflection corrections to apply (default applies the polarization correction).<br />
|-<br />
| <code>modify.cosym</code> || — || Resolve an indexing ambiguity across the ensemble using the Gildea/Winter (Brehm/Diederichs) cosym method, optionally anchored to <code>scaling.model</code>.<br />
|-<br />
| <code>modify.reindex_to_reference</code> / <code>reindex_to_abc</code> || — || Alternative re-indexing operators (to a reference model, or an explicit operator).<br />
|}<br />
<br />
=== Scaling (<code>scaling</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>scaling.model</code> || None || Reference dataset: a PDB file (uses the CRYST1 cell and computes model intensities, including bulk solvent) or an MTZ from a previous cycle. Required for <code>mark0</code> scaling.<br />
|-<br />
| <code>scaling.algorithm</code> || mark0 || <code>mark0</code>: per-image scaling against the reference. <code>mark1</code>: no scaling, just Monte-Carlo averaging (all scale factors = 1) — used for reference-free bootstrap.<br />
|-<br />
| <code>scaling.unit_cell</code> / <code>scaling.space_group</code> || None || Cell/symmetry to use when no model is provided (i.e. <code>mark1</code>).<br />
|-<br />
| <code>scaling.resolution_scalar</code> || 0.969 || <code>merging.d_min</code> is multiplied by this when deciding which reflections fall within the resolution limit, generating a few extra reference reflections to account for per-shot cell variation (e.g. set 0.96 for a few more).<br />
|-<br />
| <code>scaling.mtz.mtz_column_F</code> || fobs || Column name in the reference MTZ containing the reference structure factors (intensities or amplitudes).<br />
|}<br />
<br />
=== Post-refinement (<code>postrefinement</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>postrefinement.enable</code> || True || Enable post-refinement of per-image parameters/partialities (Sauter 2015).<br />
|-<br />
| <code>postrefinement.algorithm</code> || rs || Post-refinement protocol: <code>rs</code>, <code>rs2</code>, <code>rs_hybrid</code>, or <code>eta_deff</code>.<br />
|}<br />
<br />
=== Merging and error model (<code>merging</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>merging.d_min</code> || None || High-resolution limit for scaling and merging.<br />
|-<br />
| <code>merging.d_max</code> || None || Low-resolution limit (currently affects only CCiso).<br />
|-<br />
| <code>merging.merge_anomalous</code> || False || <code>True</code> merges Bijvoet mates together (higher multiplicity); <code>False</code> keeps them separate to preserve the anomalous signal.<br />
|-<br />
| <code>merging.minimum_multiplicity</code> || 2 || Do not produce a merged structure factor for HKLs measured fewer than this many times.<br />
|-<br />
| <code>merging.error.model</code> || mm24 || Merged-sigma error model: <code>mm24</code> (Mittan-Moreau 2025, current default), <code>ev11</code> (Evans 2011 / sdfac-refine, recommended in the README), <code>ha14</code>, or <code>errors_from_sample_residuals</code> (no refined error terms — used with <code>mark1</code> bootstrap).<br />
|-<br />
| <code>merging.set_average_unit_cell</code> || True || Output MTZ adopts the (cluster-derived) average unit cell of the data rather than the reference model's cell.<br />
|}<br />
<br />
=== Statistics (<code>statistics</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>statistics.n_bins</code> || 10 || Number of resolution bins in the statistics tables.<br />
|-<br />
| <code>statistics.cciso.mtz_file</code> || None || Isomorphous reference structure factors for R<sub>iso</sub> / CC<sub>iso</sub>.<br />
|-<br />
| <code>statistics.report_ML</code> || True || Report per-frame max-likelihood attributes (domain size, mosaicity).<br />
|}<br />
<br />
=== Output (<code>output</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>output.output_dir</code> || <code>.</code> || Directory for output files (created if it does not exist).<br />
|-<br />
| <code>output.prefix</code> || iobs || Prefix prepended to all output file names.<br />
|-<br />
| <code>output.save_experiments_and_reflections</code> || False || Dump the final experiments/reflections from the last worker as DIALS files — used for the two-stage scale-then-merge workflow.<br />
|-<br />
| <code>output.do_timing</code> || False || Log elapsed time per execution step.<br />
|-<br />
| <code>output.expanded_bookkeeping</code> || False || Store extra provenance columns (input file/reflection indices, odd/even flag, original id) in the saved reflection tables.<br />
|}<br />
<br />
=== Multiprocessing (<code>mp</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>mp.method</code> || mpi || Multiprocessing method (only <code>mpi</code> is currently supported).<br />
|-<br />
| <code>mp.debug.cProfile</code> || False || Enable code profiling; a <code>.prof</code> file is written per rank.<br />
|}<br />
<br />
== Output files ==<br />
<br />
The <code>merge</code> worker gathers all HKLs at rank 0 and writes three MTZ files, prefixed with<br />
<code>output.prefix</code> (default <code>iobs</code>):<br />
<br />
* <code>&lt;prefix&gt;_all.mtz</code> — the merged dataset using all observations (the main result for structure solution)<br />
* <code>&lt;prefix&gt;_odd.mtz</code> and <code>&lt;prefix&gt;_even.mtz</code> — the two half-datasets used for CC½ and related half-dataset statistics<br />
<br />
A main log records the per-step progress and the merging statistics tables by resolution bin:<br />
completeness, multiplicity and ⟨I/σ⟩, plus the "Table of Scaling Results" reporting CC½ (the<br />
half-dataset correlation, labelled "CC int"), CC<sub>iso</sub>, R<sub>int</sub>, R<sub>split</sub><br />
and R<sub>iso</sub>. The merged MTZ can optionally be<br />
uploaded to a shared [https://github.com/cctbx/cctbx_project/tree/master/xfel/merging/application/publish Google Drive folder] by adding the <code>publish</code> worker.<br />
<br />
== References ==<br />
<br />
* Sauter NK (2015). ''J Synchrotron Radiat'' '''22''', 239–48. (post-refinement)<br />
* Uervirojnangkoorn M, ''et al.'' (2015). ''eLife'' 10.7554/eLife.05421. (PRIME / bootstrap)<br />
* Brewster AS, ''et al.'' (2018). ''Acta Cryst.'' D '''74''', 877–894.<br />
* Brewster AS, ''et al.'' (2019a). ''Computational Crystallography Newsletter'' '''10''', 22–39. (XFEL GUI)<br />
* Brewster AS, ''et al.'' (2019b). ''Acta Cryst.'' D '''75''', 959–968. (error models)<br />
* Mittan-Moreau (2025). (mm24 error model)<br />
<br />
== See also ==<br />
<br />
* [[dials.stills_process]] — produces the integrated data that <code>cctbx.xfel.merge</code> consumes<br />
* [[cctbx.xfel GUI]] — orchestrates separate scaling and repeated merging as data are collected<br />
* The program README at <code>xfel/merging/README.md</code> in cctbx_project</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Cctbx.xfel.merge&diff=1778Cctbx.xfel.merge2026-08-21T23:15:21Z<p>Aaron: /* Merging and error model (merging) */</p>
<hr />
<div>= ''cctbx.xfel.merge'' =<br />
<br />
'''''cctbx.xfel.merge''''' is the scaling, post-refinement and merging program of cctbx.xfel. It takes<br />
the per-image integrated results produced by ''[[dials.stills_process]]'' — one crystal per still shot —<br />
and reduces the whole ensemble to a single set of merged structure-factor intensities (an MTZ file)<br />
suitable for structure solution and refinement, together with the crystallographic quality<br />
statistics (completeness, multiplicity, ⟨I/σ⟩, CC½, CC<sub>iso</sub>, R-factors, etc.).<br />
<br />
Because serial datasets routinely contain tens of thousands to millions of still shots,<br />
<code>cctbx.xfel.merge</code> is built from the ground up to run under MPI across many nodes. It is<br />
almost always launched with <code>mpirun</code>/<code>srun</code>. Note that MPI uses the term 'ranks'<br />
to distinguish independent processes that are running across one or more nodes.<br />
<br />
== Overview ==<br />
<br />
<code>cctbx.xfel.merge</code> is organized as a sequence of '''workers'''. Each worker performs one<br />
task — reading data, applying a correction, filtering, scaling, post-refining, computing a statistic,<br />
or merging — and passes its (possibly modified) experiments and reflections on to the next worker.<br />
The list of workers to run is the "step list", and it is fully configurable: you can add, remove or<br />
reorder steps with <code>dispatch.step_list</code>. This is what makes the program flexible enough to<br />
scale-and-merge in one pass, to scale subsets separately and merge later, or to run a reference-free<br />
bootstrap.<br />
<br />
The '''default''' step list is:<br />
<br />
{| class="wikitable"<br />
! # !! Worker !! Purpose<br />
|-<br />
| 1 || <code>input</code> || Read integrated experiments (<code>.expt</code>) and reflections (<code>.refl</code>) from disk in DIALS format.<br />
|-<br />
| 2 || <code>balance</code> || Redistribute the input load evenly across MPI ranks (needed when there are fewer or more input files than ranks).<br />
|-<br />
| 3 || <code>model_scaling</code> || Build the full Miller list, model intensities from the reference, and the resolution binner used for scaling and post-refinement.<br />
|-<br />
| 4 || <code>modify</code> || Apply per-reflection modifications, e.g. polarization correction (and optionally re-indexing to resolve indexing ambiguity — see <code>modify.cosym</code>/<code>reindex_to_reference</code>).<br />
|-<br />
| 5 || <code>filter</code> || Reject whole experiments (or reflections) by resolution, unit cell, correlation to the reference, energy, etc.<br />
|-<br />
| 6 || <code>scale</code> || Scale each lattice to the reference dataset (<code>mark0</code>) or simply average (<code>mark1</code>).<br />
|-<br />
| 7 || <code>postrefine</code> || Post-refine per-image crystal parameters and partialities (Sauter 2015).<br />
|-<br />
| 8 || <code>statistics_unitcell</code> || Determine and store the average unit cell.<br />
|-<br />
| 9 || <code>statistics_beam</code> || Determine and store the average wavelength.<br />
|-<br />
| 10 || <code>model_statistics</code> || Rebuild the Miller list / binner for statistics (may use the average unit cell).<br />
|-<br />
| 11 || <code>statistics_resolution</code> || Per-crystal resolution statistics.<br />
|-<br />
| 12 || <code>group</code> || Redistribute reflections across ranks so that every observation of a given HKL is gathered on the same rank prior to merging.<br />
|-<br />
| 13 || <code>errors_merge</code> || Correct the merged error estimates with a per-HKL error model (e.g. Ev11, Brewster 2019b, or MM24 (the default), Mittan-Moreau (2025)).<br />
|-<br />
| 14 || <code>statistics_intensity</code> || Intensity statistics by resolution.<br />
|-<br />
| 15 || <code>merge</code> || Merge symmetry-equivalent intensities, gather all HKLs at rank 0, and write "odd", "even" and "all" MTZ files.<br />
|-<br />
| 16 || <code>statistics_intensity_cxi</code> || Correlation and R-factor statistics (ported from cxi.merge): the "Table of Scaling Results" — CC½ (half-dataset correlation, labelled "CC int"), CC<sub>iso</sub>, R<sub>int</sub>, R<sub>split</sub>, R<sub>iso</sub>.<br />
|}<br />
<br />
Additional, non-default workers are available and can be<br />
inserted into the step list, including <code>tdata</code> (write out unit cells for clustering),<br />
<code>export</code> (batch MTZ for ''Careless''), <code>publish</code> (upload merged files to a<br />
Google Drive folder), <code>monitor</code> (resource-usage logging), <code>integrate</code><br />
(on-the-fly integration), <code>prepare</code> (SpReAD energy binning), and<br />
<code>lunus</code> (diffuse-scattering deck). A <code>deltaccint</code> worker (ΔCC½ lattice<br />
filtering) can be added after <code>group</code>. Custom workers can be supplied through the<br />
<code>XFEL_CUSTOM_WORKER_PATH</code> environment variable.<br />
<br />
== Basic usage ==<br />
<br />
<code>cctbx.xfel.merge</code> is run under MPI:<br />
<br />
<pre><br />
mpirun cctbx.xfel.merge merge.phil<br />
</pre><br />
<br />
A typical <code>merge.phil</code> (from the program README):<br />
<br />
<pre><br />
input.path=<path to data><br />
input.experiments_suffix=.expt<br />
input.reflections_suffix=.refl<br />
<br />
filter.algorithm=unit_cell<br />
filter.unit_cell.value.relative_length_tolerance=0.03<br />
filter.outlier.min_corr=0.1<br />
<br />
select.algorithm=significance_filter<br />
select.significance_filter.sigma=0.1<br />
select.significance_filter.min_ct=200<br />
select.significance_filter.max_ct=300<br />
<br />
scaling.model=<model.pdb or model.mtz><br />
scaling.resolution_scalar=0.96<br />
<br />
merging.d_min=1.8<br />
merging.error.model=ev11<br />
merging.merge_anomalous=True<br />
<br />
postrefinement.enable=True<br />
statistics.n_bins=20<br />
<br />
output.do_timing=True<br />
output.output_dir=<output folder><br />
output.prefix=<prefix for output files><br />
</pre><br />
<br />
Explore the parameters with:<br />
* <code>cctbx.xfel.merge -c</code> — show configurable parameters<br />
* <code>cctbx.xfel.merge -c -e 10 -a 2</code> — show all parameters (expert level 10) with full help text<br />
<br />
== Two-stage scaling and merging ==<br />
<br />
For large, growing datasets (the mode used by the [[cctbx.xfel GUI]]), it is more efficient to scale<br />
and post-refine each subset of data '''once''', then merge repeatedly as more data arrive. This is<br />
done by running the program twice with different step lists.<br />
<br />
'''Stage 1 — scale each subset''' (stops after per-image work and writes intermediate DIALS files):<br />
<br />
<pre><br />
dispatch.step_list=input balance model_scaling modify filter scale postrefine \<br />
statistics_unitcell statistics_beam model_statistics statistics_resolution<br />
input.parallel_file_load.method=uniform<br />
output.save_experiments_and_reflections=True<br />
</pre><br />
<br />
<code>output.save_experiments_and_reflections=True</code> dumps the scaled/post-refined data as<br />
experiment lists and reflection tables that the merge stage can read back in.<br />
<br />
'''Stage 2 — merge the scaled subsets''' (add an <code>input.path</code> for each finished subset):<br />
<br />
<pre><br />
dispatch.step_list=input model_scaling statistics_unitcell statistics_beam \<br />
model_statistics statistics_resolution group errors_merge statistics_intensity \<br />
merge statistics_intensity_cxi<br />
input.parallel_file_load.method=uniform<br />
</pre><br />
<br />
== Merging without a reference (bootstrap / mark1) ==<br />
<br />
If no scaling/merging reference is available, use a bootstrap procedure. First run the<br />
<code>mark1</code> algorithm, which simply averages the data (all per-image scale factors = 1, no<br />
post-refinement), specifying the target cell and symmetry explicitly:<br />
<br />
<pre><br />
scaling.algorithm=mark1<br />
scaling.unit_cell=<unit cell><br />
scaling.space_group=<space group><br />
merging.error.model=errors_from_sample_residuals<br />
</pre><br />
<br />
The resulting MTZ is then used as the reference (<code>scaling.model=</code>) for subsequent regular<br />
merges. Iterate a few cycles; CC½ typically improves and converges over the first few rounds.<br />
<br />
== Important parameters ==<br />
<br />
Parameters are given by their full PHIL path with defaults in parentheses. The program exposes many<br />
more parameters (see <code>-c -e 10 -a 2</code>); those below are the ones most commonly set.<br />
<br />
=== Input ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>input.path</code> || None || Path(s) to the integrated data — a folder, several folders, or individual files; wildcards allowed; may be repeated. Matching <code>.expt</code> and <code>.refl</code> files must both be present.<br />
|-<br />
| <code>input.experiments_suffix</code> || <code>_integrated.expt</code> || Only load experiment files ending with this suffix.<br />
|-<br />
| <code>input.reflections_suffix</code> || <code>_integrated.refl</code> || Only load reflection files ending with this suffix.<br />
|-<br />
| <code>input.parallel_file_load.method</code> || uniform || <code>uniform</code> spreads input files evenly over all ranks; <code>node_memory</code> distributes so per-node memory limits are respected.<br />
|-<br />
| <code>input.alist.file</code> || None || Path to a text file of experiment tags or filenames; combined with <code>input.alist.op=keep|reject</code> to merge only (or exclude) a specified subset.<br />
|-<br />
| <code>input.persistent_refl_cols</code> || None || Reflection-table columns to preserve through the pruning steps (and into saved tables).<br />
|}<br />
<br />
=== Filtering whole experiments (<code>filter</code>) ===<br />
<br />
The <code>filter</code> section accepts or rejects '''entire experiments'''.<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>filter.algorithm</code> || (none) || One or more of <code>n_obs</code>, <code>resolution</code>, <code>unit_cell</code>, <code>energy</code>, <code>bootstrap</code>, <code>experiment_prefix</code>. Nothing is filtered unless listed.<br />
|-<br />
| <code>filter.unit_cell.algorithm</code> || value || How to restrict unit cells: <code>value</code> (compare to a target), <code>range</code>, or <code>cluster</code> (use a precomputed covariance clustering).<br />
|-<br />
| <code>filter.unit_cell.value.relative_length_tolerance</code> || 0.1 || Fractional deviation of cell edge lengths allowed vs. the target cell. Often tightened (e.g. 0.03).<br />
|-<br />
| <code>filter.unit_cell.value.absolute_angle_tolerance</code> || 2.0 || Allowed cell-angle deviation (degrees) vs. the target.<br />
|-<br />
| <code>filter.unit_cell.value.target_unit_cell</code> / <code>target_space_group</code> || Auto || Target cell/symmetry; <code>Auto</code> uses the scaling model.<br />
|-<br />
| <code>filter.outlier.min_corr</code> || 0.1 || Minimum correlation of an image's intensities to the reference model; images below this are rejected. Only applied when a scaling model is present. Use <code>-1</code> to accept all images regardless of correlation.<br />
|-<br />
| <code>filter.resolution.d_min</code> || None || Reject an experiment unless some of its reflections extend beyond this resolution.<br />
|}<br />
<br />
=== Selecting individual reflections (<code>select</code>) ===<br />
<br />
The <code>select</code> section accepts or rejects '''individual reflections''' (contrast with<br />
<code>filter</code>, which acts on whole experiments).<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>select.algorithm</code> || (none) || One or more of <code>significance_filter</code>, <code>panel</code>, <code>cspad_sensor</code>, <code>isolation_forest</code>, <code>target_energy</code>.<br />
|-<br />
| <code>select.significance_filter.sigma</code> || 0.5 || Trim each lattice at the resolution where the binned ⟨I/σ⟩ drops below this value.<br />
|-<br />
| <code>select.significance_filter.min_ct</code> / <code>max_ct</code> || 10 / 50 || Target minimum/maximum mean reflections per resolution bin when choosing the number of bins. For large unit cells these are often raised (e.g. 200/300); for small cells, use the defaults.<br />
|}<br />
<br />
=== Modifying intensities (<code>modify</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>modify.algorithm</code> || polarization || Per-reflection corrections to apply (default applies the polarization correction).<br />
|-<br />
| <code>modify.cosym</code> || — || Resolve an indexing ambiguity across the ensemble using the Gildea/Winter (Brehm/Diederichs) cosym method, optionally anchored to <code>scaling.model</code>.<br />
|-<br />
| <code>modify.reindex_to_reference</code> / <code>reindex_to_abc</code> || — || Alternative re-indexing operators (to a reference model, or an explicit operator).<br />
|}<br />
<br />
=== Scaling (<code>scaling</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>scaling.model</code> || None || Reference dataset: a PDB file (uses the CRYST1 cell and computes model intensities, including bulk solvent) or an MTZ from a previous cycle. Required for <code>mark0</code> scaling.<br />
|-<br />
| <code>scaling.algorithm</code> || mark0 || <code>mark0</code>: per-image scaling against the reference. <code>mark1</code>: no scaling, just Monte-Carlo averaging (all scale factors = 1) — used for reference-free bootstrap.<br />
|-<br />
| <code>scaling.unit_cell</code> / <code>scaling.space_group</code> || None || Cell/symmetry to use when no model is provided (i.e. <code>mark1</code>).<br />
|-<br />
| <code>scaling.resolution_scalar</code> || 0.969 || <code>merging.d_min</code> is multiplied by this when deciding which reflections fall within the resolution limit, generating a few extra reference reflections to account for per-shot cell variation (e.g. set 0.96 for a few more).<br />
|-<br />
| <code>scaling.mtz.mtz_column_F</code> || fobs || Column name in the reference MTZ containing the reference structure factors (intensities or amplitudes).<br />
|}<br />
<br />
=== Post-refinement (<code>postrefinement</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>postrefinement.enable</code> || True || Enable post-refinement of per-image parameters/partialities (Sauter 2015).<br />
|-<br />
| <code>postrefinement.algorithm</code> || rs || Post-refinement protocol: <code>rs</code>, <code>rs2</code>, <code>rs_hybrid</code>, or <code>eta_deff</code>.<br />
|}<br />
<br />
=== Merging and error model (<code>merging</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>merging.d_min</code> || None || High-resolution limit for scaling and merging.<br />
|-<br />
| <code>merging.d_max</code> || None || Low-resolution limit (currently affects only CCiso).<br />
|-<br />
| <code>merging.merge_anomalous</code> || False || <code>True</code> merges Bijvoet mates together (higher multiplicity); <code>False</code> keeps them separate to preserve the anomalous signal.<br />
|-<br />
| <code>merging.minimum_multiplicity</code> || 2 || Do not produce a merged structure factor for HKLs measured fewer than this many times.<br />
|-<br />
| <code>merging.error.model</code> || mm24 || Merged-sigma error model: <code>mm24</code> (Mittan-Moreau 2025, current default), <code>ev11</code> (Evans 2011 / sdfac-refine, recommended in the README), <code>ha14</code>, or <code>errors_from_sample_residuals</code> (no refined error terms — used with <code>mark1</code> bootstrap).<br />
|-<br />
| <code>merging.set_average_unit_cell</code> || True || Output MTZ adopts the (cluster-derived) average unit cell of the data rather than the reference model's cell.<br />
|}<br />
<br />
=== Statistics (<code>statistics</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>statistics.n_bins</code> || 10 || Number of resolution bins in the statistics tables.<br />
|-<br />
| <code>statistics.cciso.mtz_file</code> || None || Isomorphous reference structure factors for R<sub>iso</sub> / CC<sub>iso</sub>.<br />
|-<br />
| <code>statistics.report_ML</code> || True || Report per-frame max-likelihood attributes (domain size, mosaicity).<br />
|}<br />
<br />
=== Output (<code>output</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>output.output_dir</code> || <code>.</code> || Directory for output files (created if it does not exist).<br />
|-<br />
| <code>output.prefix</code> || iobs || Prefix prepended to all output file names.<br />
|-<br />
| <code>output.save_experiments_and_reflections</code> || False || Dump the final experiments/reflections from the last worker as DIALS files — used for the two-stage scale-then-merge workflow.<br />
|-<br />
| <code>output.do_timing</code> || False || Log elapsed time per execution step.<br />
|-<br />
| <code>output.expanded_bookkeeping</code> || False || Store extra provenance columns (input file/reflection indices, odd/even flag, original id) in the saved reflection tables.<br />
|}<br />
<br />
=== Multiprocessing (<code>mp</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>mp.method</code> || mpi || Multiprocessing method (only <code>mpi</code> is currently supported).<br />
|-<br />
| <code>mp.debug.cProfile</code> || False || Enable code profiling; a <code>.prof</code> file is written per rank.<br />
|}<br />
<br />
== Output files ==<br />
<br />
The <code>merge</code> worker gathers all HKLs at rank 0 and writes three MTZ files, prefixed with<br />
<code>output.prefix</code> (default <code>iobs</code>):<br />
<br />
* <code>&lt;prefix&gt;_all.mtz</code> — the merged dataset using all observations (the main result for structure solution)<br />
* <code>&lt;prefix&gt;_odd.mtz</code> and <code>&lt;prefix&gt;_even.mtz</code> — the two half-datasets used for CC½ and related half-dataset statistics<br />
<br />
A main log records the per-step progress and the merging statistics tables by resolution bin:<br />
completeness, multiplicity and ⟨I/σ⟩, plus the "Table of Scaling Results" reporting CC½ (the<br />
half-dataset correlation, labelled "CC int"), CC<sub>iso</sub>, R<sub>int</sub>, R<sub>split</sub><br />
and R<sub>iso</sub>. The merged MTZ can optionally be<br />
uploaded to a shared [https://github.com/cctbx/cctbx_project/tree/master/xfel/merging/application/publish Google Drive folder] by adding the <code>publish</code> worker.<br />
<br />
== References ==<br />
<br />
* Sauter NK (2015). ''J Synchrotron Radiat'' '''22''', 239–48. (post-refinement)<br />
* Uervirojnangkoorn M, ''et al.'' (2015). ''eLife'' 10.7554/eLife.05421. (PRIME / bootstrap)<br />
* Brewster AS, ''et al.'' (2018). ''Acta Cryst.'' D '''74''', 877–894.<br />
* Brewster AS, ''et al.'' (2019a). ''Computational Crystallography Newsletter'' '''10''', 22–39. (XFEL GUI)<br />
* Brewster AS, ''et al.'' (2019b). ''Acta Cryst.'' D '''75''', 959–968. (error models)<br />
* Mittan-Moreau (2025). (mm24 error model)<br />
<br />
== See also ==<br />
<br />
* [[dials.stills_process]] — produces the integrated data that <code>cctbx.xfel.merge</code> consumes<br />
* [[cctbx.xfel GUI]] — orchestrates separate scaling and repeated merging as data are collected<br />
* The program README at <code>xfel/merging/README.md</code> in cctbx_project</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Cctbx.xfel.merge&diff=1777Cctbx.xfel.merge2026-08-21T23:14:53Z<p>Aaron: /* References */</p>
<hr />
<div>= ''cctbx.xfel.merge'' =<br />
<br />
'''''cctbx.xfel.merge''''' is the scaling, post-refinement and merging program of cctbx.xfel. It takes<br />
the per-image integrated results produced by ''[[dials.stills_process]]'' — one crystal per still shot —<br />
and reduces the whole ensemble to a single set of merged structure-factor intensities (an MTZ file)<br />
suitable for structure solution and refinement, together with the crystallographic quality<br />
statistics (completeness, multiplicity, ⟨I/σ⟩, CC½, CC<sub>iso</sub>, R-factors, etc.).<br />
<br />
Because serial datasets routinely contain tens of thousands to millions of still shots,<br />
<code>cctbx.xfel.merge</code> is built from the ground up to run under MPI across many nodes. It is<br />
almost always launched with <code>mpirun</code>/<code>srun</code>. Note that MPI uses the term 'ranks'<br />
to distinguish independent processes that are running across one or more nodes.<br />
<br />
== Overview ==<br />
<br />
<code>cctbx.xfel.merge</code> is organized as a sequence of '''workers'''. Each worker performs one<br />
task — reading data, applying a correction, filtering, scaling, post-refining, computing a statistic,<br />
or merging — and passes its (possibly modified) experiments and reflections on to the next worker.<br />
The list of workers to run is the "step list", and it is fully configurable: you can add, remove or<br />
reorder steps with <code>dispatch.step_list</code>. This is what makes the program flexible enough to<br />
scale-and-merge in one pass, to scale subsets separately and merge later, or to run a reference-free<br />
bootstrap.<br />
<br />
The '''default''' step list is:<br />
<br />
{| class="wikitable"<br />
! # !! Worker !! Purpose<br />
|-<br />
| 1 || <code>input</code> || Read integrated experiments (<code>.expt</code>) and reflections (<code>.refl</code>) from disk in DIALS format.<br />
|-<br />
| 2 || <code>balance</code> || Redistribute the input load evenly across MPI ranks (needed when there are fewer or more input files than ranks).<br />
|-<br />
| 3 || <code>model_scaling</code> || Build the full Miller list, model intensities from the reference, and the resolution binner used for scaling and post-refinement.<br />
|-<br />
| 4 || <code>modify</code> || Apply per-reflection modifications, e.g. polarization correction (and optionally re-indexing to resolve indexing ambiguity — see <code>modify.cosym</code>/<code>reindex_to_reference</code>).<br />
|-<br />
| 5 || <code>filter</code> || Reject whole experiments (or reflections) by resolution, unit cell, correlation to the reference, energy, etc.<br />
|-<br />
| 6 || <code>scale</code> || Scale each lattice to the reference dataset (<code>mark0</code>) or simply average (<code>mark1</code>).<br />
|-<br />
| 7 || <code>postrefine</code> || Post-refine per-image crystal parameters and partialities (Sauter 2015).<br />
|-<br />
| 8 || <code>statistics_unitcell</code> || Determine and store the average unit cell.<br />
|-<br />
| 9 || <code>statistics_beam</code> || Determine and store the average wavelength.<br />
|-<br />
| 10 || <code>model_statistics</code> || Rebuild the Miller list / binner for statistics (may use the average unit cell).<br />
|-<br />
| 11 || <code>statistics_resolution</code> || Per-crystal resolution statistics.<br />
|-<br />
| 12 || <code>group</code> || Redistribute reflections across ranks so that every observation of a given HKL is gathered on the same rank prior to merging.<br />
|-<br />
| 13 || <code>errors_merge</code> || Correct the merged error estimates with a per-HKL error model (e.g. Ev11, Brewster 2019b, or MM24 (the default), Mittan-Moreau (2025)).<br />
|-<br />
| 14 || <code>statistics_intensity</code> || Intensity statistics by resolution.<br />
|-<br />
| 15 || <code>merge</code> || Merge symmetry-equivalent intensities, gather all HKLs at rank 0, and write "odd", "even" and "all" MTZ files.<br />
|-<br />
| 16 || <code>statistics_intensity_cxi</code> || Correlation and R-factor statistics (ported from cxi.merge): the "Table of Scaling Results" — CC½ (half-dataset correlation, labelled "CC int"), CC<sub>iso</sub>, R<sub>int</sub>, R<sub>split</sub>, R<sub>iso</sub>.<br />
|}<br />
<br />
Additional, non-default workers are available and can be<br />
inserted into the step list, including <code>tdata</code> (write out unit cells for clustering),<br />
<code>export</code> (batch MTZ for ''Careless''), <code>publish</code> (upload merged files to a<br />
Google Drive folder), <code>monitor</code> (resource-usage logging), <code>integrate</code><br />
(on-the-fly integration), <code>prepare</code> (SpReAD energy binning), and<br />
<code>lunus</code> (diffuse-scattering deck). A <code>deltaccint</code> worker (ΔCC½ lattice<br />
filtering) can be added after <code>group</code>. Custom workers can be supplied through the<br />
<code>XFEL_CUSTOM_WORKER_PATH</code> environment variable.<br />
<br />
== Basic usage ==<br />
<br />
<code>cctbx.xfel.merge</code> is run under MPI:<br />
<br />
<pre><br />
mpirun cctbx.xfel.merge merge.phil<br />
</pre><br />
<br />
A typical <code>merge.phil</code> (from the program README):<br />
<br />
<pre><br />
input.path=<path to data><br />
input.experiments_suffix=.expt<br />
input.reflections_suffix=.refl<br />
<br />
filter.algorithm=unit_cell<br />
filter.unit_cell.value.relative_length_tolerance=0.03<br />
filter.outlier.min_corr=0.1<br />
<br />
select.algorithm=significance_filter<br />
select.significance_filter.sigma=0.1<br />
select.significance_filter.min_ct=200<br />
select.significance_filter.max_ct=300<br />
<br />
scaling.model=<model.pdb or model.mtz><br />
scaling.resolution_scalar=0.96<br />
<br />
merging.d_min=1.8<br />
merging.error.model=ev11<br />
merging.merge_anomalous=True<br />
<br />
postrefinement.enable=True<br />
statistics.n_bins=20<br />
<br />
output.do_timing=True<br />
output.output_dir=<output folder><br />
output.prefix=<prefix for output files><br />
</pre><br />
<br />
Explore the parameters with:<br />
* <code>cctbx.xfel.merge -c</code> — show configurable parameters<br />
* <code>cctbx.xfel.merge -c -e 10 -a 2</code> — show all parameters (expert level 10) with full help text<br />
<br />
== Two-stage scaling and merging ==<br />
<br />
For large, growing datasets (the mode used by the [[cctbx.xfel GUI]]), it is more efficient to scale<br />
and post-refine each subset of data '''once''', then merge repeatedly as more data arrive. This is<br />
done by running the program twice with different step lists.<br />
<br />
'''Stage 1 — scale each subset''' (stops after per-image work and writes intermediate DIALS files):<br />
<br />
<pre><br />
dispatch.step_list=input balance model_scaling modify filter scale postrefine \<br />
statistics_unitcell statistics_beam model_statistics statistics_resolution<br />
input.parallel_file_load.method=uniform<br />
output.save_experiments_and_reflections=True<br />
</pre><br />
<br />
<code>output.save_experiments_and_reflections=True</code> dumps the scaled/post-refined data as<br />
experiment lists and reflection tables that the merge stage can read back in.<br />
<br />
'''Stage 2 — merge the scaled subsets''' (add an <code>input.path</code> for each finished subset):<br />
<br />
<pre><br />
dispatch.step_list=input model_scaling statistics_unitcell statistics_beam \<br />
model_statistics statistics_resolution group errors_merge statistics_intensity \<br />
merge statistics_intensity_cxi<br />
input.parallel_file_load.method=uniform<br />
</pre><br />
<br />
== Merging without a reference (bootstrap / mark1) ==<br />
<br />
If no scaling/merging reference is available, use a bootstrap procedure. First run the<br />
<code>mark1</code> algorithm, which simply averages the data (all per-image scale factors = 1, no<br />
post-refinement), specifying the target cell and symmetry explicitly:<br />
<br />
<pre><br />
scaling.algorithm=mark1<br />
scaling.unit_cell=<unit cell><br />
scaling.space_group=<space group><br />
merging.error.model=errors_from_sample_residuals<br />
</pre><br />
<br />
The resulting MTZ is then used as the reference (<code>scaling.model=</code>) for subsequent regular<br />
merges. Iterate a few cycles; CC½ typically improves and converges over the first few rounds.<br />
<br />
== Important parameters ==<br />
<br />
Parameters are given by their full PHIL path with defaults in parentheses. The program exposes many<br />
more parameters (see <code>-c -e 10 -a 2</code>); those below are the ones most commonly set.<br />
<br />
=== Input ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>input.path</code> || None || Path(s) to the integrated data — a folder, several folders, or individual files; wildcards allowed; may be repeated. Matching <code>.expt</code> and <code>.refl</code> files must both be present.<br />
|-<br />
| <code>input.experiments_suffix</code> || <code>_integrated.expt</code> || Only load experiment files ending with this suffix.<br />
|-<br />
| <code>input.reflections_suffix</code> || <code>_integrated.refl</code> || Only load reflection files ending with this suffix.<br />
|-<br />
| <code>input.parallel_file_load.method</code> || uniform || <code>uniform</code> spreads input files evenly over all ranks; <code>node_memory</code> distributes so per-node memory limits are respected.<br />
|-<br />
| <code>input.alist.file</code> || None || Path to a text file of experiment tags or filenames; combined with <code>input.alist.op=keep|reject</code> to merge only (or exclude) a specified subset.<br />
|-<br />
| <code>input.persistent_refl_cols</code> || None || Reflection-table columns to preserve through the pruning steps (and into saved tables).<br />
|}<br />
<br />
=== Filtering whole experiments (<code>filter</code>) ===<br />
<br />
The <code>filter</code> section accepts or rejects '''entire experiments'''.<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>filter.algorithm</code> || (none) || One or more of <code>n_obs</code>, <code>resolution</code>, <code>unit_cell</code>, <code>energy</code>, <code>bootstrap</code>, <code>experiment_prefix</code>. Nothing is filtered unless listed.<br />
|-<br />
| <code>filter.unit_cell.algorithm</code> || value || How to restrict unit cells: <code>value</code> (compare to a target), <code>range</code>, or <code>cluster</code> (use a precomputed covariance clustering).<br />
|-<br />
| <code>filter.unit_cell.value.relative_length_tolerance</code> || 0.1 || Fractional deviation of cell edge lengths allowed vs. the target cell. Often tightened (e.g. 0.03).<br />
|-<br />
| <code>filter.unit_cell.value.absolute_angle_tolerance</code> || 2.0 || Allowed cell-angle deviation (degrees) vs. the target.<br />
|-<br />
| <code>filter.unit_cell.value.target_unit_cell</code> / <code>target_space_group</code> || Auto || Target cell/symmetry; <code>Auto</code> uses the scaling model.<br />
|-<br />
| <code>filter.outlier.min_corr</code> || 0.1 || Minimum correlation of an image's intensities to the reference model; images below this are rejected. Only applied when a scaling model is present. Use <code>-1</code> to accept all images regardless of correlation.<br />
|-<br />
| <code>filter.resolution.d_min</code> || None || Reject an experiment unless some of its reflections extend beyond this resolution.<br />
|}<br />
<br />
=== Selecting individual reflections (<code>select</code>) ===<br />
<br />
The <code>select</code> section accepts or rejects '''individual reflections''' (contrast with<br />
<code>filter</code>, which acts on whole experiments).<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>select.algorithm</code> || (none) || One or more of <code>significance_filter</code>, <code>panel</code>, <code>cspad_sensor</code>, <code>isolation_forest</code>, <code>target_energy</code>.<br />
|-<br />
| <code>select.significance_filter.sigma</code> || 0.5 || Trim each lattice at the resolution where the binned ⟨I/σ⟩ drops below this value.<br />
|-<br />
| <code>select.significance_filter.min_ct</code> / <code>max_ct</code> || 10 / 50 || Target minimum/maximum mean reflections per resolution bin when choosing the number of bins. For large unit cells these are often raised (e.g. 200/300); for small cells, use the defaults.<br />
|}<br />
<br />
=== Modifying intensities (<code>modify</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>modify.algorithm</code> || polarization || Per-reflection corrections to apply (default applies the polarization correction).<br />
|-<br />
| <code>modify.cosym</code> || — || Resolve an indexing ambiguity across the ensemble using the Gildea/Winter (Brehm/Diederichs) cosym method, optionally anchored to <code>scaling.model</code>.<br />
|-<br />
| <code>modify.reindex_to_reference</code> / <code>reindex_to_abc</code> || — || Alternative re-indexing operators (to a reference model, or an explicit operator).<br />
|}<br />
<br />
=== Scaling (<code>scaling</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>scaling.model</code> || None || Reference dataset: a PDB file (uses the CRYST1 cell and computes model intensities, including bulk solvent) or an MTZ from a previous cycle. Required for <code>mark0</code> scaling.<br />
|-<br />
| <code>scaling.algorithm</code> || mark0 || <code>mark0</code>: per-image scaling against the reference. <code>mark1</code>: no scaling, just Monte-Carlo averaging (all scale factors = 1) — used for reference-free bootstrap.<br />
|-<br />
| <code>scaling.unit_cell</code> / <code>scaling.space_group</code> || None || Cell/symmetry to use when no model is provided (i.e. <code>mark1</code>).<br />
|-<br />
| <code>scaling.resolution_scalar</code> || 0.969 || <code>merging.d_min</code> is multiplied by this when deciding which reflections fall within the resolution limit, generating a few extra reference reflections to account for per-shot cell variation (e.g. set 0.96 for a few more).<br />
|-<br />
| <code>scaling.mtz.mtz_column_F</code> || fobs || Column name in the reference MTZ containing the reference structure factors (intensities or amplitudes).<br />
|}<br />
<br />
=== Post-refinement (<code>postrefinement</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>postrefinement.enable</code> || True || Enable post-refinement of per-image parameters/partialities (Sauter 2015).<br />
|-<br />
| <code>postrefinement.algorithm</code> || rs || Post-refinement protocol: <code>rs</code>, <code>rs2</code>, <code>rs_hybrid</code>, or <code>eta_deff</code>.<br />
|}<br />
<br />
=== Merging and error model (<code>merging</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>merging.d_min</code> || None || High-resolution limit for scaling and merging.<br />
|-<br />
| <code>merging.d_max</code> || None || Low-resolution limit (currently affects only CCiso).<br />
|-<br />
| <code>merging.merge_anomalous</code> || False || <code>True</code> merges Bijvoet mates together (higher multiplicity); <code>False</code> keeps them separate to preserve the anomalous signal.<br />
|-<br />
| <code>merging.minimum_multiplicity</code> || 2 || Do not produce a merged structure factor for HKLs measured fewer than this many times.<br />
|-<br />
| <code>merging.error.model</code> || mm24 || Merged-sigma error model: <code>mm24</code> (Mittan-Moreau 2024, current default), <code>ev11</code> (Evans 2011 / sdfac-refine, recommended in the README), <code>ha14</code>, or <code>errors_from_sample_residuals</code> (no refined error terms — used with <code>mark1</code> bootstrap).<br />
|-<br />
| <code>merging.set_average_unit_cell</code> || True || Output MTZ adopts the (cluster-derived) average unit cell of the data rather than the reference model's cell.<br />
|}<br />
<br />
=== Statistics (<code>statistics</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>statistics.n_bins</code> || 10 || Number of resolution bins in the statistics tables.<br />
|-<br />
| <code>statistics.cciso.mtz_file</code> || None || Isomorphous reference structure factors for R<sub>iso</sub> / CC<sub>iso</sub>.<br />
|-<br />
| <code>statistics.report_ML</code> || True || Report per-frame max-likelihood attributes (domain size, mosaicity).<br />
|}<br />
<br />
=== Output (<code>output</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>output.output_dir</code> || <code>.</code> || Directory for output files (created if it does not exist).<br />
|-<br />
| <code>output.prefix</code> || iobs || Prefix prepended to all output file names.<br />
|-<br />
| <code>output.save_experiments_and_reflections</code> || False || Dump the final experiments/reflections from the last worker as DIALS files — used for the two-stage scale-then-merge workflow.<br />
|-<br />
| <code>output.do_timing</code> || False || Log elapsed time per execution step.<br />
|-<br />
| <code>output.expanded_bookkeeping</code> || False || Store extra provenance columns (input file/reflection indices, odd/even flag, original id) in the saved reflection tables.<br />
|}<br />
<br />
=== Multiprocessing (<code>mp</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>mp.method</code> || mpi || Multiprocessing method (only <code>mpi</code> is currently supported).<br />
|-<br />
| <code>mp.debug.cProfile</code> || False || Enable code profiling; a <code>.prof</code> file is written per rank.<br />
|}<br />
<br />
== Output files ==<br />
<br />
The <code>merge</code> worker gathers all HKLs at rank 0 and writes three MTZ files, prefixed with<br />
<code>output.prefix</code> (default <code>iobs</code>):<br />
<br />
* <code>&lt;prefix&gt;_all.mtz</code> — the merged dataset using all observations (the main result for structure solution)<br />
* <code>&lt;prefix&gt;_odd.mtz</code> and <code>&lt;prefix&gt;_even.mtz</code> — the two half-datasets used for CC½ and related half-dataset statistics<br />
<br />
A main log records the per-step progress and the merging statistics tables by resolution bin:<br />
completeness, multiplicity and ⟨I/σ⟩, plus the "Table of Scaling Results" reporting CC½ (the<br />
half-dataset correlation, labelled "CC int"), CC<sub>iso</sub>, R<sub>int</sub>, R<sub>split</sub><br />
and R<sub>iso</sub>. The merged MTZ can optionally be<br />
uploaded to a shared [https://github.com/cctbx/cctbx_project/tree/master/xfel/merging/application/publish Google Drive folder] by adding the <code>publish</code> worker.<br />
<br />
== References ==<br />
<br />
* Sauter NK (2015). ''J Synchrotron Radiat'' '''22''', 239–48. (post-refinement)<br />
* Uervirojnangkoorn M, ''et al.'' (2015). ''eLife'' 10.7554/eLife.05421. (PRIME / bootstrap)<br />
* Brewster AS, ''et al.'' (2018). ''Acta Cryst.'' D '''74''', 877–894.<br />
* Brewster AS, ''et al.'' (2019a). ''Computational Crystallography Newsletter'' '''10''', 22–39. (XFEL GUI)<br />
* Brewster AS, ''et al.'' (2019b). ''Acta Cryst.'' D '''75''', 959–968. (error models)<br />
* Mittan-Moreau (2025). (mm24 error model)<br />
<br />
== See also ==<br />
<br />
* [[dials.stills_process]] — produces the integrated data that <code>cctbx.xfel.merge</code> consumes<br />
* [[cctbx.xfel GUI]] — orchestrates separate scaling and repeated merging as data are collected<br />
* The program README at <code>xfel/merging/README.md</code> in cctbx_project</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Cctbx.xfel.merge&diff=1776Cctbx.xfel.merge2026-08-21T23:14:17Z<p>Aaron: /* Output files */</p>
<hr />
<div>= ''cctbx.xfel.merge'' =<br />
<br />
'''''cctbx.xfel.merge''''' is the scaling, post-refinement and merging program of cctbx.xfel. It takes<br />
the per-image integrated results produced by ''[[dials.stills_process]]'' — one crystal per still shot —<br />
and reduces the whole ensemble to a single set of merged structure-factor intensities (an MTZ file)<br />
suitable for structure solution and refinement, together with the crystallographic quality<br />
statistics (completeness, multiplicity, ⟨I/σ⟩, CC½, CC<sub>iso</sub>, R-factors, etc.).<br />
<br />
Because serial datasets routinely contain tens of thousands to millions of still shots,<br />
<code>cctbx.xfel.merge</code> is built from the ground up to run under MPI across many nodes. It is<br />
almost always launched with <code>mpirun</code>/<code>srun</code>. Note that MPI uses the term 'ranks'<br />
to distinguish independent processes that are running across one or more nodes.<br />
<br />
== Overview ==<br />
<br />
<code>cctbx.xfel.merge</code> is organized as a sequence of '''workers'''. Each worker performs one<br />
task — reading data, applying a correction, filtering, scaling, post-refining, computing a statistic,<br />
or merging — and passes its (possibly modified) experiments and reflections on to the next worker.<br />
The list of workers to run is the "step list", and it is fully configurable: you can add, remove or<br />
reorder steps with <code>dispatch.step_list</code>. This is what makes the program flexible enough to<br />
scale-and-merge in one pass, to scale subsets separately and merge later, or to run a reference-free<br />
bootstrap.<br />
<br />
The '''default''' step list is:<br />
<br />
{| class="wikitable"<br />
! # !! Worker !! Purpose<br />
|-<br />
| 1 || <code>input</code> || Read integrated experiments (<code>.expt</code>) and reflections (<code>.refl</code>) from disk in DIALS format.<br />
|-<br />
| 2 || <code>balance</code> || Redistribute the input load evenly across MPI ranks (needed when there are fewer or more input files than ranks).<br />
|-<br />
| 3 || <code>model_scaling</code> || Build the full Miller list, model intensities from the reference, and the resolution binner used for scaling and post-refinement.<br />
|-<br />
| 4 || <code>modify</code> || Apply per-reflection modifications, e.g. polarization correction (and optionally re-indexing to resolve indexing ambiguity — see <code>modify.cosym</code>/<code>reindex_to_reference</code>).<br />
|-<br />
| 5 || <code>filter</code> || Reject whole experiments (or reflections) by resolution, unit cell, correlation to the reference, energy, etc.<br />
|-<br />
| 6 || <code>scale</code> || Scale each lattice to the reference dataset (<code>mark0</code>) or simply average (<code>mark1</code>).<br />
|-<br />
| 7 || <code>postrefine</code> || Post-refine per-image crystal parameters and partialities (Sauter 2015).<br />
|-<br />
| 8 || <code>statistics_unitcell</code> || Determine and store the average unit cell.<br />
|-<br />
| 9 || <code>statistics_beam</code> || Determine and store the average wavelength.<br />
|-<br />
| 10 || <code>model_statistics</code> || Rebuild the Miller list / binner for statistics (may use the average unit cell).<br />
|-<br />
| 11 || <code>statistics_resolution</code> || Per-crystal resolution statistics.<br />
|-<br />
| 12 || <code>group</code> || Redistribute reflections across ranks so that every observation of a given HKL is gathered on the same rank prior to merging.<br />
|-<br />
| 13 || <code>errors_merge</code> || Correct the merged error estimates with a per-HKL error model (e.g. Ev11, Brewster 2019b, or MM24 (the default), Mittan-Moreau (2025)).<br />
|-<br />
| 14 || <code>statistics_intensity</code> || Intensity statistics by resolution.<br />
|-<br />
| 15 || <code>merge</code> || Merge symmetry-equivalent intensities, gather all HKLs at rank 0, and write "odd", "even" and "all" MTZ files.<br />
|-<br />
| 16 || <code>statistics_intensity_cxi</code> || Correlation and R-factor statistics (ported from cxi.merge): the "Table of Scaling Results" — CC½ (half-dataset correlation, labelled "CC int"), CC<sub>iso</sub>, R<sub>int</sub>, R<sub>split</sub>, R<sub>iso</sub>.<br />
|}<br />
<br />
Additional, non-default workers are available and can be<br />
inserted into the step list, including <code>tdata</code> (write out unit cells for clustering),<br />
<code>export</code> (batch MTZ for ''Careless''), <code>publish</code> (upload merged files to a<br />
Google Drive folder), <code>monitor</code> (resource-usage logging), <code>integrate</code><br />
(on-the-fly integration), <code>prepare</code> (SpReAD energy binning), and<br />
<code>lunus</code> (diffuse-scattering deck). A <code>deltaccint</code> worker (ΔCC½ lattice<br />
filtering) can be added after <code>group</code>. Custom workers can be supplied through the<br />
<code>XFEL_CUSTOM_WORKER_PATH</code> environment variable.<br />
<br />
== Basic usage ==<br />
<br />
<code>cctbx.xfel.merge</code> is run under MPI:<br />
<br />
<pre><br />
mpirun cctbx.xfel.merge merge.phil<br />
</pre><br />
<br />
A typical <code>merge.phil</code> (from the program README):<br />
<br />
<pre><br />
input.path=<path to data><br />
input.experiments_suffix=.expt<br />
input.reflections_suffix=.refl<br />
<br />
filter.algorithm=unit_cell<br />
filter.unit_cell.value.relative_length_tolerance=0.03<br />
filter.outlier.min_corr=0.1<br />
<br />
select.algorithm=significance_filter<br />
select.significance_filter.sigma=0.1<br />
select.significance_filter.min_ct=200<br />
select.significance_filter.max_ct=300<br />
<br />
scaling.model=<model.pdb or model.mtz><br />
scaling.resolution_scalar=0.96<br />
<br />
merging.d_min=1.8<br />
merging.error.model=ev11<br />
merging.merge_anomalous=True<br />
<br />
postrefinement.enable=True<br />
statistics.n_bins=20<br />
<br />
output.do_timing=True<br />
output.output_dir=<output folder><br />
output.prefix=<prefix for output files><br />
</pre><br />
<br />
Explore the parameters with:<br />
* <code>cctbx.xfel.merge -c</code> — show configurable parameters<br />
* <code>cctbx.xfel.merge -c -e 10 -a 2</code> — show all parameters (expert level 10) with full help text<br />
<br />
== Two-stage scaling and merging ==<br />
<br />
For large, growing datasets (the mode used by the [[cctbx.xfel GUI]]), it is more efficient to scale<br />
and post-refine each subset of data '''once''', then merge repeatedly as more data arrive. This is<br />
done by running the program twice with different step lists.<br />
<br />
'''Stage 1 — scale each subset''' (stops after per-image work and writes intermediate DIALS files):<br />
<br />
<pre><br />
dispatch.step_list=input balance model_scaling modify filter scale postrefine \<br />
statistics_unitcell statistics_beam model_statistics statistics_resolution<br />
input.parallel_file_load.method=uniform<br />
output.save_experiments_and_reflections=True<br />
</pre><br />
<br />
<code>output.save_experiments_and_reflections=True</code> dumps the scaled/post-refined data as<br />
experiment lists and reflection tables that the merge stage can read back in.<br />
<br />
'''Stage 2 — merge the scaled subsets''' (add an <code>input.path</code> for each finished subset):<br />
<br />
<pre><br />
dispatch.step_list=input model_scaling statistics_unitcell statistics_beam \<br />
model_statistics statistics_resolution group errors_merge statistics_intensity \<br />
merge statistics_intensity_cxi<br />
input.parallel_file_load.method=uniform<br />
</pre><br />
<br />
== Merging without a reference (bootstrap / mark1) ==<br />
<br />
If no scaling/merging reference is available, use a bootstrap procedure. First run the<br />
<code>mark1</code> algorithm, which simply averages the data (all per-image scale factors = 1, no<br />
post-refinement), specifying the target cell and symmetry explicitly:<br />
<br />
<pre><br />
scaling.algorithm=mark1<br />
scaling.unit_cell=<unit cell><br />
scaling.space_group=<space group><br />
merging.error.model=errors_from_sample_residuals<br />
</pre><br />
<br />
The resulting MTZ is then used as the reference (<code>scaling.model=</code>) for subsequent regular<br />
merges. Iterate a few cycles; CC½ typically improves and converges over the first few rounds.<br />
<br />
== Important parameters ==<br />
<br />
Parameters are given by their full PHIL path with defaults in parentheses. The program exposes many<br />
more parameters (see <code>-c -e 10 -a 2</code>); those below are the ones most commonly set.<br />
<br />
=== Input ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>input.path</code> || None || Path(s) to the integrated data — a folder, several folders, or individual files; wildcards allowed; may be repeated. Matching <code>.expt</code> and <code>.refl</code> files must both be present.<br />
|-<br />
| <code>input.experiments_suffix</code> || <code>_integrated.expt</code> || Only load experiment files ending with this suffix.<br />
|-<br />
| <code>input.reflections_suffix</code> || <code>_integrated.refl</code> || Only load reflection files ending with this suffix.<br />
|-<br />
| <code>input.parallel_file_load.method</code> || uniform || <code>uniform</code> spreads input files evenly over all ranks; <code>node_memory</code> distributes so per-node memory limits are respected.<br />
|-<br />
| <code>input.alist.file</code> || None || Path to a text file of experiment tags or filenames; combined with <code>input.alist.op=keep|reject</code> to merge only (or exclude) a specified subset.<br />
|-<br />
| <code>input.persistent_refl_cols</code> || None || Reflection-table columns to preserve through the pruning steps (and into saved tables).<br />
|}<br />
<br />
=== Filtering whole experiments (<code>filter</code>) ===<br />
<br />
The <code>filter</code> section accepts or rejects '''entire experiments'''.<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>filter.algorithm</code> || (none) || One or more of <code>n_obs</code>, <code>resolution</code>, <code>unit_cell</code>, <code>energy</code>, <code>bootstrap</code>, <code>experiment_prefix</code>. Nothing is filtered unless listed.<br />
|-<br />
| <code>filter.unit_cell.algorithm</code> || value || How to restrict unit cells: <code>value</code> (compare to a target), <code>range</code>, or <code>cluster</code> (use a precomputed covariance clustering).<br />
|-<br />
| <code>filter.unit_cell.value.relative_length_tolerance</code> || 0.1 || Fractional deviation of cell edge lengths allowed vs. the target cell. Often tightened (e.g. 0.03).<br />
|-<br />
| <code>filter.unit_cell.value.absolute_angle_tolerance</code> || 2.0 || Allowed cell-angle deviation (degrees) vs. the target.<br />
|-<br />
| <code>filter.unit_cell.value.target_unit_cell</code> / <code>target_space_group</code> || Auto || Target cell/symmetry; <code>Auto</code> uses the scaling model.<br />
|-<br />
| <code>filter.outlier.min_corr</code> || 0.1 || Minimum correlation of an image's intensities to the reference model; images below this are rejected. Only applied when a scaling model is present. Use <code>-1</code> to accept all images regardless of correlation.<br />
|-<br />
| <code>filter.resolution.d_min</code> || None || Reject an experiment unless some of its reflections extend beyond this resolution.<br />
|}<br />
<br />
=== Selecting individual reflections (<code>select</code>) ===<br />
<br />
The <code>select</code> section accepts or rejects '''individual reflections''' (contrast with<br />
<code>filter</code>, which acts on whole experiments).<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>select.algorithm</code> || (none) || One or more of <code>significance_filter</code>, <code>panel</code>, <code>cspad_sensor</code>, <code>isolation_forest</code>, <code>target_energy</code>.<br />
|-<br />
| <code>select.significance_filter.sigma</code> || 0.5 || Trim each lattice at the resolution where the binned ⟨I/σ⟩ drops below this value.<br />
|-<br />
| <code>select.significance_filter.min_ct</code> / <code>max_ct</code> || 10 / 50 || Target minimum/maximum mean reflections per resolution bin when choosing the number of bins. For large unit cells these are often raised (e.g. 200/300); for small cells, use the defaults.<br />
|}<br />
<br />
=== Modifying intensities (<code>modify</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>modify.algorithm</code> || polarization || Per-reflection corrections to apply (default applies the polarization correction).<br />
|-<br />
| <code>modify.cosym</code> || — || Resolve an indexing ambiguity across the ensemble using the Gildea/Winter (Brehm/Diederichs) cosym method, optionally anchored to <code>scaling.model</code>.<br />
|-<br />
| <code>modify.reindex_to_reference</code> / <code>reindex_to_abc</code> || — || Alternative re-indexing operators (to a reference model, or an explicit operator).<br />
|}<br />
<br />
=== Scaling (<code>scaling</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>scaling.model</code> || None || Reference dataset: a PDB file (uses the CRYST1 cell and computes model intensities, including bulk solvent) or an MTZ from a previous cycle. Required for <code>mark0</code> scaling.<br />
|-<br />
| <code>scaling.algorithm</code> || mark0 || <code>mark0</code>: per-image scaling against the reference. <code>mark1</code>: no scaling, just Monte-Carlo averaging (all scale factors = 1) — used for reference-free bootstrap.<br />
|-<br />
| <code>scaling.unit_cell</code> / <code>scaling.space_group</code> || None || Cell/symmetry to use when no model is provided (i.e. <code>mark1</code>).<br />
|-<br />
| <code>scaling.resolution_scalar</code> || 0.969 || <code>merging.d_min</code> is multiplied by this when deciding which reflections fall within the resolution limit, generating a few extra reference reflections to account for per-shot cell variation (e.g. set 0.96 for a few more).<br />
|-<br />
| <code>scaling.mtz.mtz_column_F</code> || fobs || Column name in the reference MTZ containing the reference structure factors (intensities or amplitudes).<br />
|}<br />
<br />
=== Post-refinement (<code>postrefinement</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>postrefinement.enable</code> || True || Enable post-refinement of per-image parameters/partialities (Sauter 2015).<br />
|-<br />
| <code>postrefinement.algorithm</code> || rs || Post-refinement protocol: <code>rs</code>, <code>rs2</code>, <code>rs_hybrid</code>, or <code>eta_deff</code>.<br />
|}<br />
<br />
=== Merging and error model (<code>merging</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>merging.d_min</code> || None || High-resolution limit for scaling and merging.<br />
|-<br />
| <code>merging.d_max</code> || None || Low-resolution limit (currently affects only CCiso).<br />
|-<br />
| <code>merging.merge_anomalous</code> || False || <code>True</code> merges Bijvoet mates together (higher multiplicity); <code>False</code> keeps them separate to preserve the anomalous signal.<br />
|-<br />
| <code>merging.minimum_multiplicity</code> || 2 || Do not produce a merged structure factor for HKLs measured fewer than this many times.<br />
|-<br />
| <code>merging.error.model</code> || mm24 || Merged-sigma error model: <code>mm24</code> (Mittan-Moreau 2024, current default), <code>ev11</code> (Evans 2011 / sdfac-refine, recommended in the README), <code>ha14</code>, or <code>errors_from_sample_residuals</code> (no refined error terms — used with <code>mark1</code> bootstrap).<br />
|-<br />
| <code>merging.set_average_unit_cell</code> || True || Output MTZ adopts the (cluster-derived) average unit cell of the data rather than the reference model's cell.<br />
|}<br />
<br />
=== Statistics (<code>statistics</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>statistics.n_bins</code> || 10 || Number of resolution bins in the statistics tables.<br />
|-<br />
| <code>statistics.cciso.mtz_file</code> || None || Isomorphous reference structure factors for R<sub>iso</sub> / CC<sub>iso</sub>.<br />
|-<br />
| <code>statistics.report_ML</code> || True || Report per-frame max-likelihood attributes (domain size, mosaicity).<br />
|}<br />
<br />
=== Output (<code>output</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>output.output_dir</code> || <code>.</code> || Directory for output files (created if it does not exist).<br />
|-<br />
| <code>output.prefix</code> || iobs || Prefix prepended to all output file names.<br />
|-<br />
| <code>output.save_experiments_and_reflections</code> || False || Dump the final experiments/reflections from the last worker as DIALS files — used for the two-stage scale-then-merge workflow.<br />
|-<br />
| <code>output.do_timing</code> || False || Log elapsed time per execution step.<br />
|-<br />
| <code>output.expanded_bookkeeping</code> || False || Store extra provenance columns (input file/reflection indices, odd/even flag, original id) in the saved reflection tables.<br />
|}<br />
<br />
=== Multiprocessing (<code>mp</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>mp.method</code> || mpi || Multiprocessing method (only <code>mpi</code> is currently supported).<br />
|-<br />
| <code>mp.debug.cProfile</code> || False || Enable code profiling; a <code>.prof</code> file is written per rank.<br />
|}<br />
<br />
== Output files ==<br />
<br />
The <code>merge</code> worker gathers all HKLs at rank 0 and writes three MTZ files, prefixed with<br />
<code>output.prefix</code> (default <code>iobs</code>):<br />
<br />
* <code>&lt;prefix&gt;_all.mtz</code> — the merged dataset using all observations (the main result for structure solution)<br />
* <code>&lt;prefix&gt;_odd.mtz</code> and <code>&lt;prefix&gt;_even.mtz</code> — the two half-datasets used for CC½ and related half-dataset statistics<br />
<br />
A main log records the per-step progress and the merging statistics tables by resolution bin:<br />
completeness, multiplicity and ⟨I/σ⟩, plus the "Table of Scaling Results" reporting CC½ (the<br />
half-dataset correlation, labelled "CC int"), CC<sub>iso</sub>, R<sub>int</sub>, R<sub>split</sub><br />
and R<sub>iso</sub>. The merged MTZ can optionally be<br />
uploaded to a shared [https://github.com/cctbx/cctbx_project/tree/master/xfel/merging/application/publish Google Drive folder] by adding the <code>publish</code> worker.<br />
<br />
== References ==<br />
<br />
* Sauter NK (2015). ''J Synchrotron Radiat'' '''22''', 239–48. (post-refinement)<br />
* Uervirojnangkoorn M, ''et al.'' (2015). ''eLife'' 10.7554/eLife.05421. (PRIME / bootstrap)<br />
* Brewster AS, ''et al.'' (2018). ''Acta Cryst.'' D '''74''', 877–894.<br />
* Brewster AS, ''et al.'' (2019a). ''Computational Crystallography Newsletter'' '''10''', 22–39. (XFEL GUI)<br />
* Brewster AS, ''et al.'' (2019b). ''Acta Cryst.'' D '''75''', 959–968. (error models)<br />
* Mittan-Moreau (2024). (mm24 error model)<br />
<br />
== See also ==<br />
<br />
* [[dials.stills_process]] — produces the integrated data that <code>cctbx.xfel.merge</code> consumes<br />
* [[cctbx.xfel GUI]] — orchestrates separate scaling and repeated merging as data are collected<br />
* The program README at <code>xfel/merging/README.md</code> in cctbx_project</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Cctbx.xfel.merge&diff=1775Cctbx.xfel.merge2026-08-21T23:12:54Z<p>Aaron: </p>
<hr />
<div>= ''cctbx.xfel.merge'' =<br />
<br />
'''''cctbx.xfel.merge''''' is the scaling, post-refinement and merging program of cctbx.xfel. It takes<br />
the per-image integrated results produced by ''[[dials.stills_process]]'' — one crystal per still shot —<br />
and reduces the whole ensemble to a single set of merged structure-factor intensities (an MTZ file)<br />
suitable for structure solution and refinement, together with the crystallographic quality<br />
statistics (completeness, multiplicity, ⟨I/σ⟩, CC½, CC<sub>iso</sub>, R-factors, etc.).<br />
<br />
Because serial datasets routinely contain tens of thousands to millions of still shots,<br />
<code>cctbx.xfel.merge</code> is built from the ground up to run under MPI across many nodes. It is<br />
almost always launched with <code>mpirun</code>/<code>srun</code>. Note that MPI uses the term 'ranks'<br />
to distinguish independent processes that are running across one or more nodes.<br />
<br />
== Overview ==<br />
<br />
<code>cctbx.xfel.merge</code> is organized as a sequence of '''workers'''. Each worker performs one<br />
task — reading data, applying a correction, filtering, scaling, post-refining, computing a statistic,<br />
or merging — and passes its (possibly modified) experiments and reflections on to the next worker.<br />
The list of workers to run is the "step list", and it is fully configurable: you can add, remove or<br />
reorder steps with <code>dispatch.step_list</code>. This is what makes the program flexible enough to<br />
scale-and-merge in one pass, to scale subsets separately and merge later, or to run a reference-free<br />
bootstrap.<br />
<br />
The '''default''' step list is:<br />
<br />
{| class="wikitable"<br />
! # !! Worker !! Purpose<br />
|-<br />
| 1 || <code>input</code> || Read integrated experiments (<code>.expt</code>) and reflections (<code>.refl</code>) from disk in DIALS format.<br />
|-<br />
| 2 || <code>balance</code> || Redistribute the input load evenly across MPI ranks (needed when there are fewer or more input files than ranks).<br />
|-<br />
| 3 || <code>model_scaling</code> || Build the full Miller list, model intensities from the reference, and the resolution binner used for scaling and post-refinement.<br />
|-<br />
| 4 || <code>modify</code> || Apply per-reflection modifications, e.g. polarization correction (and optionally re-indexing to resolve indexing ambiguity — see <code>modify.cosym</code>/<code>reindex_to_reference</code>).<br />
|-<br />
| 5 || <code>filter</code> || Reject whole experiments (or reflections) by resolution, unit cell, correlation to the reference, energy, etc.<br />
|-<br />
| 6 || <code>scale</code> || Scale each lattice to the reference dataset (<code>mark0</code>) or simply average (<code>mark1</code>).<br />
|-<br />
| 7 || <code>postrefine</code> || Post-refine per-image crystal parameters and partialities (Sauter 2015).<br />
|-<br />
| 8 || <code>statistics_unitcell</code> || Determine and store the average unit cell.<br />
|-<br />
| 9 || <code>statistics_beam</code> || Determine and store the average wavelength.<br />
|-<br />
| 10 || <code>model_statistics</code> || Rebuild the Miller list / binner for statistics (may use the average unit cell).<br />
|-<br />
| 11 || <code>statistics_resolution</code> || Per-crystal resolution statistics.<br />
|-<br />
| 12 || <code>group</code> || Redistribute reflections across ranks so that every observation of a given HKL is gathered on the same rank prior to merging.<br />
|-<br />
| 13 || <code>errors_merge</code> || Correct the merged error estimates with a per-HKL error model (e.g. Ev11, Brewster 2019b, or MM24 (the default), Mittan-Moreau (2025)).<br />
|-<br />
| 14 || <code>statistics_intensity</code> || Intensity statistics by resolution.<br />
|-<br />
| 15 || <code>merge</code> || Merge symmetry-equivalent intensities, gather all HKLs at rank 0, and write "odd", "even" and "all" MTZ files.<br />
|-<br />
| 16 || <code>statistics_intensity_cxi</code> || Correlation and R-factor statistics (ported from cxi.merge): the "Table of Scaling Results" — CC½ (half-dataset correlation, labelled "CC int"), CC<sub>iso</sub>, R<sub>int</sub>, R<sub>split</sub>, R<sub>iso</sub>.<br />
|}<br />
<br />
Additional, non-default workers are available and can be<br />
inserted into the step list, including <code>tdata</code> (write out unit cells for clustering),<br />
<code>export</code> (batch MTZ for ''Careless''), <code>publish</code> (upload merged files to a<br />
Google Drive folder), <code>monitor</code> (resource-usage logging), <code>integrate</code><br />
(on-the-fly integration), <code>prepare</code> (SpReAD energy binning), and<br />
<code>lunus</code> (diffuse-scattering deck). A <code>deltaccint</code> worker (ΔCC½ lattice<br />
filtering) can be added after <code>group</code>. Custom workers can be supplied through the<br />
<code>XFEL_CUSTOM_WORKER_PATH</code> environment variable.<br />
<br />
== Basic usage ==<br />
<br />
<code>cctbx.xfel.merge</code> is run under MPI:<br />
<br />
<pre><br />
mpirun cctbx.xfel.merge merge.phil<br />
</pre><br />
<br />
A typical <code>merge.phil</code> (from the program README):<br />
<br />
<pre><br />
input.path=<path to data><br />
input.experiments_suffix=.expt<br />
input.reflections_suffix=.refl<br />
<br />
filter.algorithm=unit_cell<br />
filter.unit_cell.value.relative_length_tolerance=0.03<br />
filter.outlier.min_corr=0.1<br />
<br />
select.algorithm=significance_filter<br />
select.significance_filter.sigma=0.1<br />
select.significance_filter.min_ct=200<br />
select.significance_filter.max_ct=300<br />
<br />
scaling.model=<model.pdb or model.mtz><br />
scaling.resolution_scalar=0.96<br />
<br />
merging.d_min=1.8<br />
merging.error.model=ev11<br />
merging.merge_anomalous=True<br />
<br />
postrefinement.enable=True<br />
statistics.n_bins=20<br />
<br />
output.do_timing=True<br />
output.output_dir=<output folder><br />
output.prefix=<prefix for output files><br />
</pre><br />
<br />
Explore the parameters with:<br />
* <code>cctbx.xfel.merge -c</code> — show configurable parameters<br />
* <code>cctbx.xfel.merge -c -e 10 -a 2</code> — show all parameters (expert level 10) with full help text<br />
<br />
== Two-stage scaling and merging ==<br />
<br />
For large, growing datasets (the mode used by the [[cctbx.xfel GUI]]), it is more efficient to scale<br />
and post-refine each subset of data '''once''', then merge repeatedly as more data arrive. This is<br />
done by running the program twice with different step lists.<br />
<br />
'''Stage 1 — scale each subset''' (stops after per-image work and writes intermediate DIALS files):<br />
<br />
<pre><br />
dispatch.step_list=input balance model_scaling modify filter scale postrefine \<br />
statistics_unitcell statistics_beam model_statistics statistics_resolution<br />
input.parallel_file_load.method=uniform<br />
output.save_experiments_and_reflections=True<br />
</pre><br />
<br />
<code>output.save_experiments_and_reflections=True</code> dumps the scaled/post-refined data as<br />
experiment lists and reflection tables that the merge stage can read back in.<br />
<br />
'''Stage 2 — merge the scaled subsets''' (add an <code>input.path</code> for each finished subset):<br />
<br />
<pre><br />
dispatch.step_list=input model_scaling statistics_unitcell statistics_beam \<br />
model_statistics statistics_resolution group errors_merge statistics_intensity \<br />
merge statistics_intensity_cxi<br />
input.parallel_file_load.method=uniform<br />
</pre><br />
<br />
== Merging without a reference (bootstrap / mark1) ==<br />
<br />
If no scaling/merging reference is available, use a bootstrap procedure. First run the<br />
<code>mark1</code> algorithm, which simply averages the data (all per-image scale factors = 1, no<br />
post-refinement), specifying the target cell and symmetry explicitly:<br />
<br />
<pre><br />
scaling.algorithm=mark1<br />
scaling.unit_cell=<unit cell><br />
scaling.space_group=<space group><br />
merging.error.model=errors_from_sample_residuals<br />
</pre><br />
<br />
The resulting MTZ is then used as the reference (<code>scaling.model=</code>) for subsequent regular<br />
merges. Iterate a few cycles; CC½ typically improves and converges over the first few rounds.<br />
<br />
== Important parameters ==<br />
<br />
Parameters are given by their full PHIL path with defaults in parentheses. The program exposes many<br />
more parameters (see <code>-c -e 10 -a 2</code>); those below are the ones most commonly set.<br />
<br />
=== Input ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>input.path</code> || None || Path(s) to the integrated data — a folder, several folders, or individual files; wildcards allowed; may be repeated. Matching <code>.expt</code> and <code>.refl</code> files must both be present.<br />
|-<br />
| <code>input.experiments_suffix</code> || <code>_integrated.expt</code> || Only load experiment files ending with this suffix.<br />
|-<br />
| <code>input.reflections_suffix</code> || <code>_integrated.refl</code> || Only load reflection files ending with this suffix.<br />
|-<br />
| <code>input.parallel_file_load.method</code> || uniform || <code>uniform</code> spreads input files evenly over all ranks; <code>node_memory</code> distributes so per-node memory limits are respected.<br />
|-<br />
| <code>input.alist.file</code> || None || Path to a text file of experiment tags or filenames; combined with <code>input.alist.op=keep|reject</code> to merge only (or exclude) a specified subset.<br />
|-<br />
| <code>input.persistent_refl_cols</code> || None || Reflection-table columns to preserve through the pruning steps (and into saved tables).<br />
|}<br />
<br />
=== Filtering whole experiments (<code>filter</code>) ===<br />
<br />
The <code>filter</code> section accepts or rejects '''entire experiments'''.<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>filter.algorithm</code> || (none) || One or more of <code>n_obs</code>, <code>resolution</code>, <code>unit_cell</code>, <code>energy</code>, <code>bootstrap</code>, <code>experiment_prefix</code>. Nothing is filtered unless listed.<br />
|-<br />
| <code>filter.unit_cell.algorithm</code> || value || How to restrict unit cells: <code>value</code> (compare to a target), <code>range</code>, or <code>cluster</code> (use a precomputed covariance clustering).<br />
|-<br />
| <code>filter.unit_cell.value.relative_length_tolerance</code> || 0.1 || Fractional deviation of cell edge lengths allowed vs. the target cell. Often tightened (e.g. 0.03).<br />
|-<br />
| <code>filter.unit_cell.value.absolute_angle_tolerance</code> || 2.0 || Allowed cell-angle deviation (degrees) vs. the target.<br />
|-<br />
| <code>filter.unit_cell.value.target_unit_cell</code> / <code>target_space_group</code> || Auto || Target cell/symmetry; <code>Auto</code> uses the scaling model.<br />
|-<br />
| <code>filter.outlier.min_corr</code> || 0.1 || Minimum correlation of an image's intensities to the reference model; images below this are rejected. Only applied when a scaling model is present. Use <code>-1</code> to accept all images regardless of correlation.<br />
|-<br />
| <code>filter.resolution.d_min</code> || None || Reject an experiment unless some of its reflections extend beyond this resolution.<br />
|}<br />
<br />
=== Selecting individual reflections (<code>select</code>) ===<br />
<br />
The <code>select</code> section accepts or rejects '''individual reflections''' (contrast with<br />
<code>filter</code>, which acts on whole experiments).<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>select.algorithm</code> || (none) || One or more of <code>significance_filter</code>, <code>panel</code>, <code>cspad_sensor</code>, <code>isolation_forest</code>, <code>target_energy</code>.<br />
|-<br />
| <code>select.significance_filter.sigma</code> || 0.5 || Trim each lattice at the resolution where the binned ⟨I/σ⟩ drops below this value.<br />
|-<br />
| <code>select.significance_filter.min_ct</code> / <code>max_ct</code> || 10 / 50 || Target minimum/maximum mean reflections per resolution bin when choosing the number of bins. For large unit cells these are often raised (e.g. 200/300); for small cells, use the defaults.<br />
|}<br />
<br />
=== Modifying intensities (<code>modify</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>modify.algorithm</code> || polarization || Per-reflection corrections to apply (default applies the polarization correction).<br />
|-<br />
| <code>modify.cosym</code> || — || Resolve an indexing ambiguity across the ensemble using the Gildea/Winter (Brehm/Diederichs) cosym method, optionally anchored to <code>scaling.model</code>.<br />
|-<br />
| <code>modify.reindex_to_reference</code> / <code>reindex_to_abc</code> || — || Alternative re-indexing operators (to a reference model, or an explicit operator).<br />
|}<br />
<br />
=== Scaling (<code>scaling</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>scaling.model</code> || None || Reference dataset: a PDB file (uses the CRYST1 cell and computes model intensities, including bulk solvent) or an MTZ from a previous cycle. Required for <code>mark0</code> scaling.<br />
|-<br />
| <code>scaling.algorithm</code> || mark0 || <code>mark0</code>: per-image scaling against the reference. <code>mark1</code>: no scaling, just Monte-Carlo averaging (all scale factors = 1) — used for reference-free bootstrap.<br />
|-<br />
| <code>scaling.unit_cell</code> / <code>scaling.space_group</code> || None || Cell/symmetry to use when no model is provided (i.e. <code>mark1</code>).<br />
|-<br />
| <code>scaling.resolution_scalar</code> || 0.969 || <code>merging.d_min</code> is multiplied by this when deciding which reflections fall within the resolution limit, generating a few extra reference reflections to account for per-shot cell variation (e.g. set 0.96 for a few more).<br />
|-<br />
| <code>scaling.mtz.mtz_column_F</code> || fobs || Column name in the reference MTZ containing the reference structure factors (intensities or amplitudes).<br />
|}<br />
<br />
=== Post-refinement (<code>postrefinement</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>postrefinement.enable</code> || True || Enable post-refinement of per-image parameters/partialities (Sauter 2015).<br />
|-<br />
| <code>postrefinement.algorithm</code> || rs || Post-refinement protocol: <code>rs</code>, <code>rs2</code>, <code>rs_hybrid</code>, or <code>eta_deff</code>.<br />
|}<br />
<br />
=== Merging and error model (<code>merging</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>merging.d_min</code> || None || High-resolution limit for scaling and merging.<br />
|-<br />
| <code>merging.d_max</code> || None || Low-resolution limit (currently affects only CCiso).<br />
|-<br />
| <code>merging.merge_anomalous</code> || False || <code>True</code> merges Bijvoet mates together (higher multiplicity); <code>False</code> keeps them separate to preserve the anomalous signal.<br />
|-<br />
| <code>merging.minimum_multiplicity</code> || 2 || Do not produce a merged structure factor for HKLs measured fewer than this many times.<br />
|-<br />
| <code>merging.error.model</code> || mm24 || Merged-sigma error model: <code>mm24</code> (Mittan-Moreau 2024, current default), <code>ev11</code> (Evans 2011 / sdfac-refine, recommended in the README), <code>ha14</code>, or <code>errors_from_sample_residuals</code> (no refined error terms — used with <code>mark1</code> bootstrap).<br />
|-<br />
| <code>merging.set_average_unit_cell</code> || True || Output MTZ adopts the (cluster-derived) average unit cell of the data rather than the reference model's cell.<br />
|}<br />
<br />
=== Statistics (<code>statistics</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>statistics.n_bins</code> || 10 || Number of resolution bins in the statistics tables.<br />
|-<br />
| <code>statistics.cciso.mtz_file</code> || None || Isomorphous reference structure factors for R<sub>iso</sub> / CC<sub>iso</sub>.<br />
|-<br />
| <code>statistics.report_ML</code> || True || Report per-frame max-likelihood attributes (domain size, mosaicity).<br />
|}<br />
<br />
=== Output (<code>output</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>output.output_dir</code> || <code>.</code> || Directory for output files (created if it does not exist).<br />
|-<br />
| <code>output.prefix</code> || iobs || Prefix prepended to all output file names.<br />
|-<br />
| <code>output.save_experiments_and_reflections</code> || False || Dump the final experiments/reflections from the last worker as DIALS files — used for the two-stage scale-then-merge workflow.<br />
|-<br />
| <code>output.do_timing</code> || False || Log elapsed time per execution step.<br />
|-<br />
| <code>output.expanded_bookkeeping</code> || False || Store extra provenance columns (input file/reflection indices, odd/even flag, original id) in the saved reflection tables.<br />
|}<br />
<br />
=== Multiprocessing (<code>mp</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>mp.method</code> || mpi || Multiprocessing method (only <code>mpi</code> is currently supported).<br />
|-<br />
| <code>mp.debug.cProfile</code> || False || Enable code profiling; a <code>.prof</code> file is written per rank.<br />
|}<br />
<br />
== Output files ==<br />
<br />
The <code>merge</code> worker gathers all HKLs at rank 0 and writes three MTZ files, prefixed with<br />
<code>output.prefix</code> (default <code>iobs</code>):<br />
<br />
* <code>&lt;prefix&gt;_all.mtz</code> — the merged dataset using all observations (the main result for structure solution)<br />
* <code>&lt;prefix&gt;_odd.mtz</code> and <code>&lt;prefix&gt;_even.mtz</code> — the two half-datasets used for CC½ and related half-dataset statistics<br />
<br />
A main log records the per-step progress and the merging statistics tables (completeness,<br />
multiplicity, ⟨I/σ⟩, CC½, CC*, R-factors) by resolution bin. The merged MTZ can optionally be<br />
uploaded to a shared [https://github.com/cctbx/cctbx_project/tree/master/xfel/merging/application/publish Google Drive folder] by adding the <code>publish</code> worker.<br />
<br />
== References ==<br />
<br />
* Sauter NK (2015). ''J Synchrotron Radiat'' '''22''', 239–48. (post-refinement)<br />
* Uervirojnangkoorn M, ''et al.'' (2015). ''eLife'' 10.7554/eLife.05421. (PRIME / bootstrap)<br />
* Brewster AS, ''et al.'' (2018). ''Acta Cryst.'' D '''74''', 877–894.<br />
* Brewster AS, ''et al.'' (2019a). ''Computational Crystallography Newsletter'' '''10''', 22–39. (XFEL GUI)<br />
* Brewster AS, ''et al.'' (2019b). ''Acta Cryst.'' D '''75''', 959–968. (error models)<br />
* Mittan-Moreau (2024). (mm24 error model)<br />
<br />
== See also ==<br />
<br />
* [[dials.stills_process]] — produces the integrated data that <code>cctbx.xfel.merge</code> consumes<br />
* [[cctbx.xfel GUI]] — orchestrates separate scaling and repeated merging as data are collected<br />
* The program README at <code>xfel/merging/README.md</code> in cctbx_project</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Cctbx.xfel.merge&diff=1774Cctbx.xfel.merge2026-08-21T23:12:15Z<p>Aaron: /* Overview */</p>
<hr />
<div>= ''cctbx.xfel.merge'' =<br />
<br />
'''''cctbx.xfel.merge''''' is the scaling, post-refinement and merging program of cctbx.xfel. It takes<br />
the per-image integrated results produced by ''[[dials.stills_process]]'' — one crystal per still shot —<br />
and reduces the whole ensemble to a single set of merged structure-factor intensities (an MTZ file)<br />
suitable for structure solution and refinement, together with the crystallographic quality<br />
statistics (completeness, multiplicity, CC½, R-factors, etc.).<br />
<br />
Because serial datasets routinely contain tens of thousands to millions of still shots,<br />
<code>cctbx.xfel.merge</code> is built from the ground up to run under MPI across many nodes. It is<br />
almost always launched with <code>mpirun</code>/<code>srun</code>. Note that MPI uses the term 'ranks'<br />
to distinguish independent processes that are running across one or more nodes.<br />
<br />
== Overview ==<br />
<br />
<code>cctbx.xfel.merge</code> is organized as a sequence of '''workers'''. Each worker performs one<br />
task — reading data, applying a correction, filtering, scaling, post-refining, computing a statistic,<br />
or merging — and passes its (possibly modified) experiments and reflections on to the next worker.<br />
The list of workers to run is the "step list", and it is fully configurable: you can add, remove or<br />
reorder steps with <code>dispatch.step_list</code>. This is what makes the program flexible enough to<br />
scale-and-merge in one pass, to scale subsets separately and merge later, or to run a reference-free<br />
bootstrap.<br />
<br />
The '''default''' step list is:<br />
<br />
{| class="wikitable"<br />
! # !! Worker !! Purpose<br />
|-<br />
| 1 || <code>input</code> || Read integrated experiments (<code>.expt</code>) and reflections (<code>.refl</code>) from disk in DIALS format.<br />
|-<br />
| 2 || <code>balance</code> || Redistribute the input load evenly across MPI ranks (needed when there are fewer or more input files than ranks).<br />
|-<br />
| 3 || <code>model_scaling</code> || Build the full Miller list, model intensities from the reference, and the resolution binner used for scaling and post-refinement.<br />
|-<br />
| 4 || <code>modify</code> || Apply per-reflection modifications, e.g. polarization correction (and optionally re-indexing to resolve indexing ambiguity — see <code>modify.cosym</code>/<code>reindex_to_reference</code>).<br />
|-<br />
| 5 || <code>filter</code> || Reject whole experiments (or reflections) by resolution, unit cell, correlation to the reference, energy, etc.<br />
|-<br />
| 6 || <code>scale</code> || Scale each lattice to the reference dataset (<code>mark0</code>) or simply average (<code>mark1</code>).<br />
|-<br />
| 7 || <code>postrefine</code> || Post-refine per-image crystal parameters and partialities (Sauter 2015).<br />
|-<br />
| 8 || <code>statistics_unitcell</code> || Determine and store the average unit cell.<br />
|-<br />
| 9 || <code>statistics_beam</code> || Determine and store the average wavelength.<br />
|-<br />
| 10 || <code>model_statistics</code> || Rebuild the Miller list / binner for statistics (may use the average unit cell).<br />
|-<br />
| 11 || <code>statistics_resolution</code> || Per-crystal resolution statistics.<br />
|-<br />
| 12 || <code>group</code> || Redistribute reflections across ranks so that every observation of a given HKL is gathered on the same rank prior to merging.<br />
|-<br />
| 13 || <code>errors_merge</code> || Correct the merged error estimates with a per-HKL error model (e.g. Ev11, Brewster 2019b, or MM24 (the default), Mittan-Moreau (2025)).<br />
|-<br />
| 14 || <code>statistics_intensity</code> || Intensity statistics by resolution.<br />
|-<br />
| 15 || <code>merge</code> || Merge symmetry-equivalent intensities, gather all HKLs at rank 0, and write "odd", "even" and "all" MTZ files.<br />
|-<br />
| 16 || <code>statistics_intensity_cxi</code> || Correlation and R-factor statistics (ported from cxi.merge): the "Table of Scaling Results" — CC½ (half-dataset correlation, labelled "CC int"), CC<sub>iso</sub>, R<sub>int</sub>, R<sub>split</sub>, R<sub>iso</sub>.<br />
|}<br />
<br />
Additional, non-default workers are available and can be<br />
inserted into the step list, including <code>tdata</code> (write out unit cells for clustering),<br />
<code>export</code> (batch MTZ for ''Careless''), <code>publish</code> (upload merged files to a<br />
Google Drive folder), <code>monitor</code> (resource-usage logging), <code>integrate</code><br />
(on-the-fly integration), <code>prepare</code> (SpReAD energy binning), and<br />
<code>lunus</code> (diffuse-scattering deck). A <code>deltaccint</code> worker (ΔCC½ lattice<br />
filtering) can be added after <code>group</code>. Custom workers can be supplied through the<br />
<code>XFEL_CUSTOM_WORKER_PATH</code> environment variable.<br />
<br />
== Basic usage ==<br />
<br />
<code>cctbx.xfel.merge</code> is run under MPI:<br />
<br />
<pre><br />
mpirun cctbx.xfel.merge merge.phil<br />
</pre><br />
<br />
A typical <code>merge.phil</code> (from the program README):<br />
<br />
<pre><br />
input.path=<path to data><br />
input.experiments_suffix=.expt<br />
input.reflections_suffix=.refl<br />
<br />
filter.algorithm=unit_cell<br />
filter.unit_cell.value.relative_length_tolerance=0.03<br />
filter.outlier.min_corr=0.1<br />
<br />
select.algorithm=significance_filter<br />
select.significance_filter.sigma=0.1<br />
select.significance_filter.min_ct=200<br />
select.significance_filter.max_ct=300<br />
<br />
scaling.model=<model.pdb or model.mtz><br />
scaling.resolution_scalar=0.96<br />
<br />
merging.d_min=1.8<br />
merging.error.model=ev11<br />
merging.merge_anomalous=True<br />
<br />
postrefinement.enable=True<br />
statistics.n_bins=20<br />
<br />
output.do_timing=True<br />
output.output_dir=<output folder><br />
output.prefix=<prefix for output files><br />
</pre><br />
<br />
Explore the parameters with:<br />
* <code>cctbx.xfel.merge -c</code> — show configurable parameters<br />
* <code>cctbx.xfel.merge -c -e 10 -a 2</code> — show all parameters (expert level 10) with full help text<br />
<br />
== Two-stage scaling and merging ==<br />
<br />
For large, growing datasets (the mode used by the [[cctbx.xfel GUI]]), it is more efficient to scale<br />
and post-refine each subset of data '''once''', then merge repeatedly as more data arrive. This is<br />
done by running the program twice with different step lists.<br />
<br />
'''Stage 1 — scale each subset''' (stops after per-image work and writes intermediate DIALS files):<br />
<br />
<pre><br />
dispatch.step_list=input balance model_scaling modify filter scale postrefine \<br />
statistics_unitcell statistics_beam model_statistics statistics_resolution<br />
input.parallel_file_load.method=uniform<br />
output.save_experiments_and_reflections=True<br />
</pre><br />
<br />
<code>output.save_experiments_and_reflections=True</code> dumps the scaled/post-refined data as<br />
experiment lists and reflection tables that the merge stage can read back in.<br />
<br />
'''Stage 2 — merge the scaled subsets''' (add an <code>input.path</code> for each finished subset):<br />
<br />
<pre><br />
dispatch.step_list=input model_scaling statistics_unitcell statistics_beam \<br />
model_statistics statistics_resolution group errors_merge statistics_intensity \<br />
merge statistics_intensity_cxi<br />
input.parallel_file_load.method=uniform<br />
</pre><br />
<br />
== Merging without a reference (bootstrap / mark1) ==<br />
<br />
If no scaling/merging reference is available, use a bootstrap procedure. First run the<br />
<code>mark1</code> algorithm, which simply averages the data (all per-image scale factors = 1, no<br />
post-refinement), specifying the target cell and symmetry explicitly:<br />
<br />
<pre><br />
scaling.algorithm=mark1<br />
scaling.unit_cell=<unit cell><br />
scaling.space_group=<space group><br />
merging.error.model=errors_from_sample_residuals<br />
</pre><br />
<br />
The resulting MTZ is then used as the reference (<code>scaling.model=</code>) for subsequent regular<br />
merges. Iterate a few cycles; CC½ typically improves and converges over the first few rounds.<br />
<br />
== Important parameters ==<br />
<br />
Parameters are given by their full PHIL path with defaults in parentheses. The program exposes many<br />
more parameters (see <code>-c -e 10 -a 2</code>); those below are the ones most commonly set.<br />
<br />
=== Input ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>input.path</code> || None || Path(s) to the integrated data — a folder, several folders, or individual files; wildcards allowed; may be repeated. Matching <code>.expt</code> and <code>.refl</code> files must both be present.<br />
|-<br />
| <code>input.experiments_suffix</code> || <code>_integrated.expt</code> || Only load experiment files ending with this suffix.<br />
|-<br />
| <code>input.reflections_suffix</code> || <code>_integrated.refl</code> || Only load reflection files ending with this suffix.<br />
|-<br />
| <code>input.parallel_file_load.method</code> || uniform || <code>uniform</code> spreads input files evenly over all ranks; <code>node_memory</code> distributes so per-node memory limits are respected.<br />
|-<br />
| <code>input.alist.file</code> || None || Path to a text file of experiment tags or filenames; combined with <code>input.alist.op=keep|reject</code> to merge only (or exclude) a specified subset.<br />
|-<br />
| <code>input.persistent_refl_cols</code> || None || Reflection-table columns to preserve through the pruning steps (and into saved tables).<br />
|}<br />
<br />
=== Filtering whole experiments (<code>filter</code>) ===<br />
<br />
The <code>filter</code> section accepts or rejects '''entire experiments'''.<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>filter.algorithm</code> || (none) || One or more of <code>n_obs</code>, <code>resolution</code>, <code>unit_cell</code>, <code>energy</code>, <code>bootstrap</code>, <code>experiment_prefix</code>. Nothing is filtered unless listed.<br />
|-<br />
| <code>filter.unit_cell.algorithm</code> || value || How to restrict unit cells: <code>value</code> (compare to a target), <code>range</code>, or <code>cluster</code> (use a precomputed covariance clustering).<br />
|-<br />
| <code>filter.unit_cell.value.relative_length_tolerance</code> || 0.1 || Fractional deviation of cell edge lengths allowed vs. the target cell. Often tightened (e.g. 0.03).<br />
|-<br />
| <code>filter.unit_cell.value.absolute_angle_tolerance</code> || 2.0 || Allowed cell-angle deviation (degrees) vs. the target.<br />
|-<br />
| <code>filter.unit_cell.value.target_unit_cell</code> / <code>target_space_group</code> || Auto || Target cell/symmetry; <code>Auto</code> uses the scaling model.<br />
|-<br />
| <code>filter.outlier.min_corr</code> || 0.1 || Minimum correlation of an image's intensities to the reference model; images below this are rejected. Only applied when a scaling model is present. Use <code>-1</code> to accept all images regardless of correlation.<br />
|-<br />
| <code>filter.resolution.d_min</code> || None || Reject an experiment unless some of its reflections extend beyond this resolution.<br />
|}<br />
<br />
=== Selecting individual reflections (<code>select</code>) ===<br />
<br />
The <code>select</code> section accepts or rejects '''individual reflections''' (contrast with<br />
<code>filter</code>, which acts on whole experiments).<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>select.algorithm</code> || (none) || One or more of <code>significance_filter</code>, <code>panel</code>, <code>cspad_sensor</code>, <code>isolation_forest</code>, <code>target_energy</code>.<br />
|-<br />
| <code>select.significance_filter.sigma</code> || 0.5 || Trim each lattice at the resolution where the binned ⟨I/σ⟩ drops below this value.<br />
|-<br />
| <code>select.significance_filter.min_ct</code> / <code>max_ct</code> || 10 / 50 || Target minimum/maximum mean reflections per resolution bin when choosing the number of bins. For large unit cells these are often raised (e.g. 200/300); for small cells, use the defaults.<br />
|}<br />
<br />
=== Modifying intensities (<code>modify</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>modify.algorithm</code> || polarization || Per-reflection corrections to apply (default applies the polarization correction).<br />
|-<br />
| <code>modify.cosym</code> || — || Resolve an indexing ambiguity across the ensemble using the Gildea/Winter (Brehm/Diederichs) cosym method, optionally anchored to <code>scaling.model</code>.<br />
|-<br />
| <code>modify.reindex_to_reference</code> / <code>reindex_to_abc</code> || — || Alternative re-indexing operators (to a reference model, or an explicit operator).<br />
|}<br />
<br />
=== Scaling (<code>scaling</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>scaling.model</code> || None || Reference dataset: a PDB file (uses the CRYST1 cell and computes model intensities, including bulk solvent) or an MTZ from a previous cycle. Required for <code>mark0</code> scaling.<br />
|-<br />
| <code>scaling.algorithm</code> || mark0 || <code>mark0</code>: per-image scaling against the reference. <code>mark1</code>: no scaling, just Monte-Carlo averaging (all scale factors = 1) — used for reference-free bootstrap.<br />
|-<br />
| <code>scaling.unit_cell</code> / <code>scaling.space_group</code> || None || Cell/symmetry to use when no model is provided (i.e. <code>mark1</code>).<br />
|-<br />
| <code>scaling.resolution_scalar</code> || 0.969 || <code>merging.d_min</code> is multiplied by this when deciding which reflections fall within the resolution limit, generating a few extra reference reflections to account for per-shot cell variation (e.g. set 0.96 for a few more).<br />
|-<br />
| <code>scaling.mtz.mtz_column_F</code> || fobs || Column name in the reference MTZ containing the reference structure factors (intensities or amplitudes).<br />
|}<br />
<br />
=== Post-refinement (<code>postrefinement</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>postrefinement.enable</code> || True || Enable post-refinement of per-image parameters/partialities (Sauter 2015).<br />
|-<br />
| <code>postrefinement.algorithm</code> || rs || Post-refinement protocol: <code>rs</code>, <code>rs2</code>, <code>rs_hybrid</code>, or <code>eta_deff</code>.<br />
|}<br />
<br />
=== Merging and error model (<code>merging</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>merging.d_min</code> || None || High-resolution limit for scaling and merging.<br />
|-<br />
| <code>merging.d_max</code> || None || Low-resolution limit (currently affects only CCiso).<br />
|-<br />
| <code>merging.merge_anomalous</code> || False || <code>True</code> merges Bijvoet mates together (higher multiplicity); <code>False</code> keeps them separate to preserve the anomalous signal.<br />
|-<br />
| <code>merging.minimum_multiplicity</code> || 2 || Do not produce a merged structure factor for HKLs measured fewer than this many times.<br />
|-<br />
| <code>merging.error.model</code> || mm24 || Merged-sigma error model: <code>mm24</code> (Mittan-Moreau 2024, current default), <code>ev11</code> (Evans 2011 / sdfac-refine, recommended in the README), <code>ha14</code>, or <code>errors_from_sample_residuals</code> (no refined error terms — used with <code>mark1</code> bootstrap).<br />
|-<br />
| <code>merging.set_average_unit_cell</code> || True || Output MTZ adopts the (cluster-derived) average unit cell of the data rather than the reference model's cell.<br />
|}<br />
<br />
=== Statistics (<code>statistics</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>statistics.n_bins</code> || 10 || Number of resolution bins in the statistics tables.<br />
|-<br />
| <code>statistics.cciso.mtz_file</code> || None || Isomorphous reference structure factors for R<sub>iso</sub> / CC<sub>iso</sub>.<br />
|-<br />
| <code>statistics.report_ML</code> || True || Report per-frame max-likelihood attributes (domain size, mosaicity).<br />
|}<br />
<br />
=== Output (<code>output</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>output.output_dir</code> || <code>.</code> || Directory for output files (created if it does not exist).<br />
|-<br />
| <code>output.prefix</code> || iobs || Prefix prepended to all output file names.<br />
|-<br />
| <code>output.save_experiments_and_reflections</code> || False || Dump the final experiments/reflections from the last worker as DIALS files — used for the two-stage scale-then-merge workflow.<br />
|-<br />
| <code>output.do_timing</code> || False || Log elapsed time per execution step.<br />
|-<br />
| <code>output.expanded_bookkeeping</code> || False || Store extra provenance columns (input file/reflection indices, odd/even flag, original id) in the saved reflection tables.<br />
|}<br />
<br />
=== Multiprocessing (<code>mp</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>mp.method</code> || mpi || Multiprocessing method (only <code>mpi</code> is currently supported).<br />
|-<br />
| <code>mp.debug.cProfile</code> || False || Enable code profiling; a <code>.prof</code> file is written per rank.<br />
|}<br />
<br />
== Output files ==<br />
<br />
The <code>merge</code> worker gathers all HKLs at rank 0 and writes three MTZ files, prefixed with<br />
<code>output.prefix</code> (default <code>iobs</code>):<br />
<br />
* <code>&lt;prefix&gt;_all.mtz</code> — the merged dataset using all observations (the main result for structure solution)<br />
* <code>&lt;prefix&gt;_odd.mtz</code> and <code>&lt;prefix&gt;_even.mtz</code> — the two half-datasets used for CC½ and related half-dataset statistics<br />
<br />
A main log records the per-step progress and the merging statistics tables (completeness,<br />
multiplicity, ⟨I/σ⟩, CC½, CC*, R-factors) by resolution bin. The merged MTZ can optionally be<br />
uploaded to a shared [https://github.com/cctbx/cctbx_project/tree/master/xfel/merging/application/publish Google Drive folder] by adding the <code>publish</code> worker.<br />
<br />
== References ==<br />
<br />
* Sauter NK (2015). ''J Synchrotron Radiat'' '''22''', 239–48. (post-refinement)<br />
* Uervirojnangkoorn M, ''et al.'' (2015). ''eLife'' 10.7554/eLife.05421. (PRIME / bootstrap)<br />
* Brewster AS, ''et al.'' (2018). ''Acta Cryst.'' D '''74''', 877–894.<br />
* Brewster AS, ''et al.'' (2019a). ''Computational Crystallography Newsletter'' '''10''', 22–39. (XFEL GUI)<br />
* Brewster AS, ''et al.'' (2019b). ''Acta Cryst.'' D '''75''', 959–968. (error models)<br />
* Mittan-Moreau (2024). (mm24 error model)<br />
<br />
== See also ==<br />
<br />
* [[dials.stills_process]] — produces the integrated data that <code>cctbx.xfel.merge</code> consumes<br />
* [[cctbx.xfel GUI]] — orchestrates separate scaling and repeated merging as data are collected<br />
* The program README at <code>xfel/merging/README.md</code> in cctbx_project</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Cctbx.xfel.merge&diff=1773Cctbx.xfel.merge2026-08-21T23:01:36Z<p>Aaron: /* cctbx.xfel.merge */</p>
<hr />
<div>= ''cctbx.xfel.merge'' =<br />
<br />
'''''cctbx.xfel.merge''''' is the scaling, post-refinement and merging program of cctbx.xfel. It takes<br />
the per-image integrated results produced by ''[[dials.stills_process]]'' — one crystal per still shot —<br />
and reduces the whole ensemble to a single set of merged structure-factor intensities (an MTZ file)<br />
suitable for structure solution and refinement, together with the crystallographic quality<br />
statistics (completeness, multiplicity, CC½, R-factors, etc.).<br />
<br />
Because serial datasets routinely contain tens of thousands to millions of still shots,<br />
<code>cctbx.xfel.merge</code> is built from the ground up to run under MPI across many nodes. It is<br />
almost always launched with <code>mpirun</code>/<code>srun</code>. Note that MPI uses the term 'ranks'<br />
to distinguish independent processes that are running across one or more nodes.<br />
<br />
== Overview ==<br />
<br />
<code>cctbx.xfel.merge</code> is organized as a sequence of '''workers'''. Each worker performs one<br />
task — reading data, applying a correction, filtering, scaling, post-refining, computing a statistic,<br />
or merging — and passes its (possibly modified) experiments and reflections on to the next worker.<br />
The list of workers to run is the "step list", and it is fully configurable: you can add, remove or<br />
reorder steps with <code>dispatch.step_list</code>. This is what makes the program flexible enough to<br />
scale-and-merge in one pass, to scale subsets separately and merge later, or to run a reference-free<br />
bootstrap.<br />
<br />
The '''default''' step list is:<br />
<br />
{| class="wikitable"<br />
! # !! Worker !! Purpose<br />
|-<br />
| 1 || <code>input</code> || Read integrated experiments (<code>.expt</code>) and reflections (<code>.refl</code>) from disk in DIALS format.<br />
|-<br />
| 2 || <code>balance</code> || Redistribute the input load evenly across MPI ranks (needed when there are fewer or more input files than ranks).<br />
|-<br />
| 3 || <code>model_scaling</code> || Build the full Miller list, model intensities from the reference, and the resolution binner used for scaling and post-refinement.<br />
|-<br />
| 4 || <code>modify</code> || Apply per-reflection modifications, e.g. polarization correction (and optionally re-indexing to resolve indexing ambiguity — see <code>modify.cosym</code>/<code>reindex_to_reference</code>).<br />
|-<br />
| 5 || <code>filter</code> || Reject whole experiments (or reflections) by resolution, unit cell, correlation to the reference, energy, etc.<br />
|-<br />
| 6 || <code>scale</code> || Scale each lattice to the reference dataset (<code>mark0</code>) or simply average (<code>mark1</code>).<br />
|-<br />
| 7 || <code>postrefine</code> || Post-refine per-image crystal parameters and partialities (Sauter 2015).<br />
|-<br />
| 8 || <code>statistics_unitcell</code> || Determine and store the average unit cell.<br />
|-<br />
| 9 || <code>statistics_beam</code> || Determine and store the average wavelength.<br />
|-<br />
| 10 || <code>model_statistics</code> || Rebuild the Miller list / binner for statistics (may use the average unit cell).<br />
|-<br />
| 11 || <code>statistics_resolution</code> || Per-crystal resolution statistics.<br />
|-<br />
| 12 || <code>group</code> || Redistribute reflections across ranks so that every observation of a given HKL is gathered on the same rank prior to merging.<br />
|-<br />
| 13 || <code>errors_merge</code> || Correct the merged error estimates with a per-HKL error model (e.g. Ev11, Brewster 2019b).<br />
|-<br />
| 14 || <code>statistics_intensity</code> || Intensity statistics by resolution.<br />
|-<br />
| 15 || <code>merge</code> || Merge symmetry-equivalent intensities, gather all HKLs at rank 0, and write "odd", "even" and "all" MTZ files.<br />
|-<br />
| 16 || <code>statistics_intensity_cxi</code> || CC½ / CC* and related statistics (ported from cxi.merge).<br />
|}<br />
<br />
Additional, non-default workers are available in <code>xfel/merging/application/</code> and can be<br />
inserted into the step list, including <code>tdata</code> (write out unit cells for clustering),<br />
<code>export</code> (batch MTZ for ''Careless''), <code>publish</code> (upload merged files to a<br />
Google Drive folder), <code>monitor</code> (resource-usage logging), <code>integrate</code><br />
(on-the-fly integration from psana2/XTC2), <code>prepare</code> (SpReAD energy binning), and<br />
<code>lunus</code> (diffuse-scattering deck). A <code>deltaccint</code> worker (ΔCC½ lattice<br />
filtering) can be added after <code>group</code>. Custom workers can be supplied through the<br />
<code>XFEL_CUSTOM_WORKER_PATH</code> environment variable.<br />
<br />
== Basic usage ==<br />
<br />
<code>cctbx.xfel.merge</code> is run under MPI:<br />
<br />
<pre><br />
mpirun cctbx.xfel.merge merge.phil<br />
</pre><br />
<br />
A typical <code>merge.phil</code> (from the program README):<br />
<br />
<pre><br />
input.path=<path to data><br />
input.experiments_suffix=.expt<br />
input.reflections_suffix=.refl<br />
<br />
filter.algorithm=unit_cell<br />
filter.unit_cell.value.relative_length_tolerance=0.03<br />
filter.outlier.min_corr=0.1<br />
<br />
select.algorithm=significance_filter<br />
select.significance_filter.sigma=0.1<br />
select.significance_filter.min_ct=200<br />
select.significance_filter.max_ct=300<br />
<br />
scaling.model=<model.pdb or model.mtz><br />
scaling.resolution_scalar=0.96<br />
<br />
merging.d_min=1.8<br />
merging.error.model=ev11<br />
merging.merge_anomalous=True<br />
<br />
postrefinement.enable=True<br />
statistics.n_bins=20<br />
<br />
output.do_timing=True<br />
output.output_dir=<output folder><br />
output.prefix=<prefix for output files><br />
</pre><br />
<br />
Explore the parameters with:<br />
* <code>cctbx.xfel.merge -c</code> — show configurable parameters<br />
* <code>cctbx.xfel.merge -c -e 10 -a 2</code> — show all parameters (expert level 10) with full help text<br />
<br />
== Two-stage scaling and merging ==<br />
<br />
For large, growing datasets (the mode used by the [[cctbx.xfel GUI]]), it is more efficient to scale<br />
and post-refine each subset of data '''once''', then merge repeatedly as more data arrive. This is<br />
done by running the program twice with different step lists.<br />
<br />
'''Stage 1 — scale each subset''' (stops after per-image work and writes intermediate DIALS files):<br />
<br />
<pre><br />
dispatch.step_list=input balance model_scaling modify filter scale postrefine \<br />
statistics_unitcell statistics_beam model_statistics statistics_resolution<br />
input.parallel_file_load.method=uniform<br />
output.save_experiments_and_reflections=True<br />
</pre><br />
<br />
<code>output.save_experiments_and_reflections=True</code> dumps the scaled/post-refined data as<br />
experiment lists and reflection tables that the merge stage can read back in.<br />
<br />
'''Stage 2 — merge the scaled subsets''' (add an <code>input.path</code> for each finished subset):<br />
<br />
<pre><br />
dispatch.step_list=input model_scaling statistics_unitcell statistics_beam \<br />
model_statistics statistics_resolution group errors_merge statistics_intensity \<br />
merge statistics_intensity_cxi<br />
input.parallel_file_load.method=uniform<br />
</pre><br />
<br />
== Merging without a reference (bootstrap / mark1) ==<br />
<br />
If no scaling/merging reference is available, use a bootstrap procedure. First run the<br />
<code>mark1</code> algorithm, which simply averages the data (all per-image scale factors = 1, no<br />
post-refinement), specifying the target cell and symmetry explicitly:<br />
<br />
<pre><br />
scaling.algorithm=mark1<br />
scaling.unit_cell=<unit cell><br />
scaling.space_group=<space group><br />
merging.error.model=errors_from_sample_residuals<br />
</pre><br />
<br />
The resulting MTZ is then used as the reference (<code>scaling.model=</code>) for subsequent regular<br />
merges. Iterate a few cycles; CC½ typically improves and converges over the first few rounds.<br />
<br />
== Important parameters ==<br />
<br />
Parameters are given by their full PHIL path with defaults in parentheses. The program exposes many<br />
more parameters (see <code>-c -e 10 -a 2</code>); those below are the ones most commonly set.<br />
<br />
=== Input ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>input.path</code> || None || Path(s) to the integrated data — a folder, several folders, or individual files; wildcards allowed; may be repeated. Matching <code>.expt</code> and <code>.refl</code> files must both be present.<br />
|-<br />
| <code>input.experiments_suffix</code> || <code>_integrated.expt</code> || Only load experiment files ending with this suffix.<br />
|-<br />
| <code>input.reflections_suffix</code> || <code>_integrated.refl</code> || Only load reflection files ending with this suffix.<br />
|-<br />
| <code>input.parallel_file_load.method</code> || uniform || <code>uniform</code> spreads input files evenly over all ranks; <code>node_memory</code> distributes so per-node memory limits are respected.<br />
|-<br />
| <code>input.alist.file</code> || None || Path to a text file of experiment tags or filenames; combined with <code>input.alist.op=keep|reject</code> to merge only (or exclude) a specified subset.<br />
|-<br />
| <code>input.persistent_refl_cols</code> || None || Reflection-table columns to preserve through the pruning steps (and into saved tables).<br />
|}<br />
<br />
=== Filtering whole experiments (<code>filter</code>) ===<br />
<br />
The <code>filter</code> section accepts or rejects '''entire experiments'''.<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>filter.algorithm</code> || (none) || One or more of <code>n_obs</code>, <code>resolution</code>, <code>unit_cell</code>, <code>energy</code>, <code>bootstrap</code>, <code>experiment_prefix</code>. Nothing is filtered unless listed.<br />
|-<br />
| <code>filter.unit_cell.algorithm</code> || value || How to restrict unit cells: <code>value</code> (compare to a target), <code>range</code>, or <code>cluster</code> (use a precomputed covariance clustering).<br />
|-<br />
| <code>filter.unit_cell.value.relative_length_tolerance</code> || 0.1 || Fractional deviation of cell edge lengths allowed vs. the target cell. Often tightened (e.g. 0.03).<br />
|-<br />
| <code>filter.unit_cell.value.absolute_angle_tolerance</code> || 2.0 || Allowed cell-angle deviation (degrees) vs. the target.<br />
|-<br />
| <code>filter.unit_cell.value.target_unit_cell</code> / <code>target_space_group</code> || Auto || Target cell/symmetry; <code>Auto</code> uses the scaling model.<br />
|-<br />
| <code>filter.outlier.min_corr</code> || 0.1 || Minimum correlation of an image's intensities to the reference model; images below this are rejected. Only applied when a scaling model is present. Use <code>-1</code> to accept all images regardless of correlation.<br />
|-<br />
| <code>filter.resolution.d_min</code> || None || Reject an experiment unless some of its reflections extend beyond this resolution.<br />
|}<br />
<br />
=== Selecting individual reflections (<code>select</code>) ===<br />
<br />
The <code>select</code> section accepts or rejects '''individual reflections''' (contrast with<br />
<code>filter</code>, which acts on whole experiments).<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>select.algorithm</code> || (none) || One or more of <code>significance_filter</code>, <code>panel</code>, <code>cspad_sensor</code>, <code>isolation_forest</code>, <code>target_energy</code>.<br />
|-<br />
| <code>select.significance_filter.sigma</code> || 0.5 || Trim each lattice at the resolution where the binned ⟨I/σ⟩ drops below this value.<br />
|-<br />
| <code>select.significance_filter.min_ct</code> / <code>max_ct</code> || 10 / 50 || Target minimum/maximum mean reflections per resolution bin when choosing the number of bins. For large unit cells these are often raised (e.g. 200/300); for small cells, use the defaults.<br />
|}<br />
<br />
=== Modifying intensities (<code>modify</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>modify.algorithm</code> || polarization || Per-reflection corrections to apply (default applies the polarization correction).<br />
|-<br />
| <code>modify.cosym</code> || — || Resolve an indexing ambiguity across the ensemble using the Gildea/Winter (Brehm/Diederichs) cosym method, optionally anchored to <code>scaling.model</code>.<br />
|-<br />
| <code>modify.reindex_to_reference</code> / <code>reindex_to_abc</code> || — || Alternative re-indexing operators (to a reference model, or an explicit operator).<br />
|}<br />
<br />
=== Scaling (<code>scaling</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>scaling.model</code> || None || Reference dataset: a PDB file (uses the CRYST1 cell and computes model intensities, including bulk solvent) or an MTZ from a previous cycle. Required for <code>mark0</code> scaling.<br />
|-<br />
| <code>scaling.algorithm</code> || mark0 || <code>mark0</code>: per-image scaling against the reference. <code>mark1</code>: no scaling, just Monte-Carlo averaging (all scale factors = 1) — used for reference-free bootstrap.<br />
|-<br />
| <code>scaling.unit_cell</code> / <code>scaling.space_group</code> || None || Cell/symmetry to use when no model is provided (i.e. <code>mark1</code>).<br />
|-<br />
| <code>scaling.resolution_scalar</code> || 0.969 || <code>merging.d_min</code> is multiplied by this when deciding which reflections fall within the resolution limit, generating a few extra reference reflections to account for per-shot cell variation (e.g. set 0.96 for a few more).<br />
|-<br />
| <code>scaling.mtz.mtz_column_F</code> || fobs || Column name in the reference MTZ containing the reference structure factors (intensities or amplitudes).<br />
|}<br />
<br />
=== Post-refinement (<code>postrefinement</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>postrefinement.enable</code> || True || Enable post-refinement of per-image parameters/partialities (Sauter 2015).<br />
|-<br />
| <code>postrefinement.algorithm</code> || rs || Post-refinement protocol: <code>rs</code>, <code>rs2</code>, <code>rs_hybrid</code>, or <code>eta_deff</code>.<br />
|}<br />
<br />
=== Merging and error model (<code>merging</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>merging.d_min</code> || None || High-resolution limit for scaling and merging.<br />
|-<br />
| <code>merging.d_max</code> || None || Low-resolution limit (currently affects only CCiso).<br />
|-<br />
| <code>merging.merge_anomalous</code> || False || <code>True</code> merges Bijvoet mates together (higher multiplicity); <code>False</code> keeps them separate to preserve the anomalous signal.<br />
|-<br />
| <code>merging.minimum_multiplicity</code> || 2 || Do not produce a merged structure factor for HKLs measured fewer than this many times.<br />
|-<br />
| <code>merging.error.model</code> || mm24 || Merged-sigma error model: <code>mm24</code> (Mittan-Moreau 2024, current default), <code>ev11</code> (Evans 2011 / sdfac-refine, recommended in the README), <code>ha14</code>, or <code>errors_from_sample_residuals</code> (no refined error terms — used with <code>mark1</code> bootstrap).<br />
|-<br />
| <code>merging.set_average_unit_cell</code> || True || Output MTZ adopts the (cluster-derived) average unit cell of the data rather than the reference model's cell.<br />
|}<br />
<br />
=== Statistics (<code>statistics</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>statistics.n_bins</code> || 10 || Number of resolution bins in the statistics tables.<br />
|-<br />
| <code>statistics.cciso.mtz_file</code> || None || Isomorphous reference structure factors for R<sub>iso</sub> / CC<sub>iso</sub>.<br />
|-<br />
| <code>statistics.report_ML</code> || True || Report per-frame max-likelihood attributes (domain size, mosaicity).<br />
|}<br />
<br />
=== Output (<code>output</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>output.output_dir</code> || <code>.</code> || Directory for output files (created if it does not exist).<br />
|-<br />
| <code>output.prefix</code> || iobs || Prefix prepended to all output file names.<br />
|-<br />
| <code>output.save_experiments_and_reflections</code> || False || Dump the final experiments/reflections from the last worker as DIALS files — used for the two-stage scale-then-merge workflow.<br />
|-<br />
| <code>output.do_timing</code> || False || Log elapsed time per execution step.<br />
|-<br />
| <code>output.expanded_bookkeeping</code> || False || Store extra provenance columns (input file/reflection indices, odd/even flag, original id) in the saved reflection tables.<br />
|}<br />
<br />
=== Multiprocessing (<code>mp</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>mp.method</code> || mpi || Multiprocessing method (only <code>mpi</code> is currently supported).<br />
|-<br />
| <code>mp.debug.cProfile</code> || False || Enable code profiling; a <code>.prof</code> file is written per rank.<br />
|}<br />
<br />
== Output files ==<br />
<br />
The <code>merge</code> worker gathers all HKLs at rank 0 and writes three MTZ files, prefixed with<br />
<code>output.prefix</code> (default <code>iobs</code>):<br />
<br />
* <code>&lt;prefix&gt;_all.mtz</code> — the merged dataset using all observations (the main result for structure solution)<br />
* <code>&lt;prefix&gt;_odd.mtz</code> and <code>&lt;prefix&gt;_even.mtz</code> — the two half-datasets used for CC½ and related half-dataset statistics<br />
<br />
A main log records the per-step progress and the merging statistics tables (completeness,<br />
multiplicity, ⟨I/σ⟩, CC½, CC*, R-factors) by resolution bin. The merged MTZ can optionally be<br />
uploaded to a shared [https://github.com/cctbx/cctbx_project/tree/master/xfel/merging/application/publish Google Drive folder] by adding the <code>publish</code> worker.<br />
<br />
== References ==<br />
<br />
* Sauter NK (2015). ''J Synchrotron Radiat'' '''22''', 239–48. (post-refinement)<br />
* Uervirojnangkoorn M, ''et al.'' (2015). ''eLife'' 10.7554/eLife.05421. (PRIME / bootstrap)<br />
* Brewster AS, ''et al.'' (2018). ''Acta Cryst.'' D '''74''', 877–894.<br />
* Brewster AS, ''et al.'' (2019a). ''Computational Crystallography Newsletter'' '''10''', 22–39. (XFEL GUI)<br />
* Brewster AS, ''et al.'' (2019b). ''Acta Cryst.'' D '''75''', 959–968. (error models)<br />
* Mittan-Moreau (2024). (mm24 error model)<br />
<br />
== See also ==<br />
<br />
* [[dials.stills_process]] — produces the integrated data that <code>cctbx.xfel.merge</code> consumes<br />
* [[cctbx.xfel GUI]] — orchestrates separate scaling and repeated merging as data are collected<br />
* The program README at <code>xfel/merging/README.md</code> in cctbx_project</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Cctbx.xfel.merge&diff=1772Cctbx.xfel.merge2026-08-21T23:00:59Z<p>Aaron: /* cctbx.xfel.merge */</p>
<hr />
<div>= ''cctbx.xfel.merge'' =<br />
<br />
'''''cctbx.xfel.merge''''' is the scaling, post-refinement and merging program of cctbx.xfel. It takes<br />
the per-image integrated results produced by ''[[dials.stills_process]]'' — one crystal per still shot —<br />
and reduces the whole ensemble to a single set of merged structure-factor intensities (an MTZ file)<br />
suitable for structure solution and refinement, together with the crystallographic quality<br />
statistics (completeness, multiplicity, CC½, R-factors, etc.).<br />
<br />
Because serial datasets routinely contain tens of thousands to millions of still shots,<br />
<code>cctbx.xfel.merge</code> is built from the ground up to run under MPI across many nodes. It is<br />
almost always launched with <code>mpirun</code>/<code>srun</code>. Note that MPI uses the term 'ranks'<br />
to distinguish independent processes that are running across more than one node.<br />
<br />
== Overview ==<br />
<br />
<code>cctbx.xfel.merge</code> is organized as a sequence of '''workers'''. Each worker performs one<br />
task — reading data, applying a correction, filtering, scaling, post-refining, computing a statistic,<br />
or merging — and passes its (possibly modified) experiments and reflections on to the next worker.<br />
The list of workers to run is the "step list", and it is fully configurable: you can add, remove or<br />
reorder steps with <code>dispatch.step_list</code>. This is what makes the program flexible enough to<br />
scale-and-merge in one pass, to scale subsets separately and merge later, or to run a reference-free<br />
bootstrap.<br />
<br />
The '''default''' step list is:<br />
<br />
{| class="wikitable"<br />
! # !! Worker !! Purpose<br />
|-<br />
| 1 || <code>input</code> || Read integrated experiments (<code>.expt</code>) and reflections (<code>.refl</code>) from disk in DIALS format.<br />
|-<br />
| 2 || <code>balance</code> || Redistribute the input load evenly across MPI ranks (needed when there are fewer or more input files than ranks).<br />
|-<br />
| 3 || <code>model_scaling</code> || Build the full Miller list, model intensities from the reference, and the resolution binner used for scaling and post-refinement.<br />
|-<br />
| 4 || <code>modify</code> || Apply per-reflection modifications, e.g. polarization correction (and optionally re-indexing to resolve indexing ambiguity — see <code>modify.cosym</code>/<code>reindex_to_reference</code>).<br />
|-<br />
| 5 || <code>filter</code> || Reject whole experiments (or reflections) by resolution, unit cell, correlation to the reference, energy, etc.<br />
|-<br />
| 6 || <code>scale</code> || Scale each lattice to the reference dataset (<code>mark0</code>) or simply average (<code>mark1</code>).<br />
|-<br />
| 7 || <code>postrefine</code> || Post-refine per-image crystal parameters and partialities (Sauter 2015).<br />
|-<br />
| 8 || <code>statistics_unitcell</code> || Determine and store the average unit cell.<br />
|-<br />
| 9 || <code>statistics_beam</code> || Determine and store the average wavelength.<br />
|-<br />
| 10 || <code>model_statistics</code> || Rebuild the Miller list / binner for statistics (may use the average unit cell).<br />
|-<br />
| 11 || <code>statistics_resolution</code> || Per-crystal resolution statistics.<br />
|-<br />
| 12 || <code>group</code> || Redistribute reflections across ranks so that every observation of a given HKL is gathered on the same rank prior to merging.<br />
|-<br />
| 13 || <code>errors_merge</code> || Correct the merged error estimates with a per-HKL error model (e.g. Ev11, Brewster 2019b).<br />
|-<br />
| 14 || <code>statistics_intensity</code> || Intensity statistics by resolution.<br />
|-<br />
| 15 || <code>merge</code> || Merge symmetry-equivalent intensities, gather all HKLs at rank 0, and write "odd", "even" and "all" MTZ files.<br />
|-<br />
| 16 || <code>statistics_intensity_cxi</code> || CC½ / CC* and related statistics (ported from cxi.merge).<br />
|}<br />
<br />
Additional, non-default workers are available in <code>xfel/merging/application/</code> and can be<br />
inserted into the step list, including <code>tdata</code> (write out unit cells for clustering),<br />
<code>export</code> (batch MTZ for ''Careless''), <code>publish</code> (upload merged files to a<br />
Google Drive folder), <code>monitor</code> (resource-usage logging), <code>integrate</code><br />
(on-the-fly integration from psana2/XTC2), <code>prepare</code> (SpReAD energy binning), and<br />
<code>lunus</code> (diffuse-scattering deck). A <code>deltaccint</code> worker (ΔCC½ lattice<br />
filtering) can be added after <code>group</code>. Custom workers can be supplied through the<br />
<code>XFEL_CUSTOM_WORKER_PATH</code> environment variable.<br />
<br />
== Basic usage ==<br />
<br />
<code>cctbx.xfel.merge</code> is run under MPI:<br />
<br />
<pre><br />
mpirun cctbx.xfel.merge merge.phil<br />
</pre><br />
<br />
A typical <code>merge.phil</code> (from the program README):<br />
<br />
<pre><br />
input.path=<path to data><br />
input.experiments_suffix=.expt<br />
input.reflections_suffix=.refl<br />
<br />
filter.algorithm=unit_cell<br />
filter.unit_cell.value.relative_length_tolerance=0.03<br />
filter.outlier.min_corr=0.1<br />
<br />
select.algorithm=significance_filter<br />
select.significance_filter.sigma=0.1<br />
select.significance_filter.min_ct=200<br />
select.significance_filter.max_ct=300<br />
<br />
scaling.model=<model.pdb or model.mtz><br />
scaling.resolution_scalar=0.96<br />
<br />
merging.d_min=1.8<br />
merging.error.model=ev11<br />
merging.merge_anomalous=True<br />
<br />
postrefinement.enable=True<br />
statistics.n_bins=20<br />
<br />
output.do_timing=True<br />
output.output_dir=<output folder><br />
output.prefix=<prefix for output files><br />
</pre><br />
<br />
Explore the parameters with:<br />
* <code>cctbx.xfel.merge -c</code> — show configurable parameters<br />
* <code>cctbx.xfel.merge -c -e 10 -a 2</code> — show all parameters (expert level 10) with full help text<br />
<br />
== Two-stage scaling and merging ==<br />
<br />
For large, growing datasets (the mode used by the [[cctbx.xfel GUI]]), it is more efficient to scale<br />
and post-refine each subset of data '''once''', then merge repeatedly as more data arrive. This is<br />
done by running the program twice with different step lists.<br />
<br />
'''Stage 1 — scale each subset''' (stops after per-image work and writes intermediate DIALS files):<br />
<br />
<pre><br />
dispatch.step_list=input balance model_scaling modify filter scale postrefine \<br />
statistics_unitcell statistics_beam model_statistics statistics_resolution<br />
input.parallel_file_load.method=uniform<br />
output.save_experiments_and_reflections=True<br />
</pre><br />
<br />
<code>output.save_experiments_and_reflections=True</code> dumps the scaled/post-refined data as<br />
experiment lists and reflection tables that the merge stage can read back in.<br />
<br />
'''Stage 2 — merge the scaled subsets''' (add an <code>input.path</code> for each finished subset):<br />
<br />
<pre><br />
dispatch.step_list=input model_scaling statistics_unitcell statistics_beam \<br />
model_statistics statistics_resolution group errors_merge statistics_intensity \<br />
merge statistics_intensity_cxi<br />
input.parallel_file_load.method=uniform<br />
</pre><br />
<br />
== Merging without a reference (bootstrap / mark1) ==<br />
<br />
If no scaling/merging reference is available, use a bootstrap procedure. First run the<br />
<code>mark1</code> algorithm, which simply averages the data (all per-image scale factors = 1, no<br />
post-refinement), specifying the target cell and symmetry explicitly:<br />
<br />
<pre><br />
scaling.algorithm=mark1<br />
scaling.unit_cell=<unit cell><br />
scaling.space_group=<space group><br />
merging.error.model=errors_from_sample_residuals<br />
</pre><br />
<br />
The resulting MTZ is then used as the reference (<code>scaling.model=</code>) for subsequent regular<br />
merges. Iterate a few cycles; CC½ typically improves and converges over the first few rounds.<br />
<br />
== Important parameters ==<br />
<br />
Parameters are given by their full PHIL path with defaults in parentheses. The program exposes many<br />
more parameters (see <code>-c -e 10 -a 2</code>); those below are the ones most commonly set.<br />
<br />
=== Input ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>input.path</code> || None || Path(s) to the integrated data — a folder, several folders, or individual files; wildcards allowed; may be repeated. Matching <code>.expt</code> and <code>.refl</code> files must both be present.<br />
|-<br />
| <code>input.experiments_suffix</code> || <code>_integrated.expt</code> || Only load experiment files ending with this suffix.<br />
|-<br />
| <code>input.reflections_suffix</code> || <code>_integrated.refl</code> || Only load reflection files ending with this suffix.<br />
|-<br />
| <code>input.parallel_file_load.method</code> || uniform || <code>uniform</code> spreads input files evenly over all ranks; <code>node_memory</code> distributes so per-node memory limits are respected.<br />
|-<br />
| <code>input.alist.file</code> || None || Path to a text file of experiment tags or filenames; combined with <code>input.alist.op=keep|reject</code> to merge only (or exclude) a specified subset.<br />
|-<br />
| <code>input.persistent_refl_cols</code> || None || Reflection-table columns to preserve through the pruning steps (and into saved tables).<br />
|}<br />
<br />
=== Filtering whole experiments (<code>filter</code>) ===<br />
<br />
The <code>filter</code> section accepts or rejects '''entire experiments'''.<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>filter.algorithm</code> || (none) || One or more of <code>n_obs</code>, <code>resolution</code>, <code>unit_cell</code>, <code>energy</code>, <code>bootstrap</code>, <code>experiment_prefix</code>. Nothing is filtered unless listed.<br />
|-<br />
| <code>filter.unit_cell.algorithm</code> || value || How to restrict unit cells: <code>value</code> (compare to a target), <code>range</code>, or <code>cluster</code> (use a precomputed covariance clustering).<br />
|-<br />
| <code>filter.unit_cell.value.relative_length_tolerance</code> || 0.1 || Fractional deviation of cell edge lengths allowed vs. the target cell. Often tightened (e.g. 0.03).<br />
|-<br />
| <code>filter.unit_cell.value.absolute_angle_tolerance</code> || 2.0 || Allowed cell-angle deviation (degrees) vs. the target.<br />
|-<br />
| <code>filter.unit_cell.value.target_unit_cell</code> / <code>target_space_group</code> || Auto || Target cell/symmetry; <code>Auto</code> uses the scaling model.<br />
|-<br />
| <code>filter.outlier.min_corr</code> || 0.1 || Minimum correlation of an image's intensities to the reference model; images below this are rejected. Only applied when a scaling model is present. Use <code>-1</code> to accept all images regardless of correlation.<br />
|-<br />
| <code>filter.resolution.d_min</code> || None || Reject an experiment unless some of its reflections extend beyond this resolution.<br />
|}<br />
<br />
=== Selecting individual reflections (<code>select</code>) ===<br />
<br />
The <code>select</code> section accepts or rejects '''individual reflections''' (contrast with<br />
<code>filter</code>, which acts on whole experiments).<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>select.algorithm</code> || (none) || One or more of <code>significance_filter</code>, <code>panel</code>, <code>cspad_sensor</code>, <code>isolation_forest</code>, <code>target_energy</code>.<br />
|-<br />
| <code>select.significance_filter.sigma</code> || 0.5 || Trim each lattice at the resolution where the binned ⟨I/σ⟩ drops below this value.<br />
|-<br />
| <code>select.significance_filter.min_ct</code> / <code>max_ct</code> || 10 / 50 || Target minimum/maximum mean reflections per resolution bin when choosing the number of bins. For large unit cells these are often raised (e.g. 200/300); for small cells, use the defaults.<br />
|}<br />
<br />
=== Modifying intensities (<code>modify</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>modify.algorithm</code> || polarization || Per-reflection corrections to apply (default applies the polarization correction).<br />
|-<br />
| <code>modify.cosym</code> || — || Resolve an indexing ambiguity across the ensemble using the Gildea/Winter (Brehm/Diederichs) cosym method, optionally anchored to <code>scaling.model</code>.<br />
|-<br />
| <code>modify.reindex_to_reference</code> / <code>reindex_to_abc</code> || — || Alternative re-indexing operators (to a reference model, or an explicit operator).<br />
|}<br />
<br />
=== Scaling (<code>scaling</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>scaling.model</code> || None || Reference dataset: a PDB file (uses the CRYST1 cell and computes model intensities, including bulk solvent) or an MTZ from a previous cycle. Required for <code>mark0</code> scaling.<br />
|-<br />
| <code>scaling.algorithm</code> || mark0 || <code>mark0</code>: per-image scaling against the reference. <code>mark1</code>: no scaling, just Monte-Carlo averaging (all scale factors = 1) — used for reference-free bootstrap.<br />
|-<br />
| <code>scaling.unit_cell</code> / <code>scaling.space_group</code> || None || Cell/symmetry to use when no model is provided (i.e. <code>mark1</code>).<br />
|-<br />
| <code>scaling.resolution_scalar</code> || 0.969 || <code>merging.d_min</code> is multiplied by this when deciding which reflections fall within the resolution limit, generating a few extra reference reflections to account for per-shot cell variation (e.g. set 0.96 for a few more).<br />
|-<br />
| <code>scaling.mtz.mtz_column_F</code> || fobs || Column name in the reference MTZ containing the reference structure factors (intensities or amplitudes).<br />
|}<br />
<br />
=== Post-refinement (<code>postrefinement</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>postrefinement.enable</code> || True || Enable post-refinement of per-image parameters/partialities (Sauter 2015).<br />
|-<br />
| <code>postrefinement.algorithm</code> || rs || Post-refinement protocol: <code>rs</code>, <code>rs2</code>, <code>rs_hybrid</code>, or <code>eta_deff</code>.<br />
|}<br />
<br />
=== Merging and error model (<code>merging</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>merging.d_min</code> || None || High-resolution limit for scaling and merging.<br />
|-<br />
| <code>merging.d_max</code> || None || Low-resolution limit (currently affects only CCiso).<br />
|-<br />
| <code>merging.merge_anomalous</code> || False || <code>True</code> merges Bijvoet mates together (higher multiplicity); <code>False</code> keeps them separate to preserve the anomalous signal.<br />
|-<br />
| <code>merging.minimum_multiplicity</code> || 2 || Do not produce a merged structure factor for HKLs measured fewer than this many times.<br />
|-<br />
| <code>merging.error.model</code> || mm24 || Merged-sigma error model: <code>mm24</code> (Mittan-Moreau 2024, current default), <code>ev11</code> (Evans 2011 / sdfac-refine, recommended in the README), <code>ha14</code>, or <code>errors_from_sample_residuals</code> (no refined error terms — used with <code>mark1</code> bootstrap).<br />
|-<br />
| <code>merging.set_average_unit_cell</code> || True || Output MTZ adopts the (cluster-derived) average unit cell of the data rather than the reference model's cell.<br />
|}<br />
<br />
=== Statistics (<code>statistics</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>statistics.n_bins</code> || 10 || Number of resolution bins in the statistics tables.<br />
|-<br />
| <code>statistics.cciso.mtz_file</code> || None || Isomorphous reference structure factors for R<sub>iso</sub> / CC<sub>iso</sub>.<br />
|-<br />
| <code>statistics.report_ML</code> || True || Report per-frame max-likelihood attributes (domain size, mosaicity).<br />
|}<br />
<br />
=== Output (<code>output</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>output.output_dir</code> || <code>.</code> || Directory for output files (created if it does not exist).<br />
|-<br />
| <code>output.prefix</code> || iobs || Prefix prepended to all output file names.<br />
|-<br />
| <code>output.save_experiments_and_reflections</code> || False || Dump the final experiments/reflections from the last worker as DIALS files — used for the two-stage scale-then-merge workflow.<br />
|-<br />
| <code>output.do_timing</code> || False || Log elapsed time per execution step.<br />
|-<br />
| <code>output.expanded_bookkeeping</code> || False || Store extra provenance columns (input file/reflection indices, odd/even flag, original id) in the saved reflection tables.<br />
|}<br />
<br />
=== Multiprocessing (<code>mp</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>mp.method</code> || mpi || Multiprocessing method (only <code>mpi</code> is currently supported).<br />
|-<br />
| <code>mp.debug.cProfile</code> || False || Enable code profiling; a <code>.prof</code> file is written per rank.<br />
|}<br />
<br />
== Output files ==<br />
<br />
The <code>merge</code> worker gathers all HKLs at rank 0 and writes three MTZ files, prefixed with<br />
<code>output.prefix</code> (default <code>iobs</code>):<br />
<br />
* <code>&lt;prefix&gt;_all.mtz</code> — the merged dataset using all observations (the main result for structure solution)<br />
* <code>&lt;prefix&gt;_odd.mtz</code> and <code>&lt;prefix&gt;_even.mtz</code> — the two half-datasets used for CC½ and related half-dataset statistics<br />
<br />
A main log records the per-step progress and the merging statistics tables (completeness,<br />
multiplicity, ⟨I/σ⟩, CC½, CC*, R-factors) by resolution bin. The merged MTZ can optionally be<br />
uploaded to a shared [https://github.com/cctbx/cctbx_project/tree/master/xfel/merging/application/publish Google Drive folder] by adding the <code>publish</code> worker.<br />
<br />
== References ==<br />
<br />
* Sauter NK (2015). ''J Synchrotron Radiat'' '''22''', 239–48. (post-refinement)<br />
* Uervirojnangkoorn M, ''et al.'' (2015). ''eLife'' 10.7554/eLife.05421. (PRIME / bootstrap)<br />
* Brewster AS, ''et al.'' (2018). ''Acta Cryst.'' D '''74''', 877–894.<br />
* Brewster AS, ''et al.'' (2019a). ''Computational Crystallography Newsletter'' '''10''', 22–39. (XFEL GUI)<br />
* Brewster AS, ''et al.'' (2019b). ''Acta Cryst.'' D '''75''', 959–968. (error models)<br />
* Mittan-Moreau (2024). (mm24 error model)<br />
<br />
== See also ==<br />
<br />
* [[dials.stills_process]] — produces the integrated data that <code>cctbx.xfel.merge</code> consumes<br />
* [[cctbx.xfel GUI]] — orchestrates separate scaling and repeated merging as data are collected<br />
* The program README at <code>xfel/merging/README.md</code> in cctbx_project</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Cctbx.xfel.merge&diff=1771Cctbx.xfel.merge2026-08-21T23:00:04Z<p>Aaron: /* Overview */</p>
<hr />
<div>= ''cctbx.xfel.merge'' =<br />
<br />
'''''cctbx.xfel.merge''''' is the scaling, post-refinement and merging program of cctbx.xfel. It takes<br />
the per-image integrated results produced by ''[[dials.stills_process]]'' — one crystal per still shot —<br />
and reduces the whole ensemble to a single set of merged structure-factor intensities (an MTZ file)<br />
suitable for structure solution and refinement, together with the crystallographic quality<br />
statistics (completeness, multiplicity, CC½, R-factors, etc.).<br />
<br />
Because serial datasets routinely contain tens of thousands to millions of still shots,<br />
<code>cctbx.xfel.merge</code> is built from the ground up to run under MPI across many nodes. It is<br />
almost always launched with <code>mpirun</code>/<code>srun</code>.<br />
<br />
== Overview ==<br />
<br />
<code>cctbx.xfel.merge</code> is organized as a sequence of '''workers'''. Each worker performs one<br />
task — reading data, applying a correction, filtering, scaling, post-refining, computing a statistic,<br />
or merging — and passes its (possibly modified) experiments and reflections on to the next worker.<br />
The list of workers to run is the "step list", and it is fully configurable: you can add, remove or<br />
reorder steps with <code>dispatch.step_list</code>. This is what makes the program flexible enough to<br />
scale-and-merge in one pass, to scale subsets separately and merge later, or to run a reference-free<br />
bootstrap.<br />
<br />
The '''default''' step list is:<br />
<br />
{| class="wikitable"<br />
! # !! Worker !! Purpose<br />
|-<br />
| 1 || <code>input</code> || Read integrated experiments (<code>.expt</code>) and reflections (<code>.refl</code>) from disk in DIALS format.<br />
|-<br />
| 2 || <code>balance</code> || Redistribute the input load evenly across MPI ranks (needed when there are fewer or more input files than ranks).<br />
|-<br />
| 3 || <code>model_scaling</code> || Build the full Miller list, model intensities from the reference, and the resolution binner used for scaling and post-refinement.<br />
|-<br />
| 4 || <code>modify</code> || Apply per-reflection modifications, e.g. polarization correction (and optionally re-indexing to resolve indexing ambiguity — see <code>modify.cosym</code>/<code>reindex_to_reference</code>).<br />
|-<br />
| 5 || <code>filter</code> || Reject whole experiments (or reflections) by resolution, unit cell, correlation to the reference, energy, etc.<br />
|-<br />
| 6 || <code>scale</code> || Scale each lattice to the reference dataset (<code>mark0</code>) or simply average (<code>mark1</code>).<br />
|-<br />
| 7 || <code>postrefine</code> || Post-refine per-image crystal parameters and partialities (Sauter 2015).<br />
|-<br />
| 8 || <code>statistics_unitcell</code> || Determine and store the average unit cell.<br />
|-<br />
| 9 || <code>statistics_beam</code> || Determine and store the average wavelength.<br />
|-<br />
| 10 || <code>model_statistics</code> || Rebuild the Miller list / binner for statistics (may use the average unit cell).<br />
|-<br />
| 11 || <code>statistics_resolution</code> || Per-crystal resolution statistics.<br />
|-<br />
| 12 || <code>group</code> || Redistribute reflections across ranks so that every observation of a given HKL is gathered on the same rank prior to merging.<br />
|-<br />
| 13 || <code>errors_merge</code> || Correct the merged error estimates with a per-HKL error model (e.g. Ev11, Brewster 2019b).<br />
|-<br />
| 14 || <code>statistics_intensity</code> || Intensity statistics by resolution.<br />
|-<br />
| 15 || <code>merge</code> || Merge symmetry-equivalent intensities, gather all HKLs at rank 0, and write "odd", "even" and "all" MTZ files.<br />
|-<br />
| 16 || <code>statistics_intensity_cxi</code> || CC½ / CC* and related statistics (ported from cxi.merge).<br />
|}<br />
<br />
Additional, non-default workers are available in <code>xfel/merging/application/</code> and can be<br />
inserted into the step list, including <code>tdata</code> (write out unit cells for clustering),<br />
<code>export</code> (batch MTZ for ''Careless''), <code>publish</code> (upload merged files to a<br />
Google Drive folder), <code>monitor</code> (resource-usage logging), <code>integrate</code><br />
(on-the-fly integration from psana2/XTC2), <code>prepare</code> (SpReAD energy binning), and<br />
<code>lunus</code> (diffuse-scattering deck). A <code>deltaccint</code> worker (ΔCC½ lattice<br />
filtering) can be added after <code>group</code>. Custom workers can be supplied through the<br />
<code>XFEL_CUSTOM_WORKER_PATH</code> environment variable.<br />
<br />
== Basic usage ==<br />
<br />
<code>cctbx.xfel.merge</code> is run under MPI:<br />
<br />
<pre><br />
mpirun cctbx.xfel.merge merge.phil<br />
</pre><br />
<br />
A typical <code>merge.phil</code> (from the program README):<br />
<br />
<pre><br />
input.path=<path to data><br />
input.experiments_suffix=.expt<br />
input.reflections_suffix=.refl<br />
<br />
filter.algorithm=unit_cell<br />
filter.unit_cell.value.relative_length_tolerance=0.03<br />
filter.outlier.min_corr=0.1<br />
<br />
select.algorithm=significance_filter<br />
select.significance_filter.sigma=0.1<br />
select.significance_filter.min_ct=200<br />
select.significance_filter.max_ct=300<br />
<br />
scaling.model=<model.pdb or model.mtz><br />
scaling.resolution_scalar=0.96<br />
<br />
merging.d_min=1.8<br />
merging.error.model=ev11<br />
merging.merge_anomalous=True<br />
<br />
postrefinement.enable=True<br />
statistics.n_bins=20<br />
<br />
output.do_timing=True<br />
output.output_dir=<output folder><br />
output.prefix=<prefix for output files><br />
</pre><br />
<br />
Explore the parameters with:<br />
* <code>cctbx.xfel.merge -c</code> — show configurable parameters<br />
* <code>cctbx.xfel.merge -c -e 10 -a 2</code> — show all parameters (expert level 10) with full help text<br />
<br />
== Two-stage scaling and merging ==<br />
<br />
For large, growing datasets (the mode used by the [[cctbx.xfel GUI]]), it is more efficient to scale<br />
and post-refine each subset of data '''once''', then merge repeatedly as more data arrive. This is<br />
done by running the program twice with different step lists.<br />
<br />
'''Stage 1 — scale each subset''' (stops after per-image work and writes intermediate DIALS files):<br />
<br />
<pre><br />
dispatch.step_list=input balance model_scaling modify filter scale postrefine \<br />
statistics_unitcell statistics_beam model_statistics statistics_resolution<br />
input.parallel_file_load.method=uniform<br />
output.save_experiments_and_reflections=True<br />
</pre><br />
<br />
<code>output.save_experiments_and_reflections=True</code> dumps the scaled/post-refined data as<br />
experiment lists and reflection tables that the merge stage can read back in.<br />
<br />
'''Stage 2 — merge the scaled subsets''' (add an <code>input.path</code> for each finished subset):<br />
<br />
<pre><br />
dispatch.step_list=input model_scaling statistics_unitcell statistics_beam \<br />
model_statistics statistics_resolution group errors_merge statistics_intensity \<br />
merge statistics_intensity_cxi<br />
input.parallel_file_load.method=uniform<br />
</pre><br />
<br />
== Merging without a reference (bootstrap / mark1) ==<br />
<br />
If no scaling/merging reference is available, use a bootstrap procedure. First run the<br />
<code>mark1</code> algorithm, which simply averages the data (all per-image scale factors = 1, no<br />
post-refinement), specifying the target cell and symmetry explicitly:<br />
<br />
<pre><br />
scaling.algorithm=mark1<br />
scaling.unit_cell=<unit cell><br />
scaling.space_group=<space group><br />
merging.error.model=errors_from_sample_residuals<br />
</pre><br />
<br />
The resulting MTZ is then used as the reference (<code>scaling.model=</code>) for subsequent regular<br />
merges. Iterate a few cycles; CC½ typically improves and converges over the first few rounds.<br />
<br />
== Important parameters ==<br />
<br />
Parameters are given by their full PHIL path with defaults in parentheses. The program exposes many<br />
more parameters (see <code>-c -e 10 -a 2</code>); those below are the ones most commonly set.<br />
<br />
=== Input ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>input.path</code> || None || Path(s) to the integrated data — a folder, several folders, or individual files; wildcards allowed; may be repeated. Matching <code>.expt</code> and <code>.refl</code> files must both be present.<br />
|-<br />
| <code>input.experiments_suffix</code> || <code>_integrated.expt</code> || Only load experiment files ending with this suffix.<br />
|-<br />
| <code>input.reflections_suffix</code> || <code>_integrated.refl</code> || Only load reflection files ending with this suffix.<br />
|-<br />
| <code>input.parallel_file_load.method</code> || uniform || <code>uniform</code> spreads input files evenly over all ranks; <code>node_memory</code> distributes so per-node memory limits are respected.<br />
|-<br />
| <code>input.alist.file</code> || None || Path to a text file of experiment tags or filenames; combined with <code>input.alist.op=keep|reject</code> to merge only (or exclude) a specified subset.<br />
|-<br />
| <code>input.persistent_refl_cols</code> || None || Reflection-table columns to preserve through the pruning steps (and into saved tables).<br />
|}<br />
<br />
=== Filtering whole experiments (<code>filter</code>) ===<br />
<br />
The <code>filter</code> section accepts or rejects '''entire experiments'''.<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>filter.algorithm</code> || (none) || One or more of <code>n_obs</code>, <code>resolution</code>, <code>unit_cell</code>, <code>energy</code>, <code>bootstrap</code>, <code>experiment_prefix</code>. Nothing is filtered unless listed.<br />
|-<br />
| <code>filter.unit_cell.algorithm</code> || value || How to restrict unit cells: <code>value</code> (compare to a target), <code>range</code>, or <code>cluster</code> (use a precomputed covariance clustering).<br />
|-<br />
| <code>filter.unit_cell.value.relative_length_tolerance</code> || 0.1 || Fractional deviation of cell edge lengths allowed vs. the target cell. Often tightened (e.g. 0.03).<br />
|-<br />
| <code>filter.unit_cell.value.absolute_angle_tolerance</code> || 2.0 || Allowed cell-angle deviation (degrees) vs. the target.<br />
|-<br />
| <code>filter.unit_cell.value.target_unit_cell</code> / <code>target_space_group</code> || Auto || Target cell/symmetry; <code>Auto</code> uses the scaling model.<br />
|-<br />
| <code>filter.outlier.min_corr</code> || 0.1 || Minimum correlation of an image's intensities to the reference model; images below this are rejected. Only applied when a scaling model is present. Use <code>-1</code> to accept all images regardless of correlation.<br />
|-<br />
| <code>filter.resolution.d_min</code> || None || Reject an experiment unless some of its reflections extend beyond this resolution.<br />
|}<br />
<br />
=== Selecting individual reflections (<code>select</code>) ===<br />
<br />
The <code>select</code> section accepts or rejects '''individual reflections''' (contrast with<br />
<code>filter</code>, which acts on whole experiments).<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>select.algorithm</code> || (none) || One or more of <code>significance_filter</code>, <code>panel</code>, <code>cspad_sensor</code>, <code>isolation_forest</code>, <code>target_energy</code>.<br />
|-<br />
| <code>select.significance_filter.sigma</code> || 0.5 || Trim each lattice at the resolution where the binned ⟨I/σ⟩ drops below this value.<br />
|-<br />
| <code>select.significance_filter.min_ct</code> / <code>max_ct</code> || 10 / 50 || Target minimum/maximum mean reflections per resolution bin when choosing the number of bins. For large unit cells these are often raised (e.g. 200/300); for small cells, use the defaults.<br />
|}<br />
<br />
=== Modifying intensities (<code>modify</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>modify.algorithm</code> || polarization || Per-reflection corrections to apply (default applies the polarization correction).<br />
|-<br />
| <code>modify.cosym</code> || — || Resolve an indexing ambiguity across the ensemble using the Gildea/Winter (Brehm/Diederichs) cosym method, optionally anchored to <code>scaling.model</code>.<br />
|-<br />
| <code>modify.reindex_to_reference</code> / <code>reindex_to_abc</code> || — || Alternative re-indexing operators (to a reference model, or an explicit operator).<br />
|}<br />
<br />
=== Scaling (<code>scaling</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>scaling.model</code> || None || Reference dataset: a PDB file (uses the CRYST1 cell and computes model intensities, including bulk solvent) or an MTZ from a previous cycle. Required for <code>mark0</code> scaling.<br />
|-<br />
| <code>scaling.algorithm</code> || mark0 || <code>mark0</code>: per-image scaling against the reference. <code>mark1</code>: no scaling, just Monte-Carlo averaging (all scale factors = 1) — used for reference-free bootstrap.<br />
|-<br />
| <code>scaling.unit_cell</code> / <code>scaling.space_group</code> || None || Cell/symmetry to use when no model is provided (i.e. <code>mark1</code>).<br />
|-<br />
| <code>scaling.resolution_scalar</code> || 0.969 || <code>merging.d_min</code> is multiplied by this when deciding which reflections fall within the resolution limit, generating a few extra reference reflections to account for per-shot cell variation (e.g. set 0.96 for a few more).<br />
|-<br />
| <code>scaling.mtz.mtz_column_F</code> || fobs || Column name in the reference MTZ containing the reference structure factors (intensities or amplitudes).<br />
|}<br />
<br />
=== Post-refinement (<code>postrefinement</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>postrefinement.enable</code> || True || Enable post-refinement of per-image parameters/partialities (Sauter 2015).<br />
|-<br />
| <code>postrefinement.algorithm</code> || rs || Post-refinement protocol: <code>rs</code>, <code>rs2</code>, <code>rs_hybrid</code>, or <code>eta_deff</code>.<br />
|}<br />
<br />
=== Merging and error model (<code>merging</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>merging.d_min</code> || None || High-resolution limit for scaling and merging.<br />
|-<br />
| <code>merging.d_max</code> || None || Low-resolution limit (currently affects only CCiso).<br />
|-<br />
| <code>merging.merge_anomalous</code> || False || <code>True</code> merges Bijvoet mates together (higher multiplicity); <code>False</code> keeps them separate to preserve the anomalous signal.<br />
|-<br />
| <code>merging.minimum_multiplicity</code> || 2 || Do not produce a merged structure factor for HKLs measured fewer than this many times.<br />
|-<br />
| <code>merging.error.model</code> || mm24 || Merged-sigma error model: <code>mm24</code> (Mittan-Moreau 2024, current default), <code>ev11</code> (Evans 2011 / sdfac-refine, recommended in the README), <code>ha14</code>, or <code>errors_from_sample_residuals</code> (no refined error terms — used with <code>mark1</code> bootstrap).<br />
|-<br />
| <code>merging.set_average_unit_cell</code> || True || Output MTZ adopts the (cluster-derived) average unit cell of the data rather than the reference model's cell.<br />
|}<br />
<br />
=== Statistics (<code>statistics</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>statistics.n_bins</code> || 10 || Number of resolution bins in the statistics tables.<br />
|-<br />
| <code>statistics.cciso.mtz_file</code> || None || Isomorphous reference structure factors for R<sub>iso</sub> / CC<sub>iso</sub>.<br />
|-<br />
| <code>statistics.report_ML</code> || True || Report per-frame max-likelihood attributes (domain size, mosaicity).<br />
|}<br />
<br />
=== Output (<code>output</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>output.output_dir</code> || <code>.</code> || Directory for output files (created if it does not exist).<br />
|-<br />
| <code>output.prefix</code> || iobs || Prefix prepended to all output file names.<br />
|-<br />
| <code>output.save_experiments_and_reflections</code> || False || Dump the final experiments/reflections from the last worker as DIALS files — used for the two-stage scale-then-merge workflow.<br />
|-<br />
| <code>output.do_timing</code> || False || Log elapsed time per execution step.<br />
|-<br />
| <code>output.expanded_bookkeeping</code> || False || Store extra provenance columns (input file/reflection indices, odd/even flag, original id) in the saved reflection tables.<br />
|}<br />
<br />
=== Multiprocessing (<code>mp</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>mp.method</code> || mpi || Multiprocessing method (only <code>mpi</code> is currently supported).<br />
|-<br />
| <code>mp.debug.cProfile</code> || False || Enable code profiling; a <code>.prof</code> file is written per rank.<br />
|}<br />
<br />
== Output files ==<br />
<br />
The <code>merge</code> worker gathers all HKLs at rank 0 and writes three MTZ files, prefixed with<br />
<code>output.prefix</code> (default <code>iobs</code>):<br />
<br />
* <code>&lt;prefix&gt;_all.mtz</code> — the merged dataset using all observations (the main result for structure solution)<br />
* <code>&lt;prefix&gt;_odd.mtz</code> and <code>&lt;prefix&gt;_even.mtz</code> — the two half-datasets used for CC½ and related half-dataset statistics<br />
<br />
A main log records the per-step progress and the merging statistics tables (completeness,<br />
multiplicity, ⟨I/σ⟩, CC½, CC*, R-factors) by resolution bin. The merged MTZ can optionally be<br />
uploaded to a shared [https://github.com/cctbx/cctbx_project/tree/master/xfel/merging/application/publish Google Drive folder] by adding the <code>publish</code> worker.<br />
<br />
== References ==<br />
<br />
* Sauter NK (2015). ''J Synchrotron Radiat'' '''22''', 239–48. (post-refinement)<br />
* Uervirojnangkoorn M, ''et al.'' (2015). ''eLife'' 10.7554/eLife.05421. (PRIME / bootstrap)<br />
* Brewster AS, ''et al.'' (2018). ''Acta Cryst.'' D '''74''', 877–894.<br />
* Brewster AS, ''et al.'' (2019a). ''Computational Crystallography Newsletter'' '''10''', 22–39. (XFEL GUI)<br />
* Brewster AS, ''et al.'' (2019b). ''Acta Cryst.'' D '''75''', 959–968. (error models)<br />
* Mittan-Moreau (2024). (mm24 error model)<br />
<br />
== See also ==<br />
<br />
* [[dials.stills_process]] — produces the integrated data that <code>cctbx.xfel.merge</code> consumes<br />
* [[cctbx.xfel GUI]] — orchestrates separate scaling and repeated merging as data are collected<br />
* The program README at <code>xfel/merging/README.md</code> in cctbx_project</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Cctbx.xfel.merge&diff=1770Cctbx.xfel.merge2026-08-21T22:58:36Z<p>Aaron: /* cctbx.xfel.merge */</p>
<hr />
<div>= ''cctbx.xfel.merge'' =<br />
<br />
'''''cctbx.xfel.merge''''' is the scaling, post-refinement and merging program of cctbx.xfel. It takes<br />
the per-image integrated results produced by ''[[dials.stills_process]]'' — one crystal per still shot —<br />
and reduces the whole ensemble to a single set of merged structure-factor intensities (an MTZ file)<br />
suitable for structure solution and refinement, together with the crystallographic quality<br />
statistics (completeness, multiplicity, CC½, R-factors, etc.).<br />
<br />
Because serial datasets routinely contain tens of thousands to millions of still shots,<br />
<code>cctbx.xfel.merge</code> is built from the ground up to run under MPI across many nodes. It is<br />
almost always launched with <code>mpirun</code>/<code>srun</code>.<br />
<br />
== Overview ==<br />
<br />
<code>cctbx.xfel.merge</code> is organized as a sequence of '''workers'''. Each worker performs one<br />
task — reading data, applying a correction, filtering, scaling, post-refining, computing a statistic,<br />
or merging — and passes its (possibly modified) experiments and reflections on to the next worker.<br />
The list of workers to run is the "step list", and it is fully configurable: you can add, remove or<br />
reorder steps with <code>dispatch.step_list</code>. This is what makes the program flexible enough to<br />
scale-and-merge in one pass, to scale subsets separately and merge later, or to run a reference-free<br />
bootstrap.<br />
<br />
The '''default''' step list is:<br />
<br />
{| class="wikitable"<br />
! # !! Worker !! Purpose<br />
|-<br />
| 1 || <code>input</code> || Read integrated experiments (<code>.expt</code>) and reflections (<code>.refl</code>) from disk in DIALS format.<br />
|-<br />
| 2 || <code>balance</code> || Redistribute the input load evenly across MPI ranks (needed when there are fewer input files than ranks).<br />
|-<br />
| 3 || <code>model_scaling</code> || Build the full Miller list, model intensities from the reference, and the resolution binner used for scaling and post-refinement.<br />
|-<br />
| 4 || <code>modify</code> || Apply per-reflection modifications, e.g. polarization correction (and optionally re-indexing to resolve indexing ambiguity — see <code>modify.cosym</code>/<code>reindex_to_reference</code>).<br />
|-<br />
| 5 || <code>filter</code> || Reject whole experiments (or reflections) by resolution, unit cell, correlation to the reference, energy, etc.<br />
|-<br />
| 6 || <code>scale</code> || Scale each lattice to the reference dataset (<code>mark0</code>) or simply average (<code>mark1</code>).<br />
|-<br />
| 7 || <code>postrefine</code> || Post-refine per-image crystal parameters and partialities (Sauter 2015).<br />
|-<br />
| 8 || <code>statistics_unitcell</code> || Determine and store the average unit cell.<br />
|-<br />
| 9 || <code>statistics_beam</code> || Determine and store the average wavelength.<br />
|-<br />
| 10 || <code>model_statistics</code> || Rebuild the Miller list / binner for statistics (may use the average unit cell).<br />
|-<br />
| 11 || <code>statistics_resolution</code> || Per-crystal resolution statistics.<br />
|-<br />
| 12 || <code>group</code> || Redistribute reflections across ranks so that every observation of a given HKL is gathered on the same rank prior to merging.<br />
|-<br />
| 13 || <code>errors_merge</code> || Correct the merged error estimates with a per-HKL error model (e.g. Ev11, Brewster 2019b).<br />
|-<br />
| 14 || <code>statistics_intensity</code> || Intensity statistics by resolution.<br />
|-<br />
| 15 || <code>merge</code> || Merge symmetry-equivalent intensities, gather all HKLs at rank 0, and write "odd", "even" and "all" MTZ files.<br />
|-<br />
| 16 || <code>statistics_intensity_cxi</code> || CC½ / CC* and related statistics (ported from cxi.merge).<br />
|}<br />
<br />
Additional, non-default workers are available in <code>xfel/merging/application/</code> and can be<br />
inserted into the step list, including <code>tdata</code> (write out unit cells for clustering),<br />
<code>export</code> (batch MTZ for ''Careless''), <code>publish</code> (upload merged files to a<br />
Google Drive folder), <code>monitor</code> (resource-usage logging), <code>integrate</code><br />
(on-the-fly integration from psana2/XTC2), <code>prepare</code> (SpReAD energy binning), and<br />
<code>lunus</code> (diffuse-scattering deck). A <code>deltaccint</code> worker (ΔCC½ lattice<br />
filtering) can be added after <code>group</code>. Custom workers can be supplied through the<br />
<code>XFEL_CUSTOM_WORKER_PATH</code> environment variable.<br />
<br />
== Basic usage ==<br />
<br />
<code>cctbx.xfel.merge</code> is run under MPI:<br />
<br />
<pre><br />
mpirun cctbx.xfel.merge merge.phil<br />
</pre><br />
<br />
A typical <code>merge.phil</code> (from the program README):<br />
<br />
<pre><br />
input.path=<path to data><br />
input.experiments_suffix=.expt<br />
input.reflections_suffix=.refl<br />
<br />
filter.algorithm=unit_cell<br />
filter.unit_cell.value.relative_length_tolerance=0.03<br />
filter.outlier.min_corr=0.1<br />
<br />
select.algorithm=significance_filter<br />
select.significance_filter.sigma=0.1<br />
select.significance_filter.min_ct=200<br />
select.significance_filter.max_ct=300<br />
<br />
scaling.model=<model.pdb or model.mtz><br />
scaling.resolution_scalar=0.96<br />
<br />
merging.d_min=1.8<br />
merging.error.model=ev11<br />
merging.merge_anomalous=True<br />
<br />
postrefinement.enable=True<br />
statistics.n_bins=20<br />
<br />
output.do_timing=True<br />
output.output_dir=<output folder><br />
output.prefix=<prefix for output files><br />
</pre><br />
<br />
Explore the parameters with:<br />
* <code>cctbx.xfel.merge -c</code> — show configurable parameters<br />
* <code>cctbx.xfel.merge -c -e 10 -a 2</code> — show all parameters (expert level 10) with full help text<br />
<br />
== Two-stage scaling and merging ==<br />
<br />
For large, growing datasets (the mode used by the [[cctbx.xfel GUI]]), it is more efficient to scale<br />
and post-refine each subset of data '''once''', then merge repeatedly as more data arrive. This is<br />
done by running the program twice with different step lists.<br />
<br />
'''Stage 1 — scale each subset''' (stops after per-image work and writes intermediate DIALS files):<br />
<br />
<pre><br />
dispatch.step_list=input balance model_scaling modify filter scale postrefine \<br />
statistics_unitcell statistics_beam model_statistics statistics_resolution<br />
input.parallel_file_load.method=uniform<br />
output.save_experiments_and_reflections=True<br />
</pre><br />
<br />
<code>output.save_experiments_and_reflections=True</code> dumps the scaled/post-refined data as<br />
experiment lists and reflection tables that the merge stage can read back in.<br />
<br />
'''Stage 2 — merge the scaled subsets''' (add an <code>input.path</code> for each finished subset):<br />
<br />
<pre><br />
dispatch.step_list=input model_scaling statistics_unitcell statistics_beam \<br />
model_statistics statistics_resolution group errors_merge statistics_intensity \<br />
merge statistics_intensity_cxi<br />
input.parallel_file_load.method=uniform<br />
</pre><br />
<br />
== Merging without a reference (bootstrap / mark1) ==<br />
<br />
If no scaling/merging reference is available, use a bootstrap procedure. First run the<br />
<code>mark1</code> algorithm, which simply averages the data (all per-image scale factors = 1, no<br />
post-refinement), specifying the target cell and symmetry explicitly:<br />
<br />
<pre><br />
scaling.algorithm=mark1<br />
scaling.unit_cell=<unit cell><br />
scaling.space_group=<space group><br />
merging.error.model=errors_from_sample_residuals<br />
</pre><br />
<br />
The resulting MTZ is then used as the reference (<code>scaling.model=</code>) for subsequent regular<br />
merges. Iterate a few cycles; CC½ typically improves and converges over the first few rounds.<br />
<br />
== Important parameters ==<br />
<br />
Parameters are given by their full PHIL path with defaults in parentheses. The program exposes many<br />
more parameters (see <code>-c -e 10 -a 2</code>); those below are the ones most commonly set.<br />
<br />
=== Input ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>input.path</code> || None || Path(s) to the integrated data — a folder, several folders, or individual files; wildcards allowed; may be repeated. Matching <code>.expt</code> and <code>.refl</code> files must both be present.<br />
|-<br />
| <code>input.experiments_suffix</code> || <code>_integrated.expt</code> || Only load experiment files ending with this suffix.<br />
|-<br />
| <code>input.reflections_suffix</code> || <code>_integrated.refl</code> || Only load reflection files ending with this suffix.<br />
|-<br />
| <code>input.parallel_file_load.method</code> || uniform || <code>uniform</code> spreads input files evenly over all ranks; <code>node_memory</code> distributes so per-node memory limits are respected.<br />
|-<br />
| <code>input.alist.file</code> || None || Path to a text file of experiment tags or filenames; combined with <code>input.alist.op=keep|reject</code> to merge only (or exclude) a specified subset.<br />
|-<br />
| <code>input.persistent_refl_cols</code> || None || Reflection-table columns to preserve through the pruning steps (and into saved tables).<br />
|}<br />
<br />
=== Filtering whole experiments (<code>filter</code>) ===<br />
<br />
The <code>filter</code> section accepts or rejects '''entire experiments'''.<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>filter.algorithm</code> || (none) || One or more of <code>n_obs</code>, <code>resolution</code>, <code>unit_cell</code>, <code>energy</code>, <code>bootstrap</code>, <code>experiment_prefix</code>. Nothing is filtered unless listed.<br />
|-<br />
| <code>filter.unit_cell.algorithm</code> || value || How to restrict unit cells: <code>value</code> (compare to a target), <code>range</code>, or <code>cluster</code> (use a precomputed covariance clustering).<br />
|-<br />
| <code>filter.unit_cell.value.relative_length_tolerance</code> || 0.1 || Fractional deviation of cell edge lengths allowed vs. the target cell. Often tightened (e.g. 0.03).<br />
|-<br />
| <code>filter.unit_cell.value.absolute_angle_tolerance</code> || 2.0 || Allowed cell-angle deviation (degrees) vs. the target.<br />
|-<br />
| <code>filter.unit_cell.value.target_unit_cell</code> / <code>target_space_group</code> || Auto || Target cell/symmetry; <code>Auto</code> uses the scaling model.<br />
|-<br />
| <code>filter.outlier.min_corr</code> || 0.1 || Minimum correlation of an image's intensities to the reference model; images below this are rejected. Only applied when a scaling model is present. Use <code>-1</code> to accept all images regardless of correlation.<br />
|-<br />
| <code>filter.resolution.d_min</code> || None || Reject an experiment unless some of its reflections extend beyond this resolution.<br />
|}<br />
<br />
=== Selecting individual reflections (<code>select</code>) ===<br />
<br />
The <code>select</code> section accepts or rejects '''individual reflections''' (contrast with<br />
<code>filter</code>, which acts on whole experiments).<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>select.algorithm</code> || (none) || One or more of <code>significance_filter</code>, <code>panel</code>, <code>cspad_sensor</code>, <code>isolation_forest</code>, <code>target_energy</code>.<br />
|-<br />
| <code>select.significance_filter.sigma</code> || 0.5 || Trim each lattice at the resolution where the binned ⟨I/σ⟩ drops below this value.<br />
|-<br />
| <code>select.significance_filter.min_ct</code> / <code>max_ct</code> || 10 / 50 || Target minimum/maximum mean reflections per resolution bin when choosing the number of bins. For large unit cells these are often raised (e.g. 200/300); for small cells, use the defaults.<br />
|}<br />
<br />
=== Modifying intensities (<code>modify</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>modify.algorithm</code> || polarization || Per-reflection corrections to apply (default applies the polarization correction).<br />
|-<br />
| <code>modify.cosym</code> || — || Resolve an indexing ambiguity across the ensemble using the Gildea/Winter (Brehm/Diederichs) cosym method, optionally anchored to <code>scaling.model</code>.<br />
|-<br />
| <code>modify.reindex_to_reference</code> / <code>reindex_to_abc</code> || — || Alternative re-indexing operators (to a reference model, or an explicit operator).<br />
|}<br />
<br />
=== Scaling (<code>scaling</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>scaling.model</code> || None || Reference dataset: a PDB file (uses the CRYST1 cell and computes model intensities, including bulk solvent) or an MTZ from a previous cycle. Required for <code>mark0</code> scaling.<br />
|-<br />
| <code>scaling.algorithm</code> || mark0 || <code>mark0</code>: per-image scaling against the reference. <code>mark1</code>: no scaling, just Monte-Carlo averaging (all scale factors = 1) — used for reference-free bootstrap.<br />
|-<br />
| <code>scaling.unit_cell</code> / <code>scaling.space_group</code> || None || Cell/symmetry to use when no model is provided (i.e. <code>mark1</code>).<br />
|-<br />
| <code>scaling.resolution_scalar</code> || 0.969 || <code>merging.d_min</code> is multiplied by this when deciding which reflections fall within the resolution limit, generating a few extra reference reflections to account for per-shot cell variation (e.g. set 0.96 for a few more).<br />
|-<br />
| <code>scaling.mtz.mtz_column_F</code> || fobs || Column name in the reference MTZ containing the reference structure factors (intensities or amplitudes).<br />
|}<br />
<br />
=== Post-refinement (<code>postrefinement</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>postrefinement.enable</code> || True || Enable post-refinement of per-image parameters/partialities (Sauter 2015).<br />
|-<br />
| <code>postrefinement.algorithm</code> || rs || Post-refinement protocol: <code>rs</code>, <code>rs2</code>, <code>rs_hybrid</code>, or <code>eta_deff</code>.<br />
|}<br />
<br />
=== Merging and error model (<code>merging</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>merging.d_min</code> || None || High-resolution limit for scaling and merging.<br />
|-<br />
| <code>merging.d_max</code> || None || Low-resolution limit (currently affects only CCiso).<br />
|-<br />
| <code>merging.merge_anomalous</code> || False || <code>True</code> merges Bijvoet mates together (higher multiplicity); <code>False</code> keeps them separate to preserve the anomalous signal.<br />
|-<br />
| <code>merging.minimum_multiplicity</code> || 2 || Do not produce a merged structure factor for HKLs measured fewer than this many times.<br />
|-<br />
| <code>merging.error.model</code> || mm24 || Merged-sigma error model: <code>mm24</code> (Mittan-Moreau 2024, current default), <code>ev11</code> (Evans 2011 / sdfac-refine, recommended in the README), <code>ha14</code>, or <code>errors_from_sample_residuals</code> (no refined error terms — used with <code>mark1</code> bootstrap).<br />
|-<br />
| <code>merging.set_average_unit_cell</code> || True || Output MTZ adopts the (cluster-derived) average unit cell of the data rather than the reference model's cell.<br />
|}<br />
<br />
=== Statistics (<code>statistics</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>statistics.n_bins</code> || 10 || Number of resolution bins in the statistics tables.<br />
|-<br />
| <code>statistics.cciso.mtz_file</code> || None || Isomorphous reference structure factors for R<sub>iso</sub> / CC<sub>iso</sub>.<br />
|-<br />
| <code>statistics.report_ML</code> || True || Report per-frame max-likelihood attributes (domain size, mosaicity).<br />
|}<br />
<br />
=== Output (<code>output</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>output.output_dir</code> || <code>.</code> || Directory for output files (created if it does not exist).<br />
|-<br />
| <code>output.prefix</code> || iobs || Prefix prepended to all output file names.<br />
|-<br />
| <code>output.save_experiments_and_reflections</code> || False || Dump the final experiments/reflections from the last worker as DIALS files — used for the two-stage scale-then-merge workflow.<br />
|-<br />
| <code>output.do_timing</code> || False || Log elapsed time per execution step.<br />
|-<br />
| <code>output.expanded_bookkeeping</code> || False || Store extra provenance columns (input file/reflection indices, odd/even flag, original id) in the saved reflection tables.<br />
|}<br />
<br />
=== Multiprocessing (<code>mp</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>mp.method</code> || mpi || Multiprocessing method (only <code>mpi</code> is currently supported).<br />
|-<br />
| <code>mp.debug.cProfile</code> || False || Enable code profiling; a <code>.prof</code> file is written per rank.<br />
|}<br />
<br />
== Output files ==<br />
<br />
The <code>merge</code> worker gathers all HKLs at rank 0 and writes three MTZ files, prefixed with<br />
<code>output.prefix</code> (default <code>iobs</code>):<br />
<br />
* <code>&lt;prefix&gt;_all.mtz</code> — the merged dataset using all observations (the main result for structure solution)<br />
* <code>&lt;prefix&gt;_odd.mtz</code> and <code>&lt;prefix&gt;_even.mtz</code> — the two half-datasets used for CC½ and related half-dataset statistics<br />
<br />
A main log records the per-step progress and the merging statistics tables (completeness,<br />
multiplicity, ⟨I/σ⟩, CC½, CC*, R-factors) by resolution bin. The merged MTZ can optionally be<br />
uploaded to a shared [https://github.com/cctbx/cctbx_project/tree/master/xfel/merging/application/publish Google Drive folder] by adding the <code>publish</code> worker.<br />
<br />
== References ==<br />
<br />
* Sauter NK (2015). ''J Synchrotron Radiat'' '''22''', 239–48. (post-refinement)<br />
* Uervirojnangkoorn M, ''et al.'' (2015). ''eLife'' 10.7554/eLife.05421. (PRIME / bootstrap)<br />
* Brewster AS, ''et al.'' (2018). ''Acta Cryst.'' D '''74''', 877–894.<br />
* Brewster AS, ''et al.'' (2019a). ''Computational Crystallography Newsletter'' '''10''', 22–39. (XFEL GUI)<br />
* Brewster AS, ''et al.'' (2019b). ''Acta Cryst.'' D '''75''', 959–968. (error models)<br />
* Mittan-Moreau (2024). (mm24 error model)<br />
<br />
== See also ==<br />
<br />
* [[dials.stills_process]] — produces the integrated data that <code>cctbx.xfel.merge</code> consumes<br />
* [[cctbx.xfel GUI]] — orchestrates separate scaling and repeated merging as data are collected<br />
* The program README at <code>xfel/merging/README.md</code> in cctbx_project</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Cctbx.xfel.merge&diff=1769Cctbx.xfel.merge2026-08-21T22:57:30Z<p>Aaron: Created page with "= ''cctbx.xfel.merge'' = '''''cctbx.xfel.merge''''' is the scaling, post-refinement and merging program of cctbx.xfel. It takes the per-image integrated results produced by dials.stills_process — one crystal per still shot — and reduces the whole ensemble to a single set of merged structure-factor intensities (an MTZ file) suitable for structure solution and refinement, together with the crystallographic quality statistics (completeness, multiplicity, CC½, CC*,..."</p>
<hr />
<div>= ''cctbx.xfel.merge'' =<br />
<br />
'''''cctbx.xfel.merge''''' is the scaling, post-refinement and merging program of cctbx.xfel. It takes<br />
the per-image integrated results produced by [[dials.stills_process]] — one crystal per still shot —<br />
and reduces the whole ensemble to a single set of merged structure-factor intensities (an MTZ file)<br />
suitable for structure solution and refinement, together with the crystallographic quality<br />
statistics (completeness, multiplicity, CC½, CC*, R-factors, etc.).<br />
<br />
Because serial datasets routinely contain tens of thousands to millions of still shots,<br />
<code>cctbx.xfel.merge</code> is built from the ground up to run under MPI across many nodes. It is<br />
almost always launched with <code>mpirun</code>/<code>srun</code>.<br />
<br />
== Overview ==<br />
<br />
<code>cctbx.xfel.merge</code> is organized as a sequence of '''workers'''. Each worker performs one<br />
task — reading data, applying a correction, filtering, scaling, post-refining, computing a statistic,<br />
or merging — and passes its (possibly modified) experiments and reflections on to the next worker.<br />
The list of workers to run is the "step list", and it is fully configurable: you can add, remove or<br />
reorder steps with <code>dispatch.step_list</code>. This is what makes the program flexible enough to<br />
scale-and-merge in one pass, to scale subsets separately and merge later, or to run a reference-free<br />
bootstrap.<br />
<br />
The '''default''' step list is:<br />
<br />
{| class="wikitable"<br />
! # !! Worker !! Purpose<br />
|-<br />
| 1 || <code>input</code> || Read integrated experiments (<code>.expt</code>) and reflections (<code>.refl</code>) from disk in DIALS format.<br />
|-<br />
| 2 || <code>balance</code> || Redistribute the input load evenly across MPI ranks (needed when there are fewer input files than ranks).<br />
|-<br />
| 3 || <code>model_scaling</code> || Build the full Miller list, model intensities from the reference, and the resolution binner used for scaling and post-refinement.<br />
|-<br />
| 4 || <code>modify</code> || Apply per-reflection modifications, e.g. polarization correction (and optionally re-indexing to resolve indexing ambiguity — see <code>modify.cosym</code>/<code>reindex_to_reference</code>).<br />
|-<br />
| 5 || <code>filter</code> || Reject whole experiments (or reflections) by resolution, unit cell, correlation to the reference, energy, etc.<br />
|-<br />
| 6 || <code>scale</code> || Scale each lattice to the reference dataset (<code>mark0</code>) or simply average (<code>mark1</code>).<br />
|-<br />
| 7 || <code>postrefine</code> || Post-refine per-image crystal parameters and partialities (Sauter 2015).<br />
|-<br />
| 8 || <code>statistics_unitcell</code> || Determine and store the average unit cell.<br />
|-<br />
| 9 || <code>statistics_beam</code> || Determine and store the average wavelength.<br />
|-<br />
| 10 || <code>model_statistics</code> || Rebuild the Miller list / binner for statistics (may use the average unit cell).<br />
|-<br />
| 11 || <code>statistics_resolution</code> || Per-crystal resolution statistics.<br />
|-<br />
| 12 || <code>group</code> || Redistribute reflections across ranks so that every observation of a given HKL is gathered on the same rank prior to merging.<br />
|-<br />
| 13 || <code>errors_merge</code> || Correct the merged error estimates with a per-HKL error model (e.g. Ev11, Brewster 2019b).<br />
|-<br />
| 14 || <code>statistics_intensity</code> || Intensity statistics by resolution.<br />
|-<br />
| 15 || <code>merge</code> || Merge symmetry-equivalent intensities, gather all HKLs at rank 0, and write "odd", "even" and "all" MTZ files.<br />
|-<br />
| 16 || <code>statistics_intensity_cxi</code> || CC½ / CC* and related statistics (ported from cxi.merge).<br />
|}<br />
<br />
Additional, non-default workers are available in <code>xfel/merging/application/</code> and can be<br />
inserted into the step list, including <code>tdata</code> (write out unit cells for clustering),<br />
<code>export</code> (batch MTZ for ''Careless''), <code>publish</code> (upload merged files to a<br />
Google Drive folder), <code>monitor</code> (resource-usage logging), <code>integrate</code><br />
(on-the-fly integration from psana2/XTC2), <code>prepare</code> (SpReAD energy binning), and<br />
<code>lunus</code> (diffuse-scattering deck). A <code>deltaccint</code> worker (ΔCC½ lattice<br />
filtering) can be added after <code>group</code>. Custom workers can be supplied through the<br />
<code>XFEL_CUSTOM_WORKER_PATH</code> environment variable.<br />
<br />
== Basic usage ==<br />
<br />
<code>cctbx.xfel.merge</code> is run under MPI:<br />
<br />
<pre><br />
mpirun cctbx.xfel.merge merge.phil<br />
</pre><br />
<br />
A typical <code>merge.phil</code> (from the program README):<br />
<br />
<pre><br />
input.path=<path to data><br />
input.experiments_suffix=.expt<br />
input.reflections_suffix=.refl<br />
<br />
filter.algorithm=unit_cell<br />
filter.unit_cell.value.relative_length_tolerance=0.03<br />
filter.outlier.min_corr=0.1<br />
<br />
select.algorithm=significance_filter<br />
select.significance_filter.sigma=0.1<br />
select.significance_filter.min_ct=200<br />
select.significance_filter.max_ct=300<br />
<br />
scaling.model=<model.pdb or model.mtz><br />
scaling.resolution_scalar=0.96<br />
<br />
merging.d_min=1.8<br />
merging.error.model=ev11<br />
merging.merge_anomalous=True<br />
<br />
postrefinement.enable=True<br />
statistics.n_bins=20<br />
<br />
output.do_timing=True<br />
output.output_dir=<output folder><br />
output.prefix=<prefix for output files><br />
</pre><br />
<br />
Explore the parameters with:<br />
* <code>cctbx.xfel.merge -c</code> — show configurable parameters<br />
* <code>cctbx.xfel.merge -c -e 10 -a 2</code> — show all parameters (expert level 10) with full help text<br />
<br />
== Two-stage scaling and merging ==<br />
<br />
For large, growing datasets (the mode used by the [[cctbx.xfel GUI]]), it is more efficient to scale<br />
and post-refine each subset of data '''once''', then merge repeatedly as more data arrive. This is<br />
done by running the program twice with different step lists.<br />
<br />
'''Stage 1 — scale each subset''' (stops after per-image work and writes intermediate DIALS files):<br />
<br />
<pre><br />
dispatch.step_list=input balance model_scaling modify filter scale postrefine \<br />
statistics_unitcell statistics_beam model_statistics statistics_resolution<br />
input.parallel_file_load.method=uniform<br />
output.save_experiments_and_reflections=True<br />
</pre><br />
<br />
<code>output.save_experiments_and_reflections=True</code> dumps the scaled/post-refined data as<br />
experiment lists and reflection tables that the merge stage can read back in.<br />
<br />
'''Stage 2 — merge the scaled subsets''' (add an <code>input.path</code> for each finished subset):<br />
<br />
<pre><br />
dispatch.step_list=input model_scaling statistics_unitcell statistics_beam \<br />
model_statistics statistics_resolution group errors_merge statistics_intensity \<br />
merge statistics_intensity_cxi<br />
input.parallel_file_load.method=uniform<br />
</pre><br />
<br />
== Merging without a reference (bootstrap / mark1) ==<br />
<br />
If no scaling/merging reference is available, use a bootstrap procedure. First run the<br />
<code>mark1</code> algorithm, which simply averages the data (all per-image scale factors = 1, no<br />
post-refinement), specifying the target cell and symmetry explicitly:<br />
<br />
<pre><br />
scaling.algorithm=mark1<br />
scaling.unit_cell=<unit cell><br />
scaling.space_group=<space group><br />
merging.error.model=errors_from_sample_residuals<br />
</pre><br />
<br />
The resulting MTZ is then used as the reference (<code>scaling.model=</code>) for subsequent regular<br />
merges. Iterate a few cycles; CC½ typically improves and converges over the first few rounds.<br />
<br />
== Important parameters ==<br />
<br />
Parameters are given by their full PHIL path with defaults in parentheses. The program exposes many<br />
more parameters (see <code>-c -e 10 -a 2</code>); those below are the ones most commonly set.<br />
<br />
=== Input ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>input.path</code> || None || Path(s) to the integrated data — a folder, several folders, or individual files; wildcards allowed; may be repeated. Matching <code>.expt</code> and <code>.refl</code> files must both be present.<br />
|-<br />
| <code>input.experiments_suffix</code> || <code>_integrated.expt</code> || Only load experiment files ending with this suffix.<br />
|-<br />
| <code>input.reflections_suffix</code> || <code>_integrated.refl</code> || Only load reflection files ending with this suffix.<br />
|-<br />
| <code>input.parallel_file_load.method</code> || uniform || <code>uniform</code> spreads input files evenly over all ranks; <code>node_memory</code> distributes so per-node memory limits are respected.<br />
|-<br />
| <code>input.alist.file</code> || None || Path to a text file of experiment tags or filenames; combined with <code>input.alist.op=keep|reject</code> to merge only (or exclude) a specified subset.<br />
|-<br />
| <code>input.persistent_refl_cols</code> || None || Reflection-table columns to preserve through the pruning steps (and into saved tables).<br />
|}<br />
<br />
=== Filtering whole experiments (<code>filter</code>) ===<br />
<br />
The <code>filter</code> section accepts or rejects '''entire experiments'''.<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>filter.algorithm</code> || (none) || One or more of <code>n_obs</code>, <code>resolution</code>, <code>unit_cell</code>, <code>energy</code>, <code>bootstrap</code>, <code>experiment_prefix</code>. Nothing is filtered unless listed.<br />
|-<br />
| <code>filter.unit_cell.algorithm</code> || value || How to restrict unit cells: <code>value</code> (compare to a target), <code>range</code>, or <code>cluster</code> (use a precomputed covariance clustering).<br />
|-<br />
| <code>filter.unit_cell.value.relative_length_tolerance</code> || 0.1 || Fractional deviation of cell edge lengths allowed vs. the target cell. Often tightened (e.g. 0.03).<br />
|-<br />
| <code>filter.unit_cell.value.absolute_angle_tolerance</code> || 2.0 || Allowed cell-angle deviation (degrees) vs. the target.<br />
|-<br />
| <code>filter.unit_cell.value.target_unit_cell</code> / <code>target_space_group</code> || Auto || Target cell/symmetry; <code>Auto</code> uses the scaling model.<br />
|-<br />
| <code>filter.outlier.min_corr</code> || 0.1 || Minimum correlation of an image's intensities to the reference model; images below this are rejected. Only applied when a scaling model is present. Use <code>-1</code> to accept all images regardless of correlation.<br />
|-<br />
| <code>filter.resolution.d_min</code> || None || Reject an experiment unless some of its reflections extend beyond this resolution.<br />
|}<br />
<br />
=== Selecting individual reflections (<code>select</code>) ===<br />
<br />
The <code>select</code> section accepts or rejects '''individual reflections''' (contrast with<br />
<code>filter</code>, which acts on whole experiments).<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>select.algorithm</code> || (none) || One or more of <code>significance_filter</code>, <code>panel</code>, <code>cspad_sensor</code>, <code>isolation_forest</code>, <code>target_energy</code>.<br />
|-<br />
| <code>select.significance_filter.sigma</code> || 0.5 || Trim each lattice at the resolution where the binned ⟨I/σ⟩ drops below this value.<br />
|-<br />
| <code>select.significance_filter.min_ct</code> / <code>max_ct</code> || 10 / 50 || Target minimum/maximum mean reflections per resolution bin when choosing the number of bins. For large unit cells these are often raised (e.g. 200/300); for small cells, use the defaults.<br />
|}<br />
<br />
=== Modifying intensities (<code>modify</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>modify.algorithm</code> || polarization || Per-reflection corrections to apply (default applies the polarization correction).<br />
|-<br />
| <code>modify.cosym</code> || — || Resolve an indexing ambiguity across the ensemble using the Gildea/Winter (Brehm/Diederichs) cosym method, optionally anchored to <code>scaling.model</code>.<br />
|-<br />
| <code>modify.reindex_to_reference</code> / <code>reindex_to_abc</code> || — || Alternative re-indexing operators (to a reference model, or an explicit operator).<br />
|}<br />
<br />
=== Scaling (<code>scaling</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>scaling.model</code> || None || Reference dataset: a PDB file (uses the CRYST1 cell and computes model intensities, including bulk solvent) or an MTZ from a previous cycle. Required for <code>mark0</code> scaling.<br />
|-<br />
| <code>scaling.algorithm</code> || mark0 || <code>mark0</code>: per-image scaling against the reference. <code>mark1</code>: no scaling, just Monte-Carlo averaging (all scale factors = 1) — used for reference-free bootstrap.<br />
|-<br />
| <code>scaling.unit_cell</code> / <code>scaling.space_group</code> || None || Cell/symmetry to use when no model is provided (i.e. <code>mark1</code>).<br />
|-<br />
| <code>scaling.resolution_scalar</code> || 0.969 || <code>merging.d_min</code> is multiplied by this when deciding which reflections fall within the resolution limit, generating a few extra reference reflections to account for per-shot cell variation (e.g. set 0.96 for a few more).<br />
|-<br />
| <code>scaling.mtz.mtz_column_F</code> || fobs || Column name in the reference MTZ containing the reference structure factors (intensities or amplitudes).<br />
|}<br />
<br />
=== Post-refinement (<code>postrefinement</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>postrefinement.enable</code> || True || Enable post-refinement of per-image parameters/partialities (Sauter 2015).<br />
|-<br />
| <code>postrefinement.algorithm</code> || rs || Post-refinement protocol: <code>rs</code>, <code>rs2</code>, <code>rs_hybrid</code>, or <code>eta_deff</code>.<br />
|}<br />
<br />
=== Merging and error model (<code>merging</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>merging.d_min</code> || None || High-resolution limit for scaling and merging.<br />
|-<br />
| <code>merging.d_max</code> || None || Low-resolution limit (currently affects only CCiso).<br />
|-<br />
| <code>merging.merge_anomalous</code> || False || <code>True</code> merges Bijvoet mates together (higher multiplicity); <code>False</code> keeps them separate to preserve the anomalous signal.<br />
|-<br />
| <code>merging.minimum_multiplicity</code> || 2 || Do not produce a merged structure factor for HKLs measured fewer than this many times.<br />
|-<br />
| <code>merging.error.model</code> || mm24 || Merged-sigma error model: <code>mm24</code> (Mittan-Moreau 2024, current default), <code>ev11</code> (Evans 2011 / sdfac-refine, recommended in the README), <code>ha14</code>, or <code>errors_from_sample_residuals</code> (no refined error terms — used with <code>mark1</code> bootstrap).<br />
|-<br />
| <code>merging.set_average_unit_cell</code> || True || Output MTZ adopts the (cluster-derived) average unit cell of the data rather than the reference model's cell.<br />
|}<br />
<br />
=== Statistics (<code>statistics</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>statistics.n_bins</code> || 10 || Number of resolution bins in the statistics tables.<br />
|-<br />
| <code>statistics.cciso.mtz_file</code> || None || Isomorphous reference structure factors for R<sub>iso</sub> / CC<sub>iso</sub>.<br />
|-<br />
| <code>statistics.report_ML</code> || True || Report per-frame max-likelihood attributes (domain size, mosaicity).<br />
|}<br />
<br />
=== Output (<code>output</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>output.output_dir</code> || <code>.</code> || Directory for output files (created if it does not exist).<br />
|-<br />
| <code>output.prefix</code> || iobs || Prefix prepended to all output file names.<br />
|-<br />
| <code>output.save_experiments_and_reflections</code> || False || Dump the final experiments/reflections from the last worker as DIALS files — used for the two-stage scale-then-merge workflow.<br />
|-<br />
| <code>output.do_timing</code> || False || Log elapsed time per execution step.<br />
|-<br />
| <code>output.expanded_bookkeeping</code> || False || Store extra provenance columns (input file/reflection indices, odd/even flag, original id) in the saved reflection tables.<br />
|}<br />
<br />
=== Multiprocessing (<code>mp</code>) ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>mp.method</code> || mpi || Multiprocessing method (only <code>mpi</code> is currently supported).<br />
|-<br />
| <code>mp.debug.cProfile</code> || False || Enable code profiling; a <code>.prof</code> file is written per rank.<br />
|}<br />
<br />
== Output files ==<br />
<br />
The <code>merge</code> worker gathers all HKLs at rank 0 and writes three MTZ files, prefixed with<br />
<code>output.prefix</code> (default <code>iobs</code>):<br />
<br />
* <code>&lt;prefix&gt;_all.mtz</code> — the merged dataset using all observations (the main result for structure solution)<br />
* <code>&lt;prefix&gt;_odd.mtz</code> and <code>&lt;prefix&gt;_even.mtz</code> — the two half-datasets used for CC½ and related half-dataset statistics<br />
<br />
A main log records the per-step progress and the merging statistics tables (completeness,<br />
multiplicity, ⟨I/σ⟩, CC½, CC*, R-factors) by resolution bin. The merged MTZ can optionally be<br />
uploaded to a shared [https://github.com/cctbx/cctbx_project/tree/master/xfel/merging/application/publish Google Drive folder] by adding the <code>publish</code> worker.<br />
<br />
== References ==<br />
<br />
* Sauter NK (2015). ''J Synchrotron Radiat'' '''22''', 239–48. (post-refinement)<br />
* Uervirojnangkoorn M, ''et al.'' (2015). ''eLife'' 10.7554/eLife.05421. (PRIME / bootstrap)<br />
* Brewster AS, ''et al.'' (2018). ''Acta Cryst.'' D '''74''', 877–894.<br />
* Brewster AS, ''et al.'' (2019a). ''Computational Crystallography Newsletter'' '''10''', 22–39. (XFEL GUI)<br />
* Brewster AS, ''et al.'' (2019b). ''Acta Cryst.'' D '''75''', 959–968. (error models)<br />
* Mittan-Moreau (2024). (mm24 error model)<br />
<br />
== See also ==<br />
<br />
* [[dials.stills_process]] — produces the integrated data that <code>cctbx.xfel.merge</code> consumes<br />
* [[cctbx.xfel GUI]] — orchestrates separate scaling and repeated merging as data are collected<br />
* The program README at <code>xfel/merging/README.md</code> in cctbx_project</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Dials.stills_process&diff=1768Dials.stills process2026-08-21T22:55:23Z<p>Aaron: </p>
<hr />
<div>= dials.stills_process =<br />
<br />
'''''dials.stills_process''''' is a stand-alone program for processing still diffraction images and is<br />
the core data-reduction program of ''cctbx.xfel''. It takes a set of<br />
still diffraction images — typically serial femtosecond crystallography (SFX) snapshots from an<br />
XFEL (LCLS, SACLA, EuXFEL) or still shots from a synchrotron — and analyzes each one, independently,<br />
through the full DIALS pipeline: '''import → spot-finding → indexing → refinement → integration'''.<br />
Its output (integrated intensities together with the per-image experimental models) is the input to<br />
the merging program ''[[cctbx.xfel.merge]]''.<br />
<br />
Because every image is a separate still exposure of a different crystal (rather than a wedge of<br />
a rotation series), each image is treated as its own experiment with its own crystal model. This<br />
"one crystal per shot" model is what distinguishes stills processing from conventional rotation data<br />
processing (e.g. <code>dials.import</code>, <code>dials.find_spots</code>, <code>dials.index</code>, etc.), <br />
and it drives most of the stills-specific defaults described below.<br />
<br />
== Overview ==<br />
<br />
For each image, <code>dials.stills_process</code> runs the following stages in order. Any stage can<br />
be turned off with the <code>dispatch</code> parameters; if a stage is disabled, processing stops<br />
after the last enabled stage.<br />
<br />
# '''Import''' — read the image, apply reference geometry / gain / mask overrides, and build an experiment.<br />
# '''Spot-finding''' — locate strong pixels and build strong reflections (<code>dispatch.find_spots</code>).<br />
# '''Hit finding''' — reject images with too few (or optionally too many) strong spots, so that only "hits" are indexed (<code>dispatch.hit_finder</code>).<br />
# '''Indexing''' — determine the crystal orientation and (optionally constrained) unit cell (<code>dispatch.index</code>).<br />
# '''Refinement''' — refine the crystal model and re-estimate the mosaic/domain-size parameters. Detector and beam are held fixed by default (<code>dispatch.refine</code>).<br />
# '''Integration''' — predict reflections and measure their intensities using the stills integrator (<code>dispatch.integrate</code>), with an optional significance filter.<br />
<br />
The program applies a set of stills-specific defaults on top of the standard DIALS parameters,<br />
including: indexing method <code>fft1d</code>; the stills integrator with profile fitting turned off<br />
(summation integration with a 2-D plane background); the stills refinement weighting strategy with<br />
the detector and beam models fixed; and a re-estimation of per-crystal mosaic parameters<br />
(domain size and mosaic angle) after refinement. These are sensible starting points for serial data<br />
and normally do not need to be changed.<br />
<br />
Failures on individual images are, by default, caught and logged so that processing continues to the<br />
next image (<code>dispatch.squash_errors=True</code>) — essential when reducing tens of thousands of<br />
shots of which only a fraction are hits.<br />
<br />
== Basic usage ==<br />
<br />
<pre><br />
# Process a directory of CBF stills, constraining the target cell/symmetry,<br />
# using 32 parallel processes on one node<br />
dials.stills_process /path/to/images/*.cbf process.phil mp.nproc=32<br />
</pre><br />
<br />
A typical <code>process.phil</code> for XFEL data looks like:<br />
<br />
<pre><br />
spotfinder.filter.min_spot_size=2<br />
spotfinder.threshold.dispersion.gain=1.0<br />
spotfinder.threshold.dispersion.global_threshold=100<br />
<br />
indexing.known_symmetry.space_group=P43212<br />
indexing.known_symmetry.unit_cell=78.9,78.9,38.1,90,90,90<br />
indexing.method=fft1d<br />
<br />
input.reference_geometry=refined.expt # from a previous metrology refinement<br />
<br />
output.output_dir=out<br />
mp.nproc=32<br />
</pre><br />
<br />
Useful command-line helpers:<br />
* <code>dials.stills_process -h</code> — help and usage<br />
* <code>dials.stills_process -c -e 2</code> — show the configurable parameters (expert level 2)<br />
* <code>dials.stills_process -c -a2</code> — include attributes (help text) for each parameter<br />
* <code>dials.stills_process ... input.show_image_tags=True</code> — list the image tags in a dataset without processing (useful for selecting subsets with <code>input.image_tag</code>)<br />
<br />
== Important parameters ==<br />
<br />
The program exposes the entire DIALS parameter set (spot-finding, indexing, refinement, integration,<br />
profile modelling, prediction). The parameters below are the ones most commonly adjusted for serial<br />
still data. Parameters are given by their full PHIL path with their default value.<br />
<br />
=== Input ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>input.reference_geometry</code> || None || Path to an <code>.expt</code> file (or image) with a single detector model. Data are processed using ''this'' geometry instead of the geometry in the image headers. This is how a refined detector metrology from <code>cctbx.xfel</code> ensemble refinement is applied — almost always set for XFEL data.<br />
|-<br />
| <code>input.sync_reference_geom</code> || True || When a reference geometry is supplied, copy its refined local frames onto the image's detector hierarchy node-by-node (the hierarchies must match in shape). If False, the reference detector replaces the image detector wholesale.<br />
|-<br />
| <code>input.file_list</code> || None || Path to a text file listing image paths, one per line (alternative to listing files on the command line).<br />
|-<br />
| <code>input.glob</code> || None || Wildcard path(s) for large multi-file datasets (e.g. <code>*.cbf</code>). May be given multiple times.<br />
|-<br />
| <code>input.image_tag</code> || None || Process only images with the given tag(s). For single-image files the tag is the filename; for multi-image files (HDF5) it is <code>filename_imagenumber</code> (image number is 5 digits with leading zeroes). Use <code>input.show_image_tags=True</code> to list them.<br />
|-<br />
| <code>input.max_images</code> || None || Limit the total number of images processed.<br />
|}<br />
<br />
=== Dispatch — controlling the pipeline ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>dispatch.find_spots</code> || True || Do spot-finding. Required for indexing/integration.<br />
|-<br />
| <code>dispatch.index</code> || True || Attempt to index. Requires <code>find_spots=True</code>.<br />
|-<br />
| <code>dispatch.integrate</code> || True || Integrate indexed images. Ignored if indexing/spot-finding is off.<br />
|-<br />
| <code>dispatch.refine</code> || False || Run an explicit model-refinement step after indexing (the stills indexer already refines; this is an additional pass, rarely needed).<br />
|-<br />
| <code>dispatch.squash_errors</code> || True || If an image fails, log it and continue to the next image. Set False to halt and show the full error when debugging.<br />
|-<br />
| <code>dispatch.hit_finder.enable</code> || True || Enable hit-finding. If disabled, every image is passed to indexing.<br />
|-<br />
| <code>dispatch.hit_finder.minimum_number_of_reflections</code> || 16 || Images with fewer strong spots than this are discarded as non-hits before indexing.<br />
|-<br />
| <code>dispatch.hit_finder.maximum_number_of_reflections</code> || None || If set, images with more strong spots than this are discarded (e.g. ice/junk).<br />
|-<br />
| <code>dispatch.process_percent</code> || None || Process only this percentage of events (evenly sampled). Useful for a quick look at a large dataset.<br />
|-<br />
| <code>dispatch.coset</code> || False || During integration, also integrate a sublattice coset as a negative control for regions with no Bragg diffraction (see <code>integration.coset.transformation</code>).<br />
|-<br />
| <code>dispatch.pre_import</code> || False || Import all data before processing. Only needed when processing multiple ''multi-image'' files at once (not a recommended workflow). Single-file datasets (e.g. HDF5) are always pre-imported automatically.<br />
|}<br />
<br />
=== Spot-finding ===<br />
<br />
Spot-finding uses the standard DIALS dispersion thresholding. The most impactful parameters for<br />
serial data:<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>spotfinder.filter.min_spot_size</code> || Auto || Minimum connected pixels for a spot. Auto = 3 for pixel-array detectors (SENSOR_PAD), 6 otherwise. Often lowered to 2 for sparse serial patterns.<br />
|-<br />
| <code>spotfinder.threshold.dispersion.gain</code> || None || Detector gain used in thresholding. Set this ''or'' the detector-model gain, not both (see gain note below).<br />
|-<br />
| <code>spotfinder.threshold.dispersion.sigma_strong</code> || 3 || Number of sigma above background for a pixel to be "strong".<br />
|-<br />
| <code>spotfinder.threshold.dispersion.sigma_background</code> || 6 || Number of sigma for the background/dispersion test.<br />
|-<br />
| <code>spotfinder.threshold.dispersion.global_threshold</code> || 0 || Absolute pixel intensity floor; pixels below this are never considered strong. Raising it (e.g. 100) suppresses noise on many detectors.<br />
|-<br />
| <code>spotfinder.threshold.dispersion.kernel_size</code> || 3 3 || Local-area kernel (pixels) for the dispersion calculation.<br />
|}<br />
<br />
'''Gain note:''' the detector gain should be set on the detector model ''or'' in the processing<br />
parameters, never both. If a supplied gain (<code>spotfinder.threshold.dispersion.gain</code> or<br />
<code>integration.summation.detector_gain</code>) conflicts with a non-unity gain already on the<br />
detector model, the program raises an error. Override with<br />
<code>input.ignore_gain_mismatch=True</code> only if you understand the consequences.<br />
<br />
=== Indexing ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>indexing.known_symmetry.space_group</code> || None || Target space group. Strongly recommended for serial data — constrains indexing and improves the hit/index rate.<br />
|-<br />
| <code>indexing.known_symmetry.unit_cell</code> || None || Target unit cell (a,b,c,α,β,γ). Used together with the space group as the indexing target.<br />
|-<br />
| <code>indexing.method</code> || fft1d || Basis-vector search method (<code>dials.stills_process</code> defaults to <code>fft1d</code>; <code>fft3d</code> and <code>real_space_grid_search</code> are also available).<br />
|-<br />
| <code>indexing.stills.method_list</code> || None || Ordered list of indexing methods to try in turn; if one fails, the next is attempted (e.g. <code>fft1d real_space_grid_search</code>).<br />
|-<br />
| <code>indexing.stills.known_orientations</code> || None || Paths to previously-processed experiment lists. If given, images are ''not'' re-indexed; the stored crystal orientations are reused (e.g. to re-integrate with new parameters). Supports wildcards; may be repeated.<br />
|-<br />
| <code>indexing.stills.require_known_orientation</code> || False || When known orientations are supplied but an image is not found among them, controls whether to skip it (True) or fall back to indexing from scratch (False).<br />
|-<br />
| <code>indexing.stills.reflection_subsampling.enable</code> || False || Retry indexing on random subsets of the strong spots (from <code>step_start</code>% down to <code>step_stop</code>% in <code>step_size</code>% steps). Can rescue images that fail to index on the full spot list. Especially valuble for multi-lattice images (multiple crystals in the beam) but does increase execution time!<br />
|}<br />
<br />
=== Integration ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>profile.gaussian_rs.parameters.sigma_b_cutoff</code> || 0.1 || Maximum beam divergence (sigma_b, degrees) before an indexed crystal is rejected prior to integration. Guards against over-broad, poorly-modelled lattices.<br />
|-<br />
| <code>significance_filter.enable</code> || False || Apply the stills significance filter, which trims each crystal to the resolution at which reflections remain statistically significant (see <code>significance_filter.isigi_cutoff</code>, default 1.0). Defaults to false here because it is typically applied in ''[[cctbx.xfel.merge]]''.<br />
|-<br />
| <code>integration.integration_only_overrides.trusted_range</code> || None || Override the panel trusted range [min, max] during integration only (e.g. to include pixels the header marks as untrusted).<br />
|-<br />
| <code>integration.coset.transformation</code> || 6 || Which modulus-2 sublattice transformation defines the coset negative control when <code>dispatch.coset=True</code> (0–6; 6 = body centering). See Sauter & Zwart, ''Acta Cryst.'' D (2009) '''65''':553.<br />
|}<br />
<br />
=== Output ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>output.output_dir</code> || <code>.</code> || Directory for output files.<br />
|-<br />
| <code>output.composite_output</code> || True || If True, each process writes ''one'' concatenated set of <code>.expt</code>/<code>.refl</code> files covering all its successful images. If False, a separate pair of files is written per image (generates a very large number of files). Composite output is strongly preferred for large serial datasets.<br />
|-<br />
| <code>output.logging_dir</code> || None || Directory for log files. Under MPI, per-rank logs (<code>log_rankNNNN.out</code>, etc.) are written here.<br />
|-<br />
| <code>output.logging_option</code> || normal || <code>normal</code> (all logging), <code>suppressed</code> (silences verbose DIALS refine/index output), or <code>disabled</code> (almost no logging).<br />
|-<br />
| <code>output.*_filename</code> || (templates) || Output filename templates for each stage: <code>strong</code>, <code>indexed</code>, <code>refined_experiments</code>, <code>integrated</code>, <code>integrated_experiments</code>, <code>coset</code>. A <code>%s</code> in the template is replaced with the process number for composite mode or the per-image tag for non-composite mode.<br />
|-<br />
| <code>output.integration_pickle</code> || None || Write legacy <code>cxi.merge</code>/PRIME integration pickle files (e.g. <code>int-%d-%s.pickle</code>). Only needed for legacy merging workflows.<br />
|}<br />
<br />
=== Multiprocessing ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>mp.method</code> || multiprocessing || Parallelization backend: <code>multiprocessing</code> (single node), or <code>mpi</code> for multi-node jobs (also <code>sge</code>, <code>lsf</code>, <code>pbs</code>).<br />
|-<br />
| <code>mp.nproc</code> || 1 || Number of processes (for the <code>multiprocessing</code> method). Set to the number of cores per node.<br />
|-<br />
| <code>mp.composite_stride</code> || None || For MPI composite output, how many ranks aggregate their results into one set of output files. Ideally set to the number of processes per node so that one composite file set is written per node rather than per rank.<br />
|}<br />
<br />
Under MPI with more than two ranks, the program runs a client/server model: rank 0 (or rank 2 for<br />
psana2 <code>.loc</code> inputs) hands out images to worker ranks on demand, giving good load<br />
balancing across the many fast-failing non-hits typical of serial data.<br />
<br />
== Output files ==<br />
<br />
With the default composite output, each process (or each aggregating rank under MPI) writes<br />
concatenated files such as:<br />
<br />
* <code>&lt;tag&gt;_strong.refl</code> — strong spots from spot-finding<br />
* <code>&lt;tag&gt;_indexed.refl</code> and <code>&lt;tag&gt;_refined.expt</code> — indexed reflections and their crystal models<br />
* <code>&lt;tag&gt;_integrated.refl</code> and <code>&lt;tag&gt;_integrated.expt</code> — integrated intensities and final models (the input to ''[[cctbx.xfel.merge]]'')<br />
<br />
Per-image progress and status codes are recorded in <code>debug/debug_&lt;rank&gt;.txt</code>, and<br />
overall logging goes to <code>dials.process.log</code> (or per-rank logs under MPI). These integrated<br />
<code>.expt</code>/<code>.refl</code> pairs are then scaled and merged with<br />
''[[cctbx.xfel.merge]]''.<br />
<br />
== See also ==<br />
<br />
* ''[[cctbx.xfel.merge]]'' — scaling, post-refinement and merging of the integrated data<br />
* The [http://dials.github.io DIALS documentation] for the underlying spot-finding, indexing, refinement and integration algorithms</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Dials.stills_process&diff=1767Dials.stills process2026-08-21T22:52:56Z<p>Aaron: /* Output files */</p>
<hr />
<div>= dials.stills_process =<br />
<br />
'''''dials.stills_process''''' is a stand-alone program for processing still diffraction images and is<br />
the core data-reduction program of ''cctbx.xfel''. It takes a set of<br />
still diffraction images — typically serial femtosecond crystallography (SFX) snapshots from an<br />
XFEL (LCLS, SACLA, EuXFEL) or still shots from a synchrotron — and analyzes each one, independently,<br />
through the full DIALS pipeline: '''import → spot-finding → indexing → refinement → integration'''.<br />
Its output (integrated intensities together with the per-image experimental models) is the input to<br />
the merging program [[cctbx.xfel.merge]].<br />
<br />
Because every image is a separate still exposure of a different crystal (rather than a wedge of<br />
a rotation series), each image is treated as its own experiment with its own crystal model. This<br />
"one crystal per shot" model is what distinguishes stills processing from conventional rotation data<br />
processing (e.g. <code>dials.import</code>, <code>dials.find_spots</code>, <code>dials.index</code>, etc.), <br />
and it drives most of the stills-specific defaults described below.<br />
<br />
== Overview ==<br />
<br />
For each image, <code>dials.stills_process</code> runs the following stages in order. Any stage can<br />
be turned off with the <code>dispatch</code> parameters; if a stage is disabled, processing stops<br />
after the last enabled stage.<br />
<br />
# '''Import''' — read the image, apply reference geometry / gain / mask overrides, and build an experiment.<br />
# '''Spot-finding''' — locate strong pixels and build strong reflections (<code>dispatch.find_spots</code>).<br />
# '''Hit finding''' — reject images with too few (or optionally too many) strong spots, so that only "hits" are indexed (<code>dispatch.hit_finder</code>).<br />
# '''Indexing''' — determine the crystal orientation and (optionally constrained) unit cell (<code>dispatch.index</code>).<br />
# '''Refinement''' — refine the crystal model and re-estimate the mosaic/domain-size parameters. Detector and beam are held fixed by default (<code>dispatch.refine</code>).<br />
# '''Integration''' — predict reflections and measure their intensities using the stills integrator (<code>dispatch.integrate</code>), with an optional significance filter.<br />
<br />
The program applies a set of stills-specific defaults on top of the standard DIALS parameters,<br />
including: indexing method <code>fft1d</code>; the stills integrator with profile fitting turned off<br />
(summation integration with a 2-D plane background); the stills refinement weighting strategy with<br />
the detector and beam models fixed; and a re-estimation of per-crystal mosaic parameters<br />
(domain size and mosaic angle) after refinement. These are sensible starting points for serial data<br />
and normally do not need to be changed.<br />
<br />
Failures on individual images are, by default, caught and logged so that processing continues to the<br />
next image (<code>dispatch.squash_errors=True</code>) — essential when reducing tens of thousands of<br />
shots of which only a fraction are hits.<br />
<br />
== Basic usage ==<br />
<br />
<pre><br />
# Process a directory of CBF stills, constraining the target cell/symmetry,<br />
# using 32 parallel processes on one node<br />
dials.stills_process /path/to/images/*.cbf process.phil mp.nproc=32<br />
</pre><br />
<br />
A typical <code>process.phil</code> for XFEL data looks like:<br />
<br />
<pre><br />
spotfinder.filter.min_spot_size=2<br />
spotfinder.threshold.dispersion.gain=1.0<br />
spotfinder.threshold.dispersion.global_threshold=100<br />
<br />
indexing.known_symmetry.space_group=P43212<br />
indexing.known_symmetry.unit_cell=78.9,78.9,38.1,90,90,90<br />
indexing.method=fft1d<br />
<br />
input.reference_geometry=refined.expt # from a previous metrology refinement<br />
<br />
output.output_dir=out<br />
mp.nproc=32<br />
</pre><br />
<br />
Useful command-line helpers:<br />
* <code>dials.stills_process -h</code> — help and usage<br />
* <code>dials.stills_process -c -e 2</code> — show the configurable parameters (expert level 2)<br />
* <code>dials.stills_process -c -a2</code> — include attributes (help text) for each parameter<br />
* <code>dials.stills_process ... input.show_image_tags=True</code> — list the image tags in a dataset without processing (useful for selecting subsets with <code>input.image_tag</code>)<br />
<br />
== Important parameters ==<br />
<br />
The program exposes the entire DIALS parameter set (spot-finding, indexing, refinement, integration,<br />
profile modelling, prediction). The parameters below are the ones most commonly adjusted for serial<br />
still data. Parameters are given by their full PHIL path with their default value.<br />
<br />
=== Input ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>input.reference_geometry</code> || None || Path to an <code>.expt</code> file (or image) with a single detector model. Data are processed using ''this'' geometry instead of the geometry in the image headers. This is how a refined detector metrology from <code>cctbx.xfel</code> ensemble refinement is applied — almost always set for XFEL data.<br />
|-<br />
| <code>input.sync_reference_geom</code> || True || When a reference geometry is supplied, copy its refined local frames onto the image's detector hierarchy node-by-node (the hierarchies must match in shape). If False, the reference detector replaces the image detector wholesale.<br />
|-<br />
| <code>input.file_list</code> || None || Path to a text file listing image paths, one per line (alternative to listing files on the command line).<br />
|-<br />
| <code>input.glob</code> || None || Wildcard path(s) for large multi-file datasets (e.g. <code>*.cbf</code>). May be given multiple times.<br />
|-<br />
| <code>input.image_tag</code> || None || Process only images with the given tag(s). For single-image files the tag is the filename; for multi-image files (HDF5) it is <code>filename_imagenumber</code> (image number is 5 digits with leading zeroes). Use <code>input.show_image_tags=True</code> to list them.<br />
|-<br />
| <code>input.max_images</code> || None || Limit the total number of images processed.<br />
|}<br />
<br />
=== Dispatch — controlling the pipeline ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>dispatch.find_spots</code> || True || Do spot-finding. Required for indexing/integration.<br />
|-<br />
| <code>dispatch.index</code> || True || Attempt to index. Requires <code>find_spots=True</code>.<br />
|-<br />
| <code>dispatch.integrate</code> || True || Integrate indexed images. Ignored if indexing/spot-finding is off.<br />
|-<br />
| <code>dispatch.refine</code> || False || Run an explicit model-refinement step after indexing (the stills indexer already refines; this is an additional pass, rarely needed).<br />
|-<br />
| <code>dispatch.squash_errors</code> || True || If an image fails, log it and continue to the next image. Set False to halt and show the full error when debugging.<br />
|-<br />
| <code>dispatch.hit_finder.enable</code> || True || Enable hit-finding. If disabled, every image is passed to indexing.<br />
|-<br />
| <code>dispatch.hit_finder.minimum_number_of_reflections</code> || 16 || Images with fewer strong spots than this are discarded as non-hits before indexing.<br />
|-<br />
| <code>dispatch.hit_finder.maximum_number_of_reflections</code> || None || If set, images with more strong spots than this are discarded (e.g. ice/junk).<br />
|-<br />
| <code>dispatch.process_percent</code> || None || Process only this percentage of events (evenly sampled). Useful for a quick look at a large dataset.<br />
|-<br />
| <code>dispatch.coset</code> || False || During integration, also integrate a sublattice coset as a negative control for regions with no Bragg diffraction (see <code>integration.coset.transformation</code>).<br />
|-<br />
| <code>dispatch.pre_import</code> || False || Import all data before processing. Only needed when processing multiple ''multi-image'' files at once (not a recommended workflow). Single-file datasets (e.g. HDF5) are always pre-imported automatically.<br />
|}<br />
<br />
=== Spot-finding ===<br />
<br />
Spot-finding uses the standard DIALS dispersion thresholding. The most impactful parameters for<br />
serial data:<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>spotfinder.filter.min_spot_size</code> || Auto || Minimum connected pixels for a spot. Auto = 3 for pixel-array detectors (SENSOR_PAD), 6 otherwise. Often lowered to 2 for sparse serial patterns.<br />
|-<br />
| <code>spotfinder.threshold.dispersion.gain</code> || None || Detector gain used in thresholding. Set this ''or'' the detector-model gain, not both (see gain note below).<br />
|-<br />
| <code>spotfinder.threshold.dispersion.sigma_strong</code> || 3 || Number of sigma above background for a pixel to be "strong".<br />
|-<br />
| <code>spotfinder.threshold.dispersion.sigma_background</code> || 6 || Number of sigma for the background/dispersion test.<br />
|-<br />
| <code>spotfinder.threshold.dispersion.global_threshold</code> || 0 || Absolute pixel intensity floor; pixels below this are never considered strong. Raising it (e.g. 100) suppresses noise on many detectors.<br />
|-<br />
| <code>spotfinder.threshold.dispersion.kernel_size</code> || 3 3 || Local-area kernel (pixels) for the dispersion calculation.<br />
|}<br />
<br />
'''Gain note:''' the detector gain should be set on the detector model ''or'' in the processing<br />
parameters, never both. If a supplied gain (<code>spotfinder.threshold.dispersion.gain</code> or<br />
<code>integration.summation.detector_gain</code>) conflicts with a non-unity gain already on the<br />
detector model, the program raises an error. Override with<br />
<code>input.ignore_gain_mismatch=True</code> only if you understand the consequences.<br />
<br />
=== Indexing ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>indexing.known_symmetry.space_group</code> || None || Target space group. Strongly recommended for serial data — constrains indexing and improves the hit/index rate.<br />
|-<br />
| <code>indexing.known_symmetry.unit_cell</code> || None || Target unit cell (a,b,c,α,β,γ). Used together with the space group as the indexing target.<br />
|-<br />
| <code>indexing.method</code> || fft1d || Basis-vector search method (<code>dials.stills_process</code> defaults to <code>fft1d</code>; <code>fft3d</code> and <code>real_space_grid_search</code> are also available).<br />
|-<br />
| <code>indexing.stills.method_list</code> || None || Ordered list of indexing methods to try in turn; if one fails, the next is attempted (e.g. <code>fft1d real_space_grid_search</code>).<br />
|-<br />
| <code>indexing.stills.known_orientations</code> || None || Paths to previously-processed experiment lists. If given, images are ''not'' re-indexed; the stored crystal orientations are reused (e.g. to re-integrate with new parameters). Supports wildcards; may be repeated.<br />
|-<br />
| <code>indexing.stills.require_known_orientation</code> || False || When known orientations are supplied but an image is not found among them, controls whether to skip it (True) or fall back to indexing from scratch (False).<br />
|-<br />
| <code>indexing.stills.reflection_subsampling.enable</code> || False || Retry indexing on random subsets of the strong spots (from <code>step_start</code>% down to <code>step_stop</code>% in <code>step_size</code>% steps). Can rescue images that fail to index on the full spot list. Especially valuble for multi-lattice images (multiple crystals in the beam) but does increase execution time!<br />
|}<br />
<br />
=== Integration ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>profile.gaussian_rs.parameters.sigma_b_cutoff</code> || 0.1 || Maximum beam divergence (sigma_b, degrees) before an indexed crystal is rejected prior to integration. Guards against over-broad, poorly-modelled lattices.<br />
|-<br />
| <code>significance_filter.enable</code> || False || Apply the stills significance filter, which trims each crystal to the resolution at which reflections remain statistically significant (see <code>significance_filter.isigi_cutoff</code>, default 1.0). Defaults to false here because it is typically applied in ''[[cctbx.xfel.merge]]''.<br />
|-<br />
| <code>integration.integration_only_overrides.trusted_range</code> || None || Override the panel trusted range [min, max] during integration only (e.g. to include pixels the header marks as untrusted).<br />
|-<br />
| <code>integration.coset.transformation</code> || 6 || Which modulus-2 sublattice transformation defines the coset negative control when <code>dispatch.coset=True</code> (0–6; 6 = body centering). See Sauter & Zwart, ''Acta Cryst.'' D (2009) '''65''':553.<br />
|}<br />
<br />
=== Output ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>output.output_dir</code> || <code>.</code> || Directory for output files.<br />
|-<br />
| <code>output.composite_output</code> || True || If True, each process writes ''one'' concatenated set of <code>.expt</code>/<code>.refl</code> files covering all its successful images. If False, a separate pair of files is written per image (generates a very large number of files). Composite output is strongly preferred for large serial datasets.<br />
|-<br />
| <code>output.logging_dir</code> || None || Directory for log files. Under MPI, per-rank logs (<code>log_rankNNNN.out</code>, etc.) are written here.<br />
|-<br />
| <code>output.logging_option</code> || normal || <code>normal</code> (all logging), <code>suppressed</code> (silences verbose DIALS refine/index output), or <code>disabled</code> (almost no logging).<br />
|-<br />
| <code>output.*_filename</code> || (templates) || Output filename templates for each stage: <code>strong</code>, <code>indexed</code>, <code>refined_experiments</code>, <code>integrated</code>, <code>integrated_experiments</code>, <code>coset</code>. A <code>%s</code> in the template is replaced with the process number for composite mode or the per-image tag for non-composite mode.<br />
|-<br />
| <code>output.integration_pickle</code> || None || Write legacy <code>cxi.merge</code>/PRIME integration pickle files (e.g. <code>int-%d-%s.pickle</code>). Only needed for legacy merging workflows.<br />
|}<br />
<br />
=== Multiprocessing ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>mp.method</code> || multiprocessing || Parallelization backend: <code>multiprocessing</code> (single node), or <code>mpi</code> for multi-node jobs (also <code>sge</code>, <code>lsf</code>, <code>pbs</code>).<br />
|-<br />
| <code>mp.nproc</code> || 1 || Number of processes (for the <code>multiprocessing</code> method). Set to the number of cores per node.<br />
|-<br />
| <code>mp.composite_stride</code> || None || For MPI composite output, how many ranks aggregate their results into one set of output files. Ideally set to the number of processes per node so that one composite file set is written per node rather than per rank.<br />
|}<br />
<br />
Under MPI with more than two ranks, the program runs a client/server model: rank 0 (or rank 2 for<br />
psana2 <code>.loc</code> inputs) hands out images to worker ranks on demand, giving good load<br />
balancing across the many fast-failing non-hits typical of serial data.<br />
<br />
== Output files ==<br />
<br />
With the default composite output, each process (or each aggregating rank under MPI) writes<br />
concatenated files such as:<br />
<br />
* <code>&lt;tag&gt;_strong.refl</code> — strong spots from spot-finding<br />
* <code>&lt;tag&gt;_indexed.refl</code> and <code>&lt;tag&gt;_refined.expt</code> — indexed reflections and their crystal models<br />
* <code>&lt;tag&gt;_integrated.refl</code> and <code>&lt;tag&gt;_integrated.expt</code> — integrated intensities and final models (the input to ''[[cctbx.xfel.merge]]'')<br />
<br />
Per-image progress and status codes are recorded in <code>debug/debug_&lt;rank&gt;.txt</code>, and<br />
overall logging goes to <code>dials.process.log</code> (or per-rank logs under MPI). These integrated<br />
<code>.expt</code>/<code>.refl</code> pairs are then scaled and merged with<br />
''[[cctbx.xfel.merge]]''.<br />
<br />
== See also ==<br />
<br />
* [[cctbx.xfel.merge]] — scaling, post-refinement and merging of the integrated data<br />
* [[IOTA]] — a front-end that supplies preset parameters for <code>dials.stills_process</code><br />
* [[Data Processing at SACLA]] — SACLA-specific processing workflow<br />
* The DIALS documentation for the underlying spot-finding, indexing, refinement and integration algorithms</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Dials.stills_process&diff=1766Dials.stills process2026-08-21T22:52:20Z<p>Aaron: /* Overview */</p>
<hr />
<div>= dials.stills_process =<br />
<br />
'''''dials.stills_process''''' is a stand-alone program for processing still diffraction images and is<br />
the core data-reduction program of ''cctbx.xfel''. It takes a set of<br />
still diffraction images — typically serial femtosecond crystallography (SFX) snapshots from an<br />
XFEL (LCLS, SACLA, EuXFEL) or still shots from a synchrotron — and analyzes each one, independently,<br />
through the full DIALS pipeline: '''import → spot-finding → indexing → refinement → integration'''.<br />
Its output (integrated intensities together with the per-image experimental models) is the input to<br />
the merging program [[cctbx.xfel.merge]].<br />
<br />
Because every image is a separate still exposure of a different crystal (rather than a wedge of<br />
a rotation series), each image is treated as its own experiment with its own crystal model. This<br />
"one crystal per shot" model is what distinguishes stills processing from conventional rotation data<br />
processing (e.g. <code>dials.import</code>, <code>dials.find_spots</code>, <code>dials.index</code>, etc.), <br />
and it drives most of the stills-specific defaults described below.<br />
<br />
== Overview ==<br />
<br />
For each image, <code>dials.stills_process</code> runs the following stages in order. Any stage can<br />
be turned off with the <code>dispatch</code> parameters; if a stage is disabled, processing stops<br />
after the last enabled stage.<br />
<br />
# '''Import''' — read the image, apply reference geometry / gain / mask overrides, and build an experiment.<br />
# '''Spot-finding''' — locate strong pixels and build strong reflections (<code>dispatch.find_spots</code>).<br />
# '''Hit finding''' — reject images with too few (or optionally too many) strong spots, so that only "hits" are indexed (<code>dispatch.hit_finder</code>).<br />
# '''Indexing''' — determine the crystal orientation and (optionally constrained) unit cell (<code>dispatch.index</code>).<br />
# '''Refinement''' — refine the crystal model and re-estimate the mosaic/domain-size parameters. Detector and beam are held fixed by default (<code>dispatch.refine</code>).<br />
# '''Integration''' — predict reflections and measure their intensities using the stills integrator (<code>dispatch.integrate</code>), with an optional significance filter.<br />
<br />
The program applies a set of stills-specific defaults on top of the standard DIALS parameters,<br />
including: indexing method <code>fft1d</code>; the stills integrator with profile fitting turned off<br />
(summation integration with a 2-D plane background); the stills refinement weighting strategy with<br />
the detector and beam models fixed; and a re-estimation of per-crystal mosaic parameters<br />
(domain size and mosaic angle) after refinement. These are sensible starting points for serial data<br />
and normally do not need to be changed.<br />
<br />
Failures on individual images are, by default, caught and logged so that processing continues to the<br />
next image (<code>dispatch.squash_errors=True</code>) — essential when reducing tens of thousands of<br />
shots of which only a fraction are hits.<br />
<br />
== Basic usage ==<br />
<br />
<pre><br />
# Process a directory of CBF stills, constraining the target cell/symmetry,<br />
# using 32 parallel processes on one node<br />
dials.stills_process /path/to/images/*.cbf process.phil mp.nproc=32<br />
</pre><br />
<br />
A typical <code>process.phil</code> for XFEL data looks like:<br />
<br />
<pre><br />
spotfinder.filter.min_spot_size=2<br />
spotfinder.threshold.dispersion.gain=1.0<br />
spotfinder.threshold.dispersion.global_threshold=100<br />
<br />
indexing.known_symmetry.space_group=P43212<br />
indexing.known_symmetry.unit_cell=78.9,78.9,38.1,90,90,90<br />
indexing.method=fft1d<br />
<br />
input.reference_geometry=refined.expt # from a previous metrology refinement<br />
<br />
output.output_dir=out<br />
mp.nproc=32<br />
</pre><br />
<br />
Useful command-line helpers:<br />
* <code>dials.stills_process -h</code> — help and usage<br />
* <code>dials.stills_process -c -e 2</code> — show the configurable parameters (expert level 2)<br />
* <code>dials.stills_process -c -a2</code> — include attributes (help text) for each parameter<br />
* <code>dials.stills_process ... input.show_image_tags=True</code> — list the image tags in a dataset without processing (useful for selecting subsets with <code>input.image_tag</code>)<br />
<br />
== Important parameters ==<br />
<br />
The program exposes the entire DIALS parameter set (spot-finding, indexing, refinement, integration,<br />
profile modelling, prediction). The parameters below are the ones most commonly adjusted for serial<br />
still data. Parameters are given by their full PHIL path with their default value.<br />
<br />
=== Input ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>input.reference_geometry</code> || None || Path to an <code>.expt</code> file (or image) with a single detector model. Data are processed using ''this'' geometry instead of the geometry in the image headers. This is how a refined detector metrology from <code>cctbx.xfel</code> ensemble refinement is applied — almost always set for XFEL data.<br />
|-<br />
| <code>input.sync_reference_geom</code> || True || When a reference geometry is supplied, copy its refined local frames onto the image's detector hierarchy node-by-node (the hierarchies must match in shape). If False, the reference detector replaces the image detector wholesale.<br />
|-<br />
| <code>input.file_list</code> || None || Path to a text file listing image paths, one per line (alternative to listing files on the command line).<br />
|-<br />
| <code>input.glob</code> || None || Wildcard path(s) for large multi-file datasets (e.g. <code>*.cbf</code>). May be given multiple times.<br />
|-<br />
| <code>input.image_tag</code> || None || Process only images with the given tag(s). For single-image files the tag is the filename; for multi-image files (HDF5) it is <code>filename_imagenumber</code> (image number is 5 digits with leading zeroes). Use <code>input.show_image_tags=True</code> to list them.<br />
|-<br />
| <code>input.max_images</code> || None || Limit the total number of images processed.<br />
|}<br />
<br />
=== Dispatch — controlling the pipeline ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>dispatch.find_spots</code> || True || Do spot-finding. Required for indexing/integration.<br />
|-<br />
| <code>dispatch.index</code> || True || Attempt to index. Requires <code>find_spots=True</code>.<br />
|-<br />
| <code>dispatch.integrate</code> || True || Integrate indexed images. Ignored if indexing/spot-finding is off.<br />
|-<br />
| <code>dispatch.refine</code> || False || Run an explicit model-refinement step after indexing (the stills indexer already refines; this is an additional pass, rarely needed).<br />
|-<br />
| <code>dispatch.squash_errors</code> || True || If an image fails, log it and continue to the next image. Set False to halt and show the full error when debugging.<br />
|-<br />
| <code>dispatch.hit_finder.enable</code> || True || Enable hit-finding. If disabled, every image is passed to indexing.<br />
|-<br />
| <code>dispatch.hit_finder.minimum_number_of_reflections</code> || 16 || Images with fewer strong spots than this are discarded as non-hits before indexing.<br />
|-<br />
| <code>dispatch.hit_finder.maximum_number_of_reflections</code> || None || If set, images with more strong spots than this are discarded (e.g. ice/junk).<br />
|-<br />
| <code>dispatch.process_percent</code> || None || Process only this percentage of events (evenly sampled). Useful for a quick look at a large dataset.<br />
|-<br />
| <code>dispatch.coset</code> || False || During integration, also integrate a sublattice coset as a negative control for regions with no Bragg diffraction (see <code>integration.coset.transformation</code>).<br />
|-<br />
| <code>dispatch.pre_import</code> || False || Import all data before processing. Only needed when processing multiple ''multi-image'' files at once (not a recommended workflow). Single-file datasets (e.g. HDF5) are always pre-imported automatically.<br />
|}<br />
<br />
=== Spot-finding ===<br />
<br />
Spot-finding uses the standard DIALS dispersion thresholding. The most impactful parameters for<br />
serial data:<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>spotfinder.filter.min_spot_size</code> || Auto || Minimum connected pixels for a spot. Auto = 3 for pixel-array detectors (SENSOR_PAD), 6 otherwise. Often lowered to 2 for sparse serial patterns.<br />
|-<br />
| <code>spotfinder.threshold.dispersion.gain</code> || None || Detector gain used in thresholding. Set this ''or'' the detector-model gain, not both (see gain note below).<br />
|-<br />
| <code>spotfinder.threshold.dispersion.sigma_strong</code> || 3 || Number of sigma above background for a pixel to be "strong".<br />
|-<br />
| <code>spotfinder.threshold.dispersion.sigma_background</code> || 6 || Number of sigma for the background/dispersion test.<br />
|-<br />
| <code>spotfinder.threshold.dispersion.global_threshold</code> || 0 || Absolute pixel intensity floor; pixels below this are never considered strong. Raising it (e.g. 100) suppresses noise on many detectors.<br />
|-<br />
| <code>spotfinder.threshold.dispersion.kernel_size</code> || 3 3 || Local-area kernel (pixels) for the dispersion calculation.<br />
|}<br />
<br />
'''Gain note:''' the detector gain should be set on the detector model ''or'' in the processing<br />
parameters, never both. If a supplied gain (<code>spotfinder.threshold.dispersion.gain</code> or<br />
<code>integration.summation.detector_gain</code>) conflicts with a non-unity gain already on the<br />
detector model, the program raises an error. Override with<br />
<code>input.ignore_gain_mismatch=True</code> only if you understand the consequences.<br />
<br />
=== Indexing ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>indexing.known_symmetry.space_group</code> || None || Target space group. Strongly recommended for serial data — constrains indexing and improves the hit/index rate.<br />
|-<br />
| <code>indexing.known_symmetry.unit_cell</code> || None || Target unit cell (a,b,c,α,β,γ). Used together with the space group as the indexing target.<br />
|-<br />
| <code>indexing.method</code> || fft1d || Basis-vector search method (<code>dials.stills_process</code> defaults to <code>fft1d</code>; <code>fft3d</code> and <code>real_space_grid_search</code> are also available).<br />
|-<br />
| <code>indexing.stills.method_list</code> || None || Ordered list of indexing methods to try in turn; if one fails, the next is attempted (e.g. <code>fft1d real_space_grid_search</code>).<br />
|-<br />
| <code>indexing.stills.known_orientations</code> || None || Paths to previously-processed experiment lists. If given, images are ''not'' re-indexed; the stored crystal orientations are reused (e.g. to re-integrate with new parameters). Supports wildcards; may be repeated.<br />
|-<br />
| <code>indexing.stills.require_known_orientation</code> || False || When known orientations are supplied but an image is not found among them, controls whether to skip it (True) or fall back to indexing from scratch (False).<br />
|-<br />
| <code>indexing.stills.reflection_subsampling.enable</code> || False || Retry indexing on random subsets of the strong spots (from <code>step_start</code>% down to <code>step_stop</code>% in <code>step_size</code>% steps). Can rescue images that fail to index on the full spot list. Especially valuble for multi-lattice images (multiple crystals in the beam) but does increase execution time!<br />
|}<br />
<br />
=== Integration ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>profile.gaussian_rs.parameters.sigma_b_cutoff</code> || 0.1 || Maximum beam divergence (sigma_b, degrees) before an indexed crystal is rejected prior to integration. Guards against over-broad, poorly-modelled lattices.<br />
|-<br />
| <code>significance_filter.enable</code> || False || Apply the stills significance filter, which trims each crystal to the resolution at which reflections remain statistically significant (see <code>significance_filter.isigi_cutoff</code>, default 1.0). Defaults to false here because it is typically applied in ''[[cctbx.xfel.merge]]''.<br />
|-<br />
| <code>integration.integration_only_overrides.trusted_range</code> || None || Override the panel trusted range [min, max] during integration only (e.g. to include pixels the header marks as untrusted).<br />
|-<br />
| <code>integration.coset.transformation</code> || 6 || Which modulus-2 sublattice transformation defines the coset negative control when <code>dispatch.coset=True</code> (0–6; 6 = body centering). See Sauter & Zwart, ''Acta Cryst.'' D (2009) '''65''':553.<br />
|}<br />
<br />
=== Output ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>output.output_dir</code> || <code>.</code> || Directory for output files.<br />
|-<br />
| <code>output.composite_output</code> || True || If True, each process writes ''one'' concatenated set of <code>.expt</code>/<code>.refl</code> files covering all its successful images. If False, a separate pair of files is written per image (generates a very large number of files). Composite output is strongly preferred for large serial datasets.<br />
|-<br />
| <code>output.logging_dir</code> || None || Directory for log files. Under MPI, per-rank logs (<code>log_rankNNNN.out</code>, etc.) are written here.<br />
|-<br />
| <code>output.logging_option</code> || normal || <code>normal</code> (all logging), <code>suppressed</code> (silences verbose DIALS refine/index output), or <code>disabled</code> (almost no logging).<br />
|-<br />
| <code>output.*_filename</code> || (templates) || Output filename templates for each stage: <code>strong</code>, <code>indexed</code>, <code>refined_experiments</code>, <code>integrated</code>, <code>integrated_experiments</code>, <code>coset</code>. A <code>%s</code> in the template is replaced with the process number for composite mode or the per-image tag for non-composite mode.<br />
|-<br />
| <code>output.integration_pickle</code> || None || Write legacy <code>cxi.merge</code>/PRIME integration pickle files (e.g. <code>int-%d-%s.pickle</code>). Only needed for legacy merging workflows.<br />
|}<br />
<br />
=== Multiprocessing ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>mp.method</code> || multiprocessing || Parallelization backend: <code>multiprocessing</code> (single node), or <code>mpi</code> for multi-node jobs (also <code>sge</code>, <code>lsf</code>, <code>pbs</code>).<br />
|-<br />
| <code>mp.nproc</code> || 1 || Number of processes (for the <code>multiprocessing</code> method). Set to the number of cores per node.<br />
|-<br />
| <code>mp.composite_stride</code> || None || For MPI composite output, how many ranks aggregate their results into one set of output files. Ideally set to the number of processes per node so that one composite file set is written per node rather than per rank.<br />
|}<br />
<br />
Under MPI with more than two ranks, the program runs a client/server model: rank 0 (or rank 2 for<br />
psana2 <code>.loc</code> inputs) hands out images to worker ranks on demand, giving good load<br />
balancing across the many fast-failing non-hits typical of serial data.<br />
<br />
== Output files ==<br />
<br />
With the default composite output, each process (or each aggregating rank under MPI) writes<br />
concatenated files such as:<br />
<br />
* <code>&lt;tag&gt;_strong.refl</code> — strong spots from spot-finding<br />
* <code>&lt;tag&gt;_indexed.refl</code> and <code>&lt;tag&gt;_refined.expt</code> — indexed reflections and their crystal models<br />
* <code>&lt;tag&gt;_integrated.refl</code> and <code>&lt;tag&gt;_integrated.expt</code> — integrated intensities and final models (the input to [[cctbx.xfel.merge]])<br />
<br />
Per-image progress and status codes are recorded in <code>debug/debug_&lt;rank&gt;.txt</code>, and<br />
overall logging goes to <code>dials.process.log</code> (or per-rank logs under MPI). These integrated<br />
<code>.expt</code>/<code>.refl</code> pairs are then scaled and merged with<br />
[[cctbx.xfel.merge]].<br />
<br />
== See also ==<br />
<br />
* [[cctbx.xfel.merge]] — scaling, post-refinement and merging of the integrated data<br />
* [[IOTA]] — a front-end that supplies preset parameters for <code>dials.stills_process</code><br />
* [[Data Processing at SACLA]] — SACLA-specific processing workflow<br />
* The DIALS documentation for the underlying spot-finding, indexing, refinement and integration algorithms</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Dials.stills_process&diff=1765Dials.stills process2026-08-21T22:51:29Z<p>Aaron: /* Important parameters */</p>
<hr />
<div>= dials.stills_process =<br />
<br />
'''''dials.stills_process''''' is a stand-alone program for processing still diffraction images and is<br />
the core data-reduction program of ''cctbx.xfel''. It takes a set of<br />
still diffraction images — typically serial femtosecond crystallography (SFX) snapshots from an<br />
XFEL (LCLS, SACLA, EuXFEL) or still shots from a synchrotron — and analyzes each one, independently,<br />
through the full DIALS pipeline: '''import → spot-finding → indexing → refinement → integration'''.<br />
Its output (integrated intensities together with the per-image experimental models) is the input to<br />
the merging program [[cctbx.xfel.merge]].<br />
<br />
Because every image is a separate still exposure of a different crystal (rather than a wedge of<br />
a rotation series), each image is treated as its own experiment with its own crystal model. This<br />
"one crystal per shot" model is what distinguishes stills processing from conventional rotation data<br />
processing (e.g. <code>dials.import</code>, <code>dials.find_spots</code>, <code>dials.index</code>, etc.), <br />
and it drives most of the stills-specific defaults described below.<br />
<br />
== Overview ==<br />
<br />
For each image, <code>dials.stills_process</code> runs the following stages in order. Any stage can<br />
be turned off with the <code>dispatch</code> parameters; if a stage is disabled, processing stops<br />
after the last enabled stage.<br />
<br />
# '''Import''' — read the image, apply reference geometry / gain / mask overrides, and build an experiment.<br />
# '''Spot-finding''' — locate strong pixels and build strong reflections (<code>dispatch.find_spots</code>).<br />
# '''Hit finding''' — reject images with too few (or optionally too many) strong spots, so that only "hits" are indexed (<code>dispatch.hit_finder</code>).<br />
# '''Indexing''' — determine the crystal orientation and (optionally constrained) unit cell (<code>dispatch.index</code>).<br />
# '''Refinement''' — refine the crystal model and re-estimate the mosaic/domain-size parameters. Detector and beam are held fixed by default (<code>dispatch.refine</code>).<br />
# '''Integration''' — predict reflections and measure their intensities using the stills integrator (<code>dispatch.integrate</code>), with optional Kapton absorption correction and a significance filter.<br />
<br />
The program applies a set of stills-specific defaults on top of the standard DIALS parameters,<br />
including: indexing method <code>fft1d</code>; the stills integrator with profile fitting turned off<br />
(summation integration with a 2-D plane background); the stills refinement weighting strategy with<br />
the detector and beam models fixed; and a re-estimation of per-crystal mosaic parameters<br />
(domain size and mosaic angle) after refinement. These are sensible starting points for serial data<br />
and normally do not need to be changed.<br />
<br />
Failures on individual images are, by default, caught and logged so that processing continues to the<br />
next image (<code>dispatch.squash_errors=True</code>) — essential when reducing tens of thousands of<br />
shots of which only a fraction are hits.<br />
<br />
== Basic usage ==<br />
<br />
<pre><br />
# Process a directory of CBF stills, constraining the target cell/symmetry,<br />
# using 32 parallel processes on one node<br />
dials.stills_process /path/to/images/*.cbf process.phil mp.nproc=32<br />
</pre><br />
<br />
A typical <code>process.phil</code> for XFEL data looks like:<br />
<br />
<pre><br />
spotfinder.filter.min_spot_size=2<br />
spotfinder.threshold.dispersion.gain=1.0<br />
spotfinder.threshold.dispersion.global_threshold=100<br />
<br />
indexing.known_symmetry.space_group=P43212<br />
indexing.known_symmetry.unit_cell=78.9,78.9,38.1,90,90,90<br />
indexing.method=fft1d<br />
<br />
input.reference_geometry=refined.expt # from a previous metrology refinement<br />
<br />
output.output_dir=out<br />
mp.nproc=32<br />
</pre><br />
<br />
Useful command-line helpers:<br />
* <code>dials.stills_process -h</code> — help and usage<br />
* <code>dials.stills_process -c -e 2</code> — show the configurable parameters (expert level 2)<br />
* <code>dials.stills_process -c -a2</code> — include attributes (help text) for each parameter<br />
* <code>dials.stills_process ... input.show_image_tags=True</code> — list the image tags in a dataset without processing (useful for selecting subsets with <code>input.image_tag</code>)<br />
<br />
== Important parameters ==<br />
<br />
The program exposes the entire DIALS parameter set (spot-finding, indexing, refinement, integration,<br />
profile modelling, prediction). The parameters below are the ones most commonly adjusted for serial<br />
still data. Parameters are given by their full PHIL path with their default value.<br />
<br />
=== Input ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>input.reference_geometry</code> || None || Path to an <code>.expt</code> file (or image) with a single detector model. Data are processed using ''this'' geometry instead of the geometry in the image headers. This is how a refined detector metrology from <code>cctbx.xfel</code> ensemble refinement is applied — almost always set for XFEL data.<br />
|-<br />
| <code>input.sync_reference_geom</code> || True || When a reference geometry is supplied, copy its refined local frames onto the image's detector hierarchy node-by-node (the hierarchies must match in shape). If False, the reference detector replaces the image detector wholesale.<br />
|-<br />
| <code>input.file_list</code> || None || Path to a text file listing image paths, one per line (alternative to listing files on the command line).<br />
|-<br />
| <code>input.glob</code> || None || Wildcard path(s) for large multi-file datasets (e.g. <code>*.cbf</code>). May be given multiple times.<br />
|-<br />
| <code>input.image_tag</code> || None || Process only images with the given tag(s). For single-image files the tag is the filename; for multi-image files (HDF5) it is <code>filename_imagenumber</code> (image number is 5 digits with leading zeroes). Use <code>input.show_image_tags=True</code> to list them.<br />
|-<br />
| <code>input.max_images</code> || None || Limit the total number of images processed.<br />
|}<br />
<br />
=== Dispatch — controlling the pipeline ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>dispatch.find_spots</code> || True || Do spot-finding. Required for indexing/integration.<br />
|-<br />
| <code>dispatch.index</code> || True || Attempt to index. Requires <code>find_spots=True</code>.<br />
|-<br />
| <code>dispatch.integrate</code> || True || Integrate indexed images. Ignored if indexing/spot-finding is off.<br />
|-<br />
| <code>dispatch.refine</code> || False || Run an explicit model-refinement step after indexing (the stills indexer already refines; this is an additional pass, rarely needed).<br />
|-<br />
| <code>dispatch.squash_errors</code> || True || If an image fails, log it and continue to the next image. Set False to halt and show the full error when debugging.<br />
|-<br />
| <code>dispatch.hit_finder.enable</code> || True || Enable hit-finding. If disabled, every image is passed to indexing.<br />
|-<br />
| <code>dispatch.hit_finder.minimum_number_of_reflections</code> || 16 || Images with fewer strong spots than this are discarded as non-hits before indexing.<br />
|-<br />
| <code>dispatch.hit_finder.maximum_number_of_reflections</code> || None || If set, images with more strong spots than this are discarded (e.g. ice/junk).<br />
|-<br />
| <code>dispatch.process_percent</code> || None || Process only this percentage of events (evenly sampled). Useful for a quick look at a large dataset.<br />
|-<br />
| <code>dispatch.coset</code> || False || During integration, also integrate a sublattice coset as a negative control for regions with no Bragg diffraction (see <code>integration.coset.transformation</code>).<br />
|-<br />
| <code>dispatch.pre_import</code> || False || Import all data before processing. Only needed when processing multiple ''multi-image'' files at once (not a recommended workflow). Single-file datasets (e.g. HDF5) are always pre-imported automatically.<br />
|}<br />
<br />
=== Spot-finding ===<br />
<br />
Spot-finding uses the standard DIALS dispersion thresholding. The most impactful parameters for<br />
serial data:<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>spotfinder.filter.min_spot_size</code> || Auto || Minimum connected pixels for a spot. Auto = 3 for pixel-array detectors (SENSOR_PAD), 6 otherwise. Often lowered to 2 for sparse serial patterns.<br />
|-<br />
| <code>spotfinder.threshold.dispersion.gain</code> || None || Detector gain used in thresholding. Set this ''or'' the detector-model gain, not both (see gain note below).<br />
|-<br />
| <code>spotfinder.threshold.dispersion.sigma_strong</code> || 3 || Number of sigma above background for a pixel to be "strong".<br />
|-<br />
| <code>spotfinder.threshold.dispersion.sigma_background</code> || 6 || Number of sigma for the background/dispersion test.<br />
|-<br />
| <code>spotfinder.threshold.dispersion.global_threshold</code> || 0 || Absolute pixel intensity floor; pixels below this are never considered strong. Raising it (e.g. 100) suppresses noise on many detectors.<br />
|-<br />
| <code>spotfinder.threshold.dispersion.kernel_size</code> || 3 3 || Local-area kernel (pixels) for the dispersion calculation.<br />
|}<br />
<br />
'''Gain note:''' the detector gain should be set on the detector model ''or'' in the processing<br />
parameters, never both. If a supplied gain (<code>spotfinder.threshold.dispersion.gain</code> or<br />
<code>integration.summation.detector_gain</code>) conflicts with a non-unity gain already on the<br />
detector model, the program raises an error. Override with<br />
<code>input.ignore_gain_mismatch=True</code> only if you understand the consequences.<br />
<br />
=== Indexing ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>indexing.known_symmetry.space_group</code> || None || Target space group. Strongly recommended for serial data — constrains indexing and improves the hit/index rate.<br />
|-<br />
| <code>indexing.known_symmetry.unit_cell</code> || None || Target unit cell (a,b,c,α,β,γ). Used together with the space group as the indexing target.<br />
|-<br />
| <code>indexing.method</code> || fft1d || Basis-vector search method (<code>dials.stills_process</code> defaults to <code>fft1d</code>; <code>fft3d</code> and <code>real_space_grid_search</code> are also available).<br />
|-<br />
| <code>indexing.stills.method_list</code> || None || Ordered list of indexing methods to try in turn; if one fails, the next is attempted (e.g. <code>fft1d real_space_grid_search</code>).<br />
|-<br />
| <code>indexing.stills.known_orientations</code> || None || Paths to previously-processed experiment lists. If given, images are ''not'' re-indexed; the stored crystal orientations are reused (e.g. to re-integrate with new parameters). Supports wildcards; may be repeated.<br />
|-<br />
| <code>indexing.stills.require_known_orientation</code> || False || When known orientations are supplied but an image is not found among them, controls whether to skip it (True) or fall back to indexing from scratch (False).<br />
|-<br />
| <code>indexing.stills.reflection_subsampling.enable</code> || False || Retry indexing on random subsets of the strong spots (from <code>step_start</code>% down to <code>step_stop</code>% in <code>step_size</code>% steps). Can rescue images that fail to index on the full spot list. Especially valuble for multi-lattice images (multiple crystals in the beam) but does increase execution time!<br />
|}<br />
<br />
=== Integration ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>profile.gaussian_rs.parameters.sigma_b_cutoff</code> || 0.1 || Maximum beam divergence (sigma_b, degrees) before an indexed crystal is rejected prior to integration. Guards against over-broad, poorly-modelled lattices.<br />
|-<br />
| <code>significance_filter.enable</code> || False || Apply the stills significance filter, which trims each crystal to the resolution at which reflections remain statistically significant (see <code>significance_filter.isigi_cutoff</code>, default 1.0). Defaults to false here because it is typically applied in ''[[cctbx.xfel.merge]]''.<br />
|-<br />
| <code>integration.integration_only_overrides.trusted_range</code> || None || Override the panel trusted range [min, max] during integration only (e.g. to include pixels the header marks as untrusted).<br />
|-<br />
| <code>integration.coset.transformation</code> || 6 || Which modulus-2 sublattice transformation defines the coset negative control when <code>dispatch.coset=True</code> (0–6; 6 = body centering). See Sauter & Zwart, ''Acta Cryst.'' D (2009) '''65''':553.<br />
|}<br />
<br />
=== Output ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>output.output_dir</code> || <code>.</code> || Directory for output files.<br />
|-<br />
| <code>output.composite_output</code> || True || If True, each process writes ''one'' concatenated set of <code>.expt</code>/<code>.refl</code> files covering all its successful images. If False, a separate pair of files is written per image (generates a very large number of files). Composite output is strongly preferred for large serial datasets.<br />
|-<br />
| <code>output.logging_dir</code> || None || Directory for log files. Under MPI, per-rank logs (<code>log_rankNNNN.out</code>, etc.) are written here.<br />
|-<br />
| <code>output.logging_option</code> || normal || <code>normal</code> (all logging), <code>suppressed</code> (silences verbose DIALS refine/index output), or <code>disabled</code> (almost no logging).<br />
|-<br />
| <code>output.*_filename</code> || (templates) || Output filename templates for each stage: <code>strong</code>, <code>indexed</code>, <code>refined_experiments</code>, <code>integrated</code>, <code>integrated_experiments</code>, <code>coset</code>. A <code>%s</code> in the template is replaced with the process number for composite mode or the per-image tag for non-composite mode.<br />
|-<br />
| <code>output.integration_pickle</code> || None || Write legacy <code>cxi.merge</code>/PRIME integration pickle files (e.g. <code>int-%d-%s.pickle</code>). Only needed for legacy merging workflows.<br />
|}<br />
<br />
=== Multiprocessing ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>mp.method</code> || multiprocessing || Parallelization backend: <code>multiprocessing</code> (single node), or <code>mpi</code> for multi-node jobs (also <code>sge</code>, <code>lsf</code>, <code>pbs</code>).<br />
|-<br />
| <code>mp.nproc</code> || 1 || Number of processes (for the <code>multiprocessing</code> method). Set to the number of cores per node.<br />
|-<br />
| <code>mp.composite_stride</code> || None || For MPI composite output, how many ranks aggregate their results into one set of output files. Ideally set to the number of processes per node so that one composite file set is written per node rather than per rank.<br />
|}<br />
<br />
Under MPI with more than two ranks, the program runs a client/server model: rank 0 (or rank 2 for<br />
psana2 <code>.loc</code> inputs) hands out images to worker ranks on demand, giving good load<br />
balancing across the many fast-failing non-hits typical of serial data.<br />
<br />
== Output files ==<br />
<br />
With the default composite output, each process (or each aggregating rank under MPI) writes<br />
concatenated files such as:<br />
<br />
* <code>&lt;tag&gt;_strong.refl</code> — strong spots from spot-finding<br />
* <code>&lt;tag&gt;_indexed.refl</code> and <code>&lt;tag&gt;_refined.expt</code> — indexed reflections and their crystal models<br />
* <code>&lt;tag&gt;_integrated.refl</code> and <code>&lt;tag&gt;_integrated.expt</code> — integrated intensities and final models (the input to [[cctbx.xfel.merge]])<br />
<br />
Per-image progress and status codes are recorded in <code>debug/debug_&lt;rank&gt;.txt</code>, and<br />
overall logging goes to <code>dials.process.log</code> (or per-rank logs under MPI). These integrated<br />
<code>.expt</code>/<code>.refl</code> pairs are then scaled and merged with<br />
[[cctbx.xfel.merge]].<br />
<br />
== See also ==<br />
<br />
* [[cctbx.xfel.merge]] — scaling, post-refinement and merging of the integrated data<br />
* [[IOTA]] — a front-end that supplies preset parameters for <code>dials.stills_process</code><br />
* [[Data Processing at SACLA]] — SACLA-specific processing workflow<br />
* The DIALS documentation for the underlying spot-finding, indexing, refinement and integration algorithms</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Dials.stills_process&diff=1764Dials.stills process2026-08-21T22:30:49Z<p>Aaron: /* Basic usage */</p>
<hr />
<div>= dials.stills_process =<br />
<br />
'''''dials.stills_process''''' is a stand-alone program for processing still diffraction images and is<br />
the core data-reduction program of ''cctbx.xfel''. It takes a set of<br />
still diffraction images — typically serial femtosecond crystallography (SFX) snapshots from an<br />
XFEL (LCLS, SACLA, EuXFEL) or still shots from a synchrotron — and analyzes each one, independently,<br />
through the full DIALS pipeline: '''import → spot-finding → indexing → refinement → integration'''.<br />
Its output (integrated intensities together with the per-image experimental models) is the input to<br />
the merging program [[cctbx.xfel.merge]].<br />
<br />
Because every image is a separate still exposure of a different crystal (rather than a wedge of<br />
a rotation series), each image is treated as its own experiment with its own crystal model. This<br />
"one crystal per shot" model is what distinguishes stills processing from conventional rotation data<br />
processing (e.g. <code>dials.import</code>, <code>dials.find_spots</code>, <code>dials.index</code>, etc.), <br />
and it drives most of the stills-specific defaults described below.<br />
<br />
== Overview ==<br />
<br />
For each image, <code>dials.stills_process</code> runs the following stages in order. Any stage can<br />
be turned off with the <code>dispatch</code> parameters; if a stage is disabled, processing stops<br />
after the last enabled stage.<br />
<br />
# '''Import''' — read the image, apply reference geometry / gain / mask overrides, and build an experiment.<br />
# '''Spot-finding''' — locate strong pixels and build strong reflections (<code>dispatch.find_spots</code>).<br />
# '''Hit finding''' — reject images with too few (or optionally too many) strong spots, so that only "hits" are indexed (<code>dispatch.hit_finder</code>).<br />
# '''Indexing''' — determine the crystal orientation and (optionally constrained) unit cell (<code>dispatch.index</code>).<br />
# '''Refinement''' — refine the crystal model and re-estimate the mosaic/domain-size parameters. Detector and beam are held fixed by default (<code>dispatch.refine</code>).<br />
# '''Integration''' — predict reflections and measure their intensities using the stills integrator (<code>dispatch.integrate</code>), with optional Kapton absorption correction and a significance filter.<br />
<br />
The program applies a set of stills-specific defaults on top of the standard DIALS parameters,<br />
including: indexing method <code>fft1d</code>; the stills integrator with profile fitting turned off<br />
(summation integration with a 2-D plane background); the stills refinement weighting strategy with<br />
the detector and beam models fixed; and a re-estimation of per-crystal mosaic parameters<br />
(domain size and mosaic angle) after refinement. These are sensible starting points for serial data<br />
and normally do not need to be changed.<br />
<br />
Failures on individual images are, by default, caught and logged so that processing continues to the<br />
next image (<code>dispatch.squash_errors=True</code>) — essential when reducing tens of thousands of<br />
shots of which only a fraction are hits.<br />
<br />
== Basic usage ==<br />
<br />
<pre><br />
# Process a directory of CBF stills, constraining the target cell/symmetry,<br />
# using 32 parallel processes on one node<br />
dials.stills_process /path/to/images/*.cbf process.phil mp.nproc=32<br />
</pre><br />
<br />
A typical <code>process.phil</code> for XFEL data looks like:<br />
<br />
<pre><br />
spotfinder.filter.min_spot_size=2<br />
spotfinder.threshold.dispersion.gain=1.0<br />
spotfinder.threshold.dispersion.global_threshold=100<br />
<br />
indexing.known_symmetry.space_group=P43212<br />
indexing.known_symmetry.unit_cell=78.9,78.9,38.1,90,90,90<br />
indexing.method=fft1d<br />
<br />
input.reference_geometry=refined.expt # from a previous metrology refinement<br />
<br />
output.output_dir=out<br />
mp.nproc=32<br />
</pre><br />
<br />
Useful command-line helpers:<br />
* <code>dials.stills_process -h</code> — help and usage<br />
* <code>dials.stills_process -c -e 2</code> — show the configurable parameters (expert level 2)<br />
* <code>dials.stills_process -c -a2</code> — include attributes (help text) for each parameter<br />
* <code>dials.stills_process ... input.show_image_tags=True</code> — list the image tags in a dataset without processing (useful for selecting subsets with <code>input.image_tag</code>)<br />
<br />
== Important parameters ==<br />
<br />
The program exposes the entire DIALS parameter set (spot-finding, indexing, refinement, integration,<br />
profile modelling, prediction). The parameters below are the ones most commonly adjusted for serial<br />
still data. Parameters are given by their full PHIL path with the default value in parentheses.<br />
<br />
=== Input ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>input.reference_geometry</code> || None || Path to an <code>.expt</code> file (or image) with a single detector model. Data are processed using ''this'' geometry instead of the geometry in the image headers. This is how a refined detector metrology from <code>cctbx.xfel</code> ensemble refinement is applied — almost always set for XFEL data.<br />
|-<br />
| <code>input.sync_reference_geom</code> || True || When a reference geometry is supplied, copy its refined local frames onto the image's detector hierarchy node-by-node (the hierarchies must match in shape). If False, the reference detector replaces the image detector wholesale.<br />
|-<br />
| <code>input.file_list</code> || None || Path to a text file listing image paths, one per line (alternative to listing files on the command line).<br />
|-<br />
| <code>input.glob</code> || None || Wildcard path(s) for large multi-file datasets (e.g. <code>*.cbf</code>). May be given multiple times.<br />
|-<br />
| <code>input.image_tag</code> || None || Process only images with the given tag(s). For single-image files the tag is the filename; for multi-image files (HDF5) it is <code>filename_imagenumber</code>. Use <code>input.show_image_tags=True</code> to list them.<br />
|-<br />
| <code>input.max_images</code> || None || Limit the total number of images processed.<br />
|}<br />
<br />
=== Dispatch — controlling the pipeline ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>dispatch.find_spots</code> || True || Do spot-finding. Required for indexing/integration.<br />
|-<br />
| <code>dispatch.index</code> || True || Attempt to index. Requires <code>find_spots=True</code>.<br />
|-<br />
| <code>dispatch.integrate</code> || True || Integrate indexed images. Ignored if indexing/spot-finding is off.<br />
|-<br />
| <code>dispatch.refine</code> || False || Run an explicit model-refinement step after indexing (the stills indexer already refines; this is an additional pass, rarely needed).<br />
|-<br />
| <code>dispatch.squash_errors</code> || True || If an image fails, log it and continue to the next image. Set False to halt and show the full error when debugging.<br />
|-<br />
| <code>dispatch.hit_finder.enable</code> || True || Enable hit-finding. If disabled, every image is passed to indexing.<br />
|-<br />
| <code>dispatch.hit_finder.minimum_number_of_reflections</code> || 16 || Images with fewer strong spots than this are discarded as non-hits before indexing.<br />
|-<br />
| <code>dispatch.hit_finder.maximum_number_of_reflections</code> || None || If set, images with more strong spots than this are discarded (e.g. ice/junk).<br />
|-<br />
| <code>dispatch.process_percent</code> || None || Process only this percentage of events (evenly sampled). Useful for a quick look at a large dataset.<br />
|-<br />
| <code>dispatch.coset</code> || False || During integration, also integrate a sublattice coset as a negative control for regions with no Bragg diffraction (see <code>integration.coset.transformation</code>).<br />
|-<br />
| <code>dispatch.pre_import</code> || False || Import all data before processing. Only needed when processing multiple ''multi-image'' files at once (not a recommended workflow). Single-file datasets are always pre-imported automatically.<br />
|}<br />
<br />
=== Spot-finding ===<br />
<br />
Spot-finding uses the standard DIALS dispersion thresholding. The most impactful parameters for<br />
serial data:<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>spotfinder.filter.min_spot_size</code> || Auto || Minimum connected pixels for a spot. Auto = 3 for pixel-array detectors (SENSOR_PAD), 6 otherwise. Often lowered to 2 for sparse serial patterns.<br />
|-<br />
| <code>spotfinder.threshold.dispersion.gain</code> || None || Detector gain used in thresholding. Set this ''or'' the detector-model gain, not both (see gain note below).<br />
|-<br />
| <code>spotfinder.threshold.dispersion.sigma_strong</code> || 3 || Number of sigma above background for a pixel to be "strong".<br />
|-<br />
| <code>spotfinder.threshold.dispersion.sigma_background</code> || 6 || Number of sigma for the background/dispersion test.<br />
|-<br />
| <code>spotfinder.threshold.dispersion.global_threshold</code> || 0 || Absolute pixel intensity floor; pixels below this are never considered strong. Raising it (e.g. 100) suppresses noise on many detectors.<br />
|-<br />
| <code>spotfinder.threshold.dispersion.kernel_size</code> || 3 3 || Local-area kernel (pixels) for the dispersion calculation.<br />
|}<br />
<br />
'''Gain note:''' the detector gain should be set on the detector model ''or'' in the processing<br />
parameters, never both. If a supplied gain (<code>spotfinder.threshold.dispersion.gain</code> or<br />
<code>integration.summation.detector_gain</code>) conflicts with a non-unity gain already on the<br />
detector model, the program raises an error. Override with<br />
<code>input.ignore_gain_mismatch=True</code> only if you understand the consequences.<br />
<br />
=== Indexing ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>indexing.known_symmetry.space_group</code> || None || Target space group. Strongly recommended for serial data — constrains indexing and improves the hit/index rate.<br />
|-<br />
| <code>indexing.known_symmetry.unit_cell</code> || None || Target unit cell (a,b,c,α,β,γ). Used together with the space group as the indexing target.<br />
|-<br />
| <code>indexing.method</code> || fft1d || Basis-vector search method (<code>dials.stills_process</code> defaults to <code>fft1d</code>; <code>fft3d</code> and <code>real_space_grid_search</code> are also available).<br />
|-<br />
| <code>indexing.stills.method_list</code> || None || Ordered list of indexing methods to try in turn; if one fails, the next is attempted (e.g. <code>fft1d real_space_grid_search</code>).<br />
|-<br />
| <code>indexing.stills.known_orientations</code> || None || Paths to previously-processed experiment lists. If given, images are ''not'' re-indexed; the stored crystal orientations are reused (e.g. to re-integrate with new parameters). Supports wildcards; may be repeated.<br />
|-<br />
| <code>indexing.stills.require_known_orientation</code> || False || When known orientations are supplied but an image is not found among them, controls whether to skip it (True) or fall back to indexing from scratch (False).<br />
|-<br />
| <code>indexing.stills.reflection_subsampling.enable</code> || False || Retry indexing on random subsets of the strong spots (from <code>step_start</code>% down to <code>step_stop</code>% in <code>step_size</code>% steps). Can rescue images that fail to index on the full spot list.<br />
|}<br />
<br />
=== Integration ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>profile.gaussian_rs.parameters.sigma_b_cutoff</code> || 0.1 || Maximum beam divergence (sigma_b, degrees) before an indexed crystal is rejected prior to integration. Guards against over-broad, poorly-modelled lattices.<br />
|-<br />
| <code>significance_filter.enable</code> || False || Apply the stills significance filter, which trims each crystal to the resolution at which reflections remain statistically significant (see <code>significance_filter.isigi_cutoff</code>, default 1.0).<br />
|-<br />
| <code>integration.integration_only_overrides.trusted_range</code> || None || Override the panel trusted range [min, max] during integration only (e.g. to include pixels the header marks as untrusted).<br />
|-<br />
| <code>integration.absorption_correction</code> || apply=False || Enable a Kapton absorption correction (<code>algorithm=fuller_kapton</code> or <code>kapton_2019</code>). Requires shoeboxes to be retained via <code>integration.debug</code> settings; the program checks this and errors otherwise.<br />
|-<br />
| <code>integration.coset.transformation</code> || 6 || Which modulus-2 sublattice transformation defines the coset negative control when <code>dispatch.coset=True</code> (0–6; 6 = body centering). See Sauter & Zwart, ''Acta Cryst.'' D (2009) '''65''':553.<br />
|}<br />
<br />
=== Output ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>output.output_dir</code> || <code>.</code> || Directory for output files.<br />
|-<br />
| <code>output.composite_output</code> || True || If True, each process writes ''one'' concatenated set of <code>.expt</code>/<code>.refl</code> files covering all its successful images. If False, a separate pair of files is written per image (generates a very large number of files). Composite output is strongly preferred for large serial datasets.<br />
|-<br />
| <code>output.logging_dir</code> || None || Directory for log files. Under MPI, per-rank logs (<code>log_rankNNNN.out</code>, etc.) are written here.<br />
|-<br />
| <code>output.logging_option</code> || normal || <code>normal</code> (all logging), <code>suppressed</code> (silences verbose DIALS refine/index output), or <code>disabled</code> (almost no logging).<br />
|-<br />
| <code>output.*_filename</code> || (templates) || Output filename templates for each stage: <code>strong</code>, <code>indexed</code>, <code>refined_experiments</code>, <code>integrated</code>, <code>integrated_experiments</code>, <code>coset</code>. A <code>%s</code> in the template is replaced with the per-image tag.<br />
|-<br />
| <code>output.integration_pickle</code> || None || Write legacy <code>cxi.merge</code>/PRIME integration pickle files (e.g. <code>int-%d-%s.pickle</code>). Only needed for legacy merging workflows.<br />
|}<br />
<br />
=== Multiprocessing ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>mp.method</code> || multiprocessing || Parallelization backend: <code>multiprocessing</code> (single node), or <code>mpi</code> for multi-node jobs (also <code>sge</code>, <code>lsf</code>, <code>pbs</code>).<br />
|-<br />
| <code>mp.nproc</code> || 1 || Number of processes (for the <code>multiprocessing</code> method). Set to the number of cores per node.<br />
|-<br />
| <code>mp.composite_stride</code> || None || For MPI composite output, how many ranks aggregate their results into one set of output files. Ideally set to the number of processes per node so that one composite file set is written per node rather than per rank.<br />
|}<br />
<br />
Under MPI with more than two ranks, the program runs a client/server model: rank 0 (or rank 2 for<br />
psana2 <code>.loc</code> inputs) hands out images to worker ranks on demand, giving good load<br />
balancing across the many fast-failing non-hits typical of serial data.<br />
<br />
== Output files ==<br />
<br />
With the default composite output, each process (or each aggregating rank under MPI) writes<br />
concatenated files such as:<br />
<br />
* <code>&lt;tag&gt;_strong.refl</code> — strong spots from spot-finding<br />
* <code>&lt;tag&gt;_indexed.refl</code> and <code>&lt;tag&gt;_refined.expt</code> — indexed reflections and their crystal models<br />
* <code>&lt;tag&gt;_integrated.refl</code> and <code>&lt;tag&gt;_integrated.expt</code> — integrated intensities and final models (the input to [[cctbx.xfel.merge]])<br />
<br />
Per-image progress and status codes are recorded in <code>debug/debug_&lt;rank&gt;.txt</code>, and<br />
overall logging goes to <code>dials.process.log</code> (or per-rank logs under MPI). These integrated<br />
<code>.expt</code>/<code>.refl</code> pairs are then scaled and merged with<br />
[[cctbx.xfel.merge]].<br />
<br />
== See also ==<br />
<br />
* [[cctbx.xfel.merge]] — scaling, post-refinement and merging of the integrated data<br />
* [[IOTA]] — a front-end that supplies preset parameters for <code>dials.stills_process</code><br />
* [[Data Processing at SACLA]] — SACLA-specific processing workflow<br />
* The DIALS documentation for the underlying spot-finding, indexing, refinement and integration algorithms</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Dials.stills_process&diff=1763Dials.stills process2026-08-21T22:26:16Z<p>Aaron: /* dials.stills_process */</p>
<hr />
<div>= dials.stills_process =<br />
<br />
'''''dials.stills_process''''' is a stand-alone program for processing still diffraction images and is<br />
the core data-reduction program of ''cctbx.xfel''. It takes a set of<br />
still diffraction images — typically serial femtosecond crystallography (SFX) snapshots from an<br />
XFEL (LCLS, SACLA, EuXFEL) or still shots from a synchrotron — and analyzes each one, independently,<br />
through the full DIALS pipeline: '''import → spot-finding → indexing → refinement → integration'''.<br />
Its output (integrated intensities together with the per-image experimental models) is the input to<br />
the merging program [[cctbx.xfel.merge]].<br />
<br />
Because every image is a separate still exposure of a different crystal (rather than a wedge of<br />
a rotation series), each image is treated as its own experiment with its own crystal model. This<br />
"one crystal per shot" model is what distinguishes stills processing from conventional rotation data<br />
processing (e.g. <code>dials.import</code>, <code>dials.find_spots</code>, <code>dials.index</code>, etc.), <br />
and it drives most of the stills-specific defaults described below.<br />
<br />
== Overview ==<br />
<br />
For each image, <code>dials.stills_process</code> runs the following stages in order. Any stage can<br />
be turned off with the <code>dispatch</code> parameters; if a stage is disabled, processing stops<br />
after the last enabled stage.<br />
<br />
# '''Import''' — read the image, apply reference geometry / gain / mask overrides, and build an experiment.<br />
# '''Spot-finding''' — locate strong pixels and build strong reflections (<code>dispatch.find_spots</code>).<br />
# '''Hit finding''' — reject images with too few (or optionally too many) strong spots, so that only "hits" are indexed (<code>dispatch.hit_finder</code>).<br />
# '''Indexing''' — determine the crystal orientation and (optionally constrained) unit cell (<code>dispatch.index</code>).<br />
# '''Refinement''' — refine the crystal model and re-estimate the mosaic/domain-size parameters. Detector and beam are held fixed by default (<code>dispatch.refine</code>).<br />
# '''Integration''' — predict reflections and measure their intensities using the stills integrator (<code>dispatch.integrate</code>), with optional Kapton absorption correction and a significance filter.<br />
<br />
The program applies a set of stills-specific defaults on top of the standard DIALS parameters,<br />
including: indexing method <code>fft1d</code>; the stills integrator with profile fitting turned off<br />
(summation integration with a 2-D plane background); the stills refinement weighting strategy with<br />
the detector and beam models fixed; and a re-estimation of per-crystal mosaic parameters<br />
(domain size and mosaic angle) after refinement. These are sensible starting points for serial data<br />
and normally do not need to be changed.<br />
<br />
Failures on individual images are, by default, caught and logged so that processing continues to the<br />
next image (<code>dispatch.squash_errors=True</code>) — essential when reducing tens of thousands of<br />
shots of which only a fraction are hits.<br />
<br />
== Basic usage ==<br />
<br />
<pre><br />
# Process a directory of CBF stills, constraining the target cell/symmetry,<br />
# using 32 parallel processes on one node<br />
dials.stills_process /path/to/images/*.cbf process.phil mp.nproc=32<br />
</pre><br />
<br />
A typical <code>process.phil</code> for XFEL data looks like:<br />
<br />
<pre><br />
spotfinder.filter.min_spot_size=2<br />
spotfinder.threshold.dispersion.gain=1.0<br />
spotfinder.threshold.dispersion.global_threshold=100<br />
<br />
indexing.known_symmetry.space_group=P43212<br />
indexing.known_symmetry.unit_cell=78.9,78.9,38.1,90,90,90<br />
indexing.method=fft1d<br />
<br />
integration.background.simple.outlier.algorithm=null<br />
<br />
input.reference_geometry=refined.expt # from a previous metrology refinement<br />
<br />
output.composite_output=True<br />
output.output_dir=out<br />
mp.nproc=32<br />
</pre><br />
<br />
Useful command-line helpers:<br />
* <code>dials.stills_process -h</code> — help and usage<br />
* <code>dials.stills_process -c -e 2</code> — show the configurable parameters (expert level 2)<br />
* <code>dials.stills_process -c -a2</code> — include attributes (help text) for each parameter<br />
* <code>dials.stills_process ... input.show_image_tags=True</code> — list the image tags in a dataset without processing (useful for selecting subsets with <code>input.image_tag</code>)<br />
<br />
== Important parameters ==<br />
<br />
The program exposes the entire DIALS parameter set (spot-finding, indexing, refinement, integration,<br />
profile modelling, prediction). The parameters below are the ones most commonly adjusted for serial<br />
still data. Parameters are given by their full PHIL path with the default value in parentheses.<br />
<br />
=== Input ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>input.reference_geometry</code> || None || Path to an <code>.expt</code> file (or image) with a single detector model. Data are processed using ''this'' geometry instead of the geometry in the image headers. This is how a refined detector metrology from <code>cctbx.xfel</code> ensemble refinement is applied — almost always set for XFEL data.<br />
|-<br />
| <code>input.sync_reference_geom</code> || True || When a reference geometry is supplied, copy its refined local frames onto the image's detector hierarchy node-by-node (the hierarchies must match in shape). If False, the reference detector replaces the image detector wholesale.<br />
|-<br />
| <code>input.file_list</code> || None || Path to a text file listing image paths, one per line (alternative to listing files on the command line).<br />
|-<br />
| <code>input.glob</code> || None || Wildcard path(s) for large multi-file datasets (e.g. <code>*.cbf</code>). May be given multiple times.<br />
|-<br />
| <code>input.image_tag</code> || None || Process only images with the given tag(s). For single-image files the tag is the filename; for multi-image files (HDF5) it is <code>filename_imagenumber</code>. Use <code>input.show_image_tags=True</code> to list them.<br />
|-<br />
| <code>input.max_images</code> || None || Limit the total number of images processed.<br />
|}<br />
<br />
=== Dispatch — controlling the pipeline ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>dispatch.find_spots</code> || True || Do spot-finding. Required for indexing/integration.<br />
|-<br />
| <code>dispatch.index</code> || True || Attempt to index. Requires <code>find_spots=True</code>.<br />
|-<br />
| <code>dispatch.integrate</code> || True || Integrate indexed images. Ignored if indexing/spot-finding is off.<br />
|-<br />
| <code>dispatch.refine</code> || False || Run an explicit model-refinement step after indexing (the stills indexer already refines; this is an additional pass, rarely needed).<br />
|-<br />
| <code>dispatch.squash_errors</code> || True || If an image fails, log it and continue to the next image. Set False to halt and show the full error when debugging.<br />
|-<br />
| <code>dispatch.hit_finder.enable</code> || True || Enable hit-finding. If disabled, every image is passed to indexing.<br />
|-<br />
| <code>dispatch.hit_finder.minimum_number_of_reflections</code> || 16 || Images with fewer strong spots than this are discarded as non-hits before indexing.<br />
|-<br />
| <code>dispatch.hit_finder.maximum_number_of_reflections</code> || None || If set, images with more strong spots than this are discarded (e.g. ice/junk).<br />
|-<br />
| <code>dispatch.process_percent</code> || None || Process only this percentage of events (evenly sampled). Useful for a quick look at a large dataset.<br />
|-<br />
| <code>dispatch.coset</code> || False || During integration, also integrate a sublattice coset as a negative control for regions with no Bragg diffraction (see <code>integration.coset.transformation</code>).<br />
|-<br />
| <code>dispatch.pre_import</code> || False || Import all data before processing. Only needed when processing multiple ''multi-image'' files at once (not a recommended workflow). Single-file datasets are always pre-imported automatically.<br />
|}<br />
<br />
=== Spot-finding ===<br />
<br />
Spot-finding uses the standard DIALS dispersion thresholding. The most impactful parameters for<br />
serial data:<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>spotfinder.filter.min_spot_size</code> || Auto || Minimum connected pixels for a spot. Auto = 3 for pixel-array detectors (SENSOR_PAD), 6 otherwise. Often lowered to 2 for sparse serial patterns.<br />
|-<br />
| <code>spotfinder.threshold.dispersion.gain</code> || None || Detector gain used in thresholding. Set this ''or'' the detector-model gain, not both (see gain note below).<br />
|-<br />
| <code>spotfinder.threshold.dispersion.sigma_strong</code> || 3 || Number of sigma above background for a pixel to be "strong".<br />
|-<br />
| <code>spotfinder.threshold.dispersion.sigma_background</code> || 6 || Number of sigma for the background/dispersion test.<br />
|-<br />
| <code>spotfinder.threshold.dispersion.global_threshold</code> || 0 || Absolute pixel intensity floor; pixels below this are never considered strong. Raising it (e.g. 100) suppresses noise on many detectors.<br />
|-<br />
| <code>spotfinder.threshold.dispersion.kernel_size</code> || 3 3 || Local-area kernel (pixels) for the dispersion calculation.<br />
|}<br />
<br />
'''Gain note:''' the detector gain should be set on the detector model ''or'' in the processing<br />
parameters, never both. If a supplied gain (<code>spotfinder.threshold.dispersion.gain</code> or<br />
<code>integration.summation.detector_gain</code>) conflicts with a non-unity gain already on the<br />
detector model, the program raises an error. Override with<br />
<code>input.ignore_gain_mismatch=True</code> only if you understand the consequences.<br />
<br />
=== Indexing ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>indexing.known_symmetry.space_group</code> || None || Target space group. Strongly recommended for serial data — constrains indexing and improves the hit/index rate.<br />
|-<br />
| <code>indexing.known_symmetry.unit_cell</code> || None || Target unit cell (a,b,c,α,β,γ). Used together with the space group as the indexing target.<br />
|-<br />
| <code>indexing.method</code> || fft1d || Basis-vector search method (<code>dials.stills_process</code> defaults to <code>fft1d</code>; <code>fft3d</code> and <code>real_space_grid_search</code> are also available).<br />
|-<br />
| <code>indexing.stills.method_list</code> || None || Ordered list of indexing methods to try in turn; if one fails, the next is attempted (e.g. <code>fft1d real_space_grid_search</code>).<br />
|-<br />
| <code>indexing.stills.known_orientations</code> || None || Paths to previously-processed experiment lists. If given, images are ''not'' re-indexed; the stored crystal orientations are reused (e.g. to re-integrate with new parameters). Supports wildcards; may be repeated.<br />
|-<br />
| <code>indexing.stills.require_known_orientation</code> || False || When known orientations are supplied but an image is not found among them, controls whether to skip it (True) or fall back to indexing from scratch (False).<br />
|-<br />
| <code>indexing.stills.reflection_subsampling.enable</code> || False || Retry indexing on random subsets of the strong spots (from <code>step_start</code>% down to <code>step_stop</code>% in <code>step_size</code>% steps). Can rescue images that fail to index on the full spot list.<br />
|}<br />
<br />
=== Integration ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>profile.gaussian_rs.parameters.sigma_b_cutoff</code> || 0.1 || Maximum beam divergence (sigma_b, degrees) before an indexed crystal is rejected prior to integration. Guards against over-broad, poorly-modelled lattices.<br />
|-<br />
| <code>significance_filter.enable</code> || False || Apply the stills significance filter, which trims each crystal to the resolution at which reflections remain statistically significant (see <code>significance_filter.isigi_cutoff</code>, default 1.0).<br />
|-<br />
| <code>integration.integration_only_overrides.trusted_range</code> || None || Override the panel trusted range [min, max] during integration only (e.g. to include pixels the header marks as untrusted).<br />
|-<br />
| <code>integration.absorption_correction</code> || apply=False || Enable a Kapton absorption correction (<code>algorithm=fuller_kapton</code> or <code>kapton_2019</code>). Requires shoeboxes to be retained via <code>integration.debug</code> settings; the program checks this and errors otherwise.<br />
|-<br />
| <code>integration.coset.transformation</code> || 6 || Which modulus-2 sublattice transformation defines the coset negative control when <code>dispatch.coset=True</code> (0–6; 6 = body centering). See Sauter & Zwart, ''Acta Cryst.'' D (2009) '''65''':553.<br />
|}<br />
<br />
=== Output ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>output.output_dir</code> || <code>.</code> || Directory for output files.<br />
|-<br />
| <code>output.composite_output</code> || True || If True, each process writes ''one'' concatenated set of <code>.expt</code>/<code>.refl</code> files covering all its successful images. If False, a separate pair of files is written per image (generates a very large number of files). Composite output is strongly preferred for large serial datasets.<br />
|-<br />
| <code>output.logging_dir</code> || None || Directory for log files. Under MPI, per-rank logs (<code>log_rankNNNN.out</code>, etc.) are written here.<br />
|-<br />
| <code>output.logging_option</code> || normal || <code>normal</code> (all logging), <code>suppressed</code> (silences verbose DIALS refine/index output), or <code>disabled</code> (almost no logging).<br />
|-<br />
| <code>output.*_filename</code> || (templates) || Output filename templates for each stage: <code>strong</code>, <code>indexed</code>, <code>refined_experiments</code>, <code>integrated</code>, <code>integrated_experiments</code>, <code>coset</code>. A <code>%s</code> in the template is replaced with the per-image tag.<br />
|-<br />
| <code>output.integration_pickle</code> || None || Write legacy <code>cxi.merge</code>/PRIME integration pickle files (e.g. <code>int-%d-%s.pickle</code>). Only needed for legacy merging workflows.<br />
|}<br />
<br />
=== Multiprocessing ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>mp.method</code> || multiprocessing || Parallelization backend: <code>multiprocessing</code> (single node), or <code>mpi</code> for multi-node jobs (also <code>sge</code>, <code>lsf</code>, <code>pbs</code>).<br />
|-<br />
| <code>mp.nproc</code> || 1 || Number of processes (for the <code>multiprocessing</code> method). Set to the number of cores per node.<br />
|-<br />
| <code>mp.composite_stride</code> || None || For MPI composite output, how many ranks aggregate their results into one set of output files. Ideally set to the number of processes per node so that one composite file set is written per node rather than per rank.<br />
|}<br />
<br />
Under MPI with more than two ranks, the program runs a client/server model: rank 0 (or rank 2 for<br />
psana2 <code>.loc</code> inputs) hands out images to worker ranks on demand, giving good load<br />
balancing across the many fast-failing non-hits typical of serial data.<br />
<br />
== Output files ==<br />
<br />
With the default composite output, each process (or each aggregating rank under MPI) writes<br />
concatenated files such as:<br />
<br />
* <code>&lt;tag&gt;_strong.refl</code> — strong spots from spot-finding<br />
* <code>&lt;tag&gt;_indexed.refl</code> and <code>&lt;tag&gt;_refined.expt</code> — indexed reflections and their crystal models<br />
* <code>&lt;tag&gt;_integrated.refl</code> and <code>&lt;tag&gt;_integrated.expt</code> — integrated intensities and final models (the input to [[cctbx.xfel.merge]])<br />
<br />
Per-image progress and status codes are recorded in <code>debug/debug_&lt;rank&gt;.txt</code>, and<br />
overall logging goes to <code>dials.process.log</code> (or per-rank logs under MPI). These integrated<br />
<code>.expt</code>/<code>.refl</code> pairs are then scaled and merged with<br />
[[cctbx.xfel.merge]].<br />
<br />
== See also ==<br />
<br />
* [[cctbx.xfel.merge]] — scaling, post-refinement and merging of the integrated data<br />
* [[IOTA]] — a front-end that supplies preset parameters for <code>dials.stills_process</code><br />
* [[Data Processing at SACLA]] — SACLA-specific processing workflow<br />
* The DIALS documentation for the underlying spot-finding, indexing, refinement and integration algorithms</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Dials.stills_process&diff=1762Dials.stills process2026-08-21T22:25:47Z<p>Aaron: /* dials.stills_process */</p>
<hr />
<div>= dials.stills_process =<br />
<br />
'''''dials.stills_process''''' is a stand-alone program for processing still diffraction images and is<br />
the core data-reduction program of ''cctbx.xfel''. It takes a set of<br />
still diffraction images — typically serial femtosecond crystallography (SFX) snapshots from an<br />
XFEL (LCLS, SACLA, EuXFEL) or still shots from a synchrotron — and analyzes each one, independently,<br />
through the full DIALS pipeline: '''import → spot-finding → indexing → refinement → integration'''.<br />
Its output (integrated intensities together with the per-image experimental models) is the input to<br />
the merging program [[cctbx.xfel.merge]].<br />
<br />
Because every image is a separate still exposure of a different crystal (rather than a wedge of<br />
a rotation series), each image is treated as its own experiment with its own crystal model. This<br />
"one crystal per shot" model is what distinguishes stills processing from conventional rotation data<br />
processing (e.g. <code>dials.import</code>, <code>dials.find_spots</code>, <code>dials.index</code>, etc.), <br />
and it drives most of the stills-specific defaults described below.<br />
<br />
== Overview ==<br />
<br />
For each image, <code>dials.stills_process</code> runs the following stages in order. Any stage can<br />
be turned off with the <code>dispatch</code> parameters; if a stage is disabled, processing stops<br />
after the last enabled stage.<br />
<br />
# '''Import''' — read the image, apply reference geometry / gain / mask overrides, and build an experiment.<br />
# '''Spot-finding''' — locate strong pixels and build strong reflections (<code>dispatch.find_spots</code>).<br />
# '''Hit finding''' — reject images with too few (or optionally too many) strong spots, so that only "hits" are indexed (<code>dispatch.hit_finder</code>).<br />
# '''Indexing''' — determine the crystal orientation and (optionally constrained) unit cell (<code>dispatch.index</code>).<br />
# '''Refinement''' — refine the crystal model and re-estimate the mosaic/domain-size parameters. Detector and beam are held fixed by default (<code>dispatch.refine</code>).<br />
# '''Integration''' — predict reflections and measure their intensities using the stills integrator (<code>dispatch.integrate</code>), with optional Kapton absorption correction and a significance filter.<br />
<br />
The program applies a set of stills-specific defaults on top of the standard DIALS parameters,<br />
including: indexing method <code>fft1d</code>; the stills integrator with profile fitting turned off<br />
(summation integration with a 2-D plane background); the stills refinement weighting strategy with<br />
the detector and beam models fixed; and a re-estimation of per-crystal mosaic parameters<br />
(domain size and mosaic angle) after refinement. These are sensible starting points for serial data<br />
and normally do not need to be changed.<br />
<br />
Failures on individual images are, by default, caught and logged so that processing continues to the<br />
next image (<code>dispatch.squash_errors=True</code>) — essential when reducing tens of thousands of<br />
shots of which only a fraction are hits.<br />
<br />
== Basic usage ==<br />
<br />
<pre><br />
# Process a directory of CBF stills, constraining the target cell/symmetry,<br />
# using 32 parallel processes on one node<br />
dials.stills_process /path/to/images/*.cbf process.phil mp.nproc=32<br />
</pre><br />
<br />
A typical <code>process.phil</code> for XFEL data looks like:<br />
<br />
<pre><br />
spotfinder.filter.min_spot_size=2<br />
spotfinder.threshold.dispersion.gain=1.0<br />
spotfinder.threshold.dispersion.global_threshold=100<br />
<br />
indexing.known_symmetry.space_group=P43212<br />
indexing.known_symmetry.unit_cell=78.9,78.9,38.1,90,90,90<br />
indexing.method=fft1d<br />
<br />
integration.background.simple.outlier.algorithm=null<br />
<br />
input.reference_geometry=refined.expt # from a previous metrology refinement<br />
<br />
output.composite_output=True<br />
output.output_dir=out<br />
mp.nproc=32<br />
</pre><br />
<br />
Useful command-line helpers:<br />
* <code>dials.stills_process -h</code> — help and usage<br />
* <code>dials.stills_process -c -e 2</code> — show the configurable parameters (expert level 2)<br />
* <code>dials.stills_process -c -a2</code> — include attributes (help text) for each parameter<br />
* <code>dials.stills_process ... input.show_image_tags=True</code> — list the image tags in a dataset without processing (useful for selecting subsets with <code>input.image_tag</code>)<br />
<br />
== Important parameters ==<br />
<br />
The program exposes the entire DIALS parameter set (spot-finding, indexing, refinement, integration,<br />
profile modelling, prediction). The parameters below are the ones most commonly adjusted for serial<br />
still data. Parameters are given by their full PHIL path with the default value in parentheses.<br />
<br />
=== Input ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>input.reference_geometry</code> || None || Path to an <code>.expt</code> file (or image) with a single detector model. Data are processed using ''this'' geometry instead of the geometry in the image headers. This is how a refined detector metrology from <code>cctbx.xfel</code> ensemble refinement is applied — almost always set for XFEL data.<br />
|-<br />
| <code>input.sync_reference_geom</code> || True || When a reference geometry is supplied, copy its refined local frames onto the image's detector hierarchy node-by-node (the hierarchies must match in shape). If False, the reference detector replaces the image detector wholesale.<br />
|-<br />
| <code>input.file_list</code> || None || Path to a text file listing image paths, one per line (alternative to listing files on the command line).<br />
|-<br />
| <code>input.glob</code> || None || Wildcard path(s) for large multi-file datasets (e.g. <code>*.cbf</code>). May be given multiple times.<br />
|-<br />
| <code>input.image_tag</code> || None || Process only images with the given tag(s). For single-image files the tag is the filename; for multi-image files (HDF5) it is <code>filename_imagenumber</code>. Use <code>input.show_image_tags=True</code> to list them.<br />
|-<br />
| <code>input.max_images</code> || None || Limit the total number of images processed.<br />
|}<br />
<br />
=== Dispatch — controlling the pipeline ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>dispatch.find_spots</code> || True || Do spot-finding. Required for indexing/integration.<br />
|-<br />
| <code>dispatch.index</code> || True || Attempt to index. Requires <code>find_spots=True</code>.<br />
|-<br />
| <code>dispatch.integrate</code> || True || Integrate indexed images. Ignored if indexing/spot-finding is off.<br />
|-<br />
| <code>dispatch.refine</code> || False || Run an explicit model-refinement step after indexing (the stills indexer already refines; this is an additional pass, rarely needed).<br />
|-<br />
| <code>dispatch.squash_errors</code> || True || If an image fails, log it and continue to the next image. Set False to halt and show the full error when debugging.<br />
|-<br />
| <code>dispatch.hit_finder.enable</code> || True || Enable hit-finding. If disabled, every image is passed to indexing.<br />
|-<br />
| <code>dispatch.hit_finder.minimum_number_of_reflections</code> || 16 || Images with fewer strong spots than this are discarded as non-hits before indexing.<br />
|-<br />
| <code>dispatch.hit_finder.maximum_number_of_reflections</code> || None || If set, images with more strong spots than this are discarded (e.g. ice/junk).<br />
|-<br />
| <code>dispatch.process_percent</code> || None || Process only this percentage of events (evenly sampled). Useful for a quick look at a large dataset.<br />
|-<br />
| <code>dispatch.coset</code> || False || During integration, also integrate a sublattice coset as a negative control for regions with no Bragg diffraction (see <code>integration.coset.transformation</code>).<br />
|-<br />
| <code>dispatch.pre_import</code> || False || Import all data before processing. Only needed when processing multiple ''multi-image'' files at once (not a recommended workflow). Single-file datasets are always pre-imported automatically.<br />
|}<br />
<br />
=== Spot-finding ===<br />
<br />
Spot-finding uses the standard DIALS dispersion thresholding. The most impactful parameters for<br />
serial data:<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>spotfinder.filter.min_spot_size</code> || Auto || Minimum connected pixels for a spot. Auto = 3 for pixel-array detectors (SENSOR_PAD), 6 otherwise. Often lowered to 2 for sparse serial patterns.<br />
|-<br />
| <code>spotfinder.threshold.dispersion.gain</code> || None || Detector gain used in thresholding. Set this ''or'' the detector-model gain, not both (see gain note below).<br />
|-<br />
| <code>spotfinder.threshold.dispersion.sigma_strong</code> || 3 || Number of sigma above background for a pixel to be "strong".<br />
|-<br />
| <code>spotfinder.threshold.dispersion.sigma_background</code> || 6 || Number of sigma for the background/dispersion test.<br />
|-<br />
| <code>spotfinder.threshold.dispersion.global_threshold</code> || 0 || Absolute pixel intensity floor; pixels below this are never considered strong. Raising it (e.g. 100) suppresses noise on many detectors.<br />
|-<br />
| <code>spotfinder.threshold.dispersion.kernel_size</code> || 3 3 || Local-area kernel (pixels) for the dispersion calculation.<br />
|}<br />
<br />
'''Gain note:''' the detector gain should be set on the detector model ''or'' in the processing<br />
parameters, never both. If a supplied gain (<code>spotfinder.threshold.dispersion.gain</code> or<br />
<code>integration.summation.detector_gain</code>) conflicts with a non-unity gain already on the<br />
detector model, the program raises an error. Override with<br />
<code>input.ignore_gain_mismatch=True</code> only if you understand the consequences.<br />
<br />
=== Indexing ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>indexing.known_symmetry.space_group</code> || None || Target space group. Strongly recommended for serial data — constrains indexing and improves the hit/index rate.<br />
|-<br />
| <code>indexing.known_symmetry.unit_cell</code> || None || Target unit cell (a,b,c,α,β,γ). Used together with the space group as the indexing target.<br />
|-<br />
| <code>indexing.method</code> || fft1d || Basis-vector search method (<code>dials.stills_process</code> defaults to <code>fft1d</code>; <code>fft3d</code> and <code>real_space_grid_search</code> are also available).<br />
|-<br />
| <code>indexing.stills.method_list</code> || None || Ordered list of indexing methods to try in turn; if one fails, the next is attempted (e.g. <code>fft1d real_space_grid_search</code>).<br />
|-<br />
| <code>indexing.stills.known_orientations</code> || None || Paths to previously-processed experiment lists. If given, images are ''not'' re-indexed; the stored crystal orientations are reused (e.g. to re-integrate with new parameters). Supports wildcards; may be repeated.<br />
|-<br />
| <code>indexing.stills.require_known_orientation</code> || False || When known orientations are supplied but an image is not found among them, controls whether to skip it (True) or fall back to indexing from scratch (False).<br />
|-<br />
| <code>indexing.stills.reflection_subsampling.enable</code> || False || Retry indexing on random subsets of the strong spots (from <code>step_start</code>% down to <code>step_stop</code>% in <code>step_size</code>% steps). Can rescue images that fail to index on the full spot list.<br />
|}<br />
<br />
=== Integration ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>profile.gaussian_rs.parameters.sigma_b_cutoff</code> || 0.1 || Maximum beam divergence (sigma_b, degrees) before an indexed crystal is rejected prior to integration. Guards against over-broad, poorly-modelled lattices.<br />
|-<br />
| <code>significance_filter.enable</code> || False || Apply the stills significance filter, which trims each crystal to the resolution at which reflections remain statistically significant (see <code>significance_filter.isigi_cutoff</code>, default 1.0).<br />
|-<br />
| <code>integration.integration_only_overrides.trusted_range</code> || None || Override the panel trusted range [min, max] during integration only (e.g. to include pixels the header marks as untrusted).<br />
|-<br />
| <code>integration.absorption_correction</code> || apply=False || Enable a Kapton absorption correction (<code>algorithm=fuller_kapton</code> or <code>kapton_2019</code>). Requires shoeboxes to be retained via <code>integration.debug</code> settings; the program checks this and errors otherwise.<br />
|-<br />
| <code>integration.coset.transformation</code> || 6 || Which modulus-2 sublattice transformation defines the coset negative control when <code>dispatch.coset=True</code> (0–6; 6 = body centering). See Sauter & Zwart, ''Acta Cryst.'' D (2009) '''65''':553.<br />
|}<br />
<br />
=== Output ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>output.output_dir</code> || <code>.</code> || Directory for output files.<br />
|-<br />
| <code>output.composite_output</code> || True || If True, each process writes ''one'' concatenated set of <code>.expt</code>/<code>.refl</code> files covering all its successful images. If False, a separate pair of files is written per image (generates a very large number of files). Composite output is strongly preferred for large serial datasets.<br />
|-<br />
| <code>output.logging_dir</code> || None || Directory for log files. Under MPI, per-rank logs (<code>log_rankNNNN.out</code>, etc.) are written here.<br />
|-<br />
| <code>output.logging_option</code> || normal || <code>normal</code> (all logging), <code>suppressed</code> (silences verbose DIALS refine/index output), or <code>disabled</code> (almost no logging).<br />
|-<br />
| <code>output.*_filename</code> || (templates) || Output filename templates for each stage: <code>strong</code>, <code>indexed</code>, <code>refined_experiments</code>, <code>integrated</code>, <code>integrated_experiments</code>, <code>coset</code>. A <code>%s</code> in the template is replaced with the per-image tag.<br />
|-<br />
| <code>output.integration_pickle</code> || None || Write legacy <code>cxi.merge</code>/PRIME integration pickle files (e.g. <code>int-%d-%s.pickle</code>). Only needed for legacy merging workflows.<br />
|}<br />
<br />
=== Multiprocessing ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>mp.method</code> || multiprocessing || Parallelization backend: <code>multiprocessing</code> (single node), or <code>mpi</code> for multi-node jobs (also <code>sge</code>, <code>lsf</code>, <code>pbs</code>).<br />
|-<br />
| <code>mp.nproc</code> || 1 || Number of processes (for the <code>multiprocessing</code> method). Set to the number of cores per node.<br />
|-<br />
| <code>mp.composite_stride</code> || None || For MPI composite output, how many ranks aggregate their results into one set of output files. Ideally set to the number of processes per node so that one composite file set is written per node rather than per rank.<br />
|}<br />
<br />
Under MPI with more than two ranks, the program runs a client/server model: rank 0 (or rank 2 for<br />
psana2 <code>.loc</code> inputs) hands out images to worker ranks on demand, giving good load<br />
balancing across the many fast-failing non-hits typical of serial data.<br />
<br />
== Output files ==<br />
<br />
With the default composite output, each process (or each aggregating rank under MPI) writes<br />
concatenated files such as:<br />
<br />
* <code>&lt;tag&gt;_strong.refl</code> — strong spots from spot-finding<br />
* <code>&lt;tag&gt;_indexed.refl</code> and <code>&lt;tag&gt;_refined.expt</code> — indexed reflections and their crystal models<br />
* <code>&lt;tag&gt;_integrated.refl</code> and <code>&lt;tag&gt;_integrated.expt</code> — integrated intensities and final models (the input to [[cctbx.xfel.merge]])<br />
<br />
Per-image progress and status codes are recorded in <code>debug/debug_&lt;rank&gt;.txt</code>, and<br />
overall logging goes to <code>dials.process.log</code> (or per-rank logs under MPI). These integrated<br />
<code>.expt</code>/<code>.refl</code> pairs are then scaled and merged with<br />
[[cctbx.xfel.merge]].<br />
<br />
== See also ==<br />
<br />
* [[cctbx.xfel.merge]] — scaling, post-refinement and merging of the integrated data<br />
* [[IOTA]] — a front-end that supplies preset parameters for <code>dials.stills_process</code><br />
* [[Data Processing at SACLA]] — SACLA-specific processing workflow<br />
* The DIALS documentation for the underlying spot-finding, indexing, refinement and integration algorithms</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Dials.stills_process&diff=1761Dials.stills process2026-08-21T22:22:06Z<p>Aaron: Created page with "= dials.stills_process = '''dials.stills_process''' is the core data-reduction program of cctbx.xfel. It takes a set of still diffraction images — typically serial femtosecond crystallography (SFX) snapshots from an XFEL (LCLS, SACLA, EuXFEL) or still shots from a synchrotron — and reduces each one, independently, through the full DIALS pipeline: '''import → spot-finding → indexing → refinement → integration'''. Its output (integrated intensities together wi..."</p>
<hr />
<div>= dials.stills_process =<br />
<br />
'''dials.stills_process''' is the core data-reduction program of cctbx.xfel. It takes a set of<br />
still diffraction images — typically serial femtosecond crystallography (SFX) snapshots from an<br />
XFEL (LCLS, SACLA, EuXFEL) or still shots from a synchrotron — and reduces each one, independently,<br />
through the full DIALS pipeline: '''import → spot-finding → indexing → refinement → integration'''.<br />
Its output (integrated intensities together with the per-image experimental models) is the input to<br />
the merging program [[cctbx.xfel.merge]].<br />
<br />
Because every image is a separate stationary exposure of a different crystal (rather than a wedge of<br />
a rotation series), each image is treated as its own experiment with its own crystal model. This<br />
"one crystal per shot" model is what distinguishes stills processing from conventional rotation data<br />
processing (e.g. <code>dials.process</code>), and it drives most of the stills-specific defaults<br />
described below.<br />
<br />
== Overview ==<br />
<br />
For each image, <code>dials.stills_process</code> runs the following stages in order. Any stage can<br />
be turned off with the <code>dispatch</code> parameters; if a stage is disabled, processing stops<br />
after the last enabled stage.<br />
<br />
# '''Import''' — read the image, apply reference geometry / gain / mask overrides, and build an experiment.<br />
# '''Spot-finding''' — locate strong pixels and build strong reflections (<code>dispatch.find_spots</code>).<br />
# '''Hit finding''' — reject images with too few (or optionally too many) strong spots, so that only "hits" are indexed (<code>dispatch.hit_finder</code>).<br />
# '''Indexing''' — determine the crystal orientation and (optionally constrained) unit cell (<code>dispatch.index</code>).<br />
# '''Refinement''' — refine the crystal model and re-estimate the mosaic/domain-size parameters. Detector and beam are held fixed by default (<code>dispatch.refine</code>).<br />
# '''Integration''' — predict reflections and measure their intensities using the stills integrator (<code>dispatch.integrate</code>), with optional Kapton absorption correction and a significance filter.<br />
<br />
The program applies a set of stills-specific defaults on top of the standard DIALS parameters,<br />
including: indexing method <code>fft1d</code>; the stills integrator with profile fitting turned off<br />
(summation integration with a 2-D plane background); the stills refinement weighting strategy with<br />
the detector and beam models fixed; and a re-estimation of per-crystal mosaic parameters<br />
(domain size and mosaic angle) after refinement. These are sensible starting points for serial data<br />
and normally do not need to be changed.<br />
<br />
Failures on individual images are, by default, caught and logged so that processing continues to the<br />
next image (<code>dispatch.squash_errors=True</code>) — essential when reducing tens of thousands of<br />
shots of which only a fraction are hits.<br />
<br />
== Basic usage ==<br />
<br />
<pre><br />
# Process a directory of CBF stills, constraining the target cell/symmetry,<br />
# using 32 parallel processes on one node<br />
dials.stills_process /path/to/images/*.cbf process.phil mp.nproc=32<br />
</pre><br />
<br />
A typical <code>process.phil</code> for XFEL data looks like:<br />
<br />
<pre><br />
spotfinder.filter.min_spot_size=2<br />
spotfinder.threshold.dispersion.gain=1.0<br />
spotfinder.threshold.dispersion.global_threshold=100<br />
<br />
indexing.known_symmetry.space_group=P43212<br />
indexing.known_symmetry.unit_cell=78.9,78.9,38.1,90,90,90<br />
indexing.method=fft1d<br />
<br />
integration.background.simple.outlier.algorithm=null<br />
<br />
input.reference_geometry=refined.expt # from a previous metrology refinement<br />
<br />
output.composite_output=True<br />
output.output_dir=out<br />
mp.nproc=32<br />
</pre><br />
<br />
Useful command-line helpers:<br />
* <code>dials.stills_process -h</code> — help and usage<br />
* <code>dials.stills_process -c -e 2</code> — show the configurable parameters (expert level 2)<br />
* <code>dials.stills_process -c -a2</code> — include attributes (help text) for each parameter<br />
* <code>dials.stills_process ... input.show_image_tags=True</code> — list the image tags in a dataset without processing (useful for selecting subsets with <code>input.image_tag</code>)<br />
<br />
== Important parameters ==<br />
<br />
The program exposes the entire DIALS parameter set (spot-finding, indexing, refinement, integration,<br />
profile modelling, prediction). The parameters below are the ones most commonly adjusted for serial<br />
still data. Parameters are given by their full PHIL path with the default value in parentheses.<br />
<br />
=== Input ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>input.reference_geometry</code> || None || Path to an <code>.expt</code> file (or image) with a single detector model. Data are processed using ''this'' geometry instead of the geometry in the image headers. This is how a refined detector metrology from <code>cctbx.xfel</code> ensemble refinement is applied — almost always set for XFEL data.<br />
|-<br />
| <code>input.sync_reference_geom</code> || True || When a reference geometry is supplied, copy its refined local frames onto the image's detector hierarchy node-by-node (the hierarchies must match in shape). If False, the reference detector replaces the image detector wholesale.<br />
|-<br />
| <code>input.file_list</code> || None || Path to a text file listing image paths, one per line (alternative to listing files on the command line).<br />
|-<br />
| <code>input.glob</code> || None || Wildcard path(s) for large multi-file datasets (e.g. <code>*.cbf</code>). May be given multiple times.<br />
|-<br />
| <code>input.image_tag</code> || None || Process only images with the given tag(s). For single-image files the tag is the filename; for multi-image files (HDF5) it is <code>filename_imagenumber</code>. Use <code>input.show_image_tags=True</code> to list them.<br />
|-<br />
| <code>input.max_images</code> || None || Limit the total number of images processed.<br />
|}<br />
<br />
=== Dispatch — controlling the pipeline ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>dispatch.find_spots</code> || True || Do spot-finding. Required for indexing/integration.<br />
|-<br />
| <code>dispatch.index</code> || True || Attempt to index. Requires <code>find_spots=True</code>.<br />
|-<br />
| <code>dispatch.integrate</code> || True || Integrate indexed images. Ignored if indexing/spot-finding is off.<br />
|-<br />
| <code>dispatch.refine</code> || False || Run an explicit model-refinement step after indexing (the stills indexer already refines; this is an additional pass, rarely needed).<br />
|-<br />
| <code>dispatch.squash_errors</code> || True || If an image fails, log it and continue to the next image. Set False to halt and show the full error when debugging.<br />
|-<br />
| <code>dispatch.hit_finder.enable</code> || True || Enable hit-finding. If disabled, every image is passed to indexing.<br />
|-<br />
| <code>dispatch.hit_finder.minimum_number_of_reflections</code> || 16 || Images with fewer strong spots than this are discarded as non-hits before indexing.<br />
|-<br />
| <code>dispatch.hit_finder.maximum_number_of_reflections</code> || None || If set, images with more strong spots than this are discarded (e.g. ice/junk).<br />
|-<br />
| <code>dispatch.process_percent</code> || None || Process only this percentage of events (evenly sampled). Useful for a quick look at a large dataset.<br />
|-<br />
| <code>dispatch.coset</code> || False || During integration, also integrate a sublattice coset as a negative control for regions with no Bragg diffraction (see <code>integration.coset.transformation</code>).<br />
|-<br />
| <code>dispatch.pre_import</code> || False || Import all data before processing. Only needed when processing multiple ''multi-image'' files at once (not a recommended workflow). Single-file datasets are always pre-imported automatically.<br />
|}<br />
<br />
=== Spot-finding ===<br />
<br />
Spot-finding uses the standard DIALS dispersion thresholding. The most impactful parameters for<br />
serial data:<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>spotfinder.filter.min_spot_size</code> || Auto || Minimum connected pixels for a spot. Auto = 3 for pixel-array detectors (SENSOR_PAD), 6 otherwise. Often lowered to 2 for sparse serial patterns.<br />
|-<br />
| <code>spotfinder.threshold.dispersion.gain</code> || None || Detector gain used in thresholding. Set this ''or'' the detector-model gain, not both (see gain note below).<br />
|-<br />
| <code>spotfinder.threshold.dispersion.sigma_strong</code> || 3 || Number of sigma above background for a pixel to be "strong".<br />
|-<br />
| <code>spotfinder.threshold.dispersion.sigma_background</code> || 6 || Number of sigma for the background/dispersion test.<br />
|-<br />
| <code>spotfinder.threshold.dispersion.global_threshold</code> || 0 || Absolute pixel intensity floor; pixels below this are never considered strong. Raising it (e.g. 100) suppresses noise on many detectors.<br />
|-<br />
| <code>spotfinder.threshold.dispersion.kernel_size</code> || 3 3 || Local-area kernel (pixels) for the dispersion calculation.<br />
|}<br />
<br />
'''Gain note:''' the detector gain should be set on the detector model ''or'' in the processing<br />
parameters, never both. If a supplied gain (<code>spotfinder.threshold.dispersion.gain</code> or<br />
<code>integration.summation.detector_gain</code>) conflicts with a non-unity gain already on the<br />
detector model, the program raises an error. Override with<br />
<code>input.ignore_gain_mismatch=True</code> only if you understand the consequences.<br />
<br />
=== Indexing ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>indexing.known_symmetry.space_group</code> || None || Target space group. Strongly recommended for serial data — constrains indexing and improves the hit/index rate.<br />
|-<br />
| <code>indexing.known_symmetry.unit_cell</code> || None || Target unit cell (a,b,c,α,β,γ). Used together with the space group as the indexing target.<br />
|-<br />
| <code>indexing.method</code> || fft1d || Basis-vector search method (<code>dials.stills_process</code> defaults to <code>fft1d</code>; <code>fft3d</code> and <code>real_space_grid_search</code> are also available).<br />
|-<br />
| <code>indexing.stills.method_list</code> || None || Ordered list of indexing methods to try in turn; if one fails, the next is attempted (e.g. <code>fft1d real_space_grid_search</code>).<br />
|-<br />
| <code>indexing.stills.known_orientations</code> || None || Paths to previously-processed experiment lists. If given, images are ''not'' re-indexed; the stored crystal orientations are reused (e.g. to re-integrate with new parameters). Supports wildcards; may be repeated.<br />
|-<br />
| <code>indexing.stills.require_known_orientation</code> || False || When known orientations are supplied but an image is not found among them, controls whether to skip it (True) or fall back to indexing from scratch (False).<br />
|-<br />
| <code>indexing.stills.reflection_subsampling.enable</code> || False || Retry indexing on random subsets of the strong spots (from <code>step_start</code>% down to <code>step_stop</code>% in <code>step_size</code>% steps). Can rescue images that fail to index on the full spot list.<br />
|}<br />
<br />
=== Integration ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>profile.gaussian_rs.parameters.sigma_b_cutoff</code> || 0.1 || Maximum beam divergence (sigma_b, degrees) before an indexed crystal is rejected prior to integration. Guards against over-broad, poorly-modelled lattices.<br />
|-<br />
| <code>significance_filter.enable</code> || False || Apply the stills significance filter, which trims each crystal to the resolution at which reflections remain statistically significant (see <code>significance_filter.isigi_cutoff</code>, default 1.0).<br />
|-<br />
| <code>integration.integration_only_overrides.trusted_range</code> || None || Override the panel trusted range [min, max] during integration only (e.g. to include pixels the header marks as untrusted).<br />
|-<br />
| <code>integration.absorption_correction</code> || apply=False || Enable a Kapton absorption correction (<code>algorithm=fuller_kapton</code> or <code>kapton_2019</code>). Requires shoeboxes to be retained via <code>integration.debug</code> settings; the program checks this and errors otherwise.<br />
|-<br />
| <code>integration.coset.transformation</code> || 6 || Which modulus-2 sublattice transformation defines the coset negative control when <code>dispatch.coset=True</code> (0–6; 6 = body centering). See Sauter & Zwart, ''Acta Cryst.'' D (2009) '''65''':553.<br />
|}<br />
<br />
=== Output ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>output.output_dir</code> || <code>.</code> || Directory for output files.<br />
|-<br />
| <code>output.composite_output</code> || True || If True, each process writes ''one'' concatenated set of <code>.expt</code>/<code>.refl</code> files covering all its successful images. If False, a separate pair of files is written per image (generates a very large number of files). Composite output is strongly preferred for large serial datasets.<br />
|-<br />
| <code>output.logging_dir</code> || None || Directory for log files. Under MPI, per-rank logs (<code>log_rankNNNN.out</code>, etc.) are written here.<br />
|-<br />
| <code>output.logging_option</code> || normal || <code>normal</code> (all logging), <code>suppressed</code> (silences verbose DIALS refine/index output), or <code>disabled</code> (almost no logging).<br />
|-<br />
| <code>output.*_filename</code> || (templates) || Output filename templates for each stage: <code>strong</code>, <code>indexed</code>, <code>refined_experiments</code>, <code>integrated</code>, <code>integrated_experiments</code>, <code>coset</code>. A <code>%s</code> in the template is replaced with the per-image tag.<br />
|-<br />
| <code>output.integration_pickle</code> || None || Write legacy <code>cxi.merge</code>/PRIME integration pickle files (e.g. <code>int-%d-%s.pickle</code>). Only needed for legacy merging workflows.<br />
|}<br />
<br />
=== Multiprocessing ===<br />
<br />
{| class="wikitable"<br />
! Parameter !! Default !! Description<br />
|-<br />
| <code>mp.method</code> || multiprocessing || Parallelization backend: <code>multiprocessing</code> (single node), or <code>mpi</code> for multi-node jobs (also <code>sge</code>, <code>lsf</code>, <code>pbs</code>).<br />
|-<br />
| <code>mp.nproc</code> || 1 || Number of processes (for the <code>multiprocessing</code> method). Set to the number of cores per node.<br />
|-<br />
| <code>mp.composite_stride</code> || None || For MPI composite output, how many ranks aggregate their results into one set of output files. Ideally set to the number of processes per node so that one composite file set is written per node rather than per rank.<br />
|}<br />
<br />
Under MPI with more than two ranks, the program runs a client/server model: rank 0 (or rank 2 for<br />
psana2 <code>.loc</code> inputs) hands out images to worker ranks on demand, giving good load<br />
balancing across the many fast-failing non-hits typical of serial data.<br />
<br />
== Output files ==<br />
<br />
With the default composite output, each process (or each aggregating rank under MPI) writes<br />
concatenated files such as:<br />
<br />
* <code>&lt;tag&gt;_strong.refl</code> — strong spots from spot-finding<br />
* <code>&lt;tag&gt;_indexed.refl</code> and <code>&lt;tag&gt;_refined.expt</code> — indexed reflections and their crystal models<br />
* <code>&lt;tag&gt;_integrated.refl</code> and <code>&lt;tag&gt;_integrated.expt</code> — integrated intensities and final models (the input to [[cctbx.xfel.merge]])<br />
<br />
Per-image progress and status codes are recorded in <code>debug/debug_&lt;rank&gt;.txt</code>, and<br />
overall logging goes to <code>dials.process.log</code> (or per-rank logs under MPI). These integrated<br />
<code>.expt</code>/<code>.refl</code> pairs are then scaled and merged with<br />
[[cctbx.xfel.merge]].<br />
<br />
== See also ==<br />
<br />
* [[cctbx.xfel.merge]] — scaling, post-refinement and merging of the integrated data<br />
* [[IOTA]] — a front-end that supplies preset parameters for <code>dials.stills_process</code><br />
* [[Data Processing at SACLA]] — SACLA-specific processing workflow<br />
* The DIALS documentation for the underlying spot-finding, indexing, refinement and integration algorithms</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Main_Page&diff=1760Main Page2026-08-21T22:15:20Z<p>Aaron: /* cctbx.xfel resources */</p>
<hr />
<div>News! See our [[https://doi.org/10.1101/2025.05.04.652045 new preprint]]! This paper shows current best practices using ''cctbx.xfel'' with ''DIALS'' for serial crystallographic data.<br />
<br />
= Open-source tools for serial crystallographic data processing =<br />
<br />
''cctbx.xfel'' is a suite of software tools designed to process diffraction data from serial femtosecond crystallography (SFX) measurements at an X-ray free-electron laser (XFEL) or a synchrotron. Built on the Computational Crystallographic Toolbox ([https://cctbx.github.io/ ''cctbx'']), the same toolbox on which [http://www.phenix-online.org ''PHENIX''], and [http://dials.github.io DIALS] are built, it enables the user to solve difficult problems relating to processing serial crystallographic data. The programs and modules provided by ''cctbx.xfel'' can reduce a large set of still diffraction images recorded at Stanford’s Linac Coherent Light Source ([http://lcls.slac.stanford.edu LCLS]), [http://sacla.xfel.jp SACLA], the [https://www.xfel.eu/ European XFEL], or a synchrotron, to a single MTZ file containing merged reflection intensities suitable for structure solution.<br />
<br />
== ''cctbx.xfel'' resources ==<br />
<br />
The current best resource is our pre-print, [https://doi.org/10.1101/2025.05.04.652045 ''cctbx.xfel: a suite for processing serial crystallographic data.'']. This paper describes the algorithms and ideas behind the implementation and execution of the two main programs of ''cctbx.xfel'':<br />
<br />
* ''[[dials.stills_process]]'': this program runs through all the first steps of processing data using dials: import (reading image headers), spot finding, indexing, geometry refinement, and integration. Serial images are independent from each other, so this program is optimized for massively parallel systems, but can be adapted to a variety of single and multi-processing modes.<br />
* ''cctbx.xfel.merge'': this program runs through the remaining steps of processing: scaling and merging to produce a full dataset (.mtz file). For documentation, see our [https://github.com/cctbx/cctbx_project/tree/master/xfel/merging Github project]<br />
<br />
Additional resources:<br />
<br />
* Our [http://cci.lbl.gov/publications/download/CCN_2019_p22_Brewster.pdf 2019 newsletter article] describes how to use the ''cctbx.xfel'' GUI, an interactive program for handling large XFEL datasets.<br />
<br />
* [[Overview]] to the system architecture at LCLS and real-time progress monitoring of data processing<br />
* Installation: there are two ways to get ''cctbx.xfel'':<br />
** ''cctbx.xfel'' is installed for general use at SLAC. To start using an existing installation, [[Using the pre-built cctbx at LCLS|simply source the current build]]. <br />
** [https://dials.github.io/installation.html Download] a binary bundle of DIALS. Use this option if you will not be processing LCLS data because, for example, if you have serial crystallographic data from a synchrotron.<br />
** Build a psana/cctbx.xfel build [https://github.com/cctbx/cctbx_project/tree/master/xfel/conda_envs from scratch]. Use this option if you want to process LCLS data outside of LCLS or want your own build.<br />
* [[Tutorials]] on pre-processing, data reduction, and merging. <br />
<br />
Other related information:<br />
<br />
* [[Serial XFEL Crystallography References]]<br />
<br />
This project is under active development. For any assistance, please contact the authors.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=2017_cxi_merge_tutorial&diff=17592017 cxi merge tutorial2026-08-19T18:36:00Z<p>Aaron: </p>
<hr />
<div>{{Archived}}<br />
<br />
This is an updated, worked example of data merging using cxi.merge, for presentation at the Feb 17, 2017 Berkeley Lab Serial Crystallography Workshop. Previous documentation sets are [[Merging | here]] and [[Advanced Merging | here]]. Literature description is in the [http://dx.doi.org/10.1038/nmeth.2887 Hattne (2014)], the [http://dx.doi.org/10.7554/eLife.05421 PRIME paper], the [http://dx.doi.org/10.1107/S1399004714024134 Sauter (2014)] and [http://dx.doi.org/10.1107/S1600577514028203 Sauter (2015)] papers. Math derivations are further described in the source code release in file postrefinement_rs_model.pdf.<br />
<br />
== Initial characterization ==<br />
In this example, we are given integrated still-shot data collected by Danny Axford at Diamond, for P6 myoglobin, PDB code [http://www.rcsb.org/pdb/explore/explore.do?structureId=5M3S 5M3S].<br />
<br />
* /net/dials/raid1/aaron/zurich0038/jr_006_batches/split_reintegrated/extracted # cctbx-style integration pickles <br />
* /net/dials/raid1/aaron/zurich0038/jr_006_batches/sig_filter/split_reintegrated/extracted # same data, with per-image resolution cutoff during integration<br />
<br />
Unix ls reveals 5031 *.pickle files in each directory. <br />
<br />
Immediately there is a problem:<br />
<br />
$ cxi.print_pickle /net/dials/raid1/aaron/zurich0038/jr_006_batches/sig_filter/split_reintegrated/extracted/*.pickle<br />
<br />
...fails on image 0059 with a traceback; it looks like the file is corrupted.<br />
<br />
So focus on the data without integration resolution cutoff:<br />
<br />
$ cxi.print_pickle /net/dials/raid1/aaron/zurich0038/jr_006_batches/split_reintegrated/extracted/*.pickle<br />
<br />
Some conclusions with the aid of grep:<br />
* all integration pickles have space group P6 (good)<br />
* distance and beam center is fixed throughout the integrated dataset<br />
* Unit cells are variable but do seem to cluster around 91.4 91.4 45.9 90 90 120<br />
<br />
phenix.fetch_pdb --mtz 5m3s<br />
<br />
Merge command file:<br />
#!/bin/csh -f<br />
<br />
set effective_params = "d_min=DMIN \<br />
data=/net/dials/raid1/aaron/zurich0038/jr_006_batches/split_reintegrated/extracted/*.pickle \<br />
output.n_bins=10 \<br />
pixel_size=0.172 \<br />
backend=FS \<br />
nproc=1 \<br />
model=5m3s.pdb \<br />
merge_anomalous=True \<br />
plot_single_index_histograms=False \<br />
scaling.algorithm=mark0 \<br />
raw_data.sdfac_auto=False \<br />
scaling.mtz_file=5m3s.mtz \<br />
scaling.show_plots=False \<br />
scaling.log_cutoff=None \<br />
scaling.mtz_column_F=i-obs \<br />
scaling.report_ML=True \<br />
set_average_unit_cell=True \<br />
rescale_with_average_cell=False \<br />
significance_filter.apply=True \<br />
significance_filter.min_ct=30 \<br />
significance_filter.sigma=0.2 \<br />
include_negatives=NEG \<br />
postrefinement.enable=True \<br />
postrefinement.algorithm=rs \<br />
output.prefix=TAG"<br />
set tag = p6m<br />
set dmin = 2.5<br />
set neg = True<br />
set eff = `echo $effective_params|sed -e "s,FS,Flex,g"|sed -e "s,DMIN,$dmin,g"|sed -e "s,NEG,$neg,g"|sed -e "s,TAG,$tag,g"`<br />
<br />
cxi.merge ${eff}<br />
exit<br />
cxi.xmerge ${eff}<br />
phenix.xtriage ${tag}_s0_mark0.mtz scaling.input.xray_data.obs_labels=imean<br />
<br />
Initial trial nproc=1 just to see if it runs. Had to fix PDB reference. Can't use *.pickle on the data= line<br />
<br />
Scale-up trial nproc=60, no postrefinement.<br />
set the MTZ flag = i_obs<br />
4493 of 5031 integration files were accepted<br />
0 rejected due to wrong Bravais group<br />
11 rejected for unit cell outliers<br />
22 rejected for low signal<br />
505 rejected due to up-front poor correlation under min_corr parameter<br />
0 rejected for file errors or no reindex matrix<br />
Usage: 5m3s.mtz does not contain any observations labelled [fobs, imean, i-obs]. Please set scaling.mtz_column_F to one of [iobs].<br />
File "/net/viper/raid1/sauter/proj-e/modules/cctbx_project/xfel/cxi/util.py", line 13, in is_odd_numbered<br />
return int(os.path.basename(file_name).split(allowable)[0][-1])%2==1<br />
ValueError: invalid literal for int() with base 10: 'd'<br />
<br />
Something is wrong in the ability to determine even/odd numbered-ness. Added "_extracted.pickle" in the code; had to put it first.<br />
<br />
Table of Scaling Results:<br />
<br />
---------------------------------------------------------------------------------------------------------<br />
CC N CC N R R R Scale Scale SpSig<br />
Bin Resolution Range Completeness int int iso iso int split iso int iso Test<br />
---------------------------------------------------------------------------------------------------------<br />
1 -1.0000 - 5.3861 [809/809] 80.0% 809 75.2% 805 61.0% 40.1% 52.9% 0.551 214.059 12489.8850<br />
2 5.3861 - 4.2749 [791/791] 54.9% 791 74.5% 791 53.0% 38.8% 49.7% 0.693 270.307 1785.4625<br />
3 4.2749 - 3.7345 [781/781] 65.8% 781 81.6% 781 46.5% 33.6% 40.7% 0.762 337.287 1149.4218<br />
4 3.7345 - 3.3930 [776/776] 63.9% 776 74.5% 776 49.3% 36.4% 48.6% 0.764 283.109 758.0388<br />
5 3.3930 - 3.1498 [765/765] 67.1% 765 81.9% 765 48.4% 35.6% 43.4% 0.795 338.091 533.7650<br />
6 3.1498 - 2.9641 [771/771] 58.6% 771 72.4% 771 49.3% 36.6% 50.7% 0.759 286.707 222.4718<br />
7 2.9641 - 2.8156 [765/765] 56.0% 765 72.3% 765 48.5% 35.3% 46.7% 0.765 320.954 154.5299<br />
8 2.8156 - 2.6930 [746/746] 63.0% 746 76.1% 746 46.4% 34.3% 42.6% 0.867 357.183 99.4430<br />
9 2.6930 - 2.5894 [790/790] 52.1% 790 69.4% 790 50.4% 37.4% 47.5% 0.814 314.326 113.1264<br />
10 2.5894 - 2.5000 [757/757] 54.9% 757 78.6% 757 52.4% 38.9% 44.4% 0.794 306.403 109.0768<br />
<br />
All [7751/7751] 74.9% 7751 78.8% 7747 51.9% 36.9% 50.1% 0.680 266.538 1298.0<br />
---------------------------------------------------------------------------------------------------------<br />
<br />
Of course we know the data do not scale because this is a polar space group, and data must be sorted by Brehm/Diederichs method.<br />
<br />
== Breaking the indexing ambiguity ==<br />
<br />
Take note of our detail instructions on [[Resolving an Indexing Ambiguity]]. Do this in three steps:<br />
<br />
=== 1) Generate a database of observations ===<br />
<br />
step1.csh:<br />
<br />
<pre><br />
#!/bin/csh -f<br />
<br />
set effective_params = "d_min=DMIN \<br />
data=/net/dials/raid1/aaron/zurich0038/jr_006_batches/split_reintegrated/extracted \<br />
output.n_bins=10 \<br />
pixel_size=0.172 \<br />
backend=FS \<br />
nproc=60 \<br />
merge_anomalous=True \<br />
plot_single_index_histograms=False \<br />
scaling.algorithm=mark1 \<br />
target_unit_cell=91.4,91.4,45.9,90,90,120 \<br />
target_space_group=P6 \<br />
raw_data.sdfac_auto=False \<br />
include_negatives=NEG \<br />
postrefinement.enable=False \<br />
output.prefix=TAG"<br />
<br />
set tag = p6m<br />
set dmin = 2.5<br />
set neg = False<br />
set eff = `echo $effective_params|sed -e "s,FS,Flex,g"|sed -e "s,DMIN,$dmin,g"|sed -e "s,NEG,$neg,g"|sed -e "s,TAG,$tag,g"`<br />
<br />
cxi.merge ${eff}<br />
</pre><br />
<br />
This yields 4988 of 5031 integration files accepted.<br />
<br />
=== 2) Sort the lattices ===<br />
<br />
step2.csh:<br />
<pre><br />
#!/bin/csh -f<br />
<br />
set effective_params = "d_min=DMIN \<br />
pixel_size=0.172 \<br />
target_unit_cell=91.4,91.4,45.9,90,90,120 \<br />
target_space_group=P6 \<br />
backend=FS \<br />
nproc=60 \<br />
merge_anomalous=True \<br />
output.prefix=TAG"<br />
<br />
set tag = p6m<br />
set dmin = 3.5<br />
set neg = False<br />
set eff = `echo $effective_params|sed -e "s,FS,Flex,g"|sed -e "s,DMIN,$dmin,g"|sed -e "s,NEG,$neg,g"|sed -e "s,TAG,$tag,g"`<br />
<br />
cxi.brehm_diederichs ${eff}<br />
</pre><br />
<br />
14 plots total. h,k,l=2503 h,-h-k,-1=2485 total 4988<br />
<br />
=== 3) Apply reindexing operators and merge ===<br />
<br />
== cxi.merge program output ==<br />
<pre><br />
----------------------------------------------------------------------------------------<br />
<asu <obs<br />
Bin Resolution Range Completeness % multi> multi> n_meas <I> <I/sig(I)><br />
----------------------------------------------------------------------------------------<br />
1 -1.000 - 5.386 [1490/1490] 100.00 102.21 102.21 152295 103994 103.244<br />
2 5.386 - 4.275 [1500/1500] 100.00 62.76 62.76 94141 128403 95.046<br />
3 4.275 - 3.735 [1499/1499] 100.00 53.90 53.90 80795 143552 92.607<br />
4 3.735 - 3.393 [1497/1497] 100.00 47.14 47.14 70571 112723 70.575<br />
5 3.393 - 3.150 [1477/1477] 100.00 43.96 43.96 64928 76925 51.011<br />
6 3.150 - 2.964 [1488/1488] 100.00 39.87 39.87 59330 57060 37.899<br />
7 2.964 - 2.816 [1483/1483] 100.00 38.17 38.17 56611 44079 32.085<br />
8 2.816 - 2.693 [1455/1455] 100.00 36.34 36.34 52874 37117 27.460<br />
9 2.693 - 2.589 [1530/1530] 100.00 34.49 34.49 52763 30496 24.443<br />
10 2.589 - 2.500 [1476/1476] 100.00 31.83 31.83 46974 27147 21.564<br />
<br />
All [14895/14895] 100.00 49.10 49.10 731282 76275 55.681<br />
----------------------------------------------------------------------------------------<br />
</pre><br />
<br />
== cxi.xmerge program output ==<br />
<pre><br />
--------------------------------------<br />
Bin Resolution Range # images %accept<br />
--------------------------------------<br />
1 -1.0000 - 5.3861 4712 100.00<br />
2 5.3861 - 4.2749 4663 98.96<br />
3 4.2749 - 3.7345 4646 98.60<br />
4 3.7345 - 3.3930 4614 97.92<br />
5 3.3930 - 3.1498 4578 97.16<br />
6 3.1498 - 2.9641 4552 96.60<br />
7 2.9641 - 2.8156 4521 95.95<br />
8 2.8156 - 2.6930 4499 95.48<br />
9 2.6930 - 2.5894 4477 95.01<br />
10 2.5894 - 2.5000 4416 93.72<br />
<br />
All 4721<br />
--------------------------------------<br />
--------------------------------------------------------------------------------------------------------<br />
CC N CC N R R R Scale Scale SpSig<br />
Bin Resolution Range Completeness int int iso iso int split iso int iso Test<br />
--------------------------------------------------------------------------------------------------------<br />
1 -1.0000 - 5.3861 [1490/1490] 87.3% 1490 88.1% 1484 46.3% 32.9% 42.6% 0.772 300.328 8084.8580<br />
2 5.3861 - 4.2749 [1500/1500] 76.4% 1500 89.3% 1500 43.8% 30.6% 34.5% 0.761 425.498 1728.0907<br />
3 4.2749 - 3.7345 [1499/1499] 80.1% 1499 91.6% 1499 42.5% 26.7% 34.5% 0.684 430.028 1556.6316<br />
4 3.7345 - 3.3930 [1497/1497] 80.5% 1497 90.3% 1497 37.9% 27.2% 29.9% 0.846 481.795 600.5001<br />
5 3.3930 - 3.1498 [1477/1477] 84.2% 1477 90.0% 1477 37.2% 26.4% 31.4% 0.838 477.825 269.5784<br />
6 3.1498 - 2.9641 [1492/1492] 80.0% 1492 91.5% 1492 39.8% 28.6% 28.3% 0.866 511.386 165.9517<br />
7 2.9641 - 2.8156 [1483/1483] 76.7% 1483 90.0% 1483 39.3% 28.7% 30.1% 0.865 470.331 102.0659<br />
8 2.8156 - 2.6930 [1451/1451] 76.8% 1451 90.7% 1451 38.5% 28.2% 27.3% 0.883 492.758 88.6666<br />
9 2.6930 - 2.5894 [1532/1532] 76.6% 1532 89.4% 1532 40.1% 29.3% 30.5% 0.879 452.831 52.0092<br />
10 2.5894 - 2.5000 [1472/1472] 77.2% 1472 88.9% 1474 42.9% 31.4% 35.3% 0.801 393.866 52.6667<br />
<br />
All [14893/14893] 84.7% 14893 88.6% 14889 41.6% 29.0% 39.8% 0.771 378.964 804.8<br />
--------------------------------------------------------------------------------------------------------<br />
</pre><br />
<br />
== Table of results ==<br />
{| class="wikitable" <br />
| style="padding: 5px;"| Tag<br />
| style="padding: 5px;"| Method<br />
| style="padding: 5px;"| Details<br />
| style="padding: 5px;"| Resolution<br>(Angstrom)<br />
| style="padding: 5px;"| # files <br>accepted<br />
| style="padding: 5px;"| CC1/2 <br> (highest shell)<br />
| style="padding: 5px;"| CCiso <br> (highest shell)<br />
| style="padding: 5px;"| <|L|> test<br>(0.5 perfect)<br />
|-<br />
| style="padding: 10px;"| nopost<br />
| style="padding: 5px;"| no postrefinement<br />
| style="padding: 5px;"| scale only<br />
| style="padding: 5px;" | 2.5<br />
| style="padding: 5px;"| 4962 (4828)<br />
| style="padding: 5px;"| 77.5% (66.2%)<br />
| style="padding: 5px;"| 84.0% (85.8%)<br />
| style="padding: 5px;"| 0.475<br />
<br />
|-<br />
| style="padding: 10px;"| basic<br />
| style="padding: 5px;"| rs<br />
| style="padding: 5px;"| refine scale, B, rotx,roty<br />
| style="padding: 5px;" | 2.5<br />
| style="padding: 5px;"| 4942 (4650)<br />
| style="padding: 5px;"| 84.7% (77.2%)<br />
| style="padding: 5px;"| 88.6% (88.9%)<br />
| style="padding: 5px;"| 0.477<br />
|-<br />
| style="padding: 10px;"| trial1<br />
| style="padding: 5px;"| rs2<br>unit weighting<br>lorentzian lineshape<br />
| style="padding: 5px;"| analytical derivatives<br>better convergence test<br>Flex database<br />
| style="padding: 5px;" | 2.5<br />
| style="padding: 5px;"| 4719 (4458)<br />
| style="padding: 5px;"| 88.2% (74.8%)<br />
| style="padding: 5px;"| 89.5% (89.1%)<br />
| style="padding: 5px;"| 0.480<br />
|-<br />
| style="padding: 10px;"| trial2<br />
| style="padding: 5px;"| rs2<br>unit weighting<br>gaussian lineshape<br />
| style="padding: 5px;"| <br />
| style="padding: 5px;" | 2.5<br />
| style="padding: 5px;"| 4721 (4416)<br />
| style="padding: 5px;"| 90.9% (69.6%)<br />
| style="padding: 5px;"| 90.9% (89.1%)<br />
| style="padding: 5px;"| 0.508<br />
|-<br />
| style="padding: 10px;"| trial3<br />
| style="padding: 5px;"| rs_hybrid<br>gentle weighting (|I|/sigma**2)<br>gaussian lineshape<br />
| style="padding: 5px;"| rs2: LBFGS <br> LevMar to refine Rs<br />
| style="padding: 5px;" | 2.5<br />
| style="padding: 5px;"| 4059 (3783)<br />
| style="padding: 5px;"| 93.5% (37.3%)<br />
| style="padding: 5px;"| 95.4% (89.1%)<br />
| style="padding: 5px;"| 0.518<br />
|-<br />
| style="padding: 10px;"| trial3 / cycle2<br />
| style="padding: 5px;"| rs_hybrid<br>gentle weighting (|I|/sigma**2)<br>gaussian lineshape<br>recycle model<br />
| style="padding: 5px;"| Use mtz from trial 3<br> as a scaling reference<br />
| style="padding: 5px;" | 2.5<br />
| style="padding: 5px;"| 3716 (3432)<br />
| style="padding: 5px;"| 92.8% (48.1%)<br />
| style="padding: 5px;"| 93.3% (85.7%)<br />
| style="padding: 5px;"| 0.522<br />
|- <br />
|}<br />
<br />
Useful:<br />
export BOOST_ADAPTBX_FPE_DEFAULT=1<br />
nproc=1<br />
postrefinement.show_trumpet_plot=True</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Updating_cppxfel&diff=1758Updating cppxfel2026-08-19T18:35:25Z<p>Aaron: </p>
<hr />
<div>{{Archived}}<br />
<br />
In the case of updating the cppxfel installation, open a terminal and navigate to the <code>cppxfel</code> directory.<br />
<br />
<pre>cd /absolute/path/to/dials/dials-dev-508/modules/cppxfel</pre><br />
<br />
Then, pull the latest code from the repository.<br />
<br />
<pre>git pull</pre><br />
<br />
If necessary, some command line tools may need to be configured. To do this, type<br />
<br />
<pre>libtbx.configure cppxfel</pre><br />
<br />
Then, the C++ code will need recompiling.<br />
<br />
<pre><br />
cd ../../build<br />
make<br />
</pre><br />
<br />
The new distribution of cppxfel will be available to use.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Tutorials&diff=1757Tutorials2026-08-19T18:35:12Z<p>Aaron: </p>
<hr />
<div>{{Archived}}<br />
<br />
Available tutorials:<br />
<br />
* [https://github.com/phyy-nx/dials_refinement_brewster2018/wiki List of tutorials] as part of our recent paper, [https://doi.org/10.1107/S2059798318009191 Brewster 2018].<br />
* [[2017_dials.stills_process | dials.stills_process]] for processing synchrotron data, including detector position refinement</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Setup&diff=1756Setup2026-08-19T18:34:53Z<p>Aaron: </p>
<hr />
<div>{{Archived}}<br />
<br />
<!-- XXX Should probably rename this page to "Set up ''cctbx.xfel''" or some such --><br />
<br />
<br />
== Prerequisities ==<br />
<br />
The installation root directories of ''cctbx.xfel'' affect the absolute paths used in the following instructions and differ from site to site. Some common installation root directories are listed below.<br />
{| class="wikitable sortable"<br />
|-<br />
! Site<br />
!class="unsortable"| Installation build directory<br />
!class="unsortable"| Installation source directory<br />
!class="unsortable"| Notes<br />
|-<br />
| BNL<br />
| <code>/usr/local/crys_test/cctbx/cctbx_build</code><br />
| <code>/usr/local/crys_test/cctbx</code><br />
|<br />
|-<br />
| CCI, LBNL<br />
|<br />
|<br />
| No central, shared installation available<br />
|-<br />
| NERSC<br />
|<br />
|<br />
| No central, shared installation available<br />
|-<br />
| SLAC<br />
| <code>/reg/g/cctbx/build</code><br />
| <code>/reg/g/cctbx/sources</code><br />
|<br />
|}<br />
''cctbx.xfel'' can be installed anywhere on the filesystem where write access is granted. Developers will generally have ''cctbx.xfel'' installed in their home directory. Even if ''cctbx.xfel'' is already installed, the environment must first be [[Set up PSDM software | prepared for the PSDM software distribution]].<br />
<br />
== ''cctbx.xfel'' setup ==<br />
<br />
To start using ''cctbx.xfel'', the environment has to be configured. The details depend on the ''cctbx.xfel'' installation root directories and the user's shell.<br />
In what follows <code><b><i>/path/to/cctbx/build</i></b></code> and <code><b><i>/path/to/cctbx/sources</i></b></code> should be replaced with the actual build and source directories of the ''cctbx.xfel'' installation. bash-users can prepare the environment for ''cctbx.xfel'' using<br />
$ . <b><i>/path/to/cctbx/build</i></b>/setpaths.sh<br />
From a csh, the corresponding command is<br />
% source <b><i>/path/to/cctbx/build</i></b>/setpaths.csh<br />
Sourcing <code>setpaths</code> only modifies the environment of the <em>current shell</em>, and would have to be repeated every time a new shell is started. To make the changes persistent, add<br />
test -r <b><i>/reg/g/cctbx/build</i></b>/setpaths.sh && \<br />
. <b><i>/reg/g/cctbx/build</i></b>/setpaths.sh<br />
to <code>~/.bashrc</code>, or<br />
test -r <b><i>/reg/g/cctbx/build</i></b>/setpaths.csh && \<br />
source <b><i>/reg/g/cctbx/build</i></b>/setpaths.csh<br />
to <code>~/.cshrc</code> as appropriate.<br />
<br />
To make the ''cctbx.xfel'' analysis modules available to PSDM's ''pyana'',<br />
$ cd <b><i>/path/to/test/release</i></b><br />
$ sit_setup<br />
$ cd <b><i>my_ana_pkg</i></b><br />
$ ln -fns <b><i>/path/to/cctbx/sources</i></b>/cctbx_project/xfel/cxi/cspad_ana src<br />
$ cd ..<br />
$ scons<br />
where <code><b><i>/path/to/test/release</i></b></code> and <code><b><i>my_ana_pkg</i></b></code> are the path to the test release and the name of the analysis package chosen while [[Set up PSDM software | setting up the PSDM software distribution]]. The last step compiles the ''cctbx.xfel'' analysis modules. These steps need only be performed once.<br />
<br />
<!--<br />
At SLAC, test that it is working by <i>e.g.</i><br />
$ cxi.pyana -c /reg/g/cctbx/tutorials/setup/test.cfg /reg/d/ana11/cxi/data/Mar2013calib/xtc/e236-r0004-s00-c00.xtc<br />
This should start a viewer, displaying dark-subtracted averages from an XTC stream. Press the green run button to see the images. This particular stream has a series of misses at its beginning; hits will appear presently. More specifically, this command starts ''pyana'' with a configuration file (<code>-c</code>) and an XTC file. For more information, see the [[Overview]] and other tutorial pages.<br />
--></div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Set_up_PSDM_software&diff=1755Set up PSDM software2026-08-19T18:33:59Z<p>Aaron: </p>
<hr />
<div>{{Archived}}<br />
<br />
THIS IS DEPRECATED. psana is now provided using conda. [https://exafel.github.io/docs/psana-cctbx-instal Directions].<br />
<br />
Note that throughout the instructions <code>$</code> denotes the prompt of any unprivileged user on the system. Certain shells may be configured to use <code>%</code> or <code>&gt;</code> for that purpose. The prompt should <em>not</em> be included in the commands typed to the shell.<br />
<br />
== Prerequisites ==<br />
<br />
The [https://confluence.slac.stanford.edu/display/PSDM/Software+Distribution Photon Science Data Management (PSDM)] software distribution must be installed. The PSDM software distribution provides ''psana'', which is a critical dependency of ''cctbx.xfel''. Its installation root directory affects the absolute paths used in the following instructions and differs from site to site. Some common installation root directories are listed below.<br />
{| class="wikitable sortable"<br />
|-<br />
! Site<br />
! class="unsortable"| Installation root directory<br />
! class="unsortable"| Notes<br />
|-<br />
| BNL<br />
| <code>/usr/local/crys/psdm</code><br />
|<br />
|-<br />
| CCI, LBNL<br />
| <code>/net/viper/raid1/psdm</code><br />
| Only available on viper.lbl.gov<br />
|-<br />
| NERSC<br />
| <code>/global/project/projectdirs/lcls/psdm-carver</code><br />
| Only to be used on the carver system<br />
|-<br />
| SLAC<br />
| <code>/reg/g/psdm</code><br />
|<br />
|}<br />
The PSDM software distribution is supported on the most popular flavors of Linux and is installed at several sites worldwide. Separate installation and maintenance instructions are provided by the PSDM project at [https://confluence.slac.stanford.edu/display/PSDM/Software+Distribution PSDM software distribution]. The package can be installed anywhere on the filesystem, and it is generally possible to install it without superuser privileges.<br />
<br />
== Create a test release and an analysis package ==<br />
<br />
A <i>test release</i>, which refers to a particular release of the PSDM distribution, is represented by a directory in the file system, often called <code>analysis-rel</code> or <code>myrelease</code>. An analysis package within the test release in turn refers to the files implementing the analysis modules of the package. Details about test releases and analysis packages are covered in [https://confluence.slac.stanford.edu/display/PCDS/Analysis+Workbook.+Quick+Tour Analysis Workbook. Quick Tour]. To prepare the environment for using the PSDM software distribution, bash-users should source <code>ana_env.sh</code>,<br />
$ . <b><i>/path/to/psdm</i></b>/etc/ana_env.sh<br />
where <code><b><i>/path/to/psdm</i></b></code> should be replaced with the path to the installation root directory. csh-users should source <code>ana_env.csh</code> instead,<br />
% source <b><i>/path/to/psdm</i></b>/etc/ana_env.csh<br />
Sourcing <code>ana_env.sh</code> or <code>ana_env.csh</code> only modifies the environment of the <em>current shell</em>, and would have to be repeated every time a new shell is started. To make the changes persistent, add<br />
test -r <b><i>/path/to/psdm</i></b>/etc/ana_env.sh && . <b><i>/path/to/psdm</i></b>/etc/ana_env.sh<br />
to <code>~/.bashrc</code>, or<br />
test -r <b><i>/path/to/psdm</i></b>/etc/ana_env.csh && source <b><i>/path/to/psdm</i></b>/etc/ana_env.csh<br />
to <code>~/.cshrc</code> as appropriate. This ensures the environment is properly prepared for non-interactive shells, <i>e.g.</i> the shells that are used to run jobs submitted to the cluster. To also be able to use the PSDM software from interactive shells, it is recommended that bash-users put<br />
test -r ~/.bashrc && . ~/.bashrc<br />
near the top of <code>~/.bash_profile</code>. There is no need to edit any additional files for csh-users, because csh always reads <code>~/.cshrc</code>.<br />
<br />
A test released based on the most current PSDM distribution is then set up in the current working directory using<br />
$ newrel ana-current <i>myrelease</i><br />
$ cd <i>myrelease</i><br />
$ sit_setup<br />
$ newpkg <i>my_ana_pkg</i><br />
Note that the names for the test release and the analysis package, <i>myrelease</i> and <i>my_ana_pkg</i> above, are common, but nevertheless arbitrary, choices.<br />
<br />
These steps only need to be performed once.<br />
<br />
== External links ==<br />
<br />
* [https://confluence.slac.stanford.edu/display/PSDM/psana+-+User+Manual Psana User Manual]<br />
* [https://confluence.slac.stanford.edu/display/PSDM/Packages+and+Releases Packages and Releases]</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Resolving_an_Indexing_Ambiguity&diff=1754Resolving an Indexing Ambiguity2026-08-19T18:33:35Z<p>Aaron: </p>
<hr />
<div>{{Archived}}<br />
<br />
Here we describe the use of [http://dx.doi.org/10.1107/S1399004713025431 Brehm & Diederichs algorithm 2] to resolve the indexing ambiguity for XFEL data. This is applicable for all polar space groups (where the Bravais symmetry is higher than the space group symmetry) and also for cases with pseudo symmetry (e.g., a monoclinic cell with a near 90-degree beta angle).<br />
<br />
== Brief Description of the Workflow ==<br />
<br />
It is assumed that the reader is familiar with the tutorial for [[Merging]]. The usual workflow of <code>cxi.merge</code> followed by <code>cxi.xmerge</code> will fail in the xmerge step if there are reindexing operators (not H,K,L) that relate the individual indexed lattices to one another. We resolve this by (1) running the <code>cxi.merge</code> step to generate a database with all the observations; (2) use Brehm-Diederichs algorithm 2 to identify the reindexing operators with <code>cxi.brehm_diederichs</code>; (3) re-run the <code>cxi.merge + cxi.xmerge</code> process with the additional list of reindexing operators as input.<br />
<br />
== Detailed Step-By-Step Instructions ==<br />
<br />
=== Create a Database of Observations ===<br />
<br />
<pre><br />
$ vi myoglobin_step1.csh<br />
#!/bin/csh -f<br />
set trial=${1}<br />
<br />
set runs = 127,130,132,134,135,140,141,142,144<br />
set datastring = \<br />
`python -c "print ' '.join(['data=/my_results/L785/r%04d/${trial}/integration'%i for i in [${runs}]])"`<br />
set tag = myoglobin_${trial}<br />
<br />
set effective_params = "d_min=2.0 \<br />
output.n_bins=20 \<br />
${datastring} \<br />
scaling.algorithm=mark1 \<br />
target_unit_cell=90.3,90.3,45.2,90,90,120 \<br />
target_space_group=P6 \<br />
nproc=16 \<br />
merge_anomalous=True \<br />
mysql.runtag=${tag} \<br />
mysql.passwd=terp888 \<br />
mysql.user=nick \<br />
mysql.database=xfelnks \<br />
scaling.mtz_file=fake_filename.mtz \<br />
pixel_size = 0.079346 \<br />
output.prefix=${tag}"<br />
<br />
cxi.merge ${effective_params} # Note the xmerge script is NOT run here<br />
<br />
$./myoglobin_step1.csh 009 # create the database from trial 009<br />
</pre><br />
<br />
Noteworthy parameters:<br />
* <code>d_min</code> is the high-resolution limit to be used for resolving the indexing ambiguity, not final merging. There is a trade-off: including too many data slows down the determination of lattice-lattice correlation coefficients, which scales as the number of common Miller indices. Including too few data makes the determination of correlation coefficients unreliable or undefined. Future: the program will print out <L>, the average number of observation pairs per correlation coefficient so the resolution limit can be sensibly adjusted. <br />
* <code>scaling.algorithm</code> is set to mark1 here (no scaling) to illustrate how data would be processed from an unknown structure. As a consequence (see the [[Advanced Merging]] page) we set the <code>target_unit_cell</code> and <code>target_space_group</code> but do not provide a PDB <code>model</code>. Also <code>scaling.mtz_file</code> is set to a dummy value.<br />
* <code>nproc</code> is the number of processors to be used for writing the database. Use as many single-host processors as are available up to a limit of about 16, beyond which there is no further benefit, at least on Linux.<br />
* <code>merge_anomalous=True</code>. Since we are trying to maximize the number of common Miller indices for each lattice pair we want to merge the Bijvoet pairs, even if we will look for dispersive differences in the final merging step.<br />
* <code>backend</code>. Use whatever backend is available on your system; choices are FS, MySQL (default), and SQLite.<br />
<br />
=== Sort the Lattices ===<br />
<br />
<pre><br />
$ vi myoglobin_step2.csh<br />
#!/bin/csh -f<br />
set trial=${1}<br />
set tag = myoglobin_${trial}<br />
<br />
set effective_params = "d_min=2.0 \<br />
target_unit_cell=90.3,90.3,45.2,90,90,120 \<br />
target_space_group=P6 \<br />
nproc=32 \<br />
merge_anomalous=True \<br />
mysql.runtag=${tag} \<br />
mysql.passwd=terp888 \<br />
mysql.user=nick \<br />
mysql.database=xfelnks \<br />
pixel_size = 0.079346 \<br />
output.prefix=${tag}"<br />
<br />
cxi.brehm_diederichs ${effective_params}<br />
<br />
$./myoglobin_step2.csh 009 # sort the lattices from trial 009<br />
</pre><br />
<br />
Noteworthy parameters:<br />
* <code>d_min</code> is the high-resolution limit to be used for resolving the indexing ambiguity, not final merging. There is a trade-off: including too many data slows down the determination of lattice-lattice correlation coefficients, which scales as the number of common Miller indices. Including too few data makes the determination of correlation coefficients unreliable or undefined. Future: the program will print out <L>, the average number of observation pairs per correlation coefficient so the resolution limit can be sensibly adjusted. <br />
* <code>nproc</code> is the number of processors used on a single host. Must either be set to 1 or >=5; there is no implementation for 2-4. If there are a large number of images (>2000) it is advantageous to set this number as high as possible [we use 64 on our AMD Linux machine]. BUT IF THERE ARE FEWER THAN ~500 IMAGES, you must use nproc=1.<br />
* <code>target_unit_cell</code>, <code>target_space_group</code> and <code>merge_anomalous</code> are mandatory and should be set to the same values used in the myoglobin_step1.csh script above. <br />
* Likewise, <code>mysql.runtag</code> and <code>output.prefix</code> should use the same values.<br />
<br />
Output files from this script:<br />
* <b>${tag}_intensities_presort.pickle</b> contains a Python tuple with two Miller arrays, the first giving all intensites input into the calculation, and the second assigning each intensity to a lattice [crystal] number. For development, not actually used for anything.<br />
* <b>${tag}_lookup.pickle</b> is a Python dictionary whose keys are the reindexing operators, and values are lists of lattice [crystal] numbers to be reindexed accordingly.<br />
* <b>${tag}_reverse_lookup.pickle</b> is a Python dictionary whose keys are the original integrated data pickle file names, and values are the reindexing operator to be applied to each file. This output is actually used in the next step.<br />
<br />
=== Apply Reindexing Operators and Merge ===<br />
<br />
<pre><br />
$ vi myoglobin_step3.csh<br />
#!/bin/csh -f<br />
set trial=${1}<br />
<br />
set runs = 127,130,132,134,135,140,141,142,144<br />
set datastring = \<br />
`python -c "print ' '.join(['data=/my_results/L785/r%04d/${trial}/integration'%i for i in [${runs}]])"`<br />
set tag = myoglobin_${trial}_step3<br />
<br />
set effective_params = "d_min=1.8 \<br />
d_max=6.0 \<br />
output.n_bins=20 \<br />
${datastring} \<br />
merging.reverse_lookup=myoglobin_${trial}_reverse_lookup.pickle<br />
scaling.algorithm=mark0 \<br />
model=3u3e.pdb \<br />
model_reindex_op=h,-h-k,-l \<br />
nproc=16 \<br />
merge_anomalous=False \<br />
mysql.runtag=${tag} \<br />
mysql.passwd=terp888 \<br />
mysql.user=nick \<br />
mysql.database=xfelnks \<br />
scaling.mtz_file=3u3e.data.mtz \<br />
scaling.mtz_column_F=fmodel \<br />
pixel_size = 0.079346 \<br />
output.prefix=${tag}"<br />
<br />
cxi.merge ${effective_params} <br />
cxi.xmerge ${effective_params} <br />
<br />
$./myoglobin_step3.csh 009 # apply the reindexing operators and merge<br />
</pre><br />
<br />
Noteworthy parameters:<br />
* <code>d_min</code> is now set to the highest-resolution desired for the merging step.<br />
* <code>merge_anomalous=False</code>. If looking for dispersive differences, set this to False.<br />
* <code>scaling.algorithm</code> is set to mark0 here to illustrate scaling of a P6 myoglobin XFEL data set against an isomorphous structure already in the PDB.<br />
* <code>model</code> The PDB file used for scaling individual crystals in the cxi.merge step. <br />
* <code>model_reindex_op</code> A total kludge. Since there is an indexing ambiguity and we've sorted it out for the XFEL data, there is still the possibility that the reference model is indexed the "wrong" way. This parameter lets us reindex the reference to coincide with the XFEL data. Default value is "h,k,l", no reindexing.<br />
* <code>scaling.mtz_file</code>. For the statistics in cxi.xmerge, this is set to an mtz file with Fcalc's from <em>phenix.fmodel 3u3e.pdb high_resolution=1.78 output.type=real</em>. <br />
* <code>scaling.mtz_column_F</code> is set to whatever column name contains the reference F's in this case "fmodel".<br />
* <code>d_max</code> is set so that isomorphous correlation coefficients are not calculated on the very low resolution reflections, since the Fcalc's have no solvent contribution. If an observed reference data set were used for the mtz_file this precaution would be unnecessary.<br />
* <code>nproc</code> is the number of processors to be used for merging. Use as many single-host processors as are available up to a limit of about 16, beyond which there is no further benefit, at least on Linux.<br />
* <code>merging.reverse_lookup</code> Sets the cxi.merge script to reindex all the integrated data files according to the operators determined in step 2, and to ignore any integrated data for which no reindexing operator was determined. In the cxi.xmerge script, each reindexing operator in turn is considered in order to report isomorphous CC and R statistics against the reference data set. It is up to the user to look at the output to find the reindexing that gives the highest CCiso values.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Progress_monitoring&diff=1753Progress monitoring2026-08-19T18:33:20Z<p>Aaron: </p>
<hr />
<div>{{Archived}}<br />
<br />
If configured to do so, ''cctbx.xfel'' can use a mysql database during hit finding to log the hit rate and other useful parameters, enabling real-time monitoring via a graphical display. <br />
<br />
== Configuring trial monitoring ==<br />
Add to the mod_hitfind section of your pyana config file these lines:<br />
<pre><br />
# cxi.lsf will replace this with its deterimined id, otherwise<br />
# set it to the ID you wish. 0 is the default.<br />
trial_id = 0<br />
db_logging = True<br />
sql_buffer_size = 5<br />
</pre><br />
<br />
These three parameters turn on and configure database-backed logging of hitfinding results:<br />
* trial_id: we find it useful to organize processing in numbered trials. Whenever we go back and change the parameters of our processing, adjusting the target cell or spotfinder parameters for example, we re-process under a different trial ID. When you submit your processing job using cxi.lsf, you can specify the trial ID on the command line with –t. What you specify will overwrite what is listed here in the cfg file. If you use bsub, then what is in the cfg file is what’s used. Note, it isn’t a good idea to re-use the trial_id when re-prossing the data if database logging is being used, simply because the hits will be logged twice which will clutter future analysis. Finally, if you want to see which trials you have used for which runs, execute '''cxi.list_db_metadata'''<br />
* db_logging: Switch for turning on and off logging<br />
* sql_buffer_size: a queue size for regulating how often to add data to the database. In our experience, 5 has been just fine.<br />
<br />
== Monitoring hit finding in real time ==<br />
After submitting a job with db logging enabled, use cxi.monitor_trials <trial number> to display the hit rate during processing. Other command line options and their defaults are listed below, including the ability to display multiple runs at once. The default is to display the 5 latest runs by timestamp.<br />
* t_wait = 8000: amount of time in ms between queries to the database<br />
* hit_cutoff = 16: the number of bragg spots before a frame is considered a hit. If this is different than what’s in your config file, then the monitor will be using a different cutoff to display which frames are hits.<br />
* average_window = 1000: how many frames are in the sliding averaging window shown in the hit % graph<br />
* n_points = 1000: how many total points to display on the graph. Set it to 0 to display all the data points (can be computationally intensive).<br />
* display_time = 1800: number of seconds of experiment time to display. Displays this number of seconds unless that would add up to more than 5 runs.<br />
* run_num = display only this run<br />
* run_min = start at this run<br />
* run_max = end with this run</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Processing_L498_thermolysin&diff=1752Processing L498 thermolysin2026-08-19T18:32:59Z<p>Aaron: </p>
<hr />
<div>{{Archived}}<br />
<br />
This page contains instructions for reproducing the results reported in the 2014 ''cctbx.xfel'' paper<ref name="Hattne:2014">[http://dx.doi.org/10.1038/nmeth.2887 Hattne, J <i>et al.</i> Accurate macromolecular structures using minimal measurements from X-ray free-electron lasers. <i>Nat. Methods</i> <b>In press</b> (2014).]</ref>. The data are available from the [http://cxidb.org Coherent X-ray Imaging Data Bank] ([http://cxidb.org/id-23 CXIDB ID 23]; [http://cxidb.org/id-23-raw.html raw XTC files], 4.0 TiB), and must be downloaded to a local disk prior to starting the analysis. Furthermore, the [https://confluence.slac.stanford.edu/display/PSDM/Software+Distribution PSDM Software Distribution] must be [[Set up PSDM software | set up]], along with a test release and an empty analysis package. A [http://www.mysql.com MySQL] database is required to merge the integrated diffraction intensities. In the interest of keeping this guide as general as possible, no particular queuing system is assumed to be available. Processing time depends strongly on the computational resources available; using 48 processors on a 64-core 1.4 GHz Opteron-based computer, the analysis takes around 24 hours. The instructions assume some familiarity with ''cctbx.xfel'' and its installation and configuration procedure.<br />
<br />
<br />
== Installing a ''cctbx.xfel'' snapshot from March 28, 2013 ==<br />
<br />
The thermolysin data for the ''cctbx.xfel'' paper was processed around March 28, 2013. Unfortunately, regular nightly releases are not available for that time, but a custom source-code bundle has been prepared. This bundle differs from the regular ''cctbx'' bundles in that [http://adder.lbl.gov/labelit ''LABELIT''] is included, and that the directory layout is identical to that of a developer installation. To download and unpack the bundle in the current directory:<br />
$ wget http://adder.lbl.gov/cctbx.xfel/downloads/cctbx.xfel-20130328.tar.gz<br />
$ tar -xpvzf cctbx.xfel-20130328.tar.gz<br />
Next, create a build directory (called <code><i>phenix-build-20130328</i></code> below, but that is an arbitrary choice), in which the sources are configured and compiled. The Python interpreter required to complete this step <em>must</em> be the one supplied by the PSDM Software Distribution. Once the PSDM software has been [[Set up PSDM software | set up]], this interpreter can be located using<br />
$ find $SIT_ROOT/sw/external/python -perm /0111 -type f -wholename "*/$SIT_ARCH/*/python"<br />
To prepare the build directory using this interpreter<br />
$ mkdir <i>phenix-build-20130328</i><br />
$ cd <i>phenix-build-20130328</i><br />
$ <b><i>python</i></b> ../phenix-src-20130328/cctbx_project/libtbx/configure.py cxi_xdr_xes xfel<br />
$ . setpaths.sh<br />
where <code><b><i>python</i></b></code> is the path to the Python interpreter located using the previous find-command. Note that csh-users should run <code>source setpaths.csh</code> instead of <code>. setpaths.sh</code>. Then compile the sources:<br />
$ make<br />
$ make<br />
Note that the <code>make</code> command may need to be run twice in order to complete the build. Once <code>make</code> does not produce any output from the compiler, the build is complete.<br />
<br />
To make the ''cctbx.xfel'' analysis modules available to PSDM's ''pyana'',<br />
$ cd <b><i>/path/to/test/release</i></b><br />
$ sit_setup<br />
$ cd <b><i>my_ana_pkg</i></b><br />
$ ln -fns <b><i>/path/to</i></b>/phenix-src-20130328/cctbx_project/xfel/cxi/cspad_ana src<br />
$ cd ..<br />
$ scons<br />
where <code><b><i>/path/to/test/release</i></b></code> and <code><b><i>my_ana_pkg</i></b></code> are the path to the test release and the name of the analysis package chosen while [[Set up PSDM software | setting up the PSDM software distribution]], respectively. <code><b><i>/path/to</i></b></code> denotes the path to the directory containing the unpacked ''cctbx.xfel'' sources. The last step compiles the ''cctbx.xfel'' analysis modules.<br />
<br />
<br />
== Creating a dark image ==<br />
<br />
To meaningfully process the thermolysin diffraction data, an average of all the images in a <i>dark run</i>—a run without any X-rays impinging on the detector—must be subtracted from the individual diffraction images. The configuration file below can be used to produce such an average, as well as an image of the standard deviation of all the pixels over the course of the run. <br />
[pyana]<br />
modules = my_ana_pkg.mod_average<br />
num-cpu = <i>4</i><br />
<br />
[my_ana_pkg.mod_average]<br />
address = CxiDs1-0|Cspad-0<br />
calib_dir = <b><i>/path/to</i></b>/phenix-src-20130328/cctbx_project/xfel/metrology/CSPad/run4/CxiDs1.0_Cspad.0<br />
avg_basename = <i>Ds1-avg</i><br />
avg_dirname = <i>r0031</i><br />
stddev_basename = <i>Ds1-stddev</i><br />
stddev_dirname = <i>r0031</i><br />
<b><i>/path/to</i></b> refers to the directory containing the unpacked ''cctbx.xfel'' sources. All other options with values set in italics can be modified without adversely affecting averaging. The above file will use four simultaneous processes, and write the average and standard deviation images to files whose names start with <code>Ds1-avg</code> and <code>Ds1-stddev</code>, respectively, both in a directory called <code><i>r0031</i></code>. <br />
<br />
The data deposited at the [http://cxidb.org/id-23 CXIDB] contains a dark run, <code>r0031</code>. To average the images in that run, save the above configuration file to disk, <i>e.g.</i> <code><i>L498-dark.cfg</i></code>, apply modifications as necessary, and execute<br />
$ cxi.pyana -c <i>L498-dark.cfg</i> <b><i>/path/to/xtc/files</i></b>/e157-r0031-*.xtc<br />
where <b><i>/path/to/xtc/files</i></b> is the path to the directory containing the raw XTC files downloaded from [http://cxidb.org/id-23 CXIDB]. The files written to the <code><i>r0031</i></code> directory will have a current date stamp appended, which can safely be removed to simplify subsequent configuration files.<br />
$ cd <i>r0031</i><br />
$ mv <i>Ds1-avg20140308104336073.pickle</i> Ds1-avg.pickle<br />
$ mv <i>Ds1-stddev20140308104336371.pickle</i> Ds1-stddev.pickle<br />
$ cd ..<br />
Note that the date stamp on the generated files above depends on the time of their creation. Further details are available on the [[Preparatory steps#Create a dark average|Create a dark image]] page of the [[Tutorials|tutorials]].<br />
<br />
<br />
== Indexing the thermolysin data ==<br />
<br />
A configuration file for processing the primary lattices in the thermolysin data is shown below.<br />
[pyana]<br />
modules = my_ana_pkg.mod_hitfind:threshold \<br />
my_ana_pkg.mod_hitfind:index<br />
num-cpu = <i>48</i><br />
<br />
[my_ana_pkg.mod_hitfind]<br />
address = CxiDs1-0|Cspad-0<br />
calib_dir = <b><i>/path/to</i></b>/phenix-src-20130328/cxi_xdr_xes/cftbx/metrology/CSPad/run4/CxiDs1.0:Cspad.0<br />
dark_path = <i>r0031/Ds1-avg.pickle</i><br />
dark_stddev = <i>r0031/Ds1-stddev.pickle</i><br />
db_logging = False<br />
detz_offset = 575<br />
<br />
[my_ana_pkg.mod_hitfind:threshold]<br />
dispatch = nop<br />
distl_flags = permissive<br />
distl_min_peaks = 16<br />
threshold = 450<br />
xtal_target = hitfind<br />
<br />
[my_ana_pkg.mod_hitfind:index]<br />
dispatch = index<br />
integration_dirname = <i>integration-first-lattice</i><br />
integration_basename = int-<br />
xtal_target = thermolysin27<br />
The configuration file above instructs <code>mod_hitfind</code> to use 48 processes. It disables all image output, which reduces the amount of disk space required to perform the analysis to about 3.4 GiB. <b><i>/path/to</i></b> again refers to the directory containing the unpacked ''cctbx.xfel'' sources, and <code>dark_path</code> as well as <code>dark_stddev</code> may have to be changed to reflect the location of the previously generated dark images. Integration results will be written to the directory <i>integration-first-lattice</i>.<br />
<br />
Due to particularities of the thermolysin measurement, processing needs to proceed in two batches. To analyze the first batch, runs 16 through 27, save the above configuration file to disk, <i>e.g.</i> <code><i>L498-indexigrate.cfg</i></code>, apply modifications as necessary, and execute<br />
$ cxi.pyana -c <i>L498-indexigrate.cfg</i> <b><i>/path/to/xtc/files</i></b>/e157-r0016-*.xtc<br />
$ cxi.pyana -c <i>L498-indexigrate.cfg</i> <b><i>/path/to/xtc/files</i></b>/e157-r0017-*.xtc<br />
$ …<br />
$ cxi.pyana -c <i>L498-indexigrate.cfg</i> <b><i>/path/to/xtc/files</i></b>/e157-r0027-*.xtc<br />
where <b><i>/path/to/xtc/files</i></b> is the path to the directory containing the raw XTC files. The second batch, runs 71 through 73, was recorded using a different distance between the interaction region and the detector. Whilst the changes to the detector position are automatically handled by ''cctbx.xfel'', the resulting difference in shadowing is not. Different areas of the detector should be ignored at the different distances, and this is accounted for by the value of the <code>xtal_target</code> option in the configuration file. To analyze this set of runs, edit <code><i>L498-indexigrate.cfg</i></code>, change <code>thermolysin27</code> to <code>thermolysin73</code>, and<br />
$ cxi.pyana -c <i>L498-indexigrate.cfg</i> <b><i>/path/to/xtc/files</i></b>/e157-r0071-*.xtc<br />
$ cxi.pyana -c <i>L498-indexigrate.cfg</i> <b><i>/path/to/xtc/files</i></b>/e157-r0072-*.xtc<br />
$ cxi.pyana -c <i>L498-indexigrate.cfg</i> <b><i>/path/to/xtc/files</i></b>/e157-r0073-*.xtc<br />
On successful completion, the number of files in the <i>integration-first-lattice</i> directory corresponds to the number of successfully integrated images. Owing to variations in hardware and compiler internals, it may deviate slightly from 11,583, the number reported in the ''cctbx.xfel'' paper<ref name="Hattne:2014"/>. Further details are available on the [[Indexing and integration|indexing and integration]] page of the [[Tutorials|tutorials]].<br />
<br />
<br />
=== Indexing the secondary lattice ===<br />
<br />
Indexing the secondary lattice is very similar to indexing the primary lattice, but requires a change to the source code. Edit <code><b><i>/path/to</i></b>/phenix-src-20130328/labelit_regression/xfel/xfel_targets.py</code>, and uncomment (<i>i.e.</i> remove the leading <code>#</code> character) <code>"outlier_detection.switch=True"</code> on line 25. Then edit the configuration file, <code><i>L498-indexigrate.cfg</i></code> above, and change <i>integration-first-lattice</i> to <i>integration-second-lattice</i> in order not to overwrite the results of the previous analysis of the primary lattice. Before reanalyzing the first batch, ensure that <code>xtal_target</code> is set to <code>thermolysin27</code>.<br />
$ cxi.pyana -c <i>L498-indexigrate.cfg</i> <b><i>/path/to/xtc/files</i></b>/e157-r0016-*.xtc<br />
$ cxi.pyana -c <i>L498-indexigrate.cfg</i> <b><i>/path/to/xtc/files</i></b>/e157-r0017-*.xtc<br />
$ …<br />
$ cxi.pyana -c <i>L498-indexigrate.cfg</i> <b><i>/path/to/xtc/files</i></b>/e157-r0027-*.xtc<br />
Then set <code>xtal_target</code> is set to <code>thermolysin73</code> in the configuration file, and process the second batch.<br />
$ cxi.pyana -c <i>L498-indexigrate.cfg</i> <b><i>/path/to/xtc/files</i></b>/e157-r0071-*.xtc<br />
$ cxi.pyana -c <i>L498-indexigrate.cfg</i> <b><i>/path/to/xtc/files</i></b>/e157-r0072-*.xtc<br />
$ cxi.pyana -c <i>L498-indexigrate.cfg</i> <b><i>/path/to/xtc/files</i></b>/e157-r0073-*.xtc<br />
The number of integrated secondary lattices should be close to 2,021, the number reported in the ''cctbx.xfel'' paper<ref name="Hattne:2014"/>.<br />
<br />
== Merging all integrated images ==<br />
<br />
The phil-file below defines values suitable for merging the primary and secondary lattices previously integrated.<br />
data = <i>integration-first-lattice</i><br />
data = <i>integration-second-lattice</i><br />
d_min = 2.10<br />
merge_anomalous = True<br />
min_corr = -1<br />
model = <i>2tli.pdb</i><br />
nproc = <i>16</i><br />
plot_single_index_histograms = False<br />
raw_data.sdfac_auto = True<br />
rescale_with_average_cell = True<br />
significance_filter.apply = True<br />
set_average_unit_cell = True<br />
pixel_size = 0.11<br />
mysql {<br />
database = <i>db_name</i><br />
passwd = <i>db_passwd</i><br />
runtag = <i>L498_thermolysin</i><br />
user = <i>db_user</i><br />
}<br />
output {<br />
n_bins = <i>10</i><br />
prefix = <i>L498_thermolysin</i><br />
}<br />
scaling {<br />
algorithm = mark0<br />
mtz_file = <i>2tli.mtz</i><br />
show_plots = False<br />
log_cutoff = 0.0<br />
}<br />
<code><i>integration-first-lattice</i></code> and <code><i>integration-second-lattice</i></code> may need to be adjusted to point to the directories where the [[#Indexing and integration | indexing and integration step]] left its results. <code><i>db_name</i></code><code>, <i>db_user</i></code>, and <code><i>db_passwd</i></code> must be substituted with the database name and access credentials to a [http://www.mysql.com MySQL] database. Databases on hosts other than the one used to merge the thermolysin data can be accessed by additionally specifying the <code>mysql.host</code> and <code>mysql.port</code> options. The model and structure factors for the scaling reference, <code>model</code> and <code>scaling.mtz_file</code> above, are both available for download from the [http://www.rcsb.org RCSB Protein Data Bank] ([http://www.rcsb.org/pdb/explore/explore.do?structureId=2tli PDB ID 2tli]). If the [http://www.phenix-online.org ''PHENIX''] suite is installed, these are conveniently obtained at the command line using<br />
$ phenix.fetch_pdb --mtz 2tli<br />
<br />
To merge the thermolysin data, save the suitably modified configuration file to <i>e.g.</i> <code>L498-merge.phil</code>, and run<br />
$ cxi.merge L498-merge.phil<br />
$ cxi.xmerge L498-merge.phil<br />
Merging statistics are printed on standard output. The merged MTZ-file is written to a file whose name is determined by the value of <code>output.prefix</code> in the configuration file (with the values shown above, the output file would be <code>L498_thermolysin.mtz</code>). Note that the version of merging programs from 28 March, 2013 do <em>not</em> not report the <i>R</i><sub>split</sub> statistic.<br />
<br />
== References ==<br />
<br />
<references/></div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Processing_Blog&diff=1751Processing Blog2026-08-19T18:32:44Z<p>Aaron: </p>
<hr />
<div>{{Archived}}<br />
<br />
Data Processing Blog<br />
<br />
== Reindexing operator needed to identify C-axis in an orthorhombic setting (6 May 2014 NKS)==<br />
The user brought us data from an orthorhombic space group. The synchrotron structure had been solved and published with this unit cell:<br />
<br />
space group P 2<sub>1</sub> 2<sub>1</sub> 2<sub>1</sub><br />
unit cell 107.5 75.0 51.3 90 90 90<br />
<br />
However, the program <code>cxi.merge</code> fails to merge any XFEL data when the synchrotron-solved PDB file is used as the "<code>model</code>" for isomorphous scaling. Knowing that the axes of an orthorhombic group can be permuted, we immediately suspected a mismatch between the XFEL unit cell and the PDB unit cell. To inspect the XFEL integration results, we use the command:<br />
<br />
<code>cxi.print_pickle [integration_result_file.pickle]</code><br />
<br />
All of the integration pickle files had this unit cell:<br />
<br />
Laue group P 2 2 2 unit cell 50.1 73.6 103.8 90 90 90<br />
<br />
Therefore we added the following item to the phil-parameters for <code>cxi.merge</code>:<br />
<br />
<code>data_reindex_op=l,k,h</code><br />
<br />
This reindexing had the desired effect of aligning the new data with the reference model.<br />
<br />
== A tricky case of merging resolution limits ==<br />
Merging some data today where the default per-frame resolution filter settings were cutting things a little too harshly, and discarding a lot more data than expected (based on manual inspection of a maximum projection pattern). So, how to change this? CXTBX.XFEL performs a per-frame resolution test at two stages: during integration, and during merging. Here, we are going to discuss editing the during merging part of things. There are two phil parameters you can play with for this: significance_filter.sigma (default 0.5), and significance_filter.max_ct (default 50). The sigma option will remove resolution bins such that all accepted bins have Mean(I/sigma) >= sigma, and the max_ct option will set the bin sizes so that the number of bins is maximized, subject to mean bin population <= max_ct. <br />
<br />
A quick inspection of the terminal output from cxi.merge showed that in many cases, the Mean(I/sigma) was jumping up and down a lot, for example: <br />
<pre><br />
Step 4. Filter on global resolution and map to asu<br />
<br />
Bin Resolution Range Compl. <I> <I/sig(I)><br />
1 22.4216 - 8.1030 [127/5229] 646.3 3.8<br />
2 8.1030 - 6.4828 [ 45/5157] 428.9 1.7<br />
3 6.4828 - 5.6785 [103/5257] 89.3 0.7<br />
4 5.6785 - 5.1663 [107/5182] 62.2 0.4<br />
5 5.1663 - 4.7999 [101/5201] 121.2 0.5<br />
6 4.7999 - 4.5193 [ 78/5189] 93.9 0.3<br />
7 4.5193 - 4.2946 [ 83/5191] 36.2 0.2<br />
8 4.2946 - 4.1089 [ 71/5156] 197.4 0.7<br />
9 4.1089 - 3.9516 [ 67/5199] 43.5 0.3<br />
10 3.9516 - 3.8159 [ 74/5170] 83.3 0.4<br />
11 3.8159 - 3.6971 [110/5194] 12.7 0.2<br />
12 3.6971 - 3.5919 [ 76/5227] 16.9 0.2<br />
</pre><br />
<br />
Using the defaults, this file would cut off at 5.7 A, but I would like it to go to 4.8, since it looks like that is a legitimate 0.5 in the I/sigI column. Changing the bin size should average this out. I therefore re-ran the merging with the following in my phil file:<br />
<br />
<pre><br />
significance_filter{ <br />
sigma=0.5<br />
max_ct=100<br />
}<br />
</pre><br />
<br />
This helped a bit, and the per-image cuttoffs were looking more sensible. I then tweaked the sigma down to 0.4, since it decreased gradually to around this point before going mental, and got even better results. <br />
<br />
When doing this, how do you know when you are just adding junk to your file? If you observe lots of variation in your intensities per image per bin, then it's fairly safe to increase max_ct. This has the effect of smoothing things out, but may also underestimate your resolution if the bins become too large. Reducing sigma, on the other hand, can easily introduce junk data. The only real way to test for this is to see if your statistics from cxi.xmerge, in particular CC1/2, get better. <br />
<br />
--[[User:Zeldin|Zeldin]] ([[User talk:Zeldin|talk]]) 23:19, 17 April 2014 (UTC)<br />
<br />
== A Traceback error in April ==<br />
<br />
15 April 2014. A user writes:<br />
<br />
I am seeing a Traceback error during integration with pyana:<br />
<pre><br />
/cctbx_project/myrelease/cxi_data/present_image_0000001<br />
<br />
Beam center is not immediately clear; rigorously retesting 9 solutions<br />
Beam x 96.6 y 97.1, initial score 9; refined rmsd: None<br />
Beam x 96.8 y 97.5, initial score 8; refined rmsd: None<br />
Beam x 96.6 y 97.0, initial score 7; refined rmsd: None<br />
Beam x 96.6 y 97.2, initial score 7; refined rmsd: None<br />
Beam x 96.7 y 97.4, initial score 6; refined rmsd: None<br />
Beam x 96.6 y 97.3, initial score 6; refined rmsd: None<br />
Beam x 96.9 y 97.0, initial score 6; refined rmsd: None<br />
Beam x 96.9 y 97.1, initial score 6; refined rmsd: None<br />
Process Process-2:<br />
<br />
Traceback (most recent call last):<br />
File "/usr/local/crys_test/psdm/sw/external/python/2.7.2/x86_64-rhel6-gcc44-opt/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap<br />
self.run()<br />
File "/usr/local/crys_test/psdm/sw/external/python/2.7.2/x86_64-rhel6-gcc44-opt/lib/python2.7/multiprocessing/process.py", line 114, in run<br />
self._target(*self._args, **self._kwargs)<br />
File "/usr/local/crys/psdm/sw/releases/ana-current/arch/x86_64-rhel6-gcc44-opt/python/pyana/pyanamod.py", line 126, in _proc<br />
for req in proto.getRequest():<br />
File "/usr/local/crys/psdm/sw/releases/ana-current/arch/x86_64-rhel6-gcc44-opt/python/pyana/mp_proto.py", line 121, in getRequest<br />
opcode = self._conn.recv_bytes()<br />
EOFError<br />
</pre><br />
<br />
Reply:<br />
<br />
* It's not clear what's causing the EOFError. However, we should at least check for memory leaks that would cause Pyana processing to draw to a standstill. The diagnostic is to use the "top" command in the Unix shell. Observe whether or not the subprocess memory use continually increases as more images are processed. Correct behavior is for the memory use to reach a stationary plateau. We recently fixed a memory leak bug (March 2014), but it is worth checking this to make sure.<br />
<br />
* You should generally not get a "rigorously retesting 9 solutions" message for XFEL indexing. Generally for XFELs we know the beam position accurately so there is no need to search for the direct beam. Your phil file should include beam_search_scope=0.5 to avoid a large search radius. However, your log indicates a narrow search, likely indicating that spot finder sees numerous spots that are unphysically close. Many things could be done to safeguard you from this:<br />
** Careful use of detector masks, as described in [[Preparatory steps]] on the tutorial wiki . This would avoid picking spots on compromised areas of the detector. Aaron is this correct?<br />
** Use distl.minimum_spot_area=1 instead of =0. This would avoid picking random local-maxima pixels as spots, but there is a tradeoff.<br />
** Set the phil file resolution cutoff (3 different places) to a value close to where you actually intend to cut off the merging. This avoids picking spots for indexing that aren't real spots.<br />
<br />
* I notice you are using integration.detector_gain=7.5. However, we've been having a lot of trouble with setting this to anything else but 1.0. Until we can sort out the code, I recommend using gain=1.0 during processing, and then in the cxi.merge phil use significance_filter.sigma=0.5. This combination should help retain as many reflections as possible.<br />
<br />
<br />
== Low resolution reflections missing ==<br />
<br />
A user was indexing using cxi.index and finding many low resolution spots not being predicted. Here, cxi.index refuses to predict spots that are overlapped. Setting integration.background_factor=1 and integration.guard_width_sq=3 can force a better outcome.<br />
<br />
<br />
== Indexing images with fewer spots ==<br />
<br />
A user was having a hard time with images that had clear diffraction patterns with between 20 to 40 spots but that didn't index with cxi.index. Here's how we resolved the problem.<br />
<br />
Add the following to the indexing phil file:<br />
<br />
1) target cell and known setting. These parameters provide constraints on the still-image indexing procedure algorithm that finds basis vectors without the benefit of additional images with rotational data. See the [[phil]] section. Example for lysozyme:<br />
target_cell = 38 74 74 90 90 90<br />
known_setting = 9<br />
<br />
2) Add distl_minimum_number_spots_for_indexing=20 and model_refinement_minimum_N=10 to the phil file. These parameters set lower limits on the number of spots needed for indexing and crystal model refinement. For patterns with fewer spots, it disables two checks for enough spots to proceed.<br />
<br />
== Wrong Bravais lattice chosen during indexing ==<br />
<br />
Some of the user's images produced integration results in the triclinic setting, but they had included a target cell in the hexagonal setting. Here are the lines at the end of the log file that show the symptom:<br />
<br />
$ cxi.index -d -n 16 target=<parameters.phil> <image.pickle><br />
<br />
New Horizons Integration results:<br />
Solution SpaceGroup Beam x y distance Resolution Mosaicity RMS<br />
12 P3 156.63 156.52 374.79 4.00 0.5 0.246<br />
:) 1 P1 156.63 156.52 374.79 11.63 0.5 0.041<br />
<br />
The :) shows that the software chose the triclinic setting for its final solution. The explanation is that LABELIT compares the ratio of RMS values from the highest to the lowest symmetry settings (RMS is the root mean squared difference between observed and predicted spot positions, in mm). The default is to allow a factor 3.5 ratio, no more. This parameter can be altered in the *.phil file or on the command line:<br />
<br />
$ cxi.index -d -n 16 target=<parameters.phil> <image.pickle> mosflm_rmsd_tolerance=10<br />
<br />
The parameter is called "mosflm" rmsd tolerance for legacy reasons: LABELIT uses MOSFLM as the integration engine; however cctbx.xfel has its own integration engine.<br />
Result:<br />
<br />
New Horizons Integration results:<br />
Solution SpaceGroup Beam x y distance Resolution Mosaicity RMS<br />
:) 12 P3 156.63 156.52 374.79 4.00 0.5 0.246<br />
1 P1 156.63 156.52 374.79 11.63 0.5 0.041<br />
<br />
Now the higher symmetry setting is chosen as the final solution.<br />
<br />
<br />
<br />
<br />
<br />
== Processing stills ==<br />
<br />
Here's a recent example of processing still from the MAR detector at XPP<br />
<br />
General reference:<br />
[[Indexing individual stills]]<br />
<br />
All the commands listed here have some help with the --help option.<br />
<br />
cd <path to directory with .mccd files from xpp><br />
cxi.image2pickle -v -c *.mccd<br />
mkdir pickles<br />
mv -v *.pickle pickles<br />
<br />
Converting the images to pickle format, using the -c (crop) option, sets them up a bit nicer for the indexer Next, I picked one of the high res images and looked at the spotfinding parameters for the program:<br />
<br />
distl.image_viewer distl.minimum_signal_height=5 distl.minimum_spot_area=2 distl.minimum_spot_height=5 pickles/<one of the image files>.pickle<br />
<br />
My initial guess (based on my experience with the MAR detector) showed a reasonable diffraction pattern picked out in red by the spotfinder.<br />
<br />
I then took a phil file that had been working well for us from another experiment, and edited it using these spotfinding parameters. Seeing as I didn't know the unit cell, I removed the target cell. I created a results directory and called this first test trial 000, and indexed all the images.<br />
<br />
mkdir -p results/000<br />
cxi.index -d -s -n 12 -o results/000 target=[[stills_target.phil]] pickles | tee results/trial_000.out<br />
<br />
This indexed around 202 images. To get an idea of the unit cell, I entered the trial 000 directory and ran<br />
cluster.unit_cell --log . Most of the images were in the following cluster:<br />
<br />
cluster_26 166 192.5(1.3 ) 192.5(1.5 ) 291.1(2.9 ) 90.00 (0.00) 90.00 (0.00) 90.00 (0.00)<br />
163 in P4, 3 in P222.<br />
<br />
The numbers in parentheses are standard deviations for those cell dimensions. Now, indexing stills is greatly helped when the symmetry is known upfront, so I added a target_cell to the phil file, like so:<br />
<br />
# Derived from average cell after scaling and merging.<br />
target_cell = 192.5 192.5 291.1 90.00 90.00 90.00<br />
known_setting = 9<br />
<br />
For the space group, cxi.index uses a known_setting parameter. P4 is in the tetragonal setting. From this webpage:<br />
<br />
[[Phil]]<br />
<br />
that means I need to use a known setting value of 9.<br />
<br />
I decided to use the LCLS computing cluster for the next indexing trial. I was logged into psana on an interactive node provided to me by the cluster, and others were logged in to the same node concurrently. The queuing system would ensure I got my own node for the job:<br />
<br />
mkdir results/001<br />
bsub -n 12 -q psanaq -o results/trial_001.out cxi.index -s -n 12 -d -o results/001 target=[[stills_target.phil]] pickles<br />
<br />
This yielded 218 indexed images.<br />
<br />
I did two rounds of merging/postrefinement with prime.postrefine. Post refinement trial 000, using indexing trial 001, in directory 001_000, I used the wrong number of amino acids per asymmetric unit, but it got me a sense of the completeness limits. Using that I remerged to an ambitious 3.3 angstroms, in directory 001_001. The files of interest are:<br />
<br />
prime/001_001/log.txt<br />
prime/001_001/postref_cycle_2_merge.mtz<br />
<br />
Command for merging:<br />
bsub -n 12 -q psanaq -o results/trial_001.out cxi.index -s -n 12 -d -o results/001 target=[[prime.phil]] pickles<br />
<br />
Documentation for prime is on the wiki.<br />
<br />
Where to go from here? I would look at which files indexed and which didn’t. See if there are some that didn’t index that should have. Maybe that could improve the completeness. One could look into prime.iota, which does a grid search for spotfinder parameters. Also, cxi.index, if you don’t supply the -d flag, will pop up a few images showing the indexing results overlaid on the image (first in the triclinic setting and then in the best symmetry setting). Red pixels are spotfinder spots, blue are integrated and yellow are background. One could run cxi.index with a single file to see how it's doing (don’t forget to supply a target, like the command above).</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Preparatory_steps_pyana_version&diff=1750Preparatory steps pyana version2026-08-19T18:32:29Z<p>Aaron: </p>
<hr />
<div>{{Archived}}<br />
<br />
This tutorial covers averaging dark and light runs, and creating mask images.<br />
<br />
<br />
== Setup your scratch area ==<br />
<br />
Create a scratch folder and populate it with some directories that will be used during this tutorial, changing <code><b><i><username></i></b></code> to your SLAC account name:<br />
<br />
$ cd /reg/g/cctbx/tutorials/scratch/<br />
$ mkdir <b><i><username></i></b><br />
$ cd <b><i><username></i></b><br />
$ mkdir darks<br />
$ mkdir averages<br />
$ mkdir results<br />
<br />
Or you may use a directory in your home folder.<br />
<br />
<br />
== Create a dark average ==<br />
<br />
The CSPAD detector at LCLS is sensitive to temperature distortions and, unlike images collected at a synchrotron source, its output is not automatically dark-subtracted. Because of this it is necessary to periodically record a dark run with the shutters closed during an LCLS experiment. The dark run should contain on the order of at least 1000 images. Before an experimental run can be processed, this dark run must be averaged so that subsequent experimental data can have the dark subtracted from each image. Furthermore, events during collection can necessitate a new dark, such as ice in the beam over-saturating creating dead pixels or a change in temperature.<br />
<br />
To create the average, start by copying the <code>dark.cfg</code> file to your pyana directory:<br />
$ cd ~/myrelease<br />
$ sit_setup<br />
$ cp /reg/g/cctbx/tutorials/preprocessing/dark.cfg .<br />
<br />
Open <code>dark.cfg</code>. It will look something like this:<br />
[pyana]<br />
modules = my_ana_pkg.mod_average<br />
num-cpu = 8<br />
<br />
[my_ana_pkg.mod_average]<br />
calib_dir = /reg/g/cctbx/sources/cctbx_project/xfel/metrology/CSPad/run4/CxiDs1.0_Cspad.0<br />
avg_dirname = /reg/g/cctbx/tutorials/scratch/<b><i><username></i></b>/darks<br />
stddev_dirname = /reg/g/cctbx/tutorials/scratch/<b><i><username></i></b>/darks<br />
max_dirname = /reg/g/cctbx/tutorials/scratch/<b><i><username></i></b>/darks<br />
detz_offset = 581<br />
address = CxiDs1-0|Cspad-0<br />
avg_basename = Ds1-r0002-avg<br />
stddev_basename = Ds1-r0002-stddev<br />
max_basename = Ds1-r0002-max<br />
<br />
The configuration file is broken into two sections, the [[#''pyana'' section | ''pyana'' section]] and the [[#Modules section | modules section]].<br />
<br />
<br />
=== ''pyana'' section ===<br />
The ''pyana'' section specifies which modules to run. In this case we are running ''cctbx.xfel'''s <code>mod_average</code> module. If we were running multiple modules, we could chain them together here, and each module would be executed in sequence using the parameters specified in the corresponding module section. We also request a number of processes to parallelize the job with. Even though averaging must be done on a single host, on that host the processing can be split up among multiple cores. Note, other aspects of ''cctbx.xfel'', such as indexing and integration, do not have this restriction.<br />
<br />
<br />
=== Module section ===<br />
The module section specifies parameters specific to averaging:<br />
;calib_dir<br />
: this directory is provided by LCLS and includes the layout information of the 64 tiles of the CSPAD detector<br />
;avg_dirname<br />
: directory to place the average image<br />
;stddev_dirname<br />
: directory to place the standard deviation image<br />
;max_dirname<br />
: directory to place the maximum projection image<br />
;detz_offset<br />
: experiment-specific number used to calculate the detector distance. This is the distance in millimeters between the sample interaction region and the far end of the detector stage. The XTC stream contains the distance from the detector stage to the detector. These two numbers are subtracted to calculate the distance between sample interaction region and the detector. We determine this number experimentally using a grid search, testing a range of potential detz_offsets and choosing the one that shows the greatest success at indexing images<br />
;address<br />
: the name of the detector in the XTC stream<br />
;avg_basename<br />
: this name is prepended to a date stamp to name the average image<br />
;stddev_basename<br />
: this name is prepended to a date stamp to name the standard deviation image<br />
;max_basename<br />
: this name is prepended to a date stamp to name the maximum projection image image<br />
<br />
<br />
=== Running and testing the command ===<br />
<br />
Before starting the averaging job the command, replace <code><b><i><username></i></b></code> in the configuration file with your user name. To average run 2 using the modified configuration file, execute<br />
$ cxi.lsf -c ~/myrelease/dark.cfg -i /reg/d/ana11/cxi/data/Mar2013calib/xtc \<br />
-o /reg/g/cctbx/tutorials/scratch/<b><i><username></i></b> -p 6 -q psanaq -r 2 -s<br />
This will request 6 processors, overriding the <code>num-cpu</code> parameter set in the configuration file. The directory specified in the <code>-i</code> option will be searched for XTC streams, and output will be arranged in the directory defined by the value of the <code>-o</code> option. The <code>-s</code> option tells ''cxi.lsf'' to process all the streams on the same host—this is a requirement for averaging. The job will be scheduled in the queue called <code>psanaq</code>, which is one of several available queues. Note that the job must be submitted from the <code>~/myrelease</code> directory.<br />
<br />
Type <code>bjobs</code> to see if the job is running. When it is, follow the output thusly:<br />
$ tail -f /reg/g/cctbx/tutorials/scratch/<b><i><username></i></b>/r0002/000/stdout/sNN.out<br />
Note the presence of the <code>000</code> directory. ''cxi.lsf'' places the output of each job in a separate <i>trial</i> directory, and prints this directory just before exiting. By default, trial numbers start at zero and increase by one with each subsequent analysis of the same run. To use any other, previously unused, trial number, use ''cxi.lsf'''s <code>-t</code> option.<br />
<br />
When the average has been completed, the output can be inspected using the ''cctbx'' image viewer:<br />
$ cctbx.image_viewer /reg/g/cctbx/tutorials/scratch/<b><i><username></i></b>/r0002/000/out/Ds1-r0002-*.pickle<br />
<br />
== Averaging an experimental light run ==<br />
<br />
Once the dark average is completed, experimental, lighted, runs can be averaged to see the extent of diffraction and potential pathologies in the run. Copy the <code>average.cfg</code> file from the tutorials directory to your pyana directory:<br />
$ cd ~/myrelease<br />
$ cp /reg/g/cctbx/tutorials/preprocessing/average.cfg .<br />
<br />
The configuration file will look like this:<br />
[pyana]<br />
modules = my_ana_pkg.mod_average<br />
num-cpu = 8<br />
<br />
[my_ana_pkg.mod_average]<br />
calib_dir = /reg/g/cctbx/sources/cctbx_project/xfel/metrology/CSPad/run4/CxiDs1.0_Cspad.0<br />
dark_path = /reg/g/cctbx/tutorials/scratch/<b><i><username></i></b>/r0002/000/out/<b><i><name of dark average></i></b>.pickle<br />
dark_stddev = /reg/g/cctbx/tutorials/scratch/<b><i><username></i></b>/r0002/000/out/<b><i><name of dark stddev></i></b>.pickle<br />
avg_dirname = /reg/g/cctbx/tutorials/scratch/<b><i><username></i></b>/averages<br />
stddev_dirname = /reg/g/cctbx/tutorials/scratch/<b><i><username></i></b>/averages<br />
max_dirname = /reg/g/cctbx/tutorials/scratch/<b><i><username></i></b>/averages<br />
detz_offset = 581<br />
address = CxiDs1-0|Cspad-0<br />
avg_basename = Ds1-r0003-avg<br />
stddev_basename = Ds1-r0003-stddev<br />
max_basename = Ds1-r0003-max<br />
The only difference is the addition of two parameters:<br />
;dark_path<br />
: path to the average dark image created before<br />
;dark_stddev<br />
: path to the standard deviation dark image created before<br />
<br />
The job is ran as before using ''cxi.lsf'', again remembering to change all instances of <code><b><i><username></i></b></code> to your own SLAC user name. Here you must also change <code><b><i><name of dark average></i></b></code> and <code><b><i><name of dark stddev></i></b></code> to the names of the average and standard deviation images created in the previous step. The result of the run will be three new images that can be viewed with ''cctbx.image_viewer'' in the same manner as the darks were.<br />
<br />
<br />
== Information contained in the light average ==<br />
<br />
Firstly, the light and dark images can be used to mask of any bad pixels. Inactive and non-bonded pixels, as well as hypersensitive pixels and any beam stop (if used) should be masked out. The light images (maximum in particular) also serve as a virtual, dark-subtracted, powder pattern that can used to inspect the data for diffraction. The light maximum should also flag up obvious errors in metrology, because rings will be non-continuous if the quadrants are misaligned. Intensity values from the corners of the light average or maximum are also useful to estimate background when taking first guesses at thresholds for hit-finding and integration.<br />
<br />
<br />
== Creating a mask image ==<br />
''cctbx.xfel'' uses three images to create a mask. Those images, and the purposes of them are listed below:<br />
* Average image from a dark run: pixels ≤ 0 are considered dead, pixels > 2000 are too sensitive<br />
* Standard deviation from a dark run: pixels ≤ 0 are considered dead, pixels ≥ 10 are too uncertain<br />
* Maximum projection from a lighted run (''i.e.'' an experimental run): pixels < 300 are considered non-bonded or in a shadow. [N.B.–this assumes a lot–300 may be appropriate for XPP data collected at atmosphere; not CXI data <i>in vacuo</i>–NKS.] The presence of diffracting data is not needed, but also will not interfere.<br />
<br />
To create the mask, execute these commands:<br />
$ cd /reg/g/cctbx/tutorials/scratch/<b><i><username></i></b>/<br />
$ mkdir masks<br />
$ cd masks<br />
$ cxi.make_mask -m 10 <b><i><name of dark average></i></b>.pickle <b><i><name of dark standard deviation></i></b>.pickle \<br />
<b><i><name of light maximum projection></i></b>.pickle<br />
<br />
A new image is created where each pixel has one of two values: -2 (masked out pixels), and 0 (good pixels). Note, the maximum projection minimum pixel value is overridden here to a value that is more sensible for the tutorial's lysozyme data. You will likely need to adjust this parameter based on your data (see below for details and for further overrides). <br />
<br />
Use the resultant mask image during processing with ''cctbx.xfel'' by adding this flag to your pyana config file:<br />
mask_path = <b><i><path to mask file></i></b><br />
In addition, a polygon and a circle can be specified using the <code>--poly_mask</code> and <code>--circle_mask</code> parameters:<br />
$ cxi.make_mask --poly_mask=17,854,76,967,760,982,762,879 --circle_mask=855,939,103 Cspad-avg.pickle Cspad-stddev.pickle Cspad-max.pickle<br />
This example creates a mask image with a polygon [with vertices (17,854), (76,967), (760,982), and (762,879)] masked out, and a circle with center (855,939) and radius 103 masked out. Only one polygon and circle can be specified at the moment.<br />
<br />
The mask can be examined with ''cctbx.image_viewer''.<br />
<br />
==== Applying the mask ====<br />
<br />
Once the mask is created, it needs to be referenced by your configuration file:<br />
<br />
mask_path = <absolute path to mask file><br />
<br />
Further, the indexing software needs to know what value to use as the masked-out value. Add the following to your phil files for indexing:<br />
<br />
integration.mask_pixel_value=-2<br />
<br />
Where -2 is the default mask value, but it can be whatever you specify using the -x option cxi.make_mask (see below).<br />
<br />
=== Options ===<br />
<br />
Some additional options are available for specifying the details of the masking operation:<br />
;-a, --avg_max (default 2000)<br />
: specifies the maximum value in ADU a pixel in the average image can have before it's masked out. The minimum is always zero.<br />
;-s, --stddev_max (default 10)<br />
: specifies the maximum value in ADU a pixel in the standard deviation image can have before it's masked out. The minimum is always zero.<br />
;-m, --maxproj_min (default 300)<br />
: specifies the minimum value in ADU a pixel in the maximum projection image can have before it's masked out. Of the three parameters for controlling cutoffs in ''cxi.make_mask'', the -m option is unique from the -a and -s options in that it will likely vary the according to your sample's background. Carefully examine the corners of your lighted maximum projection and choose a value lower than the ADU values displayed. Also note that there is no cutoff on the high end of the maximum projection image specified in the mask, as that is defined as the saturation value for the detector.<br />
;-x, --mask_pix_val (default -2)<br />
: specifies the value to use when masking out a pixel. In other words, bad pixels in images in the XTC stream will be replaced with this value.<br />
;-o, --output (default mask_.pickle)<br />
: specifies the output file path, should be *.pickle, can include directory.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Preparatory_steps&diff=1749Preparatory steps2026-08-19T18:32:08Z<p>Aaron: </p>
<hr />
<div>{{Archived}}<br />
<br />
This tutorial covers averaging dark and light runs, and creating mask images.<br />
<br />
== Setup your scratch area ==<br />
<br />
Create a scratch folder and populate it with some directories that will be used during this tutorial. Usually you will work within the ftc directory of your experiment, for example:<br />
<br />
/reg/d/psdm/cxi/cxi49812/ftc<br />
<br />
In that directory, add a directory for your username, changing <code><b><I>expname</i></b></code> to your experiment name and <code><b><i><username></i></b></code> to your SLAC account name:<br />
<br />
$ cd /reg/d/psdm/cxi/<b><i><expname></i></b>/ftc<br />
$ mkdir <b><i><username></i></b><br />
$ cd <b><i><username></i></b><br />
$ mkdir darks<br />
$ mkdir averages<br />
$ mkdir results<br />
<br />
You may use a directory in your home folder for saving results but this is not advised as it will fill quickly.<br />
<br />
Finally, create a directory in your myrelease folder to hold config files and phil files needed during processing:<br />
$ cd ~/myrelease<br />
$ mkdir <b><i><expname></i></b><br />
<br />
== Create a dark average ==<br />
<br />
The CSPAD detector at LCLS is sensitive to temperature distortions and, unlike images collected at a synchrotron source, its output is not automatically dark-subtracted. Because of this it is necessary to periodically record a dark run with the shutters closed during an LCLS experiment. The dark run should contain on the order of at least 1000 images. Before an experimental run can be processed, this dark run must be averaged so that subsequent experimental data can have the dark subtracted from each image. Furthermore, events during collection can necessitate a new dark, such as ice in the beam over-saturating creating dead pixels or a change in temperature.<br />
<br />
To create the average, you will need this config file: [[dark.cfg]]. Save it to your myrelease/<b><i><expname></i></b> folder.<br />
<br />
The configuration file is broken into two sections, the [[#psana section | ''psana'' section]] and the [[#Modules section | modules section]].<br />
<br />
=== ''psana'' section ===<br />
The ''psana'' section specifies which modules to run. In this case we are running the built in module <code>CSPadPixCoords.CSPadNDArrProducer</code>. This module is fully documented in [https://confluence.slac.stanford.edu/display/PSDM/psana+-+Module+Catalog#psana-ModuleCatalog-ModuleCSPadPixCoords::CSPadNDArrProducer SLAC's module catalog]. If we were running multiple modules, we could chain them together here, and each module would be executed in sequence using the parameters specified in the corresponding module section. <br />
<br />
=== Module section ===<br />
The module section specifies parameters specific to the CSPadPixCoords.CSPadNDArrProducer. The user should only ever need to change the source, which dictates which detector is going to be averaged. This string can be obtained from the beamline operator.<br />
<br />
=== Running and testing the command ===<br />
<br />
To average a dark run, the user can use cxi.mpi_average. Run it with no parameters to get a full help listing with examples. To test it on 10 events before using the queuing system, run it as follows:<br />
<br />
$ cxi.mpi_average -c '''''expname'''''/dark.cfg -x '''''expname''''' -r '''''runnumber''''' -a '''''address''''' -d '''''detzoffset''''' -n 10 -p<br />
<br />
The options are described as follows:<br />
<br />
* -c: path to the dark config file<br />
* -x: your experiment name<br />
* -r: the dark run number<br />
* -a: the address string for the detector. Will be something like CxiDs1.0:Cspad.0<br />
* -d: The detz offset. This is the offset (in mm) from the sample interaction region to the back the CSPAD detector rail at CXI. Typically it's on the order of between 560 and 580 mm. The beamline operator will provide you with a value. At XPP you must specify the detector distance directly as their CSPAD is on a robot arm.<br />
* -p: write the output as image pickle files. Currently the standard file format for ''cctbx.xfel''.<br />
<br />
When complete, three new .pickle files will be present in your directory. The dark average will be named '''''expname'''''_avg-r'''''runnum'''''.pickle. A similar stddev and max file will be found as well. Use the -o option to put these averages somewhere else, such as a location available for all users to access for your experiment, e.g. /reg/d/psdm/'''''instrumentname'''''/'''''expname'''''/res/darks<br />
<br />
Now, to submit the averaging job to the processing queue using many processor and all the events, use bsub:<br />
<br />
$ bsub -a mympi -n 100 -q psanacsq -o dark_average.log cxi.mpi_average -c '''''expname'''''/dark.cfg -x '''''expname''''' -r '''''runnumber''''' -a '''''address''''' -d '''''detzoffset''''' -p<br />
<br />
The job will be scheduled in the queue called <code>psanacsq</code>, which is one of several available queues, using 100 cores. Note that the job must be submitted from the <code>~/myrelease</code> directory.<br />
<br />
Type <code>bjobs</code> to see if the job is running. When it is, follow the output thusly:<br />
$ tail -f dark_average.log<br />
<br />
When the average has been completed, the output can be inspected using the ''cctbx'' image viewer:<br />
$ cctbx.image_viewer <path to pickle files><br />
<br />
== Averaging an experimental light run ==<br />
<br />
Experimental, lighted, runs can be averaged to see the extent of diffraction and potential pathologies in the run. Here, you will need this config file [[average.cfg]]. Copy the file from to your myrelease/'''''expname''''' directory.<br />
<br />
The configuration file uses two new modules, [https://confluence.slac.stanford.edu/display/PSDM/psana+-+Module+Catalog#psana-ModuleCatalog-Modulecspad_mod.CsPadCalib cspad_mod.CsPadCalib] and pyimgalgos.cspad_gainmap (currently undocumented). The former uses darks prepared by [[calibman]] for dark subtraction and performs common mode correction. The latter applies any high/low gain pixel mask applied during data collection. Note how the data from one module is passed to the next module using inkey and outkey. cxi.mpi_average looks for the final data using the outkey ''image0''.<br />
<br />
Submit the job and view the results exactly as before, with the exception of using this average.cfg file instead.<br />
<br />
== Information contained in the light average ==<br />
<br />
Firstly, the light and dark images can be used to mask of any bad pixels. Inactive and non-bonded pixels, as well as hypersensitive pixels and any beam stop (if used) should be masked out. The light images (maximum in particular) also serve as a virtual, dark-subtracted, powder pattern that can used to inspect the data for diffraction. The light maximum should also flag up obvious errors in metrology, because rings will be non-continuous if the quadrants are misaligned. Intensity values from the corners of the light average or maximum are also useful to estimate background when taking first guesses at thresholds for hit-finding and integration.<br />
<br />
<br />
== Creating a mask image ==<br />
''cctbx.xfel'' uses three images to create a mask. Those images, and the purposes of them are listed below:<br />
* Average image from a dark run: pixels ≤ 0 are considered dead, pixels > 2000 are too sensitive<br />
* Standard deviation from a dark run: pixels ≤ 0 are considered dead, pixels ≥ 10 are too uncertain<br />
* Maximum projection from a lighted run (''i.e.'' an experimental run): pixels < 300 are considered non-bonded or in a shadow. [N.B.–this assumes a lot–300 may be appropriate for XPP data collected at atmosphere; not CXI data <i>in vacuo</i>–NKS.] The presence of diffracting data is not needed, but also will not interfere.<br />
<br />
To create the mask, execute this command:<br />
$ cxi.make_mask -m 10 <b><i><name of dark average></i></b>.pickle <b><i><name of dark standard deviation></i></b>.pickle \<br />
<b><i><name of light maximum projection></i></b>.pickle<br />
<br />
A new image is created where each pixel has one of two values: -2 (masked out pixels), and 0 (good pixels). Note, the maximum projection minimum pixel value is overridden here to a value that is more sensible for the tutorial's lysozyme data. You will likely need to adjust this parameter based on your data (see below for details and for further overrides). <br />
<br />
Use the resultant mask image during processing with ''cctbx.xfel'' by adding this flag to your pyana config file:<br />
mask_path = <b><i><path to mask file></i></b><br />
In addition, a polygon and a circle can be specified using the <code>--poly_mask</code> and <code>--circle_mask</code> parameters:<br />
$ cxi.make_mask --poly_mask=17,854,76,967,760,982,762,879 --circle_mask=855,939,103 Cspad-avg.pickle Cspad-stddev.pickle Cspad-max.pickle<br />
This example creates a mask image with a polygon [with vertices (17,854), (76,967), (760,982), and (762,879)] masked out, and a circle with center (855,939) and radius 103 masked out. Only one polygon and circle can be specified at the moment.<br />
<br />
The mask can be examined with ''cctbx.image_viewer''.<br />
<br />
==== Applying the mask ====<br />
<br />
Once the mask is created, it needs to be referenced by your configuration file:<br />
<br />
mask_path = <absolute path to mask file><br />
<br />
Further, the indexing software needs to know what value to use as the masked-out value. Add the following to your phil files for indexing:<br />
<br />
integration.mask_pixel_value=-2<br />
<br />
Where -2 is the default mask value, but it can be whatever you specify using the -x option cxi.make_mask (see below).<br />
<br />
=== Options ===<br />
<br />
Some additional options are available for specifying the details of the masking operation:<br />
;-a, --avg_max (default 2000)<br />
: specifies the maximum value in ADU a pixel in the average image can have before it's masked out. The minimum is always zero.<br />
;-s, --stddev_max (default 10)<br />
: specifies the maximum value in ADU a pixel in the standard deviation image can have before it's masked out. The minimum is always zero.<br />
;-m, --maxproj_min (default 300)<br />
: specifies the minimum value in ADU a pixel in the maximum projection image can have before it's masked out. Of the three parameters for controlling cutoffs in ''cxi.make_mask'', the -m option is unique from the -a and -s options in that it will likely vary the according to your sample's background. Carefully examine the corners of your lighted maximum projection and choose a value lower than the ADU values displayed. Also note that there is no cutoff on the high end of the maximum projection image specified in the mask, as that is defined as the saturation value for the detector.<br />
;-x, --mask_pix_val (default -2)<br />
: specifies the value to use when masking out a pixel. In other words, bad pixels in images in the XTC stream will be replaced with this value.<br />
;-o, --output (default mask_.pickle)<br />
: specifies the output file path, should be *.pickle, can include directory.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Phil&diff=1748Phil2026-08-19T18:31:39Z<p>Aaron: </p>
<hr />
<div>{{Archived}}<br />
<br />
As described in the [[overview]], ''phil'' files contain parameters used during hitfinding, indexing and integration. This tutorial uses two ''phil'' files during the indexing and integration step: Ls04-lysozyme.phil and metrology-7.1.phil. The former specifies parameters specific to the processing run, while the latter specifies whole-pixel and sub-pixel metrology corrections applied to the 64 CSPAD sensor tiles.<br />
<br />
== Hitfinding/indexing/integration ''phil'' file ==<br />
<br />
For the tutorial, Ls04-lysozyme.phil, stored in the /reg/d/ffb/cxi/temp/cctbx/tutorials/indexing directory but copied to your ~/myrelease directory during use, contains configuration settings we worked out that best process this data. The file will look like this:<br />
<br />
<pre><br />
# -*- mode: Conf -*-<br />
<br />
include file metrology-7.1.phil<br />
<br />
# From looking at 35 images integrated with detz_offset = 581 and<br />
# without target_cell set.<br />
target_cell = 38 79 79 90 90 90<br />
target_cell_centring_type = P<br />
known_setting = 9<br />
<br />
distl_highres_limit = 2.0<br />
force_method2_resolution_limit = 2.0<br />
<br />
mosaicity_limit = 1<br />
<br />
# Set to True to pick up second lattice, if present.<br />
#outlier_detection_switch = True<br />
<br />
# TEST<br />
distl_minimum_number_spots_for_indexing = 20<br />
<br />
distl {<br />
res.outer = 2.0<br />
minimum_signal_height = 5<br />
#minimum_spot_height = 10<br />
minimum_spot_height = 5<br />
minimum_spot_area = 1<br />
spot_area_maximum_factor = 20<br />
compactness_filter = False<br />
#method2_cutoff_percentage = 5<br />
method2_cutoff_percentage = 2.5<br />
<br />
# Avoids intensity filter.<br />
#peak_intensity_maximum_factor = 10000<br />
peak_intensity_maximum_factor = 100<br />
}<br />
<br />
indexing {<br />
# Set to True to generate correction vectors.<br />
verbose_cv = True<br />
}<br />
<br />
integration {<br />
background_factor = 2<br />
<br />
#detector_gain = 7.5<br />
detector_gain = 1.0<br />
<br />
#model = use_case_3_simulated_annealing_7<br />
model = user_supplied<br />
<br />
signal_penetration = 0.5<br />
#spot_shape_verbose = False [This time consuming option prints out a detailed shape analysis of every spot]<br />
spotfinder_subset = spots_non-ice<br />
}<br />
</pre><br />
<br />
Several lines of parameters are given, then a few parameter blocks are specified, enclosed in {} brackets. The parameters in detail:<br />
* include file metrology-7.1.phil: this line specifies another ''phil'' file to include. In this case, the included file specifies metrology corrections (see below).<br />
* target_cell: the known unit cell for this sample. In the form a, b, c, alpha, beta, gamma. Must be in the primitive setting, or specify a target_cell_centring_type. Or, convert the dimensions first to the primitive setting with ''iotbx.lattice_symmetry''.<br />
* target_cell_centring_type: the centering type of the Bravais lattice: P (default), C, I, R, or F.<br />
* known_setting: this is the setting number of the true Bravais symmetry as output by the autoindexing procedure in ''LABELIT''. The indexing process fundamentally determines the triclinic cell, but also a list of possible Bravais lattices. Normally we associate the following setting numbers with the crystal systems:<br />
**Triclinic=1<br />
**Monoclinic=2<br />
**Orthorhombic=5<br />
**Rhombohedral=5<br />
**Tetragonal=9<br />
**Hexagonal=12<br />
**Cubic=22<br />
**However, the assumptions break down if there is pseudosymmetry, i.e., if a higher-symmetry Bravais lattice is close to the true Bravais symmetry. In these cases, a careful inspection of the ''LABELIT'' output is necessary to choose the setting, and it may be necessary to implement new code in ''cctbx.xfel'' for this (contact the authors).<br />
* distl_highres_limit and force_method2_resolution_limit: only process to this resolution limit. [Note these are #1 and #2 of 3 resolution flags that all must be set together]<br />
* mosaicity_limit: maximum moisicity before a frame is rejected<br />
* distl_minimum_number_spots_for_indexing: indexing will not proceed unless there are at least this many good spots found on the image<br />
* Subcategory <code>distl</code>: parameters specific to spot finding. '''NOTE: More extensive documentation on spotfinder and other parameters is at the [http://cci.lbl.gov/labelit/html/spotfinder.html spot finder web site]'''<br />
** res.outer: resolution limit for ''spotfinder'' [Note this is #3 of 3 resolution flags that all must be set together]<br />
** minimum_signal_height: in units of background noise sigma, how much signal is needed for a spot<br />
** minimum_spot_height: minimum height for a pixel to be considered a maximum (after it's determined to be signal)<br />
** minimum_spot_area: minimum area in pixels for each spot. NOTE: there is a longstanding bug in Spotfinder; increment this number by +1 to get the actually-used minimum area.<br />
** spot_area_maximum_factor: in multiples of minimum spot area, how large spots are allowed to be<br />
** compactness_filter: (Use False). This is an experimental algorithm that insists that spots need to be compact; e.g., a line 4 pixels long is not an acceptable spot. However, it turns out that many XFEL spots are not compact, so this algorithm degrades XFEL performance. <br />
** method2_cutoff_percentage: Controls how method2 (see the Spotfinder paper) chooses the resolution cutoff. Synchrotron data: 25 is OK. XFEL stills: requires low values, try 5. Lower numbers result in the determination of more expansive values for the resolution cutoff. <br />
** peak_intensity_maximum_factor: a peak intensity filter<br />
* Subcategory <code>indexing</code>:<br />
** verbose_cv: if true, correction vectors are generated. This is verbose output required for the metrology cutoff. Not for routine users.<br />
* Subcategory <code>integration</code>:<br />
** background_factor: ratio of # pixels used for background subtraction : # pixels in spot<br />
** detector_gain: ADU units per photon<br />
** model: ''labelit'' has several integration models, and allows users to provide their own. The model listed here is custom for these xfel applications<br />
** signal_penetration: thickness of the CSPAD sensors.<br />
** spotfinder_subset: which spots found by ''spotfinder'' to use. Choose from: < add ><br />
<br />
== Metrology phil ==<br />
<br />
'''Update 12/11/13: quad translations and tile translations should no longer be specified in the phil file. These live in the code for now. We are re-working the file storage type and in the future we won't need these at all.'''<br />
<br />
''cctbx.xfel'' applies 4 levels of metrology:<br />
* Optical metrology: this is supplied by LCLS. The pyana config file specifies a path to a directory where initial tile placements are specified.<br />
* Quadrant translations: adjustments to each of the 4 quadrants as a whole. ''(12/11/13: specified in the codebase)''<br />
* Unit-pixel or whole-pixel metrology: a series of whole-pixel translations applied to each tile. ''(12/11/13: specified in the codebase)''<br />
* Sub-pixel metrology: fractional corrections including translations and rotations of each tile. Used during integration.<br />
<br />
Sub-pixel metrology corrections are contained in the metrology ''phil'' file. For this tutorial that is metrology-7.1.phil:<br />
<br />
<pre><br />
distl {<br />
detector_tiling = None<br />
peripheral_margin = 1<br />
)<br />
integration {<br />
# This is L785_119.<br />
subpixel_joint_model {<br />
rotations = \<br />
0.1056719175 0.1080664532 -0.2151669703 -0.2099097279 \<br />
...<br />
translations = \<br />
0.2700895641 -0.6117579705 -0.5111648376 -0.8676594815 \<br />
...<br />
}<br />
}<br />
</pre><br />
<br />
These parameters are in detail:<br />
* distl subsection: quadrant translations and single-pixel tile translations<br />
** detector_tiling: < add ><br />
** peripheral_margin: how many pixels to leave as a border around the final image<br />
* integration subsection: sub-pixel adjustments applied during reflection integration:<br />
** subpixel_joint_model: root tag for the sub-pixel corrections<br />
*** rotations: angles in degrees for each tile (64)<br />
*** translation: x/y translations for each tile (128)<br />
<br />
== Comparing two phil files ==<br />
<br />
Often phil files are edited repeatedly by users. cxi.mpi_submit saves copies of phil files as they are used as a record of which trials were processed using which parameters. Now, say you have two different phil files that you want to see the differences between them. Due to comments or rearrangements, diffing the two files isn't making your life easier. Do this:<br />
<br />
$ cxi.parameters > default.phil<br />
$ libtbx.phil --diff default.phil file1.phil > a.txt<br />
$ libtbx.phil --diff default.phil file2.phil > b.txt<br />
$ diff a.txt b.txt<br />
<br />
This will show the true differences between the phil files. Note also that cxi.parameters is a useful tool for seeing all the default parameters available in ''cctbx.xfel''.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Mod_image_dict&diff=1747Mod image dict2026-08-19T18:31:05Z<p>Aaron: </p>
<hr />
<div>{{Archived}}<br />
<br />
<pre><br />
[psana]<br />
modules = my_ana_pkg.mod_image_dict<br />
[my_ana_pkg.mod_image_dict]<br />
address = MfxEndstation-0|Rayonix-0<br />
dark_path = None<br />
dark_stddev = None<br />
mask_path = None<br />
detz_offset = 89.0<br />
gain_map_path = None<br />
gain_map_level = None<br />
override_beam_x = 962.0<br />
override_beam_y = 963.0<br />
override_energy = None<br />
bin_size = 2<br />
crop_rayonix = False<br />
</pre><br />
<br />
These parameters are for a Rayonix image. Use crop_rayonix to trim the image such that it is square and centered on the beam center. Useful for the LABELIT backend of ''cctbx.xfel'' but not needed for DIALS.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Metrology_refinement&diff=1746Metrology refinement2026-08-19T18:30:47Z<p>Aaron: </p>
<hr />
<div>{{Archived}}<br />
<br />
Accurate data integration requires highly precise knowledge of pixel positions in laboratory space (metrology). Gaining this knowledge is especially difficult due to the [[CSPAD Pattern | segmented nature of the CSPAD detector]], which is tiled into 64 application-specific integrated circuits (ASICs). The 64 ASICs are arranged in quadrants that are approximately 4-fold rotationally symmetric, with 8 sensors per quadrant and 2 ASICs per sensor. The sensors are field-serviceable, and may therefore change from Run to Run. <br />
<br />
We thus need to determine positions and rotations for all 64 tiles, ideally down to an accuracy on order of 10 microns. As a general overview, cctbx takes the following steps:<br />
<br />
* Tile placement in physical space is measured by the beamline operators optically using electron microscopy. This is done at the per-sensor level (2 ASICs per sensor). <br />
<br />
* Relative positions of the quadrants are determined coarsely by considering powder rings.<br />
<br />
* Sensor positions are refined based on Bragg spot diffraction, allowing for whole-pixel translations in x and y.<br />
<br />
* ASIC positions are refined to subpixel accuracy based on Bragg diffraction, allow for sub-pixel translations and rotations.<br />
<br />
Most of these steps may be performed by general users, but some require editing the source code. Therefore, it may be suitable to consult the software authors. Please don't hesitate to ask for assistance regarding any aspect of this procedure.<br />
<br />
This page is superseded by the Aug. 2014 tutorial documented here: [[L498_Thermolysin]]<br />
<br />
== Optical Metrology ==<br />
<br />
LCLS provides initial tile placements. Periodically the detector is disassembled and re-built, necessitating new calibration information. The optical measurements are included in the ''cctbx.xfel'' in the source code and are reference by your ''pyana'' config file. See [[Preparatory_steps]] for more information.<br />
<br />
== Quadrant positioning ==<br />
<br />
===Problem statement===<br />
<br />
The optical metrology step above determines only the sensor positions relative to the frames of reference of each quadrant; but not the absolute position of each quadrant in space. At the XPP instrument, the LCLS-provided initial tile placements may additionally place the quadrants, but this should be confirmed. At the CXI instrument, the forward detector DS1 has rail-mounted quadrants to allow re-sizing of the central hole. At CXI, quadrant placement should be assessed for both the forward and back (DS2) detectors. <br />
<br />
===Method===<br />
cctbx includes a special GUI to adjust quadrant positions visually, based on connectivity of concentric rings on a powder pattern. True powders can be used (silver behenate), or a powder pattern can be synthesized by averaging protein crystal Bragg patterns from an entire run (see [[Preparatory_steps]]). If protein patterns are averaged, it is most useful to view powder rings on a standard deviation or composite maximum image; rings on the average image itself tend to be faint.<br />
<br />
Once a suitable image is obtained, start the calibration tool with this command:<br />
<br />
# For silver behenate:<br />
cxi.view <powder pattern image path> viewer.calibrate_silver=true distl.detector_format_version="CXI 7.1"<br />
<br />
Note, the flag distl.detector_format_version is a legacy requirement that will be phased out.<br />
<br />
Usually the standard deviation or maximum projection will show virtual powder ring. An image with strong rings to moderate resolution is desired. <br />
<br />
Once loaded, you will see your image and many red rings, incidentally corresponding to the diffraction pattern of silver behenate. Additionally, a separate settings panel will be present with 8 input boxes corresponding to the x and y offsets of each of the four quadrants, UL, UR, LL, and LR (upper and lower, left and right). Manually adjust the offsets until the powder rings are circular through the whole image, using the red rings as guides. These corrections will but put into your phil parameters file in the distl quad_translations entry in this form:<br />
<br />
distl {<br />
quad_translations = 2 -6 3 -6 -7 0 -1 -4<br />
}<br />
<br />
Where the numbers are listed in this order: ULX ULY, URX URY, LLX LLY, LRX, LRY<br />
<br />
== Unit pixel metrology corrections ==<br />
<br />
''cctbx.xfel'' then refines tile positions to whole pixel, and then sub-pixel precision by calculating spot positions on a highly diffracting dataset and refining tile locations versus actual spot positions for each tile. <br />
<br />
<br />
== Sub-pixel metrology corrections ==<br />
<br />
distl.tile_flags</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Merging&diff=1745Merging2026-08-19T18:30:24Z<p>Aaron: </p>
<hr />
<div>{{Archived}}<br />
<br />
The result of [[Indexing and integration]] is a set of Python pickle files, each of which essentially contains a table of Miller indices of the observed reflections, their integrated intensities, and estimated errors. In the general case, these files reflect the measurements from single shots, each exposing different crystals with a unique pulse of X-rays. ''Merging'' refers to the procedure applied to unite all these observations into a single data set. During merging, a distinct multiplicative factor, which accounts for the variance in pulse intensity and crystal size, is applied to the observations from a single shot to bring all the observations onto a common scale. After rejecting possible outliers, the intensities for individual reflections are summed, and their errors are propagated in quadrature. The result of merging is an mtz file suited for further processing, ''e.g.'' molecular replacement.<br />
<br />
<br />
== Merging a set of integration files ==<br />
In ''cctbx.xfel'' the per-image scale factors are determined using a ''scaling reference''. This scaling reference is expected to be a previously solved, isomorphous data set. The scale factor is determined by a least-squares fit of the observations to the reference intensities, after applying corrections for polarization<ref>[http://dx.doi.org/10.1107/S0021889882012060 Kahn, R, <i>et al.</i> Macromolecular Crystallography with Synchrotron Radiation: Photographic Data Collection and Polarization Correction. <i>J Appl Cryst</i> <b>15</b>, 330–337 (1982).]</ref>, and a significance filter, which limits the resolution of each diffraction pattern based on the signal-to-noise ratio. Lattices not conforming to the Bravais symmetry of the scaling reference are rejected. By default non-isomorphous lattices with cell lengths differing by more than 10% in length from the mean, or 2° in the angles are also rejected, as are lattices that correlate poorly with the scaling reference.<br />
<br />
''cxi.merge'' may take several passes over the integrated images. In order to speed up processing, ''cxi.merge'' will write the scaled data to a back end database during each pass. Currently three database back ends are implemented.<br />
<br />
* <code>FS</code> is the simplest back end. It stores the scaled intensities, their Miller indices, and information about the shots they were observed during in three flat files on the file system.<br />
<br />
* The <code>MySQL</code> back end stores data in a MySQL database. The database must be set up beforehand, and credentials to access it must be supplied in the parameters passed to ''cxi.merge''.<br />
<br />
* The <code>SQLite</code> back end uses a simple SQLite database, which is written to a single file on the file system. It is easier to use than the <code>MySQL</code> back end and more efficient than the <code>FS</code> backend. Regrettably, the <code>SQLite</code> back end does not appear to work on the Lustre file system.<br />
<br />
In ''cctbx.xfel'' images are merged using the <code>cxi.merge</code> command. <code>cxi.merge</code> accepts several options which can conveniently be assembled in a ''phil'' file. In this tutorial only a subset of the available options are defined.<br />
<br />
; backend<br />
: Back end database; <code>FS</code> for flat-file ASCII data storage, <code>MySQL</code> and <code>SQLite</code> for the respective proper database back ends. <br />
; d_min<br />
: Limiting resolution for scaling and merging<br />
; pixel_size<br />
: set the pixel size for the detector used, this is a single value (0.11 for CSPad)<br />
; data<br />
: Directory containing integrated data in pickle format. Repeat to specify additional directories.<br />
; merge_anomalous<br />
: <code>True</code> to merge anomalous contributors (''i.e.'' Bijvoet mates), <code>False</code> to preserve them<br />
; min_corr<br />
: Correlation cutoff for rejecting individual frames<br />
; model<br />
: The scaling reference, PDB filename containing atomic coordinates and isomorphous <code>CRYST1</code> record. Note: quick command to download a pdb file from the Internet is: <code> phenix.fetch_pdb [pdb_code]</code><br />
; nproc<br />
: Specifies the number of scaling processes ''cxi.merge'' may have running at any one time<br />
; output.prefix<br />
: Prefix for all output file names<br />
; rawdata.sdfac_auto<br />
: <code>True</code> to apply <code>SDFAC</code> correction to each image, assuming negative intensities are normally distributed noise<br />
; rescale_with_average_cell<br />
: Rescale the images a second time, requiring images to conform to the average unit cell. If set to <code>True</code>, <code>set_average_unit_cell</code> must also be set to <code>True</code>.<br />
; set_average_unit_cell<br />
: If <code>True</code> set the unit cell of the merged data to the average of the merged images, otherwise use the unit cell of the scaling reference<br />
<br />
<br />
The merged data is written to an mtz-file named <code><i>output.prefix</i>.mtz</code>. Standard output, redirected to <code>merge.out</code> above, mainly consists of statistics for each individual image as it is scaled. The output concludes with a section labelled <code>FINISHED MERGING</code>, which first lists the number of accepted and rejected images,<br />
<pre><br />
962 of 1185 integration files were accepted<br />
127 rejected due to wrong Bravais group<br />
1 rejected for unit cell outliers<br />
12 rejected for low signal<br />
83 rejected due to poor correlation<br />
</pre><br />
This is followed by a histograms of the unit cell distribution, and the merging table:<br />
<pre><br />
---------------------------------------------------------------------------------<br />
<asu <obs<br />
Bin Resolution Range Completeness redun> redun> n_meas <I> <I/sig(I)><br />
---------------------------------------------------------------------------------<br />
1 -1.0000 - 6.4633 [306/309] 67.61 68.28 20893 35621 27.044<br />
2 6.4633 - 5.1299 [285/285] 43.18 43.18 12305 19272 12.490<br />
3 5.1299 - 4.4814 [275/275] 46.77 46.77 12862 24391 14.261<br />
4 4.4814 - 4.0716 [269/269] 43.38 43.38 11670 29436 14.819<br />
5 4.0716 - 3.7798 [249/249] 35.50 35.50 8840 29811 12.716<br />
6 3.7798 - 3.5569 [267/267] 32.07 32.07 8562 24212 10.523<br />
7 3.5569 - 3.3787 [274/274] 20.99 20.99 5750 23626 8.171<br />
8 3.3787 - 3.2317 [256/256] 15.96 15.96 4085 24168 6.746<br />
9 3.2317 - 3.1072 [263/264] 12.16 12.20 3209 23936 5.859<br />
10 3.1072 - 3.0000 [263/263] 8.74 8.74 2298 20613 4.556<br />
<br />
All [2707/2711] 33.37 33.42 90474 25594 11.978<br />
----------------------------------------------------------------------------------<br />
</pre><br />
In this case, the overall completeness to 3.0 Å is 2,707 / 2,711, or approximately 100%, and each observed reflection is measured 33.42 times on average (<code><obs redun></code>). In total 90,474 observations, with an average scaled and integrated intensity of 25,594 analog-to-digital units (ADU), were merged and the mean <i>I</i> / σ(<i>I</i>) was determined to be 11.978.<br />
<br />
== Additional merging statistics ==<br />
''cxi.xmerge'' retrieves the scaled, unmerged intensities from the database back end, and calculates the <i>CC</i><sub>1/2</sub><ref>[http://dx.doi.org/10.1126/science.1218231 Karplus, P. A. & Diederichs, K. Linking Crystallographic Model and Data Quality. <i>Science</i> <b>336</b>, 1030–1033 (2012).]</ref> and <i>CC</i><sub>iso</sub> statistics. <i>CC</i><sub>1/2</sub> is defined as Pearson's correlation coefficient between two sets, such that for each unique reflection the average intensities of two randomly chosen halves of its independent observations are assigned to different set. <i>CC</i><sub>iso</sub> is the correlation coefficient between the merged data and the isomorphous scaling reference. Both statistics are computed in each resolution bin, as well as for the full set of reflections.<br />
<br />
Once the database has been populated using ''cxi.merge'', ''cxi.xmerge'' can be run, using the parameters defined in a ''phil''-file, <code>Ls-04-lysozyme-xmerge.phil</code>.<br />
<br />
<pre>$ cxi.xmerge Ls04-lysozyme-xmerge.phil</pre><br />
<br />
The options used in this tutorial not already described in [[#Merging a set of integration files|Merging a set of integration files]] are explained below.<br />
; scaling.mtz_file<br />
: mtz-file containing experimentally determined reference structure factors, such as synchrotron measurements, to be used for CC<sub>iso</sub>. Experimental measurements, when available, can be downloaded from the PDB using <code>phenix.fetch_pdb –-mtz [pdb_code]</code>. <br />
; scaling.mtz_column_F<br />
: In the experimental measurements mtz-file, the column name containing the reference data; can be either amplitudes or intensities. The column name should be given '''in lower case'''. An MTZ-format file can be inspected to see which columns it contains with <code>phenix.mtz.dump [code].mtz</code>. Quick tip: if experimental measurements are unavailable, mock experimental structure factors can be generated with <br><code>phenix.fmodel <pdb file> high_resolution=<value> output.type=real</code> in which case one would use <code>scaling.mtz_column_F=fmodel</code>.<br />
; scaling.log_cutoff<br />
: Intensities less than <i>e</i><sup><code>scaling.log_cutoff</code></sup> will not be included in the calculation of CC<sub>iso</sub> and R<sub>iso</sub> (however, these measurements will still be included in the final merged data). <br />
<br />
The results are printed on standard output and conclude with a table including the <i>CC</i><br />
<br />
<pre><br />
---------------------------------------------------------------------------------------<br />
CC CC R R Scale Scale<br />
Bin Resolution Range Completeness int N iso N int iso int iso<br />
---------------------------------------------------------------------------------------<br />
1 -1.0000 - 6.4633 [233/309] 45.1% 233 56.3% 142 57.5% 45.6% 0.749 3.084<br />
2 6.4633 - 5.1299 [184/285] 22.5% 184 56.7% 62 57.2% 31.5% 0.862 2.841<br />
3 5.1299 - 4.4814 [203/275] 43.7% 203 57.6% 85 55.3% 37.0% 0.830 2.977<br />
4 4.4814 - 4.0716 [220/269] 39.0% 220 59.7% 101 51.4% 30.3% 0.805 3.296<br />
5 4.0716 - 3.7798 [187/249] 77.5% 187 33.6% 63 54.3% 39.4% 0.775 4.243<br />
6 3.7798 - 3.5569 [198/267] 35.4% 198 47.2% 56 56.8% 27.5% 0.903 3.587<br />
7 3.5569 - 3.3787 [198/274] 32.0% 198 70.7% 40 59.9% 31.4% 0.814 3.846<br />
8 3.3787 - 3.2317 [163/256] 30.6% 163 24.5% 28 63.9% 49.1% 0.734 4.278<br />
9 3.2317 - 3.1072 [164/264] 40.6% 164 53.6% 7 80.4% 32.1% 0.622 5.543<br />
10 3.1072 - 3.0000 [137/261] 8.2% 137 93.9% 5 85.6% 33.4% 0.221 4.372<br />
<br />
All 34.7% 1887 47.7% 589 62.5% 39.5% 0.754 3.157<br />
---------------------------------------------------------------------------------------<br />
</pre><br />
In this case the overall <i>CC</i><sub>1/2</sub> (<code>CC int</code>) is 34.7%, and the correlation coefficient to the isomorphous reference, <i>CC</i><sub>iso</sub> (<code>CC iso</code>) is 47.7%.<br />
Note that the completeness for the cxi.xmerge output appears lower than that for the cxi.merge output. This is because the cxi.xmerge final table shows the completeness for the data used in the calculation of correlation coefficients, and so the completeness in the previous table should be used for general statistics.<br />
<br />
== Merging the tutorial data ==<br />
<br />
Compared to indexing and integration, merging is a relatively quick procedure. However, particularly for large datasets, it may significantly strain computational resources. Therefore, it is recommended to merge data on SLAC's interactive nodes.<br />
<pre><br />
$ ssh psanacs.slac.stanford.edu<br />
$ cd myrelease<br />
$ cp /reg/g/cctbx/tutorials/merging/Ls04-lysozyme-*merge.phil .<br />
</pre><br />
Before running the merging programs, the ''phil''-files may need to be edited. In particular, the location of the directories containing pickle files with the integrated intensities may need to be adjusted. Since both <code>cxi.merge</code> and <code>cxi.xmerge</code> log to standard output, it is advisable to redirect the stream to a file.<br />
<pre><br />
$ cxi.merge Ls04-lysozyme-merge.phil > merge.out<br />
$ cxi.xmerge Ls04-lysozyme-xmerge.phil > xmerge.out<br />
</pre><br />
The merged reflections, suitable for further processing, are written <code>Ls04-lysozyme.mtz</code>. Note that although the data for this tutorial extends to 2.2 Å, the statistics become exceedingly poor around 3.0 Å.<br />
<br />
== References ==<br />
<references/></div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Maintaining_an_installation&diff=1744Maintaining an installation2026-08-18T23:12:46Z<p>Aaron: </p>
<hr />
<div>{{Archived}}<br />
<br />
== Keeping up to date with PSDM releases==<br />
<br />
<!-- XXX Separate installation and maintenance instructions are provided by the PSDM project at [https://confluence.slac.stanford.edu/display/PSDM/Software+Distribution PSDM software distribution]. XXX Also missing bit on how to update cctbx for non-developers. --><br />
<br />
Since the ''cctbx.xfel'' modules do not critically depend on any particular version of ''pyana'', it is generally safe to closely track the latest version of the PSDM software distribution. To upgrade to the current PSDM release,<br />
$ cd <b><i>/path/to/test/release</i></b><br />
$ relupgrade ana-current<br />
where <code><b><i>/path/to/test/release</i></b></code> must be substituted for path of the test release. The ''cctbx.xfel'' modules should then be rebuilt using<br />
$ scons -c<br />
$ scons<br />
It is possible to manually edit <code><b><i>/path/to/test/release</i></b>/.sit_release</code> to contain <code>ana-current</code> instead of a particular version number. This ensures that the latest version of the PSDM software distribution is used during each session.<br />
<br />
== Keeping a developer installation up to date ==<br />
<br />
Given the rapid development of the ''PHENIX'' suite, it is recommended to update the sources and rebuild often. Certain changes may require the configure command (or <code>libtbx.refresh</code>) to be rerun.<br />
$ cd <b><i>/path/to/cctbx/sources</i></b><br />
$ ./allsvn update<br />
$ cd <b><i>/path/to/cctbx/build</i></b><br />
$ make<br />
<!-- XXX How often should one do something about the hot tarballs (i.e. annlib_hot.tar.gz, boost_hot.tar.gz, and scons_hot.tar.gz)? XXX See also Nick’s note at the end of document about the New framework for specifying crystal targets. --></div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=LB67_Thermolysin&diff=1743LB67 Thermolysin2026-08-18T23:12:04Z<p>Aaron: </p>
<hr />
<div>{{Archived}}<br />
<br />
A short tutorial for processing thermolysin data from LB67 with a gain mask applied. The data for this tutorial is at /reg/d/psdm/cxi/cxi84914/xtc/e350. Config files and phil files are available at /reg/d/psdm/cxi/cxi84914/scratch/cctbx-tutorial/lb67thermolysin.<br />
<br />
IMPORTANT. Throughout this tutorial and in the config and phil files referenced, we are using $USER to represent your username. This will substitute your username in the command line, but not in the phil or config files. Be sure to replace $USER with your username in the config and phil files.<br />
<br />
== Dark average ==<br />
<br />
First off, use mod_average to make a dark from LB67 run 24. Copy dark.cfg from /reg/d/psdm/cxi/cxi84914/scratch/cctbx-tutorial/lb67thermolysin into your myrelease folder. Then, from the myrelease folder, submit the averaging job: <br />
cxi.lsf -c dark.cfg -o /reg/d/psdm/cxi/cxi84914/scratch/$USER/averages -x cxi84914 -r 24 -s -q psanacsq -p 6 -i /reg/d/psdm/cxi/cxi84914/xtc/e350<br />
<br />
You can track the progress of your job using a command like this:<br />
tail -f /reg/d/psdm/cxi/cxi84914/scratch/$USER/averages/r0024/000/stdout/sNN.out<br />
<br />
When the average is complete, you can inspect them thusly:<br />
cctbx.image_viewer /reg/d/psdm/cxi/cxi84914/scratch/$USER/averages/r0024/000/out/*<br />
<br />
Nine images are created, 3 for each of the front CSPAD (Ds1), the back CSPAD (Dsd) and the XES spectrometer that was attached to this experiment (Sc1). For each detector the average (avg), standard deviation (stddev) and maximum projection (max) are created.<br />
<br />
== Gain map ==<br />
<br />
As is likely clear from examining the averages for the Ds1 detector, during this experiment a gain mask was applied. At high resolution, the individual pixels were set in high gain mode (more sensitive to photons) and the low resolution pixels were set to low gain mode (low sensitivity to photons). This allows more accurate recording of low resolution reflections without saturating the pixels, while preserving the ability to record weak data at high resolution.<br />
<br />
We treat this data by multiplying the low gain pixels by a constant factor to put them on the same scale as the high gain pixels. To do this we apply a mask. To generate this mask, we need to convert the raw mask used by the beamline operator to a format usable to cctbx.xfel. First, let's extract some useful information from one of the dark averages:<br />
<br />
cxi.print_pickle /reg/d/psdm/cxi/cxi84914/scratch/$USER/averages/r0024/000/out/avg-Ds1-r0024.pickle<br />
<br />
Detector format version: CXI 8.2<br />
DISTANCE 378.0042<br />
DETECTOR_ADDRESS CxiDs1-0|Cspad-0<br />
SIZE1 1765<br />
SIZE2 1765<br />
TIMESTAMP 2014-03-06T00:05Z19.502<br />
CCD_IMAGE_SATURATION 16383<br />
SATURATED_VALUE 16383<br />
64 active areas, first one: [715, 439, 909, 624]<br />
PIXEL_SIZE 0.11<br />
BEAM_CENTER_Y 96.965825<br />
BEAM_CENTER_X 96.99305<br />
WAVELENGTH 1.75860511344<br />
xtal_target None<br />
DATA len=3115225 max=16383.000000 min=0.000000<br />
SEQUENCE_NUMBER 0<br />
<br />
We will use some of the information here to convert the gain mask to cctbx.xfel format. Copy the file cxib6714_1.gain from /reg/d/psdm/cxi/cxi84914/scratch/cctbx-tutorial/lb67thermolysin to your myrelease folder, then run:<br />
cxi.gain_map -o cxib6714_1.pickle -d 378 -w 1.75860511344 cxib6714_1.gain -f "CXI 8.2"<br />
<br />
Here -o is the output file name, -d is the distance, -w is the wavelength and -f is the detector format version. You can view the resultant cxib6714_1.pickle with cctbx.image_viewer<br />
<br />
Finally, we test the map is working using a light average. Copy average.cfg from /reg/d/psdm/cxi/cxi84914/scratch/cctbx-tutorial/lb67thermolysin to your myrelease directory. Make sure the paths to the dark average and standard deviation are correct, as well as the gain maak pickle file. Submit the averaging job:<br />
cxi.lsf -c cxi84914/average.cfg -o /reg/d/psdm/cxi/cxi84914/scratch/$USER/averages -x cxi84914 -r 25 -s -q psanacsq -p 8 -i /reg/d/psdm/cxi/cxi84914/xtc/e350<br />
<br />
Again, you can follow the log file and view the result as describe above.<br />
<br />
== Indexing ==<br />
<br />
Now that the darks and gain mask are in place, the data is indexed straightforwardly. Copy all the phil files and LB67-thermolysin.cfg to your myrelease folder, then submit the indexing job:<br />
<br />
cxi.lsf -c LB67-thermolysin.cfg -o /reg/d/psdm/cxi/cxi84914/scratch/$USER/thermolysin -p 16 -q psanacsq -r 25 -x cxi84914 -i /reg/d/psdm/cxi/cxi84914/xtc/e350<br />
<br />
You can follow the log of one of the streams using<br />
tail -f /reg/d/psdm/cxi/cxi84914/scratch/$USER/thermolysin/r0024/000/stdout/s00.out<br />
<br />
Individual indexed frames will be deposited in /reg/d/psdm/cxi/cxi84914/scratch/$USER/thermolysin/r0024/000/out and the integrated results will be in /reg/d/psdm/cxi/cxi84914/scratch/$USER/thermolysin/r0024/000/integration. For the parameters provided, around 1923 indexed images are expected.<br />
<br />
== Discovery ==<br />
<br />
'''UNDER CONSTRUCTION'''<br />
<br />
The indexing results above present already optimized processing parameters. The user will need to explore their data, testing a variety of parameters to extract the best reduced signal. Over the next few sections, techniques for discovering necessary parameters are presented.<br />
<br />
=== Detector distance ===<br />
<br />
The best way to be sure your detz offset is correct is through a series of trials, testing different values. Create a dist_trials directory in your myrelease folder. cd into it, then generate config files with different candidate detz offsets, thusly:<br />
<br />
for i in `seq 565 580`; do vi -c "%s/578/$i/g" -c "w LB67-thermolysin_$i.cfg" \<br />
-c q\! ../LB67-thermolysin.cfg ; done<br />
<br />
Here, a vi command is executed repeatedly that searches for the number 578 in your config file and replaces it with a number from 565 to 580, then writes out the new file with an appropriate file name. If this seems like a poor window to search, note that the number was originally 572, and has already been optimized to 578 by the authors.<br />
<br />
Next, submit indexing jobs for each candidate detz offset from your myrelease folder:<br />
<br />
for i in `seq 565 580`; do cxi.lsf -c dist_trials/LB67-thermolysin_$i.cfg \<br />
-o /reg/d/psdm/cxi/cxib6714/ftc/$USER/dist_trials/ -x cxib6714 -r 30 -q psanacsq -p 8 -t $i; done<br />
<br />
When complete, go to your results directory:<br />
cd /reg/d/psdm/cxi/cxib6714/scratch/$USER/dist_trials/r0025<br />
<br />
Then determine which detz offset is best:<br />
for i in `ls`; do echo -n "$i "; ls $i/out | wc -l; done<br />
<br />
=== Find candidate hits ===<br />
<br />
=== Spotfinding grid searches ===</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=L498_Thermolysin&diff=1742L498 Thermolysin2026-08-18T23:11:51Z<p>Aaron: </p>
<hr />
<div>{{Archived}}<br />
<br />
In this tutorial, we assume that we are handed an SFX dataset containing thermolysin diffraction, but are not told anything else. We will have to go through all the data runs, figure out which one is to be used for dark subtraction, and account for untrusted pixels and detector metrology. At this point, we will be prepared to integrate and merge the data. Finally, we will perform simple molecular replacement and ask whether there is any Zn signal in the anomalous difference Fourier. <br />
<br />
=Discovery of data collection parameters=<br />
Log in to pslogin.slac.stanford.edu, and then to psana. Carry through flags so that X-windows will work<br />
<br />
ssh -YAC $USER@pslogin.slac.stanford.edu<br />
ssh -YAC psana<br />
<br />
Go in to the working directory and source the package manager:<br />
<br />
cd ~/myrelease<br />
sit_setup<br />
<br />
Create a subdirectory for the 2014 tutorial files if not already done:<br />
mkdir -p cxi84914<br />
<br />
List out the thermolysin XTC files:<br />
ls /reg/d/psdm/cxi/cxi84914/xtc/e157<br />
<br />
<br />
Notice that there are numerous runs in the directory. Now we will create composite averages for each run. Grab this configuration file: [[mkdark.cfg]] and put it in your cxi84914 directory. For one run only:<br />
<br />
cxi.lsf -c ~/myrelease/cxi84914/mkdark.cfg \<br />
-o /reg/d/psdm/cxi/cxi84914/scratch/$USER/initial_dark/e157/ \<br />
-i /reg/d/psdm/cxi/cxi84914/xtc/e157 -q psanacsq -s -p 8 -x 157 -r 16 -t 0<br />
Take note:<br />
* -c configuration file<br />
* -o output directory (will be created)<br />
* -i input files (directory containing the XTC streams)<br />
* -q which batch queue to use<br />
* -s funnel all streams for the run into one node (takes longer, but necessary for averaging)<br />
* -p number of cores to use on the node<br />
* -x which experiment number<br />
* -r which run number<br />
* -t which processing trial (auto increments from 0 if not given)<br />
<br />
For all the runs in the thermolysin data set:<br />
kinit<br />
aklog<br />
for m in 16 17 18 19 20 21 22 23 24 25 26 27 31 71 72 73; \<br />
do echo $m; cxi.lsf -c ~/myrelease/cxi84914/mkdark.cfg \<br />
-o /reg/d/psdm/cxi/cxi84914/scratch/$USER/initial_dark/e157/ \<br />
-i /reg/d/psdm/cxi/cxi84914/xtc/e157 -q psanacsq -s -p 8 -x 157 -r ${m} -t 0; done<br />
<br />
bjobs lists all your batch jobs; use this form for more information including other-user load:<br />
bjobs -w -u all -q psanacsq<br />
bkill [number] # stops unwanted job<br />
<br />
Some runs take up to 2 hrs wall time to average. Find the averages, view the max-composites, and list out header information:<br />
ls /reg/d/psdm/cxi/cxi84914/scratch/$USER/initial_dark/e157/r*/000/out/*.pickle<br />
cctbx.image_viewer `find /reg/d/psdm/cxi/cxi84914/scratch/$USER/initial_dark/e157/r*/000 -name "max*.pickle"`<br />
for m in `find /reg/d/psdm/cxi/cxi84914/scratch/$USER/initial_dark/e157/r*/000 -name "max*.pickle"`; <br />
do echo $m; cxi.print_pickle $m; echo; done<br />
<br />
Let's make a table of the results:<br />
{| class="wikitable" <br />
| style="padding: 2px;"| Run<br />
| style="padding: 2px;"| Distance<br />
| style="padding: 2px;"| Wavelength<br />
| style="padding: 2px;"| Diffraction<br />
| style="padding: 2px;"| Comments<br />
| style="padding: 2px;"| Events<br />
| style="padding: 2px;"| Wall clock <br> 000 psanacsq <br> 6 cores<br />
| style="padding: 2px;"| Wall clock <br> 001 psanaq <br> 6 cores<br />
|-<br />
| style="padding: 2px;"| 16<br />
| style="padding: 2px;"| 271.0<br />
| style="padding: 2px;"| 1.2686<br />
| style="padding: 2px;"| weak powder<br />
| style="padding: 2px;"| <br />
| style="padding: 2px;"| 39313<br />
| style="padding: 2px;"| 60:36<br />
| style="padding: 2px;"| 36:20<br />
|-<br />
| style="padding: 2px;"| 17<br />
| style="padding: 2px;"| 221.0<br />
| style="padding: 2px;"| 1.2686<br />
| style="padding: 2px;"| weak powder<br />
| style="padding: 2px;"| <br />
| style="padding: 2px;"| 66647<br />
| style="padding: 2px;"| 72:32<br />
| style="padding: 2px;"| 62:16 <br />
|-<br />
| style="padding: 2px;"| 18<br />
| style="padding: 2px;"| 221.0<br />
| style="padding: 2px;"| 1.2686<br />
| style="padding: 2px;"| weak powder<br />
| style="padding: 2px;"| <br />
| style="padding: 2px;"| 47297<br />
| style="padding: 2px;"| 77:21<br />
| style="padding: 2px;"| 45:28<br />
|-<br />
| style="padding: 2px;"| 19<br />
| style="padding: 2px;"| 221.0<br />
| style="padding: 2px;"| 1.2686<br />
| style="padding: 2px;"| weak powder<br />
| style="padding: 2px;"| <br />
|-<br />
| style="padding: 2px;"| 20<br />
| style="padding: 2px;"| 221.0<br />
| style="padding: 2px;"| 1.2686<br />
| style="padding: 2px;"| strong powder<br />
| style="padding: 2px;"| <br />
|-<br />
| style="padding: 2px;"| 21<br />
| style="padding: 2px;"| 171.0<br />
| style="padding: 2px;"| 1.2686<br />
| style="padding: 2px;"| strong powder<br />
| style="padding: 2px;"| shadow<br />
|-<br />
| style="padding: 2px;"| 22<br />
| style="padding: 2px;"| 171.0<br />
| style="padding: 2px;"| 1.2686<br />
| style="padding: 2px;"| strong powder<br />
| style="padding: 2px;"| shadow<br />
|-<br />
| style="padding: 2px;"| 23<br />
| style="padding: 2px;"| 171.0<br />
| style="padding: 2px;"| 1.2686<br />
| style="padding: 2px;"| strong powder<br />
| style="padding: 2px;"| shadow<br />
|-<br />
| style="padding: 2px;"| 24<br />
| style="padding: 2px;"| 171.0<br />
| style="padding: 2px;"| 1.2686<br />
| style="padding: 2px;"| strong powder<br />
| style="padding: 2px;"| shadow<br />
|-<br />
| style="padding: 2px;"| 25<br />
| style="padding: 2px;"| 171.0<br />
| style="padding: 2px;"| 1.2686<br />
| style="padding: 2px;"| strong powder<br />
| style="padding: 2px;"| shadow<br />
|-<br />
| style="padding: 2px;"| 26<br />
| style="padding: 2px;"| 171.0<br />
| style="padding: 2px;"| 1.2686<br />
| style="padding: 2px;"| strong powder<br />
| style="padding: 2px;"| shadow<br />
|-<br />
| style="padding: 2px;"| 27<br />
| style="padding: 2px;"| 171.0<br />
| style="padding: 2px;"| 1.2686<br />
| style="padding: 2px;"| strong powder<br />
| style="padding: 2px;"| shadow<br />
|-<br />
| style="padding: 2px;"| 31<br />
| style="padding: 2px;"| 570.9<br />
| style="padding: 2px;"| 1.2686<br />
| style="padding: 2px;"| Dark<br />
| style="padding: 2px;"| <br />
|-<br />
| style="padding: 2px;"| 71<br />
| style="padding: 2px;"| 271.0<br />
| style="padding: 2px;"| 1.2966<br />
| style="padding: 2px;"| weak powder<br />
| style="padding: 2px;"| <br />
|-<br />
| style="padding: 2px;"| 72<br />
| style="padding: 2px;"| 131.0<br />
| style="padding: 2px;"| 1.2966<br />
| style="padding: 2px;"| weak powder<br />
| style="padding: 2px;"| shadow<br />
|-<br />
| style="padding: 2px;"| 73<br />
| style="padding: 2px;"| 131<br />
| style="padding: 2px;"| 1.2966<br />
| style="padding: 2px;"| strong powder<br />
| style="padding: 2px;"| shadow<br />
|-<br />
|}<br />
<br />
Some conclusions: <br />
<br />
* Run 31 was the dark run. We'll use the average and standard deviation for further processing. Tutorial students can take result from the instructor's directory:<br />
/reg/d/psdm/cxi/cxi84914/scratch/nksauter/initial_dark/e157/r0031/000/out/avg-r0031.pickle<br />
/reg/d/psdm/cxi/cxi84914/scratch/nksauter/initial_dark/e157/r0031/000/out/stddev-r0031.pickle<br />
<br />
* We are interested in getting the Zn anomalous signal from thermolysin, therefore we'll discard runs 71-73 collected at 1.2966 Angstroms, lower energy than the Zn K-edge at 9659 eV or 1.2836 Angstroms. See the [http://xdb.lbl.gov X-ray handbook].<br />
<br />
* We'll also discard run 16 as the diffraction was relatively weak and the unique detector distance would require separate detector calibration. <br />
<br />
* We'll accept runs 17-20 ("calibration17"), and runs 21-27 ("calibration21").<br />
<br />
=Prepare to mask out the untrusted pixels=<br />
We'll now calculate dark-subtracted averages. <br />
for m in 17 18 19 20 21 22 23 24 25 26 27; \<br />
do echo $m; cxi.lsf -c ~/myrelease/cxi84914/mkavg_e157.cfg \<br />
-o /reg/d/psdm/cxi/cxi84914/scratch/$USER/averages/e157/ \<br />
-i /reg/d/psdm/cxi/cxi84914/xtc/e157 -q psanacsq -s -p 8 -x 157 -r ${m} -t 0; done<br />
<br />
This repeats exactly the same averaging calculations as before, except the dark average from run 31 is subtracted. <br />
The dark image to be subtracted (along with its std deviation) is defined in the configuration file [[mkavg_e157.cfg]].<br />
<br />
Now we'll figure out which pixels are untrusted, and thus should not be integrated. Three criteria will be used:<br />
* Hot pixels--on the average-dark the pixel values exceed 1350 (should be fine tuned by inspecting the dark & using trial and error)<br />
* Hot pixels--on the standard deviation-dark the stddev exceeds 5 and therefore unreliable (also should be fine tuned by trial and error)<br />
* Cold pixels or shadows--on a maximum-composite data image, inspect values and set a minimum threshold value (we choose 20 here)<br />
<br />
cxi.make_mask -v --maxproj_min 20 --avg_max 1350 --stddev_max 5 --output mask_base.pickle \<br />
/reg/d/psdm/cxi/cxi84914/scratch/$USER/initial_dark/e157/r0031/000/out/avg-r0031.pickle \<br />
/reg/d/psdm/cxi/cxi84914/scratch/$USER/initial_dark/e157/r0031/000/out/stddev-r0031.pickle \<br />
/reg/d/psdm/cxi/cxi84914/scratch/$USER/averages/e157/r0020/000/out/max-r0020.pickle<br />
<br />
Inspect the mask:<br />
cctbx.image_viewer mask_base.pickle show_untrusted=true<br />
<br />
Simple thresholding wasn't sufficient for capturing some shadow areas and also an apparent untrusted region in one of the ASICs. Calculate some additional masks using the polygon feature; tied to our current metrology. Reading off polygon vertices from the cctbx.image_viewer, be sure to read coordinates as fast,slow (reverse order).<br />
<br />
cxi.make_mask -v --poly_mask=1258,83,1148,9,1258,9 --output=mask_inactive.pickle -a 3000 -s 100 -m 0 \<br />
/reg/d/psdm/cxi/cxi84914/scratch/$USER/initial_dark/e157/r0031/000/out/avg-r0031.pickle \<br />
/reg/d/psdm/cxi/cxi84914/scratch/$USER/initial_dark/e157/r0031/000/out/stddev-r0031.pickle \<br />
/reg/d/psdm/cxi/cxi84914/scratch/$USER/averages/e157/r0020/000/out/max-r0020.pickle<br />
<br />
Shadow on runs 17-20:<br />
<br />
cxi.make_mask -v --poly_mask=915,1696,712,1719,506,1750,915,1750 --output=mask_shadow17.pickle -a 3000 -s 100 -m 0 \<br />
/reg/d/psdm/cxi/cxi84914/scratch/$USER/initial_dark/e157/r0031/000/out/avg-r0031.pickle \<br />
/reg/d/psdm/cxi/cxi84914/scratch/$USER/initial_dark/e157/r0031/000/out/stddev-r0031.pickle \<br />
/reg/d/psdm/cxi/cxi84914/scratch/$USER/averages/e157/r0020/000/out/max-r0020.pickle<br />
<br />
Shadow on runs 21-27:<br />
cxi.make_mask -v --output=mask_shadow21.pickle -a 3000 -s 100 -m 0 \<br />
--poly_mask=1741,1536,1540,1518,1331,1506,1120,1505,918,1515,712,1550,496,1576,286,1629,86,1685,86,1754,1741,1754 \<br />
/reg/d/psdm/cxi/cxi84914/scratch/$USER/initial_dark/e157/r0031/000/out/avg-r0031.pickle \<br />
/reg/d/psdm/cxi/cxi84914/scratch/$USER/initial_dark/e157/r0031/000/out/stddev-r0031.pickle \<br />
/reg/d/psdm/cxi/cxi84914/scratch/$USER/averages/e157/r0021/000/out/max-r0021.pickle<br />
<br />
Combine three masks together for runs 17-20:<br />
cxi.or_mask mask_base.pickle mask_inactive.pickle mask_shadow17.pickle \<br />
/reg/d/psdm/cxi/cxi84914/scratch/$USER/averages/e157/mask_calibrate17.pickle<br />
<br />
Combine three masks together for runs 21-27:<br />
cxi.or_mask mask_base.pickle mask_inactive.pickle mask_shadow21.pickle \<br />
/reg/d/psdm/cxi/cxi84914/scratch/$USER/averages/e157/mask_calibrate21.pickle<br />
<br />
=Correct the detector metrology=<br />
Accurate data integration requires highly precise knowledge of pixel positions in laboratory space (metrology). Gaining this knowledge is especially difficult due to the [[CSPAD Pattern | segmented nature of the CSPAD detector]], which is tiled into 64 application-specific integrated circuits (ASICs). The 64 ASICs are arranged in quadrants that are approximately 4-fold rotationally symmetric, with 8 sensors per quadrant and 2 ASICs per sensor. The sensors are field-serviceable, and may therefore change from Run to Run. <br />
<br />
We thus need to determine positions and rotations for all 64 tiles, ideally down to an accuracy on order of 10 microns. As a general overview, cctbx takes the following steps:<br />
<br />
* Tile placement in physical space is measured by the beamline operators optically using electron microscopy. This is done at the per-sensor level (2 ASICs per sensor). This is already hard-coded; nothing for the user to do.<br />
<br />
* Relative positions of the quadrants are determined coarsely by considering powder rings. <br />
<br />
* Sensor positions are refined based on Bragg spot diffraction, allowing for whole-pixel translations in x and y.<br />
<br />
* ASIC positions are refined to subpixel accuracy based on Bragg diffraction, allow for sub-pixel translations and rotations.<br />
<br />
==Quadrant positions==<br />
The electron microscopy step above determines only the sensor positions relative to the frames of reference of each quadrant; but not the absolute position of each quadrant in space. At the CXI instrument, the forward detector DS1 has rail-mounted quadrants to allow re-sizing of the central hole. The quadrant placement should be assessed for both the forward and back (DS2) detectors. <br />
<br />
For thermolysin we examine one of the strong images (maximum composite). Students may use the instructor's files ($USER=nksauter). We first determine that our image has a timestamp that identifies it within cctbx as being from run 5 (2011):<br />
<br />
cxi.print_pickle /reg/d/psdm/cxi/cxi84914/scratch/$USER/averages/e157/r0021/000/out/max-r0021.pickle <br />
> Detector format version: CXI 5.1<br />
cxi.detector_format_versions<br />
cxi.detector_format_versions "CXI 5.1"<br />
cxi.view /reg/d/psdm/cxi/cxi84914/scratch/$USER/averages/e157/r0021/000/out/max-r0021.pickle \<br />
distl.detector_format_version="CXI 5.1" viewer.calibrate_unitcell.d_min=10 \<br />
viewer.calibrate_unitcell.unitcell=94,94,130.8,90,90,120 viewer.calibrate_unitcell.spacegroup=P6122<br />
<br />
The "Settings" GUI panel shows detector distance as well as the hard-coded quadrant positions corresponding to "CXI 5.1", namely -3,-1,-1,-5,-13,2,-7,-4. Tile translations have been zeroed out in the code. The settings can be changed in the panel, or alternately given as a separate command line parameter (distl.quad_translations=-3,-1,-1,-5,-13,2,-7,-4). The object is to align the powder pattern with the predicted rings (red circles) based on the unit cell parameters. It can be seen that the alignment is not quite perfect.<br />
<br />
Since we have well-formed powder rings, we can run the automatic quadrant positioning tool:<br />
<br />
cspad.quadrants /reg/d/psdm/cxi/cxi84914/scratch/$USER/averages/e157/r0021/000/out/max-r0021.pickle \<br />
distl.detector_format_version="CXI 5.1"<br />
> The NEW QUAD translations are: [-4, -4, -1, -7, -13, 1, -7, -6]<br />
<br />
Try a few different max-composites from runs 21-24:<br />
<br />
The NEW QUAD translations are: [-4, -5, -2, -7, -12, 0, -7, -5]<br />
<br />
The NEW QUAD translations are: [-3, -4, -1, -7, -12, 0, -7, -5]<br />
<br />
The NEW QUAD translations are: [-3, -3, -4, -6, -12, 0, -7, -5]<br />
<br />
The NEW QUAD translations are: [-3, -3, -1, -7, -11, 0, -6, -3]<br />
<br />
These can be pasted on to the command line for graphical review:<br />
cxi.view /reg/d/psdm/cxi/cxi84914/scratch/$USER/averages/e157/r0021/000/out/max-r0021.pickle \<br />
distl.detector_format_version="CXI 5.1" viewer.calibrate_unitcell.d_min=10 \<br />
viewer.calibrate_unitcell.unitcell=94,94,130.8,90,90,120 viewer.calibrate_unitcell.spacegroup=P6122 \<br />
distl.quad_translations=-3,-4,-3,-7,-12,0,-7,-5<br />
<br />
This looks slightly better. From the GUI it also appears that distance=176 fits better than distance=171; meaning that the detz_offset for the configuration file should be 576, not 571.<br />
<br />
==Unit-pixel tile positions==<br />
Now we will index the data to derive model lattices. [The configuration file [[L498-thermolysin-17.cfg]] names our phil parameter file [[L498-thermolysin-t000.phil]] (t000 means trial 0)]. We'll then compare model and observation, from which we can deduce better metrology.<br />
<br />
for m in 17 18 19 20; \<br />
do echo $m; cxi.lsf -c ~/myrelease/cxi84914/L498-thermolysin-17.cfg \<br />
-o /reg/d/psdm/cxi/cxi84914/scratch/$USER/results/e157/ \<br />
-i /reg/d/psdm/cxi/cxi84914/xtc/e157 -q psanacsq -p 8 -x 157 -r ${m} -t 0; done<br />
<br />
for m in 21 22 23 24 25 26 27; \<br />
do echo $m; cxi.lsf -c ~/myrelease/cxi84914/[[L498-thermolysin-21.cfg]] \<br />
-o /reg/d/psdm/cxi/cxi84914/scratch/$USER/results/e157/ \<br />
-i /reg/d/psdm/cxi/cxi84914/xtc/e157 -q psanacsq -p 8 -x 157 -r ${m} -t 0; done<br />
<br />
bkill 0 # stop all jobs; wrong file path<br />
<br />
A quick command to count how many integration files there are:<br />
for m in `seq 17 27`; \<br />
do echo $m `find /reg/d/psdm/cxi/cxi84914/scratch/$USER/results/e157/r00$m/000/integration -name "int*.pickle"|wc -l` ; done; \<br />
find /reg/d/psdm/cxi/cxi84914/scratch/$USER/results/e157/*/000/integration -name "int*.pickle"|wc -l<br />
<br />
Determine whole-pixel translations for all sensors on the CSPAD. <br />
cspad.metrology data=/reg/d/psdm/cxi/cxi84914/scratch/$USER/results/e157/r*/000/integration bravais_setting_id=12 max_frames=1000 min_count=25 detector_format_version="CXI 5.1" cxi84914/L498-thermolysin-t000.phil | tee L498-17-t000.unit<br />
List out the new unit translations:<br />
cat L498-17-t000.unit |grep -A21 "Unit translations"<br />
Results from three refinement cycles are listed; capture the last one and incorporate it into a new version of the integration phil file, [[L498-thermolysin-t001.phil]]. Also in this phil file, for the next integration round we'll increase our integration limits (3 places) to 1.8 Angstroms. After editing the configuration file to use this new phil file, submit the new round of integration jobs.<br />
<br />
For the purpose of the tutorial, we'll drop runs 17-20 since they contribute only 1095 lattices compared with >18000 for runs 21-27. Fine-tuning of the metrology for runs 17-20 would have to be performed separately since the detector is at a different distance.<br />
<br />
for m in 21 22 23 24 25 26 27; \<br />
do echo $m; cxi.lsf -c ~/myrelease/cxi84914/L498-thermolysin-21.cfg \<br />
-o /reg/d/psdm/cxi/cxi84914/scratch/$USER/results/e157/ \<br />
-i /reg/d/psdm/cxi/cxi84914/xtc/e157 -q psanacsq -p 8 -x 157 -r ${m} -t 1; done<br />
<br />
Another round of metrology refinement, this time using trial 001 as the basis:<br />
cspad.metrology data=/reg/d/psdm/cxi/cxi84914/scratch/$USER/results/e157/r002[1-7]/001/integration bravais_setting_id=12 max_frames=1000 min_count=25 detector_format_version="CXI 5.1" cxi84914/L498-thermolysin-t001.phil | tee L498-21-t001.unit<br />
<br />
…proving we have not yet converged. Take this output and construct trial 002: [[L498-thermolysin-t002.phil]]; edit the *.cfg file. Submit another round of integration jobs:<br />
for m in 21 22 23 24 25 26 27; \<br />
do echo $m; cxi.lsf -c ~/myrelease/cxi84914/L498-thermolysin-21.cfg \<br />
-o /reg/d/psdm/cxi/cxi84914/scratch/$USER/results/e157/ \<br />
-i /reg/d/psdm/cxi/cxi84914/xtc/e157 -q psanacsq -p 8 -x 157 -r ${m} -t 2; done<br />
<br />
Evaluate metrology again:<br />
cspad.metrology data=/reg/d/psdm/cxi/cxi84914/scratch/$USER/results/e157/r002[1-7]/002/integration bravais_setting_id=12 max_frames=1500 min_count=25 detector_format_version="CXI 5.1" cxi84914/L498-thermolysin-t002.phil | tee L498-21-t002.unit<br />
<br />
==Add sub-pixel corrections== <br />
There are negligible changes in the unit-translations (non-zero only at large radius), and the rmsd is now 0.96 pixels. We'll leave the unit-translations exactly where they are now, and add subpixel translations and rotations to the phil file. These are taken from the very end of the log file (L498-21-t002.unit). Incorporate these into [[L498-thermolysin-t003.phil]].<br />
<br />
=Integrate the data=<br />
We're ready for the final integration trial, which will be t003. Again edit the configuration file L498-thermolysin-21.cfg so that it points to the latest phil file, L498-thermolysin-t003.phil. Submit integration jobs:<br />
for m in 21 22 23 24 25 26 27; \<br />
do echo $m; cxi.lsf -c ~/myrelease/cxi84914/L498-thermolysin-21.cfg \<br />
-o /reg/d/psdm/cxi/cxi84914/scratch/$USER/results/e157/ \<br />
-i /reg/d/psdm/cxi/cxi84914/xtc/e157 -q psanacsq -p 8 -x 157 -r ${m} -t 3; done<br />
<br />
…19980 lattices indexed and integrated. If the second lattice is desired (about 10% of images have two) set indexing.outlier_detection.switch=True in the phil file, and integrate again <em> under a new trial number (-t 4)</em>.<br />
<br />
Trial 005: Try first-lattice again, correcting an omission of the mask_pixel_value=-2 phil parameter in trial 003.<br />
<br />
…19987 lattices indexed and integrated in trial 005.<br />
<br />
=Merge the data=<br />
on pslogin, download the reference structure 2tli from the pdb (must be done on pslogin, the only outward-facing host). 2tli.pdb will be used for per-image scaling; 2tli.mtz will be used for reporting correlation to isomorphous synchrotron data.<br />
<br />
phenix.fetch_pdb --mtz 2tli<br />
<br />
back to psana. Get the command file [[mergethermo.csh]]. <br />
mkdir /reg/d/psdm/cxi/cxi84914/scratch/$USER/merge/e157<br />
cd /reg/d/psdm/cxi/cxi84914/scratch/$USER/merge/e157<br />
./mergethermo.csh<br />
<br />
[[mergethermo.log | Log file]]<br />
<br />
[[mergethermo005.log | Better log file from trial 005--using untrusted pixel mask]]<br />
<br />
=Solve the structure=<br />
<br />
Some quick commands to evaluate the data. Here the result *.mtz files must be moved back to your $HOME directory and transferred to your laptop to run PHENIX:<br />
<br />
cp /reg/d/psdm/cxi/cxi84914/scratch/$USER/merge/e157/thermoanom_2tli_s0_mark0.mtz $HOME<br />
cp /reg/d/psdm/cxi/cxi84914/scratch/$USER/merge/e157/thermonoanom_2tli_s0_mark0.mtz $HOME<br />
<br />
Always use the "s0" files; "s1" and "s2" are the semi-datasets used only to calculate CC1/2.<br />
phenix.fetch_pdb --mtz 2tli<br />
phenix.xtriage thermonoanom_2tli_s0_mark0.mtz scaling.input.xray_data.obs_labels=Iobs > triage_noanom.log<br />
<br />
Wilson B factor:14.8<br />
<br />
phenix.automr 2tli.pdb thermonoanom_2tli_s0_mark0.mtz seq_file=2tli.fa identity=100 copies=1 build=False<br />
<br />
The MR-placed model is in ./AutoMR_run_1_/MR.1.pdb.<br />
<br />
Obtain the set of R-free-flags used in the Nature Methods paper<br />
wget http://cci.lbl.gov/publications/download/4ow3_original_iobs_flags.mtz<br />
<br />
Refine the model<br />
phenix.refine ./AutoMR_run_1_/MR.1.pdb refinement.output.prefix=001 \<br />
xray_data.file_name=thermonoanom_2tli_s0_mark0.mtz \<br />
xray_data.r_free_flags.file_name=4ow3_original_iobs_flags.mtz \<br />
xray_data.r_free_flags.label=R-free-flags \<br />
main.number_of_macro_cycles=6 optimize_xyz_weight=True \<br />
optimize_adp_weight=True nproc=20 refinement.input.xray_data.labels=IMEAN \<br />
ordered_solvent=true ordered_solvent.mode=every_macro_cycle<br />
trial 003: RWORK = 23.1% RFREE = 26.5% out to 2.1 Angstrom<br />
<br />
trial 005: RWORK = 22.0% RFREE = 26.3% out to 2.1 Angstrom (taking untrusted pixels into account)<br />
<br />
Run an ersatz script (provided by Nat Echols) to measure the peak heights of the anomalous scatterers.<br />
<br />
libtbx.python [[map_height_at_atoms.py]] \<br />
001_001.pdb thermoanom_2tli_s0_mark0.mtz \<br />
input.xray_data.labels=Iobs \<br />
xray_data.r_free_flags.file_name=4ow3_original_iobs_flags.mtz \<br />
xray_data.r_free_flags.label=R-free-flags<br />
<br />
Promising results for the Zn and one Ca:<br />
<pre><br />
pdb="ZN ZN A 317 " : 5.97 sigma<br />
pdb="CA CA A 318 " : 0.93 sigma<br />
pdb="CA CA A 319 " : 2.74 sigma<br />
pdb="CA CA A 320 " : 1.08 sigma<br />
pdb="CA CA A 321 " : 0.20 sigma<br />
</pre><br />
<br />
Better results from trial 005:<br />
<pre><br />
pdb="ZN ZN A 317 " : 7.66 sigma<br />
pdb="CA CA A 318 " : 1.10 sigma<br />
pdb="CA CA A 319 " : 2.52 sigma<br />
pdb="CA CA A 320 " : 1.40 sigma<br />
pdb="CA CA A 321 " : 1.01 sigma<br />
<br />
</pre></div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Installation&diff=1741Installation2026-08-18T23:11:37Z<p>Aaron: </p>
<hr />
<div>{{Archived}}<br />
<br />
Incoming!</div>Aaron