Cctbx.xfel.merge

cctbx.xfel.merge

cctbx.xfel.merge is the scaling, post-refinement and merging program of cctbx.xfel. It takes the per-image integrated results produced by dials.stills_process — one crystal per still shot — and reduces the whole ensemble to a single set of merged structure-factor intensities (an MTZ file) suitable for structure solution and refinement, together with the crystallographic quality statistics (completeness, multiplicity, ⟨I/σ⟩, CC½, CC_iso, R-factors, etc.).

Because serial datasets routinely contain tens of thousands to millions of still shots, cctbx.xfel.merge is built from the ground up to run under MPI across many nodes. It is almost always launched with mpirun/srun. Note that MPI uses the term 'ranks' to distinguish independent processes that are running across one or more nodes.

Overview

cctbx.xfel.merge is organized as a sequence of workers. Each worker performs one task — reading data, applying a correction, filtering, scaling, post-refining, computing a statistic, or merging — and passes its (possibly modified) experiments and reflections on to the next worker. The list of workers to run is the "step list", and it is fully configurable: you can add, remove or reorder steps with dispatch.step_list. This is what makes the program flexible enough to scale-and-merge in one pass, to scale subsets separately and merge later, or to run a reference-free bootstrap.

The default step list is:

#	Worker	Purpose
1	input	Read integrated experiments (.expt) and reflections (.refl) from disk in DIALS format.
2	balance	Redistribute the input load evenly across MPI ranks (needed when there are fewer or more input files than ranks).
3	model_scaling	Build the full Miller list, model intensities from the reference, and the resolution binner used for scaling and post-refinement.
4	modify	Apply per-reflection modifications, e.g. polarization correction (and optionally re-indexing to resolve indexing ambiguity — see modify.cosym/reindex_to_reference).
5	filter	Reject whole experiments (or reflections) by resolution, unit cell, correlation to the reference, energy, etc.
6	scale	Scale each lattice to the reference dataset (mark0) or simply average (mark1).
7	postrefine	Post-refine per-image crystal parameters and partialities (Sauter 2015).
8	statistics_unitcell	Determine and store the average unit cell.
9	statistics_beam	Determine and store the average wavelength.
10	model_statistics	Rebuild the Miller list / binner for statistics (may use the average unit cell).
11	statistics_resolution	Per-crystal resolution statistics.
12	group	Redistribute reflections across ranks so that every observation of a given HKL is gathered on the same rank prior to merging.
13	errors_merge	Correct the merged error estimates with a per-HKL error model (e.g. Ev11, Brewster 2019b, or MM24 (the default), Mittan-Moreau (2025)).
14	statistics_intensity	Intensity statistics by resolution.
15	merge	Merge symmetry-equivalent intensities, gather all HKLs at rank 0, and write "odd", "even" and "all" MTZ files.
16	statistics_intensity_cxi	Correlation and R-factor statistics (ported from cxi.merge): the "Table of Scaling Results" — CC½ (half-dataset correlation, labelled "CC int"), CCiso, Rint, Rsplit, Riso.

Additional, non-default workers are available and can be inserted into the step list, including tdata (write out unit cells for clustering), export (batch MTZ for Careless), publish (upload merged files to a Google Drive folder), monitor (resource-usage logging), integrate (on-the-fly integration), prepare (SpReAD energy binning), and lunus (diffuse-scattering deck). A deltaccint worker (ΔCC½ lattice filtering) can be added after group. Custom workers can be supplied through the XFEL_CUSTOM_WORKER_PATH environment variable.

Basic usage

cctbx.xfel.merge is run under MPI:

mpirun cctbx.xfel.merge merge.phil

A typical merge.phil:

input.path=<path to data>
input.experiments_suffix=.expt
input.reflections_suffix=.refl

filter.algorithm=unit_cell
filter.unit_cell.value.relative_length_tolerance=0.03
filter.outlier.min_corr=0.1

select.algorithm=significance_filter
select.significance_filter.sigma=0.1
select.significance_filter.min_ct=200
select.significance_filter.max_ct=300

scaling.model=<model.pdb or model.mtz>
scaling.resolution_scalar=0.96

merging.d_min=1.8
merging.error.model=mm24
merging.merge_anomalous=True

postrefinement.enable=True
statistics.n_bins=20

output.do_timing=True
output.output_dir=<output folder>
output.prefix=<prefix for output files>

Explore the parameters with:

• cctbx.xfel.merge -c — show configurable parameters
• cctbx.xfel.merge -c -e 10 -a 2 — show all parameters (expert level 10) with full help text

Two-stage scaling and merging

For large, growing datasets (the mode used by the cctbx.xfel GUI), it is more efficient to scale and post-refine each subset of data once, then merge repeatedly as more data arrive. This is done by running the program twice with different step lists.

Stage 1 — scale each subset (stops after per-image work and writes intermediate DIALS files):

dispatch.step_list=input balance model_scaling modify filter scale postrefine \
    statistics_unitcell statistics_beam model_statistics statistics_resolution
output.save_experiments_and_reflections=True

output.save_experiments_and_reflections=True dumps the scaled/post-refined data as experiment lists and reflection tables that the merge stage can read back in.

Stage 2 — merge the scaled subsets (add an input.path for each finished subset):

dispatch.step_list=input model_scaling statistics_unitcell statistics_beam \
    model_statistics statistics_resolution group errors_merge statistics_intensity \
    merge statistics_intensity_cxi

Merging without a reference (bootstrap / mark1)

If no scaling/merging reference is available, use a bootstrap procedure. First run the mark1 algorithm, which simply averages the data (all per-image scale factors = 1, no post-refinement), specifying the target cell and symmetry explicitly:

scaling.algorithm=mark1
scaling.unit_cell=<unit cell>
scaling.space_group=<space group>
merging.error.model=errors_from_sample_residuals

The resulting MTZ is then used as the reference (scaling.model=) for subsequent regular merges. Iterate a few cycles; CC½ typically improves and converges over the first few rounds.

Important parameters

Parameters are given by their full PHIL path with defaults in parentheses. The program exposes many more parameters (see -c -e 10 -a 2); those below are the ones most commonly set.

Input

Parameter	Default	Description
input.path	None	Path(s) to the integrated data — a folder, several folders, or individual files; wildcards allowed; may be repeated. Matching .expt and .refl files must both be present.
input.experiments_suffix	_integrated.expt	Only load experiment files ending with this suffix.
input.reflections_suffix	_integrated.refl	Only load reflection files ending with this suffix.
input.parallel_file_load.method	uniform	uniform spreads input files evenly over all ranks; node_memory distributes so per-node memory limits are respected.
input.alist.file	None	reject to merge only (or exclude) a specified subset.
input.persistent_refl_cols	None	Reflection-table columns to preserve through the pruning steps (and into saved tables).

Filtering whole experiments (filter)

The filter section accepts or rejects entire experiments.

Parameter	Default	Description
filter.algorithm	(none)	One or more of n_obs, resolution, unit_cell, energy, bootstrap, experiment_prefix. Nothing is filtered unless listed.
filter.unit_cell.algorithm	value	How to restrict unit cells: value (compare to a target), range, or cluster (use a precomputed covariance clustering).
filter.unit_cell.value.relative_length_tolerance	0.1	Fractional deviation of cell edge lengths allowed vs. the target cell. Often tightened (e.g. 0.03).
filter.unit_cell.value.absolute_angle_tolerance	2.0	Allowed cell-angle deviation (degrees) vs. the target.
filter.unit_cell.value.target_unit_cell / target_space_group	Auto	Target cell/symmetry; Auto uses the scaling model.
filter.outlier.min_corr	0.1	Minimum correlation of an image's intensities to the reference model; images below this are rejected. Only applied when a scaling model is present. Use -1 to accept all images regardless of correlation.
filter.resolution.d_min	None	Reject an experiment unless some of its reflections extend beyond this resolution.

Selecting individual reflections (select)

The select section accepts or rejects individual reflections (contrast with filter, which acts on whole experiments).

Parameter	Default	Description
select.algorithm	(none)	One or more of significance_filter, panel, cspad_sensor, isolation_forest, target_energy.
select.significance_filter.sigma	0.5	Trim each lattice at the resolution where the binned ⟨I/σ⟩ drops below this value. Often reduced to 0.1.
select.significance_filter.min_ct / max_ct	10 / 50	Target minimum/maximum mean reflections per resolution bin when choosing the number of bins. For large unit cells these are often raised (e.g. 200/300); for small cells, use the defaults.

Modifying intensities (modify)

Parameter	Default	Description
modify.algorithm	polarization	Per-reflection corrections to apply (default applies the polarization correction).
modify.cosym	—	Resolve an indexing ambiguity across the ensemble using the Gildea/Winter (Brehm/Diederichs) cosym method, optionally anchored to scaling.model.
modify.reindex_to_reference / reindex_to_abc	—	Alternative re-indexing operators (to a reference model, or an explicit operator).

Scaling (scaling)

Parameter	Default	Description
scaling.model	None	Reference dataset: a PDB file (uses the CRYST1 cell and computes model intensities, including bulk solvent) or an MTZ from a previous cycle. Required for mark0 scaling.
scaling.algorithm	mark0	mark0: per-image scaling against the reference. mark1: no scaling, just Monte-Carlo averaging (all scale factors = 1) — used for reference-free bootstrap.
scaling.unit_cell / scaling.space_group	None	Cell/symmetry to use when no model is provided (i.e. mark1).
scaling.resolution_scalar	0.969	merging.d_min is multiplied by this when deciding which reflections fall within the resolution limit, generating a few extra reference reflections to account for per-shot cell variation (e.g. set 0.96 for a few more).
scaling.mtz.mtz_column_F	fobs	Column name in the reference MTZ containing the reference structure factors (intensities or amplitudes).

Post-refinement (postrefinement)

Parameter	Default	Description
postrefinement.enable	True	Enable post-refinement of per-image parameters/partialities (Sauter 2015).
postrefinement.algorithm	rs	Post-refinement protocol: rs, rs2, rs_hybrid, or eta_deff.

Merging and error model (merging)

Parameter	Default	Description
merging.d_min	None	High-resolution limit for scaling and merging.
merging.d_max	None	Low-resolution limit (currently affects only CCiso).
merging.merge_anomalous	False	True merges Bijvoet mates together (higher multiplicity); False keeps them separate to preserve the anomalous signal.
merging.minimum_multiplicity	2	Do not produce a merged structure factor for HKLs measured fewer than this many times.
merging.error.model	mm24	Merged-sigma error model: mm24 (Mittan-Moreau 2025, current default), ev11 (Evans 2011 / sdfac-refine), ha14, or errors_from_sample_residuals (no refined error terms — used with mark1 bootstrap).
merging.set_average_unit_cell	True	Output MTZ adopts the average unit cell of the data rather than the reference model's cell.

Statistics (statistics)

Parameter	Default	Description
statistics.n_bins	10	Number of resolution bins in the statistics tables. Often raised to 20.
statistics.cciso.mtz_file	None	Isomorphous reference structure factors for Riso / CCiso.
statistics.report_ML	True	Report per-frame max-likelihood attributes (domain size, mosaicity).

Output (output)

Parameter	Default	Description
output.output_dir	.	Directory for output files (created if it does not exist).
output.prefix	iobs	Prefix prepended to all output file names.
output.save_experiments_and_reflections	False	Dump the final experiments/reflections from the last worker as DIALS files — used for the two-stage scale-then-merge workflow.
output.do_timing	False	Log elapsed time per execution step.
output.expanded_bookkeeping	False	Store extra provenance columns (input file/reflection indices, odd/even flag, original id) in the saved reflection tables.

Multiprocessing (mp)

Parameter	Default	Description
mp.method	mpi	Multiprocessing method (only mpi is currently supported).
mp.debug.cProfile	False	Enable code profiling; a .prof file is written per rank.

Output files

The merge worker gathers all HKLs at rank 0 and writes three MTZ files, prefixed with output.prefix (default iobs):

• <prefix>_all.mtz — the merged dataset using all observations (the main result for structure solution)
• <prefix>_odd.mtz and <prefix>_even.mtz — the two half-datasets used for CC½ and related half-dataset statistics

A main log records the per-step progress and the merging statistics tables by resolution bin: completeness, multiplicity and ⟨I/σ⟩, plus the "Table of Scaling Results" reporting CC½ (the half-dataset correlation, labelled "CC int"), CC_iso, R_int, R_split and R_iso. The merged MTZ can optionally be uploaded to a shared Google Drive folder by adding the publish worker.

References

• Sauter NK (2015). J Synchrotron Radiat 22, 239–48. (post-refinement)
• Uervirojnangkoorn M, et al. (2015). eLife 10.7554/eLife.05421. (PRIME / bootstrap)
• Brewster AS, et al. (2018). Acta Cryst. D 74, 877–894.
• Brewster AS, et al. (2019a). Computational Crystallography Newsletter 10, 22–39. (XFEL GUI)
• Brewster AS, et al. (2019b). Acta Cryst. D 75, 959–968. (error models)
• Mittan-Moreau (2025). (mm24 error model)

See also

• dials.stills_process — produces the integrated data that cctbx.xfel.merge consumes
• cctbx.xfel GUI — orchestrates separate scaling and repeated merging as data are collected
• The program README at xfel/merging/README.md in cctbx_project