Cctbx.xfel.merge - Revision history

Aaron at 23:44, 21 August 2026

2026-08-21T23:44:27Z

← Older revision		Revision as of 23:44, 21 August 2026
Line 1:		Line 1:
	= ''cctbx.xfel.merge'' =		{{DISPLAYTITLE:''cctbx.xfel.merge''}}

	'''''cctbx.xfel.merge''''' is the scaling, post-refinement and merging program of cctbx.xfel. It takes		'''''cctbx.xfel.merge''''' is the scaling, post-refinement and merging program of cctbx.xfel. It takes
Line 12:		Line 12:
	to distinguish independent processes that are running across one or more nodes.		to distinguish independent processes that are running across one or more nodes.

	== Overview ==		= Overview =

	<code>cctbx.xfel.merge</code> is organized as a sequence of '''workers'''. Each worker performs one		<code>cctbx.xfel.merge</code> is organized as a sequence of '''workers'''. Each worker performs one
Line 69:		Line 69:
	<code>XFEL_CUSTOM_WORKER_PATH</code> environment variable.		<code>XFEL_CUSTOM_WORKER_PATH</code> environment variable.

	== Basic usage ==		= Basic usage =

	<code>cctbx.xfel.merge</code> is run under MPI:		<code>cctbx.xfel.merge</code> is run under MPI:
Line 111:		Line 111:
	* <code>cctbx.xfel.merge -c -e 10 -a 2</code> — show all parameters (expert level 10) with full help text		* <code>cctbx.xfel.merge -c -e 10 -a 2</code> — show all parameters (expert level 10) with full help text

	== Two-stage scaling and merging ==		= Two-stage scaling and merging =

	For large, growing datasets (the mode used by the [[cctbx.xfel GUI]]), it is more efficient to scale		For large, growing datasets (the mode used by the [[cctbx.xfel GUI]]), it is more efficient to scale
Line 136:		Line 136:
	</pre>		</pre>

	== Merging without a reference (bootstrap / mark1) ==		= Merging without a reference (bootstrap / mark1) =

	If no scaling/merging reference is available, use a bootstrap procedure. First run the		If no scaling/merging reference is available, use a bootstrap procedure. First run the
Line 152:		Line 152:
	merges. Iterate a few cycles; CC½ typically improves and converges over the first few rounds.		merges. Iterate a few cycles; CC½ typically improves and converges over the first few rounds.

	== Important parameters ==		= Important parameters =

	Parameters are given by their full PHIL path with defaults in parentheses. The program exposes many		Parameters are given by their full PHIL path with defaults in parentheses. The program exposes many
	more parameters (see <code>-c -e 10 -a 2</code>); those below are the ones most commonly set.		more parameters (see <code>-c -e 10 -a 2</code>); those below are the ones most commonly set.

	=== Input ===		== Input ==

	{\| class="wikitable"		{\| class="wikitable"
Line 175:		Line 175:
	\|}		\|}

	=== Filtering whole experiments (<code>filter</code>) ===		== Filtering whole experiments (<code>filter</code>) ==

	The <code>filter</code> section accepts or rejects '''entire experiments'''.		The <code>filter</code> section accepts or rejects '''entire experiments'''.
Line 197:		Line 197:
	\|}		\|}

	=== Selecting individual reflections (<code>select</code>) ===		== Selecting individual reflections (<code>select</code>) ==

	The <code>select</code> section accepts or rejects '''individual reflections''' (contrast with		The <code>select</code> section accepts or rejects '''individual reflections''' (contrast with
Line 212:		Line 212:
	\|}		\|}

	=== Modifying intensities (<code>modify</code>) ===		== Modifying intensities (<code>modify</code>) ==

	{\| class="wikitable"		{\| class="wikitable"
Line 224:		Line 224:
	\|}		\|}

	=== Scaling (<code>scaling</code>) ===		== Scaling (<code>scaling</code>) ==

	{\| class="wikitable"		{\| class="wikitable"
Line 240:		Line 240:
	\|}		\|}

	=== Post-refinement (<code>postrefinement</code>) ===		== Post-refinement (<code>postrefinement</code>) ==

	{\| class="wikitable"		{\| class="wikitable"
Line 250:		Line 250:
	\|}		\|}

	=== Merging and error model (<code>merging</code>) ===		== Merging and error model (<code>merging</code>) ==

	{\| class="wikitable"		{\| class="wikitable"
Line 268:		Line 268:
	\|}		\|}

	=== Statistics (<code>statistics</code>) ===		== Statistics (<code>statistics</code>) ==

	{\| class="wikitable"		{\| class="wikitable"
Line 280:		Line 280:
	\|}		\|}

	=== Output (<code>output</code>) ===		== Output (<code>output</code>) ==

	{\| class="wikitable"		{\| class="wikitable"
Line 296:		Line 296:
	\|}		\|}

	=== Multiprocessing (<code>mp</code>) ===		== Multiprocessing (<code>mp</code>) ==

	{\| class="wikitable"		{\| class="wikitable"
Line 306:		Line 306:
	\|}		\|}

	== Output files ==		= Output files =

	The <code>merge</code> worker gathers all HKLs at rank 0 and writes three MTZ files, prefixed with		The <code>merge</code> worker gathers all HKLs at rank 0 and writes three MTZ files, prefixed with
Line 320:		Line 320:
	uploaded to a shared [https://github.com/cctbx/cctbx_project/tree/master/xfel/merging/application/publish Google Drive folder] by adding the <code>publish</code> worker.		uploaded to a shared [https://github.com/cctbx/cctbx_project/tree/master/xfel/merging/application/publish Google Drive folder] by adding the <code>publish</code> worker.

	== References ==		= References =

	* Sauter NK (2015). ''J Synchrotron Radiat'' '''22''', 239–48. (post-refinement)		* Sauter NK (2015). ''J Synchrotron Radiat'' '''22''', 239–48. (post-refinement)
Line 329:		Line 329:
	* Mittan-Moreau (2025). (mm24 error model)		* Mittan-Moreau (2025). (mm24 error model)

	== See also ==		= See also =

	* [[dials.stills_process]] — produces the integrated data that <code>cctbx.xfel.merge</code> consumes		* [[dials.stills_process]] — produces the integrated data that <code>cctbx.xfel.merge</code> consumes
	* [[cctbx.xfel GUI]] — orchestrates separate scaling and repeated merging as data are collected		* [[cctbx.xfel GUI]] — orchestrates separate scaling and repeated merging as data are collected
	* The program README at <code>xfel/merging/README.md</code> in cctbx_project		* The program README at <code>xfel/merging/README.md</code> in cctbx_project

Aaron: /* Basic usage */

2026-08-21T23:26:17Z

Basic usage

← Older revision		Revision as of 23:26, 21 August 2026
Line 103:		Line 103:
	statistics.n_bins=20		statistics.n_bins=20

	~~output.do_timing=True~~
	output.output_dir=<output folder>		output.output_dir=<output folder>
	output.prefix=<prefix for output files>		output.prefix=<prefix for output files>

Aaron: /* Statistics (statistics) */

2026-08-21T23:25:39Z

Statistics (statistics)

← Older revision		Revision as of 23:25, 21 August 2026
Line 274:		Line 274:
	! Parameter !! Default !! Description		! Parameter !! Default !! Description
	\|-		\|-
	\| <code>statistics.n_bins</code> \|\| 10 \|\| Number of resolution bins in the statistics tables.		\| <code>statistics.n_bins</code> \|\| 10 \|\| Number of resolution bins in the statistics tables. Often raised to 20.
	\|-		\|-
	\| <code>statistics.cciso.mtz_file</code> \|\| None \|\| Isomorphous reference structure factors for R<sub>iso</sub> / CC<sub>iso</sub>.		\| <code>statistics.cciso.mtz_file</code> \|\| None \|\| Isomorphous reference structure factors for R<sub>iso</sub> / CC<sub>iso</sub>.

Aaron: /* Merging and error model (merging) */

2026-08-21T23:24:52Z

Merging and error model (merging)

← Older revision		Revision as of 23:24, 21 August 2026
Line 266:		Line 266:
	\| <code>merging.error.model</code> \|\| mm24 \|\| Merged-sigma error model: <code>mm24</code> (Mittan-Moreau 2025, current default), <code>ev11</code> (Evans 2011 / sdfac-refine), <code>ha14</code>, or <code>errors_from_sample_residuals</code> (no refined error terms — used with <code>mark1</code> bootstrap).		\| <code>merging.error.model</code> \|\| mm24 \|\| Merged-sigma error model: <code>mm24</code> (Mittan-Moreau 2025, current default), <code>ev11</code> (Evans 2011 / sdfac-refine), <code>ha14</code>, or <code>errors_from_sample_residuals</code> (no refined error terms — used with <code>mark1</code> bootstrap).
	\|-		\|-
	\| <code>merging.set_average_unit_cell</code> \|\| True \|\| Output MTZ adopts the ~~(cluster-derived)~~ average unit cell of the data rather than the reference model's cell.		\| <code>merging.set_average_unit_cell</code> \|\| True \|\| Output MTZ adopts the average unit cell of the data rather than the reference model's cell.
	\|}		\|}

Aaron: /* Merging and error model (merging) */

2026-08-21T23:24:39Z

Merging and error model (merging)

← Older revision		Revision as of 23:24, 21 August 2026
Line 264:		Line 264:
	\| <code>merging.minimum_multiplicity</code> \|\| 2 \|\| Do not produce a merged structure factor for HKLs measured fewer than this many times.		\| <code>merging.minimum_multiplicity</code> \|\| 2 \|\| Do not produce a merged structure factor for HKLs measured fewer than this many times.
	\|-		\|-
	\| <code>merging.error.model</code> \|\| mm24 \|\| Merged-sigma error model: <code>mm24</code> (Mittan-Moreau 2025, current default), <code>ev11</code> (Evans 2011 / sdfac-refine~~, recommended in the README~~), <code>ha14</code>, or <code>errors_from_sample_residuals</code> (no refined error terms — used with <code>mark1</code> bootstrap).		\| <code>merging.error.model</code> \|\| mm24 \|\| Merged-sigma error model: <code>mm24</code> (Mittan-Moreau 2025, current default), <code>ev11</code> (Evans 2011 / sdfac-refine), <code>ha14</code>, or <code>errors_from_sample_residuals</code> (no refined error terms — used with <code>mark1</code> bootstrap).
	\|-		\|-
	\| <code>merging.set_average_unit_cell</code> \|\| True \|\| Output MTZ adopts the (cluster-derived) average unit cell of the data rather than the reference model's cell.		\| <code>merging.set_average_unit_cell</code> \|\| True \|\| Output MTZ adopts the (cluster-derived) average unit cell of the data rather than the reference model's cell.

Aaron: /* Selecting individual reflections (select) */

2026-08-21T23:22:00Z

Selecting individual reflections (select)

← Older revision		Revision as of 23:22, 21 August 2026
Line 208:		Line 208:
	\| <code>select.algorithm</code> \|\| (none) \|\| One or more of <code>significance_filter</code>, <code>panel</code>, <code>cspad_sensor</code>, <code>isolation_forest</code>, <code>target_energy</code>.		\| <code>select.algorithm</code> \|\| (none) \|\| One or more of <code>significance_filter</code>, <code>panel</code>, <code>cspad_sensor</code>, <code>isolation_forest</code>, <code>target_energy</code>.
	\|-		\|-
	\| <code>select.significance_filter.sigma</code> \|\| 0.5 \|\| Trim each lattice at the resolution where the binned ⟨I/σ⟩ drops below this value.		\| <code>select.significance_filter.sigma</code> \|\| 0.5 \|\| Trim each lattice at the resolution where the binned ⟨I/σ⟩ drops below this value. Often reduced to 0.1.
	\|-		\|-
	\| <code>select.significance_filter.min_ct</code> / <code>max_ct</code> \|\| 10 / 50 \|\| Target minimum/maximum mean reflections per resolution bin when choosing the number of bins. For large unit cells these are often raised (e.g. 200/300); for small cells, use the defaults.		\| <code>select.significance_filter.min_ct</code> / <code>max_ct</code> \|\| 10 / 50 \|\| Target minimum/maximum mean reflections per resolution bin when choosing the number of bins. For large unit cells these are often raised (e.g. 200/300); for small cells, use the defaults.

Aaron: /* Two-stage scaling and merging */

2026-08-21T23:18:48Z

Two-stage scaling and merging

← Older revision		Revision as of 23:18, 21 August 2026
Line 123:		Line 123:
	dispatch.step_list=input balance model_scaling modify filter scale postrefine \		dispatch.step_list=input balance model_scaling modify filter scale postrefine \
	statistics_unitcell statistics_beam model_statistics statistics_resolution		statistics_unitcell statistics_beam model_statistics statistics_resolution
	~~input.parallel_file_load.method=uniform~~
	output.save_experiments_and_reflections=True		output.save_experiments_and_reflections=True
	</pre>		</pre>
Line 136:		Line 135:
	model_statistics statistics_resolution group errors_merge statistics_intensity \		model_statistics statistics_resolution group errors_merge statistics_intensity \
	merge statistics_intensity_cxi		merge statistics_intensity_cxi
	~~input.parallel_file_load.method=uniform~~
	</pre>		</pre>

Aaron: /* Basic usage */

2026-08-21T23:17:17Z

Basic usage

← Older revision		Revision as of 23:17, 21 August 2026
Line 77:		Line 77:
	</pre>		</pre>

	A typical <code>merge.phil</code> ~~(from the program README)~~:		A typical <code>merge.phil</code>:

	<pre>		<pre>
Line 97:		Line 97:

	merging.d_min=1.8		merging.d_min=1.8
	merging.error.model=~~ev11~~		merging.error.model=mm24
	merging.merge_anomalous=True		merging.merge_anomalous=True

Aaron: /* Merging and error model (merging) */

2026-08-21T23:15:21Z

Merging and error model (merging)

← Older revision		Revision as of 23:15, 21 August 2026
Line 266:		Line 266:
	\| <code>merging.minimum_multiplicity</code> \|\| 2 \|\| Do not produce a merged structure factor for HKLs measured fewer than this many times.		\| <code>merging.minimum_multiplicity</code> \|\| 2 \|\| Do not produce a merged structure factor for HKLs measured fewer than this many times.
	\|-		\|-
	\| <code>merging.error.model</code> \|\| mm24 \|\| Merged-sigma error model: <code>mm24</code> (Mittan-Moreau ~~2024~~, current default), <code>ev11</code> (Evans 2011 / sdfac-refine, recommended in the README), <code>ha14</code>, or <code>errors_from_sample_residuals</code> (no refined error terms — used with <code>mark1</code> bootstrap).		\| <code>merging.error.model</code> \|\| mm24 \|\| Merged-sigma error model: <code>mm24</code> (Mittan-Moreau 2025, current default), <code>ev11</code> (Evans 2011 / sdfac-refine, recommended in the README), <code>ha14</code>, or <code>errors_from_sample_residuals</code> (no refined error terms — used with <code>mark1</code> bootstrap).
	\|-		\|-
	\| <code>merging.set_average_unit_cell</code> \|\| True \|\| Output MTZ adopts the (cluster-derived) average unit cell of the data rather than the reference model's cell.		\| <code>merging.set_average_unit_cell</code> \|\| True \|\| Output MTZ adopts the (cluster-derived) average unit cell of the data rather than the reference model's cell.

Aaron: /* References */

2026-08-21T23:14:53Z

References

← Older revision		Revision as of 23:14, 21 August 2026
Line 330:		Line 330:
	* Brewster AS, ''et al.'' (2019a). ''Computational Crystallography Newsletter'' '''10''', 22–39. (XFEL GUI)		* Brewster AS, ''et al.'' (2019a). ''Computational Crystallography Newsletter'' '''10''', 22–39. (XFEL GUI)
	* Brewster AS, ''et al.'' (2019b). ''Acta Cryst.'' D '''75''', 959–968. (error models)		* Brewster AS, ''et al.'' (2019b). ''Acta Cryst.'' D '''75''', 959–968. (error models)
	* Mittan-Moreau (~~2024~~). (mm24 error model)		* Mittan-Moreau (2025). (mm24 error model)

	== See also ==		== See also ==