Dials.stills process - Revision history

Aaron at 23:40, 21 August 2026

2026-08-21T23:40:33Z

← Older revision		Revision as of 23:40, 21 August 2026
Line 1:		Line 1:
	{{DISPLAYTITLE:dials.stills_process}}		{{DISPLAYTITLE:''dials.stills_process''}}

	'''''dials.stills_process''''' is a stand-alone program for processing still diffraction images and is		'''''dials.stills_process''''' is a stand-alone program for processing still diffraction images and is

Aaron at 23:39, 21 August 2026

2026-08-21T23:39:39Z

← Older revision		Revision as of 23:39, 21 August 2026
Line 1:		Line 1:
	= dials.stills_process =		{{DISPLAYTITLE:dials.stills_process}}

	'''''dials.stills_process''''' is a stand-alone program for processing still diffraction images and is		'''''dials.stills_process''''' is a stand-alone program for processing still diffraction images and is
Line 15:		Line 15:
	and it drives most of the stills-specific defaults described below.		and it drives most of the stills-specific defaults described below.

	== Overview ==		= Overview =

	For each image, <code>dials.stills_process</code> runs the following stages in order. Any stage can		For each image, <code>dials.stills_process</code> runs the following stages in order. Any stage can
Line 39:		Line 39:
	shots of which only a fraction are hits.		shots of which only a fraction are hits.

	== Basic usage ==		= Basic usage =

	<pre>		<pre>
Line 70:		Line 70:
	* <code>dials.stills_process ... input.show_image_tags=True</code> — list the image tags in a dataset without processing (useful for selecting subsets with <code>input.image_tag</code>)		* <code>dials.stills_process ... input.show_image_tags=True</code> — list the image tags in a dataset without processing (useful for selecting subsets with <code>input.image_tag</code>)

	== Important parameters ==		= Important parameters =

	The program exposes the entire DIALS parameter set (spot-finding, indexing, refinement, integration,		The program exposes the entire DIALS parameter set (spot-finding, indexing, refinement, integration,
Line 76:		Line 76:
	still data. Parameters are given by their full PHIL path with their default value.		still data. Parameters are given by their full PHIL path with their default value.

	=== Input ===		== Input ==

	{\| class="wikitable"		{\| class="wikitable"
Line 94:		Line 94:
	\|}		\|}

	=== Dispatch — controlling the pipeline ===		== Dispatch — controlling the pipeline ==

	{\| class="wikitable"		{\| class="wikitable"
Line 122:		Line 122:
	\|}		\|}

	=== Spot-finding ===		== Spot-finding ==

	Spot-finding uses the standard DIALS dispersion thresholding. The most impactful parameters for		Spot-finding uses the standard DIALS dispersion thresholding. The most impactful parameters for
Line 149:		Line 149:
	<code>input.ignore_gain_mismatch=True</code> only if you understand the consequences.		<code>input.ignore_gain_mismatch=True</code> only if you understand the consequences.

	=== Indexing ===		== Indexing ==

	{\| class="wikitable"		{\| class="wikitable"
Line 169:		Line 169:
	\|}		\|}

	=== Integration ===		== Integration ==

	{\| class="wikitable"		{\| class="wikitable"
Line 183:		Line 183:
	\|}		\|}

	=== Output ===		== Output ==

	{\| class="wikitable"		{\| class="wikitable"
Line 201:		Line 201:
	\|}		\|}

	=== Multiprocessing ===		== Multiprocessing ==

	{\| class="wikitable"		{\| class="wikitable"
Line 217:		Line 217:
	balancing across the many fast-failing non-hits typical of serial data.		balancing across the many fast-failing non-hits typical of serial data.

	== Output files ==		= Output files =

	With the default composite output, each process (or each aggregating rank under MPI) writes		With the default composite output, each process (or each aggregating rank under MPI) writes
Line 231:		Line 231:
	''[[cctbx.xfel.merge]]''.		''[[cctbx.xfel.merge]]''.

	== See also ==		= See also =

	* ''[[cctbx.xfel.merge]]'' — scaling, post-refinement and merging of the integrated data		* ''[[cctbx.xfel.merge]]'' — scaling, post-refinement and merging of the integrated data
	* The [http://dials.github.io DIALS documentation] for the underlying spot-finding, indexing, refinement and integration algorithms		* The [http://dials.github.io DIALS documentation] for the underlying spot-finding, indexing, refinement and integration algorithms

Aaron at 22:55, 21 August 2026

2026-08-21T22:55:23Z

← Older revision		Revision as of 22:55, 21 August 2026
Line 7:		Line 7:
	through the full DIALS pipeline: '''import → spot-finding → indexing → refinement → integration'''.		through the full DIALS pipeline: '''import → spot-finding → indexing → refinement → integration'''.
	Its output (integrated intensities together with the per-image experimental models) is the input to		Its output (integrated intensities together with the per-image experimental models) is the input to
	the merging program [[cctbx.xfel.merge]].		the merging program ''[[cctbx.xfel.merge]]''.

	Because every image is a separate still exposure of a different crystal (rather than a wedge of		Because every image is a separate still exposure of a different crystal (rather than a wedge of
Line 233:		Line 233:
	== See also ==		== See also ==

	* [[cctbx.xfel.merge]] — scaling, post-refinement and merging of the integrated data		* ''[[cctbx.xfel.merge]]'' — scaling, post-refinement and merging of the integrated data
	* [~~[IOTA]] — a front-end that supplies preset parameters for <code>~~dials.~~stills_process</code>~~		* The [http://dials.github.io DIALS documentation] for the underlying spot-finding, indexing, refinement and integration algorithms
	* [[Data Processing at SACLA]] — SACLA-specific processing workflow
	* The DIALS documentation for the underlying spot-finding, indexing, refinement and integration algorithms

Aaron: /* Output files */

2026-08-21T22:52:56Z

Output files

← Older revision		Revision as of 22:52, 21 August 2026
Line 224:		Line 224:
	* <code><tag>_strong.refl</code> — strong spots from spot-finding		* <code><tag>_strong.refl</code> — strong spots from spot-finding
	* <code><tag>_indexed.refl</code> and <code><tag>_refined.expt</code> — indexed reflections and their crystal models		* <code><tag>_indexed.refl</code> and <code><tag>_refined.expt</code> — indexed reflections and their crystal models
	* <code><tag>_integrated.refl</code> and <code><tag>_integrated.expt</code> — integrated intensities and final models (the input to [[cctbx.xfel.merge]])		* <code><tag>_integrated.refl</code> and <code><tag>_integrated.expt</code> — integrated intensities and final models (the input to ''[[cctbx.xfel.merge]]'')

	Per-image progress and status codes are recorded in <code>debug/debug_<rank>.txt</code>, and		Per-image progress and status codes are recorded in <code>debug/debug_<rank>.txt</code>, and
	overall logging goes to <code>dials.process.log</code> (or per-rank logs under MPI). These integrated		overall logging goes to <code>dials.process.log</code> (or per-rank logs under MPI). These integrated
	<code>.expt</code>/<code>.refl</code> pairs are then scaled and merged with		<code>.expt</code>/<code>.refl</code> pairs are then scaled and merged with
	[[cctbx.xfel.merge]].		''[[cctbx.xfel.merge]]''.

	== See also ==		== See also ==

Aaron: /* Overview */

2026-08-21T22:52:20Z

Overview

← Older revision		Revision as of 22:52, 21 August 2026
Line 26:		Line 26:
	# '''Indexing''' — determine the crystal orientation and (optionally constrained) unit cell (<code>dispatch.index</code>).		# '''Indexing''' — determine the crystal orientation and (optionally constrained) unit cell (<code>dispatch.index</code>).
	# '''Refinement''' — refine the crystal model and re-estimate the mosaic/domain-size parameters. Detector and beam are held fixed by default (<code>dispatch.refine</code>).		# '''Refinement''' — refine the crystal model and re-estimate the mosaic/domain-size parameters. Detector and beam are held fixed by default (<code>dispatch.refine</code>).
	# '''Integration''' — predict reflections and measure their intensities using the stills integrator (<code>dispatch.integrate</code>), with optional ~~Kapton absorption correction and a~~ significance filter.		# '''Integration''' — predict reflections and measure their intensities using the stills integrator (<code>dispatch.integrate</code>), with an optional significance filter.

	The program applies a set of stills-specific defaults on top of the standard DIALS parameters,		The program applies a set of stills-specific defaults on top of the standard DIALS parameters,

Aaron: /* Important parameters */

2026-08-21T22:51:29Z

Important parameters

Aaron: /* Basic usage */

2026-08-21T22:30:49Z

Basic usage

← Older revision		Revision as of 22:30, 21 August 2026
Line 57:		Line 57:
	indexing.known_symmetry.unit_cell=78.9,78.9,38.1,90,90,90		indexing.known_symmetry.unit_cell=78.9,78.9,38.1,90,90,90
	indexing.method=fft1d		indexing.method=fft1d

	~~integration.background.simple.outlier.algorithm=null~~

	input.reference_geometry=refined.expt # from a previous metrology refinement		input.reference_geometry=refined.expt # from a previous metrology refinement

	~~output.composite_output=True~~
	output.output_dir=out		output.output_dir=out
	mp.nproc=32		mp.nproc=32

Aaron: /* dials.stills_process */

2026-08-21T22:26:16Z

dials.stills_process

← Older revision		Revision as of 22:26, 21 August 2026
Line 2:		Line 2:

	'''''dials.stills_process''''' is a stand-alone program for processing still diffraction images and is		'''''dials.stills_process''''' is a stand-alone program for processing still diffraction images and is
	the core data-reduction program of ''cctbx.xfel''. It takes a set of		the core data-reduction program of ''cctbx.xfel''. It takes a set of
	still diffraction images — typically serial femtosecond crystallography (SFX) snapshots from an		still diffraction images — typically serial femtosecond crystallography (SFX) snapshots from an
	XFEL (LCLS, SACLA, EuXFEL) or still shots from a synchrotron — and analyzes each one, independently,		XFEL (LCLS, SACLA, EuXFEL) or still shots from a synchrotron — and analyzes each one, independently,

Aaron: /* dials.stills_process */

2026-08-21T22:25:47Z

dials.stills_process

← Older revision		Revision as of 22:25, 21 August 2026
Line 1:		Line 1:
	= dials.stills_process =		= dials.stills_process =

	'''dials.stills_process''' is the core data-reduction program of cctbx.xfel. It takes a set of		'''''dials.stills_process''''' is a stand-alone program for processing still diffraction images and is
			the core data-reduction program of ''cctbx.xfel''. It takes a set of
	still diffraction images — typically serial femtosecond crystallography (SFX) snapshots from an		still diffraction images — typically serial femtosecond crystallography (SFX) snapshots from an
	XFEL (LCLS, SACLA, EuXFEL) or still shots from a synchrotron — and ~~reduces~~ each one, independently,		XFEL (LCLS, SACLA, EuXFEL) or still shots from a synchrotron — and analyzes each one, independently,
	through the full DIALS pipeline: '''import → spot-finding → indexing → refinement → integration'''.		through the full DIALS pipeline: '''import → spot-finding → indexing → refinement → integration'''.
	Its output (integrated intensities together with the per-image experimental models) is the input to		Its output (integrated intensities together with the per-image experimental models) is the input to
	the merging program [[cctbx.xfel.merge]].		the merging program [[cctbx.xfel.merge]].

	Because every image is a separate ~~stationary~~ exposure of a different crystal (rather than a wedge of		Because every image is a separate still exposure of a different crystal (rather than a wedge of
	a rotation series), each image is treated as its own experiment with its own crystal model. This		a rotation series), each image is treated as its own experiment with its own crystal model. This
	"one crystal per shot" model is what distinguishes stills processing from conventional rotation data		"one crystal per shot" model is what distinguishes stills processing from conventional rotation data
	processing (e.g. <code>dials.~~process~~</code>), and it drives most of the stills-specific defaults		processing (e.g. <code>dials.import</code>, <code>dials.find_spots</code>, <code>dials.index</code>, etc.),
	described below.		and it drives most of the stills-specific defaults described below.

	== Overview ==		== Overview ==

Aaron: Created page with "= dials.stills_process = '''dials.stills_process''' is the core data-reduction program of cctbx.xfel. It takes a set of still diffraction images — typically serial femtosecond crystallography (SFX) snapshots from an XFEL (LCLS, SACLA, EuXFEL) or still shots from a synchrotron — and reduces each one, independently, through the full DIALS pipeline: '''import → spot-finding → indexing → refinement → integration'''. Its output (integrated intensities together wi..."

2026-08-21T22:22:06Z

Created page with "= dials.stills_process = '''dials.stills_process''' is the core data-reduction program of cctbx.xfel. It takes a set of still diffraction images — typically serial femtosecond crystallography (SFX) snapshots from an XFEL (LCLS, SACLA, EuXFEL) or still shots from a synchrotron — and reduces each one, independently, through the full DIALS pipeline: '''import → spot-finding → indexing → refinement → integration'''. Its output (integrated intensities together wi..."

New page

= dials.stills_process =

'''dials.stills_process''' is the core data-reduction program of cctbx.xfel. It takes a set of
still diffraction images — typically serial femtosecond crystallography (SFX) snapshots from an
XFEL (LCLS, SACLA, EuXFEL) or still shots from a synchrotron — and reduces each one, independently,
through the full DIALS pipeline: '''import → spot-finding → indexing → refinement → integration'''.
Its output (integrated intensities together with the per-image experimental models) is the input to
the merging program [[cctbx.xfel.merge]].

Because every image is a separate stationary exposure of a different crystal (rather than a wedge of
a rotation series), each image is treated as its own experiment with its own crystal model. This
"one crystal per shot" model is what distinguishes stills processing from conventional rotation data
processing (e.g. <code>dials.process</code>), and it drives most of the stills-specific defaults
described below.

== Overview ==

For each image, <code>dials.stills_process</code> runs the following stages in order. Any stage can
be turned off with the <code>dispatch</code> parameters; if a stage is disabled, processing stops
after the last enabled stage.

# '''Import''' — read the image, apply reference geometry / gain / mask overrides, and build an experiment.
# '''Spot-finding''' — locate strong pixels and build strong reflections (<code>dispatch.find_spots</code>).
# '''Hit finding''' — reject images with too few (or optionally too many) strong spots, so that only "hits" are indexed (<code>dispatch.hit_finder</code>).
# '''Indexing''' — determine the crystal orientation and (optionally constrained) unit cell (<code>dispatch.index</code>).
# '''Refinement''' — refine the crystal model and re-estimate the mosaic/domain-size parameters. Detector and beam are held fixed by default (<code>dispatch.refine</code>).
# '''Integration''' — predict reflections and measure their intensities using the stills integrator (<code>dispatch.integrate</code>), with optional Kapton absorption correction and a significance filter.

The program applies a set of stills-specific defaults on top of the standard DIALS parameters,
including: indexing method <code>fft1d</code>; the stills integrator with profile fitting turned off
(summation integration with a 2-D plane background); the stills refinement weighting strategy with
the detector and beam models fixed; and a re-estimation of per-crystal mosaic parameters
(domain size and mosaic angle) after refinement. These are sensible starting points for serial data
and normally do not need to be changed.

Failures on individual images are, by default, caught and logged so that processing continues to the
next image (<code>dispatch.squash_errors=True</code>) — essential when reducing tens of thousands of
shots of which only a fraction are hits.

== Basic usage ==

<pre>
# Process a directory of CBF stills, constraining the target cell/symmetry,
# using 32 parallel processes on one node
dials.stills_process /path/to/images/*.cbf process.phil mp.nproc=32
</pre>

A typical <code>process.phil</code> for XFEL data looks like:

<pre>
spotfinder.filter.min_spot_size=2
spotfinder.threshold.dispersion.gain=1.0
spotfinder.threshold.dispersion.global_threshold=100

indexing.known_symmetry.space_group=P43212
indexing.known_symmetry.unit_cell=78.9,78.9,38.1,90,90,90
indexing.method=fft1d

integration.background.simple.outlier.algorithm=null

input.reference_geometry=refined.expt # from a previous metrology refinement

output.composite_output=True
output.output_dir=out
mp.nproc=32
</pre>

Useful command-line helpers:
* <code>dials.stills_process -h</code> — help and usage
* <code>dials.stills_process -c -e 2</code> — show the configurable parameters (expert level 2)
* <code>dials.stills_process -c -a2</code> — include attributes (help text) for each parameter
* <code>dials.stills_process ... input.show_image_tags=True</code> — list the image tags in a dataset without processing (useful for selecting subsets with <code>input.image_tag</code>)

== Important parameters ==

The program exposes the entire DIALS parameter set (spot-finding, indexing, refinement, integration,
profile modelling, prediction). The parameters below are the ones most commonly adjusted for serial
still data. Parameters are given by their full PHIL path with the default value in parentheses.

=== Input ===

{| class="wikitable"
! Parameter !! Default !! Description
|-
| <code>input.reference_geometry</code> || None || Path to an <code>.expt</code> file (or image) with a single detector model. Data are processed using ''this'' geometry instead of the geometry in the image headers. This is how a refined detector metrology from <code>cctbx.xfel</code> ensemble refinement is applied — almost always set for XFEL data.
|-
| <code>input.sync_reference_geom</code> || True || When a reference geometry is supplied, copy its refined local frames onto the image's detector hierarchy node-by-node (the hierarchies must match in shape). If False, the reference detector replaces the image detector wholesale.
|-
| <code>input.file_list</code> || None || Path to a text file listing image paths, one per line (alternative to listing files on the command line).
|-
| <code>input.glob</code> || None || Wildcard path(s) for large multi-file datasets (e.g. <code>*.cbf</code>). May be given multiple times.
|-
| <code>input.image_tag</code> || None || Process only images with the given tag(s). For single-image files the tag is the filename; for multi-image files (HDF5) it is <code>filename_imagenumber</code>. Use <code>input.show_image_tags=True</code> to list them.
|-
| <code>input.max_images</code> || None || Limit the total number of images processed.
|}

=== Dispatch — controlling the pipeline ===

{| class="wikitable"
! Parameter !! Default !! Description
|-
| <code>dispatch.find_spots</code> || True || Do spot-finding. Required for indexing/integration.
|-
| <code>dispatch.index</code> || True || Attempt to index. Requires <code>find_spots=True</code>.
|-
| <code>dispatch.integrate</code> || True || Integrate indexed images. Ignored if indexing/spot-finding is off.
|-
| <code>dispatch.refine</code> || False || Run an explicit model-refinement step after indexing (the stills indexer already refines; this is an additional pass, rarely needed).
|-
| <code>dispatch.squash_errors</code> || True || If an image fails, log it and continue to the next image. Set False to halt and show the full error when debugging.
|-
| <code>dispatch.hit_finder.enable</code> || True || Enable hit-finding. If disabled, every image is passed to indexing.
|-
| <code>dispatch.hit_finder.minimum_number_of_reflections</code> || 16 || Images with fewer strong spots than this are discarded as non-hits before indexing.
|-
| <code>dispatch.hit_finder.maximum_number_of_reflections</code> || None || If set, images with more strong spots than this are discarded (e.g. ice/junk).
|-
| <code>dispatch.process_percent</code> || None || Process only this percentage of events (evenly sampled). Useful for a quick look at a large dataset.
|-
| <code>dispatch.coset</code> || False || During integration, also integrate a sublattice coset as a negative control for regions with no Bragg diffraction (see <code>integration.coset.transformation</code>).
|-
| <code>dispatch.pre_import</code> || False || Import all data before processing. Only needed when processing multiple ''multi-image'' files at once (not a recommended workflow). Single-file datasets are always pre-imported automatically.
|}

=== Spot-finding ===

Spot-finding uses the standard DIALS dispersion thresholding. The most impactful parameters for
serial data:

{| class="wikitable"
! Parameter !! Default !! Description
|-
| <code>spotfinder.filter.min_spot_size</code> || Auto || Minimum connected pixels for a spot. Auto = 3 for pixel-array detectors (SENSOR_PAD), 6 otherwise. Often lowered to 2 for sparse serial patterns.
|-
| <code>spotfinder.threshold.dispersion.gain</code> || None || Detector gain used in thresholding. Set this ''or'' the detector-model gain, not both (see gain note below).
|-
| <code>spotfinder.threshold.dispersion.sigma_strong</code> || 3 || Number of sigma above background for a pixel to be "strong".
|-
| <code>spotfinder.threshold.dispersion.sigma_background</code> || 6 || Number of sigma for the background/dispersion test.
|-
| <code>spotfinder.threshold.dispersion.global_threshold</code> || 0 || Absolute pixel intensity floor; pixels below this are never considered strong. Raising it (e.g. 100) suppresses noise on many detectors.
|-
| <code>spotfinder.threshold.dispersion.kernel_size</code> || 3 3 || Local-area kernel (pixels) for the dispersion calculation.
|}

'''Gain note:''' the detector gain should be set on the detector model ''or'' in the processing
parameters, never both. If a supplied gain (<code>spotfinder.threshold.dispersion.gain</code> or
<code>integration.summation.detector_gain</code>) conflicts with a non-unity gain already on the
detector model, the program raises an error. Override with
<code>input.ignore_gain_mismatch=True</code> only if you understand the consequences.

=== Indexing ===

{| class="wikitable"
! Parameter !! Default !! Description
|-
| <code>indexing.known_symmetry.space_group</code> || None || Target space group. Strongly recommended for serial data — constrains indexing and improves the hit/index rate.
|-
| <code>indexing.known_symmetry.unit_cell</code> || None || Target unit cell (a,b,c,α,β,γ). Used together with the space group as the indexing target.
|-
| <code>indexing.method</code> || fft1d || Basis-vector search method (<code>dials.stills_process</code> defaults to <code>fft1d</code>; <code>fft3d</code> and <code>real_space_grid_search</code> are also available).
|-
| <code>indexing.stills.method_list</code> || None || Ordered list of indexing methods to try in turn; if one fails, the next is attempted (e.g. <code>fft1d real_space_grid_search</code>).
|-
| <code>indexing.stills.known_orientations</code> || None || Paths to previously-processed experiment lists. If given, images are ''not'' re-indexed; the stored crystal orientations are reused (e.g. to re-integrate with new parameters). Supports wildcards; may be repeated.
|-
| <code>indexing.stills.require_known_orientation</code> || False || When known orientations are supplied but an image is not found among them, controls whether to skip it (True) or fall back to indexing from scratch (False).
|-
| <code>indexing.stills.reflection_subsampling.enable</code> || False || Retry indexing on random subsets of the strong spots (from <code>step_start</code>% down to <code>step_stop</code>% in <code>step_size</code>% steps). Can rescue images that fail to index on the full spot list.
|}

=== Integration ===

{| class="wikitable"
! Parameter !! Default !! Description
|-
| <code>profile.gaussian_rs.parameters.sigma_b_cutoff</code> || 0.1 || Maximum beam divergence (sigma_b, degrees) before an indexed crystal is rejected prior to integration. Guards against over-broad, poorly-modelled lattices.
|-
| <code>significance_filter.enable</code> || False || Apply the stills significance filter, which trims each crystal to the resolution at which reflections remain statistically significant (see <code>significance_filter.isigi_cutoff</code>, default 1.0).
|-
| <code>integration.integration_only_overrides.trusted_range</code> || None || Override the panel trusted range [min, max] during integration only (e.g. to include pixels the header marks as untrusted).
|-
| <code>integration.absorption_correction</code> || apply=False || Enable a Kapton absorption correction (<code>algorithm=fuller_kapton</code> or <code>kapton_2019</code>). Requires shoeboxes to be retained via <code>integration.debug</code> settings; the program checks this and errors otherwise.
|-
| <code>integration.coset.transformation</code> || 6 || Which modulus-2 sublattice transformation defines the coset negative control when <code>dispatch.coset=True</code> (0–6; 6 = body centering). See Sauter & Zwart, ''Acta Cryst.'' D (2009) '''65''':553.
|}

=== Output ===

{| class="wikitable"
! Parameter !! Default !! Description
|-
| <code>output.output_dir</code> || <code>.</code> || Directory for output files.
|-
| <code>output.composite_output</code> || True || If True, each process writes ''one'' concatenated set of <code>.expt</code>/<code>.refl</code> files covering all its successful images. If False, a separate pair of files is written per image (generates a very large number of files). Composite output is strongly preferred for large serial datasets.
|-
| <code>output.logging_dir</code> || None || Directory for log files. Under MPI, per-rank logs (<code>log_rankNNNN.out</code>, etc.) are written here.
|-
| <code>output.logging_option</code> || normal || <code>normal</code> (all logging), <code>suppressed</code> (silences verbose DIALS refine/index output), or <code>disabled</code> (almost no logging).
|-
| <code>output.*_filename</code> || (templates) || Output filename templates for each stage: <code>strong</code>, <code>indexed</code>, <code>refined_experiments</code>, <code>integrated</code>, <code>integrated_experiments</code>, <code>coset</code>. A <code>%s</code> in the template is replaced with the per-image tag.
|-
| <code>output.integration_pickle</code> || None || Write legacy <code>cxi.merge</code>/PRIME integration pickle files (e.g. <code>int-%d-%s.pickle</code>). Only needed for legacy merging workflows.
|}

=== Multiprocessing ===

{| class="wikitable"
! Parameter !! Default !! Description
|-
| <code>mp.method</code> || multiprocessing || Parallelization backend: <code>multiprocessing</code> (single node), or <code>mpi</code> for multi-node jobs (also <code>sge</code>, <code>lsf</code>, <code>pbs</code>).
|-
| <code>mp.nproc</code> || 1 || Number of processes (for the <code>multiprocessing</code> method). Set to the number of cores per node.
|-
| <code>mp.composite_stride</code> || None || For MPI composite output, how many ranks aggregate their results into one set of output files. Ideally set to the number of processes per node so that one composite file set is written per node rather than per rank.
|}

Under MPI with more than two ranks, the program runs a client/server model: rank 0 (or rank 2 for
psana2 <code>.loc</code> inputs) hands out images to worker ranks on demand, giving good load
balancing across the many fast-failing non-hits typical of serial data.

== Output files ==

With the default composite output, each process (or each aggregating rank under MPI) writes
concatenated files such as:

* <code><tag>_strong.refl</code> — strong spots from spot-finding
* <code><tag>_indexed.refl</code> and <code><tag>_refined.expt</code> — indexed reflections and their crystal models
* <code><tag>_integrated.refl</code> and <code><tag>_integrated.expt</code> — integrated intensities and final models (the input to [[cctbx.xfel.merge]])

Per-image progress and status codes are recorded in <code>debug/debug_<rank>.txt</code>, and
overall logging goes to <code>dials.process.log</code> (or per-rank logs under MPI). These integrated
<code>.expt</code>/<code>.refl</code> pairs are then scaled and merged with
[[cctbx.xfel.merge]].

== See also ==

* [[cctbx.xfel.merge]] — scaling, post-refinement and merging of the integrated data
* [[IOTA]] — a front-end that supplies preset parameters for <code>dials.stills_process</code>
* [[Data Processing at SACLA]] — SACLA-specific processing workflow
* The DIALS documentation for the underlying spot-finding, indexing, refinement and integration algorithms

← Older revision		Revision as of 22:51, 21 August 2026
Line 74:		Line 74:
	The program exposes the entire DIALS parameter set (spot-finding, indexing, refinement, integration,		The program exposes the entire DIALS parameter set (spot-finding, indexing, refinement, integration,
	profile modelling, prediction). The parameters below are the ones most commonly adjusted for serial		profile modelling, prediction). The parameters below are the ones most commonly adjusted for serial
	still data. Parameters are given by their full PHIL path with ~~the~~ default value ~~in parentheses~~.		still data. Parameters are given by their full PHIL path with their default value.

	=== Input ===		=== Input ===
Line 89:		Line 89:
	\| <code>input.glob</code> \|\| None \|\| Wildcard path(s) for large multi-file datasets (e.g. <code>*.cbf</code>). May be given multiple times.		\| <code>input.glob</code> \|\| None \|\| Wildcard path(s) for large multi-file datasets (e.g. <code>*.cbf</code>). May be given multiple times.
	\|-		\|-
	\| <code>input.image_tag</code> \|\| None \|\| Process only images with the given tag(s). For single-image files the tag is the filename; for multi-image files (HDF5) it is <code>filename_imagenumber</code>. Use <code>input.show_image_tags=True</code> to list them.		\| <code>input.image_tag</code> \|\| None \|\| Process only images with the given tag(s). For single-image files the tag is the filename; for multi-image files (HDF5) it is <code>filename_imagenumber</code> (image number is 5 digits with leading zeroes). Use <code>input.show_image_tags=True</code> to list them.
	\|-		\|-
	\| <code>input.max_images</code> \|\| None \|\| Limit the total number of images processed.		\| <code>input.max_images</code> \|\| None \|\| Limit the total number of images processed.
Line 119:		Line 119:
	\| <code>dispatch.coset</code> \|\| False \|\| During integration, also integrate a sublattice coset as a negative control for regions with no Bragg diffraction (see <code>integration.coset.transformation</code>).		\| <code>dispatch.coset</code> \|\| False \|\| During integration, also integrate a sublattice coset as a negative control for regions with no Bragg diffraction (see <code>integration.coset.transformation</code>).
	\|-		\|-
	\| <code>dispatch.pre_import</code> \|\| False \|\| Import all data before processing. Only needed when processing multiple ''multi-image'' files at once (not a recommended workflow). Single-file datasets are always pre-imported automatically.		\| <code>dispatch.pre_import</code> \|\| False \|\| Import all data before processing. Only needed when processing multiple ''multi-image'' files at once (not a recommended workflow). Single-file datasets (e.g. HDF5) are always pre-imported automatically.
	\|}		\|}

Line 166:		Line 166:
	\| <code>indexing.stills.require_known_orientation</code> \|\| False \|\| When known orientations are supplied but an image is not found among them, controls whether to skip it (True) or fall back to indexing from scratch (False).		\| <code>indexing.stills.require_known_orientation</code> \|\| False \|\| When known orientations are supplied but an image is not found among them, controls whether to skip it (True) or fall back to indexing from scratch (False).
	\|-		\|-
	\| <code>indexing.stills.reflection_subsampling.enable</code> \|\| False \|\| Retry indexing on random subsets of the strong spots (from <code>step_start</code>% down to <code>step_stop</code>% in <code>step_size</code>% steps). Can rescue images that fail to index on the full spot list.		\| <code>indexing.stills.reflection_subsampling.enable</code> \|\| False \|\| Retry indexing on random subsets of the strong spots (from <code>step_start</code>% down to <code>step_stop</code>% in <code>step_size</code>% steps). Can rescue images that fail to index on the full spot list. Especially valuble for multi-lattice images (multiple crystals in the beam) but does increase execution time!
	\|}		\|}

Line 176:		Line 176:
	\| <code>profile.gaussian_rs.parameters.sigma_b_cutoff</code> \|\| 0.1 \|\| Maximum beam divergence (sigma_b, degrees) before an indexed crystal is rejected prior to integration. Guards against over-broad, poorly-modelled lattices.		\| <code>profile.gaussian_rs.parameters.sigma_b_cutoff</code> \|\| 0.1 \|\| Maximum beam divergence (sigma_b, degrees) before an indexed crystal is rejected prior to integration. Guards against over-broad, poorly-modelled lattices.
	\|-		\|-
	\| <code>significance_filter.enable</code> \|\| False \|\| Apply the stills significance filter, which trims each crystal to the resolution at which reflections remain statistically significant (see <code>significance_filter.isigi_cutoff</code>, default 1.0).		\| <code>significance_filter.enable</code> \|\| False \|\| Apply the stills significance filter, which trims each crystal to the resolution at which reflections remain statistically significant (see <code>significance_filter.isigi_cutoff</code>, default 1.0). Defaults to false here because it is typically applied in ''[[cctbx.xfel.merge]]''.
	\|-		\|-
	\| <code>integration.integration_only_overrides.trusted_range</code> \|\| None \|\| Override the panel trusted range [min, max] during integration only (e.g. to include pixels the header marks as untrusted).		\| <code>integration.integration_only_overrides.trusted_range</code> \|\| None \|\| Override the panel trusted range [min, max] during integration only (e.g. to include pixels the header marks as untrusted).
	\|-
	\| <code>integration.absorption_correction</code> \|\| apply=False \|\| Enable a Kapton absorption correction (<code>algorithm=fuller_kapton</code> or <code>kapton_2019</code>). Requires shoeboxes to be retained via <code>integration.debug</code> settings; the program checks this and errors otherwise.
	\|-		\|-
	\| <code>integration.coset.transformation</code> \|\| 6 \|\| Which modulus-2 sublattice transformation defines the coset negative control when <code>dispatch.coset=True</code> (0–6; 6 = body centering). See Sauter & Zwart, ''Acta Cryst.'' D (2009) '''65''':553.		\| <code>integration.coset.transformation</code> \|\| 6 \|\| Which modulus-2 sublattice transformation defines the coset negative control when <code>dispatch.coset=True</code> (0–6; 6 = body centering). See Sauter & Zwart, ''Acta Cryst.'' D (2009) '''65''':553.
Line 198:		Line 196:
	\| <code>output.logging_option</code> \|\| normal \|\| <code>normal</code> (all logging), <code>suppressed</code> (silences verbose DIALS refine/index output), or <code>disabled</code> (almost no logging).		\| <code>output.logging_option</code> \|\| normal \|\| <code>normal</code> (all logging), <code>suppressed</code> (silences verbose DIALS refine/index output), or <code>disabled</code> (almost no logging).
	\|-		\|-
	\| <code>output.*_filename</code> \|\| (templates) \|\| Output filename templates for each stage: <code>strong</code>, <code>indexed</code>, <code>refined_experiments</code>, <code>integrated</code>, <code>integrated_experiments</code>, <code>coset</code>. A <code>%s</code> in the template is replaced with the per-image tag.		\| <code>output.*_filename</code> \|\| (templates) \|\| Output filename templates for each stage: <code>strong</code>, <code>indexed</code>, <code>refined_experiments</code>, <code>integrated</code>, <code>integrated_experiments</code>, <code>coset</code>. A <code>%s</code> in the template is replaced with the process number for composite mode or the per-image tag for non-composite mode.
	\|-		\|-
	\| <code>output.integration_pickle</code> \|\| None \|\| Write legacy <code>cxi.merge</code>/PRIME integration pickle files (e.g. <code>int-%d-%s.pickle</code>). Only needed for legacy merging workflows.		\| <code>output.integration_pickle</code> \|\| None \|\| Write legacy <code>cxi.merge</code>/PRIME integration pickle files (e.g. <code>int-%d-%s.pickle</code>). Only needed for legacy merging workflows.