Latest revision as of 23:40, 21 August 2026

dials.stills_process is a stand-alone program for processing still diffraction images and is the core data-reduction program of cctbx.xfel. It takes a set of still diffraction images — typically serial femtosecond crystallography (SFX) snapshots from an XFEL (LCLS, SACLA, EuXFEL) or still shots from a synchrotron — and analyzes each one, independently, through the full DIALS pipeline: import → spot-finding → indexing → refinement → integration. Its output (integrated intensities together with the per-image experimental models) is the input to the merging program cctbx.xfel.merge.

Because every image is a separate still exposure of a different crystal (rather than a wedge of a rotation series), each image is treated as its own experiment with its own crystal model. This "one crystal per shot" model is what distinguishes stills processing from conventional rotation data processing (e.g. dials.import, dials.find_spots, dials.index, etc.), and it drives most of the stills-specific defaults described below.

Overview

For each image, dials.stills_process runs the following stages in order. Any stage can be turned off with the dispatch parameters; if a stage is disabled, processing stops after the last enabled stage.

Import — read the image, apply reference geometry / gain / mask overrides, and build an experiment.
Spot-finding — locate strong pixels and build strong reflections (dispatch.find_spots).
Hit finding — reject images with too few (or optionally too many) strong spots, so that only "hits" are indexed (dispatch.hit_finder).
Indexing — determine the crystal orientation and (optionally constrained) unit cell (dispatch.index).
Refinement — refine the crystal model and re-estimate the mosaic/domain-size parameters. Detector and beam are held fixed by default (dispatch.refine).
Integration — predict reflections and measure their intensities using the stills integrator (dispatch.integrate), with an optional significance filter.

The program applies a set of stills-specific defaults on top of the standard DIALS parameters, including: indexing method fft1d; the stills integrator with profile fitting turned off (summation integration with a 2-D plane background); the stills refinement weighting strategy with the detector and beam models fixed; and a re-estimation of per-crystal mosaic parameters (domain size and mosaic angle) after refinement. These are sensible starting points for serial data and normally do not need to be changed.

Failures on individual images are, by default, caught and logged so that processing continues to the next image (dispatch.squash_errors=True) — essential when reducing tens of thousands of shots of which only a fraction are hits.

Basic usage

# Process a directory of CBF stills, constraining the target cell/symmetry,
# using 32 parallel processes on one node
dials.stills_process /path/to/images/*.cbf process.phil mp.nproc=32

A typical process.phil for XFEL data looks like:

spotfinder.filter.min_spot_size=2
spotfinder.threshold.dispersion.gain=1.0
spotfinder.threshold.dispersion.global_threshold=100

indexing.known_symmetry.space_group=P43212
indexing.known_symmetry.unit_cell=78.9,78.9,38.1,90,90,90
indexing.method=fft1d

input.reference_geometry=refined.expt   # from a previous metrology refinement

output.output_dir=out
mp.nproc=32

Useful command-line helpers:

dials.stills_process -h — help and usage
dials.stills_process -c -e 2 — show the configurable parameters (expert level 2)
dials.stills_process -c -a2 — include attributes (help text) for each parameter
dials.stills_process ... input.show_image_tags=True — list the image tags in a dataset without processing (useful for selecting subsets with input.image_tag)

Important parameters

The program exposes the entire DIALS parameter set (spot-finding, indexing, refinement, integration, profile modelling, prediction). The parameters below are the ones most commonly adjusted for serial still data. Parameters are given by their full PHIL path with their default value.

Input

Parameter	Default	Description
`input.reference_geometry`	None	Path to an `.expt` file (or image) with a single detector model. Data are processed using this geometry instead of the geometry in the image headers. This is how a refined detector metrology from `cctbx.xfel` ensemble refinement is applied — almost always set for XFEL data.
`input.sync_reference_geom`	True	When a reference geometry is supplied, copy its refined local frames onto the image's detector hierarchy node-by-node (the hierarchies must match in shape). If False, the reference detector replaces the image detector wholesale.
`input.file_list`	None	Path to a text file listing image paths, one per line (alternative to listing files on the command line).
`input.glob`	None	Wildcard path(s) for large multi-file datasets (e.g. `*.cbf`). May be given multiple times.
`input.image_tag`	None	Process only images with the given tag(s). For single-image files the tag is the filename; for multi-image files (HDF5) it is `filename_imagenumber` (image number is 5 digits with leading zeroes). Use `input.show_image_tags=True` to list them.
`input.max_images`	None	Limit the total number of images processed.

Dispatch — controlling the pipeline

Parameter	Default	Description
`dispatch.find_spots`	True	Do spot-finding. Required for indexing/integration.
`dispatch.index`	True	Attempt to index. Requires `find_spots=True`.
`dispatch.integrate`	True	Integrate indexed images. Ignored if indexing/spot-finding is off.
`dispatch.refine`	False	Run an explicit model-refinement step after indexing (the stills indexer already refines; this is an additional pass, rarely needed).
`dispatch.squash_errors`	True	If an image fails, log it and continue to the next image. Set False to halt and show the full error when debugging.
`dispatch.hit_finder.enable`	True	Enable hit-finding. If disabled, every image is passed to indexing.
`dispatch.hit_finder.minimum_number_of_reflections`	16	Images with fewer strong spots than this are discarded as non-hits before indexing.
`dispatch.hit_finder.maximum_number_of_reflections`	None	If set, images with more strong spots than this are discarded (e.g. ice/junk).
`dispatch.process_percent`	None	Process only this percentage of events (evenly sampled). Useful for a quick look at a large dataset.
`dispatch.coset`	False	During integration, also integrate a sublattice coset as a negative control for regions with no Bragg diffraction (see `integration.coset.transformation`).
`dispatch.pre_import`	False	Import all data before processing. Only needed when processing multiple multi-image files at once (not a recommended workflow). Single-file datasets (e.g. HDF5) are always pre-imported automatically.

Spot-finding

Spot-finding uses the standard DIALS dispersion thresholding. The most impactful parameters for serial data:

Parameter	Default	Description
`spotfinder.filter.min_spot_size`	Auto	Minimum connected pixels for a spot. Auto = 3 for pixel-array detectors (SENSOR_PAD), 6 otherwise. Often lowered to 2 for sparse serial patterns.
`spotfinder.threshold.dispersion.gain`	None	Detector gain used in thresholding. Set this or the detector-model gain, not both (see gain note below).
`spotfinder.threshold.dispersion.sigma_strong`	3	Number of sigma above background for a pixel to be "strong".
`spotfinder.threshold.dispersion.sigma_background`	6	Number of sigma for the background/dispersion test.
`spotfinder.threshold.dispersion.global_threshold`	0	Absolute pixel intensity floor; pixels below this are never considered strong. Raising it (e.g. 100) suppresses noise on many detectors.
`spotfinder.threshold.dispersion.kernel_size`	3 3	Local-area kernel (pixels) for the dispersion calculation.

Gain note: the detector gain should be set on the detector model or in the processing parameters, never both. If a supplied gain (spotfinder.threshold.dispersion.gain or integration.summation.detector_gain) conflicts with a non-unity gain already on the detector model, the program raises an error. Override with input.ignore_gain_mismatch=True only if you understand the consequences.

Indexing

Parameter	Default	Description
`indexing.known_symmetry.space_group`	None	Target space group. Strongly recommended for serial data — constrains indexing and improves the hit/index rate.
`indexing.known_symmetry.unit_cell`	None	Target unit cell (a,b,c,α,β,γ). Used together with the space group as the indexing target.
`indexing.method`	fft1d	Basis-vector search method (`dials.stills_process` defaults to `fft1d`; `fft3d` and `real_space_grid_search` are also available).
`indexing.stills.method_list`	None	Ordered list of indexing methods to try in turn; if one fails, the next is attempted (e.g. `fft1d real_space_grid_search`).
`indexing.stills.known_orientations`	None	Paths to previously-processed experiment lists. If given, images are not re-indexed; the stored crystal orientations are reused (e.g. to re-integrate with new parameters). Supports wildcards; may be repeated.
`indexing.stills.require_known_orientation`	False	When known orientations are supplied but an image is not found among them, controls whether to skip it (True) or fall back to indexing from scratch (False).
`indexing.stills.reflection_subsampling.enable`	False	Retry indexing on random subsets of the strong spots (from `step_start`% down to `step_stop`% in `step_size`% steps). Can rescue images that fail to index on the full spot list. Especially valuble for multi-lattice images (multiple crystals in the beam) but does increase execution time!

Integration

Parameter	Default	Description
`profile.gaussian_rs.parameters.sigma_b_cutoff`	0.1	Maximum beam divergence (sigma_b, degrees) before an indexed crystal is rejected prior to integration. Guards against over-broad, poorly-modelled lattices.
`significance_filter.enable`	False	Apply the stills significance filter, which trims each crystal to the resolution at which reflections remain statistically significant (see `significance_filter.isigi_cutoff`, default 1.0). Defaults to false here because it is typically applied in cctbx.xfel.merge.
`integration.integration_only_overrides.trusted_range`	None	Override the panel trusted range [min, max] during integration only (e.g. to include pixels the header marks as untrusted).
`integration.coset.transformation`	6	Which modulus-2 sublattice transformation defines the coset negative control when `dispatch.coset=True` (0–6; 6 = body centering). See Sauter & Zwart, Acta Cryst. D (2009) 65:553.

Output

Parameter	Default	Description
`output.output_dir`	`.`	Directory for output files.
`output.composite_output`	True	If True, each process writes one concatenated set of `.expt`/`.refl` files covering all its successful images. If False, a separate pair of files is written per image (generates a very large number of files). Composite output is strongly preferred for large serial datasets.
`output.logging_dir`	None	Directory for log files. Under MPI, per-rank logs (`log_rankNNNN.out`, etc.) are written here.
`output.logging_option`	normal	`normal` (all logging), `suppressed` (silences verbose DIALS refine/index output), or `disabled` (almost no logging).
`output.*_filename`	(templates)	Output filename templates for each stage: `strong`, `indexed`, `refined_experiments`, `integrated`, `integrated_experiments`, `coset`. A `%s` in the template is replaced with the process number for composite mode or the per-image tag for non-composite mode.
`output.integration_pickle`	None	Write legacy `cxi.merge`/PRIME integration pickle files (e.g. `int-%d-%s.pickle`). Only needed for legacy merging workflows.

Multiprocessing

Parameter	Default	Description
`mp.method`	multiprocessing	Parallelization backend: `multiprocessing` (single node), or `mpi` for multi-node jobs (also `sge`, `lsf`, `pbs`).
`mp.nproc`	1	Number of processes (for the `multiprocessing` method). Set to the number of cores per node.
`mp.composite_stride`	None	For MPI composite output, how many ranks aggregate their results into one set of output files. Ideally set to the number of processes per node so that one composite file set is written per node rather than per rank.

Under MPI with more than two ranks, the program runs a client/server model: rank 0 (or rank 2 for psana2 .loc inputs) hands out images to worker ranks on demand, giving good load balancing across the many fast-failing non-hits typical of serial data.

Output files

With the default composite output, each process (or each aggregating rank under MPI) writes concatenated files such as:

<tag>_strong.refl — strong spots from spot-finding
<tag>_indexed.refl and <tag>_refined.expt — indexed reflections and their crystal models
<tag>_integrated.refl and <tag>_integrated.expt — integrated intensities and final models (the input to cctbx.xfel.merge)

Per-image progress and status codes are recorded in debug/debug_<rank>.txt, and overall logging goes to dials.process.log (or per-rank logs under MPI). These integrated .expt/.refl pairs are then scaled and merged with cctbx.xfel.merge.

@@ Line 1: / Line 1: @@
-= dials.stills_process =
+{{DISPLAYTITLE:''dials.stills_process''}}
-'''dials.stills_process''' is the core data-reduction program of cctbx.xfel. It takes a set of
+'''''dials.stills_process''''' is a stand-alone program for processing still diffraction images and is
+the core data-reduction program of ''cctbx.xfel''. It takes a set of
 still diffraction images — typically serial femtosecond crystallography (SFX) snapshots from an
-XFEL (LCLS, SACLA, EuXFEL) or still shots from a synchrotron — and reduces each one, independently,
+XFEL (LCLS, SACLA, EuXFEL) or still shots from a synchrotron — and analyzes each one, independently,
 through the full DIALS pipeline: '''import → spot-finding → indexing → refinement → integration'''.
 Its output (integrated intensities together with the per-image experimental models) is the input to
-the merging program [[cctbx.xfel.merge]].
+the merging program ''[[cctbx.xfel.merge]]''.
-Because every image is a separate stationary exposure of a different crystal (rather than a wedge of
+Because every image is a separate still exposure of a different crystal (rather than a wedge of
 a rotation series), each image is treated as its own experiment with its own crystal model. This
 "one crystal per shot" model is what distinguishes stills processing from conventional rotation data
-processing (e.g. <code>dials.process</code>), and it drives most of the stills-specific defaults
+processing (e.g. <code>dials.import</code>, <code>dials.find_spots</code>, <code>dials.index</code>, etc.),
-described below.
+and it drives most of the stills-specific defaults described below.
-== Overview ==
+= Overview =
 For each image, <code>dials.stills_process</code> runs the following stages in order. Any stage can
@@ Line 25: / Line 26: @@
 # '''Indexing''' — determine the crystal orientation and (optionally constrained) unit cell (<code>dispatch.index</code>).
 # '''Refinement''' — refine the crystal model and re-estimate the mosaic/domain-size parameters. Detector and beam are held fixed by default (<code>dispatch.refine</code>).
-# '''Integration''' — predict reflections and measure their intensities using the stills integrator (<code>dispatch.integrate</code>), with optional Kapton absorption correction and a significance filter.
+# '''Integration''' — predict reflections and measure their intensities using the stills integrator (<code>dispatch.integrate</code>), with an optional significance filter.
 The program applies a set of stills-specific defaults on top of the standard DIALS parameters,
@@ Line 38: / Line 39: @@
 shots of which only a fraction are hits.
-== Basic usage ==
+= Basic usage =
 <pre>
@@ Line 56: / Line 57: @@
 indexing.known_symmetry.unit_cell=78.9,78.9,38.1,90,90,90
 indexing.method=fft1d
-integration.background.simple.outlier.algorithm=null
 input.reference_geometry=refined.expt   # from a previous metrology refinement
-output.composite_output=True
 output.output_dir=out
 mp.nproc=32
@@ Line 72: / Line 70: @@
 * <code>dials.stills_process ... input.show_image_tags=True</code> — list the image tags in a dataset without processing (useful for selecting subsets with <code>input.image_tag</code>)
-== Important parameters ==
+= Important parameters =
 The program exposes the entire DIALS parameter set (spot-finding, indexing, refinement, integration,
 profile modelling, prediction). The parameters below are the ones most commonly adjusted for serial
-still data. Parameters are given by their full PHIL path with the default value in parentheses.
+still data. Parameters are given by their full PHIL path with their default value.
-=== Input ===
+== Input ==
 {| class="wikitable"
@@ Line 91: / Line 89: @@
 | <code>input.glob</code> || None || Wildcard path(s) for large multi-file datasets (e.g. <code>*.cbf</code>). May be given multiple times.
 |-
-| <code>input.image_tag</code> || None || Process only images with the given tag(s). For single-image files the tag is the filename; for multi-image files (HDF5) it is <code>filename_imagenumber</code>. Use <code>input.show_image_tags=True</code> to list them.
+| <code>input.image_tag</code> || None || Process only images with the given tag(s). For single-image files the tag is the filename; for multi-image files (HDF5) it is <code>filename_imagenumber</code> (image number is 5 digits with leading zeroes). Use <code>input.show_image_tags=True</code> to list them.
 |-
 | <code>input.max_images</code> || None || Limit the total number of images processed.
 |}
-=== Dispatch — controlling the pipeline ===
+== Dispatch — controlling the pipeline ==
 {| class="wikitable"
@@ Line 121: / Line 119: @@
 | <code>dispatch.coset</code> || False || During integration, also integrate a sublattice coset as a negative control for regions with no Bragg diffraction (see <code>integration.coset.transformation</code>).
 |-
-| <code>dispatch.pre_import</code> || False || Import all data before processing. Only needed when processing multiple ''multi-image'' files at once (not a recommended workflow). Single-file datasets are always pre-imported automatically.
+| <code>dispatch.pre_import</code> || False || Import all data before processing. Only needed when processing multiple ''multi-image'' files at once (not a recommended workflow). Single-file datasets (e.g. HDF5) are always pre-imported automatically.
 |}
-=== Spot-finding ===
+== Spot-finding ==
 Spot-finding uses the standard DIALS dispersion thresholding. The most impactful parameters for
@@ Line 151: / Line 149: @@
 <code>input.ignore_gain_mismatch=True</code> only if you understand the consequences.
-=== Indexing ===
+== Indexing ==
 {| class="wikitable"
@@ Line 168: / Line 166: @@
 | <code>indexing.stills.require_known_orientation</code> || False || When known orientations are supplied but an image is not found among them, controls whether to skip it (True) or fall back to indexing from scratch (False).
 |-
-| <code>indexing.stills.reflection_subsampling.enable</code> || False || Retry indexing on random subsets of the strong spots (from <code>step_start</code>% down to <code>step_stop</code>% in <code>step_size</code>% steps). Can rescue images that fail to index on the full spot list.
+| <code>indexing.stills.reflection_subsampling.enable</code> || False || Retry indexing on random subsets of the strong spots (from <code>step_start</code>% down to <code>step_stop</code>% in <code>step_size</code>% steps). Can rescue images that fail to index on the full spot list.  Especially valuble for multi-lattice images (multiple crystals in the beam) but does increase execution time!
 |}
-=== Integration ===
+== Integration ==
 {| class="wikitable"
@@ Line 178: / Line 176: @@
 | <code>profile.gaussian_rs.parameters.sigma_b_cutoff</code> || 0.1 || Maximum beam divergence (sigma_b, degrees) before an indexed crystal is rejected prior to integration. Guards against over-broad, poorly-modelled lattices.
 |-
-| <code>significance_filter.enable</code> || False || Apply the stills significance filter, which trims each crystal to the resolution at which reflections remain statistically significant (see <code>significance_filter.isigi_cutoff</code>, default 1.0).
+| <code>significance_filter.enable</code> || False || Apply the stills significance filter, which trims each crystal to the resolution at which reflections remain statistically significant (see <code>significance_filter.isigi_cutoff</code>, default 1.0).  Defaults to false here because it is typically applied in ''[[cctbx.xfel.merge]]''.
 |-
 | <code>integration.integration_only_overrides.trusted_range</code> || None || Override the panel trusted range [min, max] during integration only (e.g. to include pixels the header marks as untrusted).
-|-
-| <code>integration.absorption_correction</code> || apply=False || Enable a Kapton absorption correction (<code>algorithm=fuller_kapton</code> or <code>kapton_2019</code>). Requires shoeboxes to be retained via <code>integration.debug</code> settings; the program checks this and errors otherwise.
 |-
 | <code>integration.coset.transformation</code> || 6 || Which modulus-2 sublattice transformation defines the coset negative control when <code>dispatch.coset=True</code> (0–6; 6 = body centering). See Sauter & Zwart, ''Acta Cryst.'' D (2009) '''65''':553.
 |}
-=== Output ===
+== Output ==
 {| class="wikitable"
@@ Line 200: / Line 196: @@
 | <code>output.logging_option</code> || normal || <code>normal</code> (all logging), <code>suppressed</code> (silences verbose DIALS refine/index output), or <code>disabled</code> (almost no logging).
 |-
-| <code>output.*_filename</code> || (templates) || Output filename templates for each stage: <code>strong</code>, <code>indexed</code>, <code>refined_experiments</code>, <code>integrated</code>, <code>integrated_experiments</code>, <code>coset</code>. A <code>%s</code> in the template is replaced with the per-image tag.
+| <code>output.*_filename</code> || (templates) || Output filename templates for each stage: <code>strong</code>, <code>indexed</code>, <code>refined_experiments</code>, <code>integrated</code>, <code>integrated_experiments</code>, <code>coset</code>. A <code>%s</code> in the template is replaced with the process number for composite mode or the per-image tag for non-composite mode.
 |-
 | <code>output.integration_pickle</code> || None || Write legacy <code>cxi.merge</code>/PRIME integration pickle files (e.g. <code>int-%d-%s.pickle</code>). Only needed for legacy merging workflows.
 |}
-=== Multiprocessing ===
+== Multiprocessing ==
 {| class="wikitable"
@@ Line 221: / Line 217: @@
 balancing across the many fast-failing non-hits typical of serial data.
-== Output files ==
+= Output files =
 With the default composite output, each process (or each aggregating rank under MPI) writes
@@ Line 228: / Line 224: @@
 * <code>&lt;tag&gt;_strong.refl</code> — strong spots from spot-finding
 * <code>&lt;tag&gt;_indexed.refl</code> and <code>&lt;tag&gt;_refined.expt</code> — indexed reflections and their crystal models
-* <code>&lt;tag&gt;_integrated.refl</code> and <code>&lt;tag&gt;_integrated.expt</code> — integrated intensities and final models (the input to [[cctbx.xfel.merge]])
+* <code>&lt;tag&gt;_integrated.refl</code> and <code>&lt;tag&gt;_integrated.expt</code> — integrated intensities and final models (the input to ''[[cctbx.xfel.merge]]'')
 Per-image progress and status codes are recorded in <code>debug/debug_&lt;rank&gt;.txt</code>, and
 overall logging goes to <code>dials.process.log</code> (or per-rank logs under MPI). These integrated
 <code>.expt</code>/<code>.refl</code> pairs are then scaled and merged with
-[[cctbx.xfel.merge]].
+''[[cctbx.xfel.merge]]''.
-== See also ==
+= See also =
-* [[cctbx.xfel.merge]] — scaling, post-refinement and merging of the integrated data
+* ''[[cctbx.xfel.merge]]'' — scaling, post-refinement and merging of the integrated data
-* [[IOTA]] — a front-end that supplies preset parameters for <code>dials.stills_process</code>
+* The [http://dials.github.io DIALS documentation] for the underlying spot-finding, indexing, refinement and integration algorithms
-* [[Data Processing at SACLA]] — SACLA-specific processing workflow
-* The DIALS documentation for the underlying spot-finding, indexing, refinement and integration algorithms

Dials.stills process: Difference between revisions

Latest revision as of 23:40, 21 August 2026

Contents

Overview

Basic usage

Important parameters

Input

Dispatch — controlling the pipeline

Spot-finding

Indexing

Integration

Output

Multiprocessing

Output files

See also

Navigation menu

Dials.stills process: Difference between revisions

Latest revision as of 23:40, 21 August 2026

Overview

Basic usage

Important parameters

Input

Dispatch — controlling the pipeline

Spot-finding

Indexing

Integration

Output

Multiprocessing

Output files

See also

Navigation menu

Search