Dials.stills process

dials.stills_process

dials.stills_process is a stand-alone program for processing still diffraction images and is the core data-reduction program of cctbx.xfel. It takes a set of still diffraction images — typically serial femtosecond crystallography (SFX) snapshots from an XFEL (LCLS, SACLA, EuXFEL) or still shots from a synchrotron — and analyzes each one, independently, through the full DIALS pipeline: import → spot-finding → indexing → refinement → integration. Its output (integrated intensities together with the per-image experimental models) is the input to the merging program cctbx.xfel.merge.

Because every image is a separate still exposure of a different crystal (rather than a wedge of a rotation series), each image is treated as its own experiment with its own crystal model. This "one crystal per shot" model is what distinguishes stills processing from conventional rotation data processing (e.g. dials.import, dials.find_spots, dials.index, etc.), and it drives most of the stills-specific defaults described below.

Overview

For each image, dials.stills_process runs the following stages in order. Any stage can be turned off with the dispatch parameters; if a stage is disabled, processing stops after the last enabled stage.

1. Import — read the image, apply reference geometry / gain / mask overrides, and build an experiment.
2. Spot-finding — locate strong pixels and build strong reflections (dispatch.find_spots).
3. Hit finding — reject images with too few (or optionally too many) strong spots, so that only "hits" are indexed (dispatch.hit_finder).
4. Indexing — determine the crystal orientation and (optionally constrained) unit cell (dispatch.index).
5. Refinement — refine the crystal model and re-estimate the mosaic/domain-size parameters. Detector and beam are held fixed by default (dispatch.refine).
6. Integration — predict reflections and measure their intensities using the stills integrator (dispatch.integrate), with optional Kapton absorption correction and a significance filter.

The program applies a set of stills-specific defaults on top of the standard DIALS parameters, including: indexing method fft1d; the stills integrator with profile fitting turned off (summation integration with a 2-D plane background); the stills refinement weighting strategy with the detector and beam models fixed; and a re-estimation of per-crystal mosaic parameters (domain size and mosaic angle) after refinement. These are sensible starting points for serial data and normally do not need to be changed.

Failures on individual images are, by default, caught and logged so that processing continues to the next image (dispatch.squash_errors=True) — essential when reducing tens of thousands of shots of which only a fraction are hits.

Basic usage

# Process a directory of CBF stills, constraining the target cell/symmetry,
# using 32 parallel processes on one node
dials.stills_process /path/to/images/*.cbf process.phil mp.nproc=32

A typical process.phil for XFEL data looks like:

spotfinder.filter.min_spot_size=2
spotfinder.threshold.dispersion.gain=1.0
spotfinder.threshold.dispersion.global_threshold=100

indexing.known_symmetry.space_group=P43212
indexing.known_symmetry.unit_cell=78.9,78.9,38.1,90,90,90
indexing.method=fft1d

input.reference_geometry=refined.expt   # from a previous metrology refinement

output.output_dir=out
mp.nproc=32

Useful command-line helpers:

• dials.stills_process -h — help and usage
• dials.stills_process -c -e 2 — show the configurable parameters (expert level 2)
• dials.stills_process -c -a2 — include attributes (help text) for each parameter
• dials.stills_process ... input.show_image_tags=True — list the image tags in a dataset without processing (useful for selecting subsets with input.image_tag)

Important parameters

The program exposes the entire DIALS parameter set (spot-finding, indexing, refinement, integration, profile modelling, prediction). The parameters below are the ones most commonly adjusted for serial still data. Parameters are given by their full PHIL path with the default value in parentheses.

Input

Parameter	Default	Description
input.reference_geometry	None	Path to an .expt file (or image) with a single detector model. Data are processed using this geometry instead of the geometry in the image headers. This is how a refined detector metrology from cctbx.xfel ensemble refinement is applied — almost always set for XFEL data.
input.sync_reference_geom	True	When a reference geometry is supplied, copy its refined local frames onto the image's detector hierarchy node-by-node (the hierarchies must match in shape). If False, the reference detector replaces the image detector wholesale.
input.file_list	None	Path to a text file listing image paths, one per line (alternative to listing files on the command line).
input.glob	None	Wildcard path(s) for large multi-file datasets (e.g. *.cbf). May be given multiple times.
input.image_tag	None	Process only images with the given tag(s). For single-image files the tag is the filename; for multi-image files (HDF5) it is filename_imagenumber. Use input.show_image_tags=True to list them.
input.max_images	None	Limit the total number of images processed.

Dispatch — controlling the pipeline

Parameter	Default	Description
dispatch.find_spots	True	Do spot-finding. Required for indexing/integration.
dispatch.index	True	Attempt to index. Requires find_spots=True.
dispatch.integrate	True	Integrate indexed images. Ignored if indexing/spot-finding is off.
dispatch.refine	False	Run an explicit model-refinement step after indexing (the stills indexer already refines; this is an additional pass, rarely needed).
dispatch.squash_errors	True	If an image fails, log it and continue to the next image. Set False to halt and show the full error when debugging.
dispatch.hit_finder.enable	True	Enable hit-finding. If disabled, every image is passed to indexing.
dispatch.hit_finder.minimum_number_of_reflections	16	Images with fewer strong spots than this are discarded as non-hits before indexing.
dispatch.hit_finder.maximum_number_of_reflections	None	If set, images with more strong spots than this are discarded (e.g. ice/junk).
dispatch.process_percent	None	Process only this percentage of events (evenly sampled). Useful for a quick look at a large dataset.
dispatch.coset	False	During integration, also integrate a sublattice coset as a negative control for regions with no Bragg diffraction (see integration.coset.transformation).
dispatch.pre_import	False	Import all data before processing. Only needed when processing multiple multi-image files at once (not a recommended workflow). Single-file datasets are always pre-imported automatically.

Spot-finding

Spot-finding uses the standard DIALS dispersion thresholding. The most impactful parameters for serial data:

Parameter	Default	Description
spotfinder.filter.min_spot_size	Auto	Minimum connected pixels for a spot. Auto = 3 for pixel-array detectors (SENSOR_PAD), 6 otherwise. Often lowered to 2 for sparse serial patterns.
spotfinder.threshold.dispersion.gain	None	Detector gain used in thresholding. Set this or the detector-model gain, not both (see gain note below).
spotfinder.threshold.dispersion.sigma_strong	3	Number of sigma above background for a pixel to be "strong".
spotfinder.threshold.dispersion.sigma_background	6	Number of sigma for the background/dispersion test.
spotfinder.threshold.dispersion.global_threshold	0	Absolute pixel intensity floor; pixels below this are never considered strong. Raising it (e.g. 100) suppresses noise on many detectors.
spotfinder.threshold.dispersion.kernel_size	3 3	Local-area kernel (pixels) for the dispersion calculation.

Gain note: the detector gain should be set on the detector model or in the processing parameters, never both. If a supplied gain (spotfinder.threshold.dispersion.gain or integration.summation.detector_gain) conflicts with a non-unity gain already on the detector model, the program raises an error. Override with input.ignore_gain_mismatch=True only if you understand the consequences.

Indexing

Parameter	Default	Description
indexing.known_symmetry.space_group	None	Target space group. Strongly recommended for serial data — constrains indexing and improves the hit/index rate.
indexing.known_symmetry.unit_cell	None	Target unit cell (a,b,c,α,β,γ). Used together with the space group as the indexing target.
indexing.method	fft1d	Basis-vector search method (dials.stills_process defaults to fft1d; fft3d and real_space_grid_search are also available).
indexing.stills.method_list	None	Ordered list of indexing methods to try in turn; if one fails, the next is attempted (e.g. fft1d real_space_grid_search).
indexing.stills.known_orientations	None	Paths to previously-processed experiment lists. If given, images are not re-indexed; the stored crystal orientations are reused (e.g. to re-integrate with new parameters). Supports wildcards; may be repeated.
indexing.stills.require_known_orientation	False	When known orientations are supplied but an image is not found among them, controls whether to skip it (True) or fall back to indexing from scratch (False).
indexing.stills.reflection_subsampling.enable	False	Retry indexing on random subsets of the strong spots (from step_start% down to step_stop% in step_size% steps). Can rescue images that fail to index on the full spot list.

Integration

Parameter	Default	Description
profile.gaussian_rs.parameters.sigma_b_cutoff	0.1	Maximum beam divergence (sigma_b, degrees) before an indexed crystal is rejected prior to integration. Guards against over-broad, poorly-modelled lattices.
significance_filter.enable	False	Apply the stills significance filter, which trims each crystal to the resolution at which reflections remain statistically significant (see significance_filter.isigi_cutoff, default 1.0).
integration.integration_only_overrides.trusted_range	None	Override the panel trusted range [min, max] during integration only (e.g. to include pixels the header marks as untrusted).
integration.absorption_correction	apply=False	Enable a Kapton absorption correction (algorithm=fuller_kapton or kapton_2019). Requires shoeboxes to be retained via integration.debug settings; the program checks this and errors otherwise.
integration.coset.transformation	6	Which modulus-2 sublattice transformation defines the coset negative control when dispatch.coset=True (0–6; 6 = body centering). See Sauter & Zwart, Acta Cryst. D (2009) 65:553.

Output

Parameter	Default	Description
output.output_dir	.	Directory for output files.
output.composite_output	True	If True, each process writes one concatenated set of .expt/.refl files covering all its successful images. If False, a separate pair of files is written per image (generates a very large number of files). Composite output is strongly preferred for large serial datasets.
output.logging_dir	None	Directory for log files. Under MPI, per-rank logs (log_rankNNNN.out, etc.) are written here.
output.logging_option	normal	normal (all logging), suppressed (silences verbose DIALS refine/index output), or disabled (almost no logging).
output.*_filename	(templates)	Output filename templates for each stage: strong, indexed, refined_experiments, integrated, integrated_experiments, coset. A %s in the template is replaced with the per-image tag.
output.integration_pickle	None	Write legacy cxi.merge/PRIME integration pickle files (e.g. int-%d-%s.pickle). Only needed for legacy merging workflows.

Multiprocessing

Parameter	Default	Description
mp.method	multiprocessing	Parallelization backend: multiprocessing (single node), or mpi for multi-node jobs (also sge, lsf, pbs).
mp.nproc	1	Number of processes (for the multiprocessing method). Set to the number of cores per node.
mp.composite_stride	None	For MPI composite output, how many ranks aggregate their results into one set of output files. Ideally set to the number of processes per node so that one composite file set is written per node rather than per rank.

Under MPI with more than two ranks, the program runs a client/server model: rank 0 (or rank 2 for psana2 .loc inputs) hands out images to worker ranks on demand, giving good load balancing across the many fast-failing non-hits typical of serial data.

Output files

With the default composite output, each process (or each aggregating rank under MPI) writes concatenated files such as:

• <tag>_strong.refl — strong spots from spot-finding
• <tag>_indexed.refl and <tag>_refined.expt — indexed reflections and their crystal models
• <tag>_integrated.refl and <tag>_integrated.expt — integrated intensities and final models (the input to cctbx.xfel.merge)

Per-image progress and status codes are recorded in debug/debug_<rank>.txt, and overall logging goes to dials.process.log (or per-rank logs under MPI). These integrated .expt/.refl pairs are then scaled and merged with cctbx.xfel.merge.

See also

• cctbx.xfel.merge — scaling, post-refinement and merging of the integrated data
• IOTA — a front-end that supplies preset parameters for dials.stills_process
• Data Processing at SACLA — SACLA-specific processing workflow
• The DIALS documentation for the underlying spot-finding, indexing, refinement and integration algorithms