2017 dials.stills process

Processing the example data from DLS (Credit: Danny Axford). Pilatus data, 11000 images. For the sake of this example, we will be in a folder named $work, and the images are in a folder named $data

Remember, all these commands have great help strings, use -h, or to see details for the parameters, use -c. If you want to see all the parameters, use -e 10. And if you want to see the descriptions of each parameters, use -a 2. Example:

 dials.import -c -a 2 -e 10 | less

Initial discovery

First, use the dials.image_viewer on 10 images to get a sense of the data

 dials.image_viewer $data/*.cbf.bz2

Next, verify beam center and diffraction extent using an average image.

 mkdir $work/averages
 cd $work/averages
 dxtbx.image_average -v -a avg.pickle -m max.pickle -s stddev.pickle -n 64 $data/*.cbf.bz2

Here, we specify output for the average, maximum and standard deviation images using -a, -m and -s. -v means verbose output, and we specify 64 cores for averaging using -n. The output images are in image pickle format, and can be displayed using the image viewer: dials.image_viewer *.pickle

Use the Actions->Show ring tool with the maximum projection image to verify the rings are circular around the beam center.

Also, in the average image, note the defects around the beam center and the 8 silicon reflections around the edges of the detector. These artifacts will need to be masked out. Hold down shift and left click to drag boxes onto the image. Click save mask to generate a mask.pickle file we will use during processing. If you want to verify the mask is good, restart the image viewer using mask=mask.pickle, then click the show mask checkbox in the settings panel. The mask will display in red.

Trial processing

We will use the following phil file to process the data. Note the use of $work to specify mask paths. Also note, this part I am currently editing, removing values here that are just defaults.

spotfinder.lookup.mask=$work/averages/mask.pickle
integration.lookup.mask=$work/averages/mask.pickle
spotfinder.filter.min_spot_size=2
verbosity=10
indexing {
  known_symmetry {
    space_group = P6
    unit_cell = 91.7 91.7 46 90 90 120
  }
  refinement_protocol.n_macro_cycles = 1
  refinement_protocol.d_min_start=2.0
  basis_vector_combinations.max_try=10
  stills.indexer=stills
  stills.method_list=fft1d real_space_grid_search
}
verbosity=10
integration {
  integrator=stills
  profile.fitting=False
  background {
    simple {
      outlier {
        algorithm = null
      }
    }
  }
}
profile {
  gaussian_rs {
    min_spots.overall = 0
  }
}

refinement {
  parameterisation {
    beam.fix=all
    detector.fix=all
    detector.hierarchy_level=0
    auto_reduction {
      action=fix
      min_nref_per_parameter=1
    }
  treat_single_image_as_still=True
  }
  reflections {
    outlier.algorithm=null
    weighting_strategy.override=stills
    weighting_strategy.delpsi_constant=1000000
  }
}

With this phil file we can process the data:

 cd $work; mkdir -p results/000; cd results/000
 dials.stills_process $data/*.cbf.bz2 $work/process.phil mp.nproc=64

Here, we put the processed results in a results folder, in a subfolder labeled 000 that represents trial 0.

When complete, we can look at integration results:

 dials.image_viewer idx-zurich0038_00036.cbf_refined_experiments.json idx-zurich0038_00036.cbf_integrated.pickle

We can look at a graph showing hitfinding and indexing rates:

 cctbx.xfel.plot_run_stats_from_experiments . hit_cutoff=40

The hit_cutoff is adjusted up a bit to be above the background (see the top plot, how the images that are essentially blank and didn't index (gray dots) still show up with ~30 or so reflections).

We can also look at unit cell histograms to see if the results are isomorphous:

 cctbx.xfel.plot_uc_cloud_from_experiments *_refined_experiments.json

Here we see a slight asymmetry in the c axis which indicates the detector position might not be accurate.

Ensemble refinement of detector position

To get a more isomorphous distribution we will refine the detector model using many crystal models as input.

 cd $work; mkdir metrology; cd metrology
 dials.combine_experiments $work/results/000/*indexed.pickle $work/results/000/*_refined_experiments.json n_subset=2000 reference_from_experiment.detector=0

Here, since we fixed the detector position in place during indexing, we use reference_from_experiment.detector=0 so that each experiment in the output combined.json file points to the same detector. If you allow the detector to refine separately for each image, you will need to average the detector positions when running dials.combine_experiments, using the parameter reference_from_experiment.average_detector=True instead.

Now, we refine the detector and all 2000 crystals at once:

 dials.refine combined_experiments.json combined_reflections.pickle refine.phil

The phil file is here:

refinement {
  parameterisation {
    auto_reduction {
      min_nref_per_parameter = 3
      action = fail fix *remove
    }
    beam {
      fix = *all in_spindle_plane out_spindle_plane wavelength
    }
    detector {
      fix_list = Tau1
    }
  }
  refinery {
    engine = SimpleLBFGS LBFGScurvs GaussNewton LevMar *SparseLevMar
  }
  reflections {
    outlier {
      algorithm = null auto mcd tukey *sauter_poon
      separate_experiments = False
      separate_panels = True
    }
  }
}

To compare the old to the new metrology:

 dxtbx.plot_detector_models combined_experiments.json refined_experiments.json

We can reprocess all the data using this updated geometry, adding this to the process.phil

 input.reference_geometry=$work/metrology/refined_experiments.json

Then create a new trial and reprocess the data.

 cd $work/results; mkdir 001; cd 001
 dials.stills_process $data/*.cbf.bz2 $work/process.phil mp.nproc=64

The results should be more isomorphous:

 cctbx.xfel.plot_uc_cloud_from_experiments *_refined_experiments.json

From here, proceed to merging:

PRIME