Cxi02416 calibration: Difference between revisions
| Line 426: | Line 426: | ||
'''cspad.detector_statistics''' tag=t003_2k | '''cspad.detector_statistics''' tag=t003_2k | ||
Warning, this dumps a lot of output. The program examines the current directory for files from cspad.cbf_metrology. For each hierarchy level 0-3, the program creates | Warning, this dumps a lot of output. The program examines the current directory for files from cspad.cbf_metrology. For each hierarchy level 0-3, the program creates three tables of statistics: | ||
1) Detector congruence. The two half datasets from each level of refinement are compared to each other, and agreement between the two independent refinement runs is reported. | 1) Detector congruence. The two half datasets from each level of refinement are compared to each other, and agreement between the two independent refinement runs is reported. | ||
2) Detector statistics. Using the two half datasets as independent measurements, statistics about the cspad are reported, such as normal vector tilts and XYZ offsets. Weighted means and standard deviations are reported as well, and can be used to estimate the overall precision of refinement. | 2) Detector statistics. Using the two half datasets as independent measurements, statistics about the cspad are reported, such as normal vector tilts and XYZ offsets. Weighted means and standard deviations are reported as well, and can be used to estimate the overall precision of refinement. | ||
3) RMSDs by detector number. For each of the half datasets, overall, radial and transverse RMSDs are shown. | |||
For this walkthrough, we will extract a few statistics only. Look for the third set of tables, delineated with | For this walkthrough, we will extract a few statistics only. Look for the third set of tables, delineated with | ||
Revision as of 21:45, 4 April 2016
Calibration of cspad using cxi02416, cctbx.xfel and DIALS
This page is not intended as a manual for processing XFEL data using cctbx.xfel and DIALS. Rather, this is a documentation of steps taken to calibrate the CSPAD using data collected at CXI in February 2016, on detector Ds2.
Program names are highlighted in bold. Generally, help can be obtained using -h or -c options to the program at the command line.
In this walkthrough, we will be calibrating the tile positions of the CSPAD detector at the CXI endstation at LCLS. We will be using a calibration dataset collected by CXI staff in February 2016, during calibration experiment cxi02614. The procedure is to index the images, refine the tile positions minimizing the difference between observed and calculated Bragg reflections, re-index the images, re-refine the tile positions, and repeat until convergence. At each step of the way, we will label our current metrology with a version number, starting at version 0:
| Metrology version | Description |
| Version 0 (v0) | Initial metrology deployed by beamline operators. The tile positions are measured using an optical microscope, but as the quadrants can move independently, they are not correctly aligned in relation to each other or to the beam center. |
| Version 1 (v1) | After collecting some data, powder rings can be seen after averaging the events in a run. Several tools are available for aligning the quadrants by eye or automatically using powder rings. |
| Version 2 (v2) | After indexing the images using v1, we will refine the tile positions to produce metrology v2. |
| Version 3 (v3) | After re-indexing the images using v2, we will re-refine the tile positions to produce metrology v3. |
| ... | And so forth until convergence |
We start with this known information provided by the beamline operators:
- Data is contained in runs 2-16
- Detector address: CxiDs2.0:Cspad.0. This string identifies the front CSPAD detector in the XTC streams.
- detz_offset (IE the distance from the sample position to the back of the detector rail): 568 mm.
Dark pedestal, common mode correction, and untrusted pixels
Subtracting the dark pedestal, applying common mode correction and masking out untrusted pixels is critical for spot picking and integration. cctbx.xfel uses masks and pedestals deployed into the experiment’s calibration store using the LCLS program calibman [1] . Dark pedestal correction, hot pixel determination, and user defined mask generation for regions of interest or for masking out shadows is all available through that program.
Common mode correction accounts for flat, per-panel fluctuations that occur non-uniformly throughout a single exposure. LCLS has several algorithms available [2]. We recommend the non-bonded pixels algorithm; the parameter files below specifies this algorithm during indexing and integration.
Aligning quadrant positions (v1)
Alignment of quadrant positions using powder rings can be done manually using cctbx.xfel or LCLS's calibman tool [3]. ctbx.xfel also provides an automatic algorithm for quadrant alignment, provided powder rings of sufficient quality. For cxi02416, the beamline operators already calibrated quadrant positions to a point where indexing can proceed. Regardless, if needed, here is how to accomplish the same task using cctbx.xfel
Averaging diffraction data to create powder patterns
The cctbx.xfel program cxi.mpi_average is used to create averages:
for i in `seq 2 16`; do bsub -n 12 -q psanaq -o avg_r$i.log \ cxi.mpi_average -x cxi02416 -r $i -a CxiDs2.0:Cspad.0 -d 568 -o . -v; done
For each of the runs with data (2-16), this command submits an averaging job using 12 processors, providing information about the experiment such as detector address and detz_offset in the form of command line arguments.
Progress can be monitored with the command bjobs.
When the averages are complete, they will consist of files named cxi02416_avg-r0002.cbf, cxi02416_stddev-r0002.cbf, and cxi02416_max-r0002.cbf for each run, representing the average, standard deviation, and maximum of all the pixel data in each of the runs. Display the data using cctbx.image_viewer:
cctbx.image_viewer *.cbf
Manual calibration using cctbx.xfel
Typically the best powder rings come from the maximum projection (example: cxi02416_max-r0013.cbf). To manually align the quadrant positions, use cctbx.image_viewer cxi02416_max-r0013.cbf. Under actions, click on 'Show quadrant calibration', then use the spinners to align the powder rings. You may find the ring tool or the unit cell tool, also under the Actions menu, to be useful visual aids during this process. When done, click 'Save current metrology' to save the changes to a .def file, which is a cbf header.
This walkthrough uses pre-aligned quadrant locations. However, if it is desired to use the metrology from manual quadrant re-alignment for indexing, it first needs to be converted to SLAC's metrology file format. Use this command:
cxi.cbfheader2slaccalib cbf_header=quadrants.cbf
This command will create a 0-end.data file. See the instructions under indexing using v2 metrology for deploying it for use.
Automatic calibration using cctbx.xfel
If a quadrant is properly placed, the pixel values for a strong powder pattern will be highly correlated after rotating it 45 degrees around the beam center. cspad.quadrants_cbf performs a grid search of XY offsets for each quadrant, searching for the position with the highest rotational autocorrelation. It then writes out a new cbf file with the adjusted header:
cspad.quadrants_cbf cxi02416_max-r0013.cbf
Specify the '-p' parameter to enable plots of the grid search results for each quadrant. Here is the output:
Doing cross-correlation on panel ARRAY_D0Q0S1A0 Searching a grid with dimensions (41, 41) max cc 0.1592 is at (0, 0) Doing cross-correlation on panel ARRAY_D0Q1S1A0 Searching a grid with dimensions (41, 41) max cc 0.1209 is at (0, 5) Doing cross-correlation on panel ARRAY_D0Q2S1A0 Searching a grid with dimensions (41, 41) max cc 0.1686 is at (0, -2) Doing cross-correlation on panel ARRAY_D0Q3S1A0 Searching a grid with dimensions (41, 41) max cc 0.1144 is at (1, 2)
The CC values are all less that 0.2 which indicates that even though run 13 shows the strongest powder diffraction, the rings are not contiguous or strong enough compared to the background to get a good rotational autocorrelation. Use the image viewer to verify the new quadrant positions are not ideal:
cctbx.image_viewer cxi02416_max-r0013_cc.cbf
It is possible that using a maximum projection of all the runs would make the rings more contiguous and brighter, leading to higher CC values. This can be done quickly using the maximum projections already made:
cxi.cspad_average *_max*.cbf -m all_max.cbf
Use cctbx.image_viewer to compare all_max.cbf to cxi02416_max-r0013.cbf. The rings are noticeably better. Now, do the grid search:
cspad.quadrants_cbf all_max.cbf
Results:
Doing cross-correlation on panel ARRAY_D0Q0S1A0 Searching a grid with dimensions (41, 41) max cc 0.2568 is at (-1, 0) Doing cross-correlation on panel ARRAY_D0Q1S1A0 Searching a grid with dimensions (41, 41) max cc 0.2022 is at (0, 4) Doing cross-correlation on panel ARRAY_D0Q2S1A0 Searching a grid with dimensions (41, 41) max cc 0.2328 is at (-4, 0) Doing cross-correlation on panel ARRAY_D0Q3S1A0 Searching a grid with dimensions (41, 41) max cc 0.1853 is at (0, 2)
The CC values are higher, and likely high enough for at least two of the quadrants to get some initial indexing results. Regardless, we recommend silver behenate powder for this automatic procedure as it gives very smooth, contiguous rings.
Again, for this walkthrough we use quadrant positions aligned by the beamline operator. However, if after automatic alignment it is desired to use the results for indexing, they first need to be converted to SLAC's metrology file format. Use this command:
cxi.cbfheader2slaccalib cbf_header=all_max_cc.cbf
This command will create a 0-end.data file. See the instructions under indexing using v2 metrology for deploying it for use.
Initial indexing
The initial metrology deployed by the beamline operator is sufficient to get initial indexing results. Indexing in cctbx.xfel typically is done in a series of trials. By this point, I had done two trials of indexing (trials 0 and 1), working out some spotfinding parameters. We join this walkthrough at trial 2.
With this information, and this phil file, we can index the data:
for i in `seq 2 16`; do cxi.mpi_submit input.experiment=cxi02416 \ output.output_dir=/reg/d/psdm/cxi/cxi02416/ftc/brewster/dials \ mp.nproc=36 mp.queue=psanaq output.split_logs=True \ input.dispatcher=cctbx.xfel.xtc_process \ input.target=cxi02416-lyso-t002.phil input.trial=2 \ input.run_num=$i; done
This command submits jobs for runs 2 through 16, using the DIALS backend of cctbx.xfel. Optionally, to save time during initial indexing and metrology refinement, use dispatch.integrate=False to skip the integration step. After indexing is completed, we got 4920 indexed images, as can be shown by this command:
cd /reg/d/psdm/cxi/cxi02416/ftc/brewster/dials ls r0*/002/out/*.json | wc -l
Refine metrology (v2)
Let's call the metrology deployed by the beamline operator version 0 (v0). After quadrant alignment, the operators updated the metrology to version 1 (v1). The following command will do an iterative joint hierarchical refinement of the components of the CSPAD detector. The new tile positions we call version 2 (v2).
bsub -q psanaq -o t002_1k.out cspad.cbf_metrology tag=t002_1k \ cxi02416-refine.phil reflections=indexed ../r0*/002/out \ n_subset=1000 split_dataset=True
The program first aggregates the requested number of images into a single dataset. Then, it refines the detector as a whole (including Z position and tilt). Using the new detector position, it refines the quadrants independently from each other, and then the 2x1 sensors, and then the individual panels. Finally, it converts the DIALS format metrology into the SLAC file format (0-end.data). The refinement is a 'joint refinement' because the information from many crystals is used to refine a single detector model.
Details about the parameters used:
- tag=t002_1k: the output files will be named t002_1k*
- cxi02416-refine.phil: this phil file contains parameters for joint hierarchical refinement.
- reflections=indexed: the indexing process produces two sets of reflection files, bright indexed reflections, and final integrated reflections (can include weak intensities). Here, I've chosen to refine only against the bright, indexed reflections.
- n_subset=1000: pick 1000 images at random to refine
- split_dataset=True: the refinement is done twice independently, using odd numbered or even numbered images, each time using n_subset images. This will be useful later for evaluating the accuracy of the metrology.
Visualize tile shifts
In order to get a sense of the magnitude of the shifts in panel position after refinement, use the program cxi.display_metrology. For example, the original detector geometry can be displayed thusly:
cxi.display_metrology \ /reg/d/psdm/cxi/cxi02416/calib/CsPad::CalibV1/CxiDs2.0:Cspad.0/geometry/0-end.data
Compare it to the refined geometry:
cxi.display_metrology 0-end.data.t002_1k_1
You will see a small change in the origin (center arrow) and obvious changes in the quadrant positions. The relative positions of the sensors to each other will not change a large amount, though if you inspect the files themselves you will see changes.
Another tool that is useful for evaluating changes in metrology is dev.dials.plot_detector_shifts. This program will plot detector shifts in the X and Y directions (also known as the fast and slow directions, a convention referring to how data is read from the byte arrays stored on disk). It will also plot shifts along the Z axis, I.E., the detector's normal axis. Let's use it to compare the unrefined geometry to the geometry refined at level 2 (I.E. at the level of the 2x1 sensors):
dev.dials.plot_detector_shifts t002_1k_1_combined_experiments.json \ t002_1k_1_refined_experiments_level2.json \ plot_type=spherical_polar tag=v1v2level2 display v1v2level2*.png
Use the spacebar to cycle between the images. The pixels are mapped to the Ewald sphere and then displayed as a function of azimuthal and elevation angles along the Ewald sphere, which distorts the detector appearance but is independent of any detector layout. The colorbar scales will show the magnitude of panel shifts.
Try running the program using t002_1k_1_refined_experiments_level0.json to see how much the detector as a whole shifted, or t002_1k_1_refined_experiments_level1.json to see shifts in the quadrants.
Finally, the program cspad.detector_shifts shows the magnitude of shifts between a reference and a moving detector in tabular form:
cspad.detector_shifts \ t002_1k_1_combined_experiments.json t002_1k_1_combined_reflections.pickle \ t002_1k_1_refined_experiments_level2.json t002_1k_1_refined_reflections_level2.pickle
Looking at the Delta XY columns, we see the detector moved 2.6 microns in the XY plane, the quadrants moved on average 223.5+/-52.0 microns and the 2x1 sensors moved 31.7+/-25.1 microns. Note how most of the change is in the quadrant locations, which is expected as these were done by eye. The 2x1 sensor positions, determined using optical microscopy, changed by an order of magnitude less than the quadrants. Also, the same program ran against the other half of the split dataset (t002_1k_2_*) will reveal very similar shifts, indicating these shifts are not due to random chance.
Full output for cspad.detector_shifts:
Found 4 hierarchy levels
Hierarchy level 0 Detector shifts
-----------------------------------------------------------------------------------------------------------
PanelG BC dist Delta XY R Offsets T Offsets Z Offsets dR Norm dT Norm Local dNorm Rot Z N Refls
ID (mm) (microns) (microns) (microns) (microns) (deg) (deg) (deg) (deg)
-----------------------------------------------------------------------------------------------------------
0 2.4 2.6 23.9 0.0 0.0 -0.0262 -0.0045 0.0266 0.0000 35366
Weighted mean 2.6 23.9 0.0 -5.3 -0.0262 -0.0045 0.0266 0.0000
Weighted stddev 0.0 0.0 0.0 0.0 0.0000 0.0000 0.0000 0.0000
-----------------------------------------------------------------------------------------------------------
Hierarchy level 1 Detector shifts
-----------------------------------------------------------------------------------------------------------
PanelG BC dist Delta XY R Offsets T Offsets Z Offsets dR Norm dT Norm Local dNorm Rot Z N Refls
ID (mm) (microns) (microns) (microns) (microns) (deg) (deg) (deg) (deg)
-----------------------------------------------------------------------------------------------------------
0 62.9 166.8 -179.3 -91.5 -91.5 -0.0796 0.0429 0.0729 0.0000 9500
1 64.3 244.0 -238.1 98.3 98.3 -0.1394 0.0577 0.1368 0.0000 8388
3 66.0 205.4 -187.9 55.9 55.9 -0.0969 -0.0916 0.1224 0.0000 9244
2 67.0 288.3 -222.8 -142.5 -142.5 0.0529 0.0676 0.0669 0.0000 8234
Weighted mean 223.5 -205.6 -19.9 1288.1 -0.0675 0.0170 0.0996 0.0000
Weighted stddev 52.0 28.0 113.8 119.3 0.0806 0.0753 0.0348 0.0000
-----------------------------------------------------------------------------------------------------------
Hierarchy level 2 Detector shifts
-----------------------------------------------------------------------------------------------------------
PanelG BC dist Delta XY R Offsets T Offsets Z Offsets dR Norm dT Norm Local dNorm Rot Z N Refls
ID (mm) (microns) (microns) (microns) (microns) (deg) (deg) (deg) (deg)
-----------------------------------------------------------------------------------------------------------
1 21.9 3.1 16.1 -6.7 -6.7 -0.1707 -0.1239 0.2046 0.0001 2051
9 23.6 28.4 7.0 18.7 18.7 0.0587 0.3254 0.3081 0.0001 1752
25 24.7 37.7 -36.9 -58.3 -58.3 -0.2511 -0.6087 0.5523 0.0006 1897
17 26.3 15.3 -6.3 20.9 20.9 0.0621 0.2271 0.1694 0.0000 1645
0 40.8 18.2 1.9 3.9 3.9 0.0204 -0.2756 0.3172 0.0006 1790
8 41.3 11.5 -1.0 2.7 2.7 0.1900 0.1839 0.3758 0.0011 1926
24 43.9 16.8 5.3 -47.4 -47.4 -0.2186 -0.4251 0.3455 0.0006 2024
16 44.5 26.1 18.6 14.3 14.3 0.1773 0.2270 0.2060 0.0000 1733
7 55.5 15.4 -0.6 -6.6 -6.6 -0.1542 0.1246 0.1117 0.0002 1928
31 57.7 23.0 -16.4 -2.7 -2.7 0.0358 -0.2582 0.2656 0.0004 1598
15 58.0 26.5 20.6 0.9 0.9 -0.0135 0.1436 0.0989 0.0001 1491
23 60.0 19.6 -7.8 25.2 25.2 -0.0851 0.0755 0.1639 0.0002 1838
11 72.2 68.7 -64.6 -18.8 -18.8 -0.6625 0.1031 0.5312 0.0001 1017
3 73.1 66.6 -47.0 -9.2 -9.2 -0.3160 0.1041 0.2524 0.0005 917
19 75.3 62.9 -67.9 -12.9 -12.9 -0.0452 0.1673 0.0967 0.0001 672
6 76.5 13.2 23.5 -9.4 -9.4 0.0184 -0.1715 0.2574 0.0006 1022
27 76.5 41.8 7.0 11.9 11.9 -0.0461 -0.2238 0.0953 0.0002 1114
2 77.7 40.9 -21.9 -6.7 -6.7 -0.2080 -0.0272 0.1273 0.0000 914
10 78.3 30.7 -26.7 -21.8 -21.8 -0.3839 0.1773 0.2850 0.0001 908
30 78.6 61.4 1.4 34.1 34.1 0.1073 -0.2332 0.3137 0.0006 737
14 78.9 27.4 20.4 -15.4 -15.4 0.1525 0.2232 0.2635 0.0001 605
22 80.6 28.1 4.5 -20.9 -20.9 0.0905 0.0922 0.0623 0.0001 923
18 81.2 39.7 -40.6 -16.3 -16.3 -0.0614 0.1521 0.1167 0.0002 739
26 81.3 126.2 117.0 19.3 19.3 0.1011 -0.1083 0.1667 0.0003 1027
4 87.6 26.3 -9.7 -4.4 -4.4 -0.2709 0.0775 0.1987 0.0003 608
12 89.8 25.2 22.6 -4.7 -4.7 0.0539 0.0448 0.1875 0.0007 527
28 90.7 63.4 -17.3 34.0 34.0 -0.3379 0.0477 0.2630 0.0006 577
20 92.1 67.6 52.5 -31.5 -31.5 0.0800 0.0600 0.0192 0.0000 496
5 104.9 14.6 31.0 -9.8 -9.8 -0.5151 0.0433 0.4343 0.0026 270
13 106.3 36.9 26.6 -33.0 -33.0 -0.0042 0.0441 0.1372 0.0003 162
29 108.0 105.5 -46.3 67.5 67.5 -0.2877 0.0256 0.2276 0.0001 270
21 108.7 59.5 42.9 -30.9 -30.9 0.1146 0.0449 0.0680 0.0000 188
Weighted mean 31.7 -0.9 -3.9 18.3 -0.0710 -0.0185 0.2433 0.0003
Weighted stddev 25.1 32.1 24.0 84.4 0.1932 0.2486 0.1307 0.0004
-----------------------------------------------------------------------------------------------------------
Hierarchy level 3 Detector shifts
-----------------------------------------------------------------------------------------------------------
PanelG BC dist Delta XY R Offsets T Offsets Z Offsets dR Norm dT Norm Local dNorm Rot Z N Refls
ID (mm) (microns) (microns) (microns) (microns) (deg) (deg) (deg) (deg)
-----------------------------------------------------------------------------------------------------------
2 15.8 0.0 0.0 0.1 0.1 -0.1048 -0.1830 0.0000 0.0001 1046
18 16.2 0.0 0.0 0.1 0.1 -0.0812 0.3206 0.0000 0.0001 929
50 19.2 0.0 0.3 0.4 0.4 0.0254 -0.6580 0.0000 0.0006 974
34 19.3 0.0 0.0 0.1 0.1 -0.0389 0.2322 0.0000 0.0000 740
3 30.7 0.0 -0.1 -0.0 -0.0 -0.1936 -0.0835 0.0000 0.0001 1005
19 33.0 0.0 -0.1 -0.0 -0.0 0.1283 0.3048 0.0000 0.0001 823
51 33.0 0.0 -0.5 -0.1 -0.1 -0.3796 -0.5381 0.0000 0.0006 923
35 35.2 0.0 -0.1 -0.1 -0.1 0.1108 0.2077 0.0000 0.0000 905
16 37.6 0.0 -0.2 0.1 0.1 0.1378 0.2257 0.0000 0.0011 947
0 37.9 0.0 0.1 0.0 0.0 0.0885 -0.2618 0.0000 0.0006 853
32 40.8 0.0 0.0 0.2 0.2 0.1146 0.2643 0.0000 0.0000 731
48 41.1 0.0 -0.1 0.3 0.3 -0.1043 -0.4665 0.0000 0.0006 977
1 46.2 0.0 -0.1 0.0 0.0 -0.0361 -0.2740 0.0000 0.0006 937
17 47.3 0.0 0.1 -0.2 -0.2 0.2236 0.1412 0.0000 0.0011 979
49 49.1 0.0 -0.0 -0.3 -0.3 -0.3012 -0.3712 0.0000 0.0006 1047
33 50.4 0.0 -0.1 -0.1 -0.1 0.2201 0.1859 0.0000 0.0000 1002
15 51.3 0.0 0.1 -0.1 -0.1 -0.1295 0.1501 0.0000 0.0002 1044
63 52.9 0.0 -0.0 -0.1 -0.1 -0.0109 -0.2605 0.0000 0.0004 814
31 54.2 0.0 0.0 -0.0 -0.0 0.0123 0.1437 0.0000 0.0001 728
47 55.7 0.0 -0.0 -0.1 -0.1 -0.0701 0.0896 0.0000 0.0002 966
14 61.4 0.0 -0.1 0.1 0.1 -0.1712 0.1001 0.0000 0.0002 884
23 61.5 0.0 0.1 -0.0 -0.0 -0.6607 0.1139 0.0000 0.0001 790
7 62.3 0.0 0.1 0.1 0.1 -0.3144 0.1086 0.0000 0.0005 686
30 63.5 0.0 -0.0 0.0 0.0 -0.0349 0.1399 0.0000 0.0001 763
62 64.0 0.0 0.0 0.1 0.1 0.0742 -0.2499 0.0000 0.0004 784
39 64.6 0.0 0.0 -0.0 -0.0 -0.0428 0.1679 0.0000 0.0001 494
55 65.7 0.0 0.0 0.0 0.0 -0.0493 -0.2231 0.0000 0.0002 779
46 65.8 0.0 -0.0 0.1 0.1 -0.0955 0.0618 0.0000 0.0002 872
5 67.7 0.0 0.1 -0.0 -0.0 -0.2092 -0.0153 0.0000 0.0000 674
21 68.5 0.0 -0.0 -0.0 -0.0 -0.3729 0.1994 0.0000 0.0001 712
53 71.2 0.0 0.0 -0.1 -0.1 0.0948 -0.1139 0.0000 0.0003 726
37 71.4 0.0 0.0 -0.1 -0.1 -0.0525 0.1554 0.0000 0.0002 545
13 73.5 0.0 -0.1 -0.0 -0.0 -0.0048 -0.1724 0.0000 0.0006 595
61 75.2 0.0 -0.1 -0.1 -0.1 0.0751 -0.2455 0.0000 0.0006 439
29 76.1 0.0 0.0 -0.0 -0.0 0.1812 0.2006 0.0000 0.0001 326
45 77.5 0.0 -0.0 -0.0 -0.0 0.1022 0.0791 0.0000 0.0001 567
9 80.3 0.0 -0.0 0.1 0.1 -0.2769 0.0524 0.0000 0.0003 403
12 80.8 0.0 0.1 0.0 0.0 0.0392 -0.1679 0.0000 0.0006 427
25 82.2 0.0 -0.1 0.1 0.1 0.0496 0.0496 0.0000 0.0007 374
28 82.9 0.0 -0.0 0.0 0.0 0.1240 0.2403 0.0000 0.0001 279
22 83.0 0.0 -0.1 0.0 0.0 -0.6637 0.0950 0.0000 0.0001 227
60 83.3 0.0 0.1 0.1 0.1 0.1350 -0.2184 0.0000 0.0006 298
57 83.5 0.0 -0.2 0.0 0.0 -0.3409 0.0165 0.0000 0.0006 406
6 83.9 0.0 -0.1 -0.1 -0.1 -0.3171 0.1006 0.0000 0.0005 231
41 84.6 0.0 0.0 0.0 0.0 0.0742 0.0671 0.0000 0.0000 352
44 85.0 0.0 0.0 0.1 0.1 0.0784 0.1027 0.0000 0.0001 356
38 86.1 0.0 -0.0 0.0 0.0 -0.0471 0.1668 0.0000 0.0001 178
54 87.3 0.0 -0.0 -0.0 -0.0 -0.0437 -0.2243 0.0000 0.0002 335
4 88.0 0.0 -0.1 0.0 0.0 -0.2065 -0.0364 0.0000 0.0000 240
20 88.3 0.0 0.0 0.0 0.0 -0.3915 0.1598 0.0000 0.0001 196
36 91.3 0.0 -0.0 0.0 0.0 -0.0681 0.1492 0.0000 0.0002 194
52 91.5 0.0 -0.0 0.1 0.1 0.1058 -0.1037 0.0000 0.0003 301
8 95.7 0.0 0.0 -0.1 -0.1 -0.2641 0.0981 0.0000 0.0003 205
24 98.1 0.0 0.0 -0.1 -0.1 0.0572 0.0406 0.0000 0.0007 153
56 98.5 0.0 0.1 -0.1 -0.1 -0.3333 0.0736 0.0000 0.0006 171
11 98.8 0.0 -0.5 0.2 0.2 -0.5170 -0.0010 0.0000 0.0026 161
27 100.0 0.0 -0.0 0.1 0.1 -0.0079 0.0436 0.0000 0.0003 111
40 100.2 0.0 -0.0 -0.0 -0.0 0.0844 0.0537 0.0000 0.0000 144
59 102.0 0.0 -0.0 0.0 0.0 -0.2889 0.0009 0.0000 0.0001 182
43 102.4 0.0 0.0 0.0 0.0 0.1103 0.0546 0.0000 0.0000 117
10 111.6 0.0 0.4 -0.3 -0.3 -0.5104 0.0823 0.0000 0.0026 109
26 113.4 0.0 0.0 -0.1 -0.1 -0.0008 0.0443 0.0000 0.0003 51
58 114.7 0.0 0.0 -0.0 -0.0 -0.2850 0.0473 0.0000 0.0001 88
42 115.7 0.0 -0.0 -0.0 -0.0 0.1176 0.0361 0.0000 0.0000 71
Weighted mean 0.0 -0.0 0.0 0.0 -0.0678 -0.0189 0.0000 0.0003
Weighted stddev 0.0 0.1 0.1 0.0 0.1950 0.2441 0.0000 0.0004
-----------------------------------------------------------------------------------------------------------
Detector shifts summary
---------------------------------------------------------------------------------------------------------------------------------------------------------------
Hierarchy Delta XY Delta XY R Offsets R Offsets T Offsets T Offsets Z Offsets Z Offsets dR Norm dR Norm dT Norm dT Norm Local dNorm Local dNorm Rot Z Rot Z
Level Sigma Sigma Sigma Sigma Sigma Sigma Sigma Sigma
(microns) (microns) (microns) (microns) (microns) (microns) (microns) (microns) (deg) (deg) (deg) (deg) (deg) (deg) (deg) (deg)
---------------------------------------------------------------------------------------------------------------------------------------------------------------
0 2.6 0.0 23.9 0.0 0.0 0.0 -5.3 0.0 -0.0262 0.0000 -0.0045 0.0000 0.0266 0.0000 0.0000 0.0000
1 223.5 52.0 -205.6 28.0 -19.9 113.8 1288.1 119.3 -0.0675 0.0806 0.0170 0.0753 0.0996 0.0348 0.0000 0.0000
2 31.7 25.1 -0.9 32.1 -3.9 24.0 18.3 84.4 -0.0710 0.1932 -0.0185 0.2486 0.2433 0.1307 0.0003 0.0004
3 0.0 0.0 -0.0 0.1 0.0 0.1 0.0 0.0 -0.0678 0.1950 -0.0189 0.2441 0.0000 0.0000 0.0003 0.0004
---------------------------------------------------------------------------------------------------------------------------------------------------------------
For each hierarchy level, the average shifts in are computed among objects at that level, weighted by the number of reflections recorded on each object. For example, for a four quadrant detector, the average Z shift will be the average of the four quadrant Z values, each weighted by the number of reflections on that quadrant.
-------------------
Column descriptions
-------------------
Individual hierarchy level tables only:
PanelG id: ID of the panel group.
BC dist: distance of the panel group from the beam center.
N Refls: number of reflections on this panel group
All tables:
Delta XY: magnitude of the shift in the local XY frame.
R, T offsets: shifts relative to the parent object's location in the radial and transverse directions (relative to the detector center).
Z offsets: relative shifts in the local frame in the local Z direction.
R, T Norm: angle between normal vectors in lab space, projected onto the radial or transverse plane.
Local dNorm: local relative angle between normal vectors.
Rot Z: rotation around detector normal in lab space
Index using v2 metrology
In order to improve our metrology we will re-index using the new tile positions. We assume we don't have write access to the geometry file for this detector, namely /reg/d/psdm/cxi/cxi02416/calib/CsPad::CalibV1/CxiDs2.0:Cspad.0/geometry/0-end.data. To that end, we need a copy of the calibration directory for the experiment so we can modify it:
cd <a subfolder in your home directory> mkdir -p cxi02416/calib cd cxi02416/calib cp -r /reg/d/psdm/cxi/cxi02416/calib/* .
Now, we can link in v2, backing up v1 first:
cd CsPad::CalibV1/CxiDs2.0:Cspad.0/geometry mv 0-end.data 0-end.data.v1 ln -fns <path to 0-end.data.t002_1k_1> 0-end.data
We optionally use softlinks here to avoid duplicating data. We can now reprocess the data as trial 3:
for i in `seq 2 16`; do cxi.mpi_submit input.experiment=cxi02416 \ output.output_dir=/reg/d/psdm/cxi/cxi02416/ftc/brewster/dials \ mp.nproc=36 mp.queue=psanaq output.split_logs=True \ input.dispatcher=cctbx.xfel.xtc_process \ input.target=cxi02416/lyso-t003.phil input.trial=3 input.run_num=$i \ input.cfg=cxi02416-calibdir.cfg; done
The input phil file is the same as before. However, to instruct psana to use the modified calibration directory, we add a psana config file named cxi02416-calibdir.cfg with these lines:
[psana] calib-dir = <a subfolder in your home directory>/cxi02416/calib
If you have write permissions to your geometry folder, you don't need to use this config file or make a copy of your calibration directory. We recommend saving the original metrology like we did in the above example, by renaming it to 0-end.data.v1, so you can return to it as needed.
Indexing with the new metrology yielded 5484 indexed images.
Refine metrology (v3)
We now refine the metrology generated from the indexed images from trial 3 and call it metrology version 3 (v3). The command is otherwise the same, with an updated tag. We also increase the number images used to 2000.
bsub -q psanaq -o t003_1k.out cspad.cbf_metrology tag=t003_1k \ cxi02416-refine.phil reflections=indexed ../r0*/003/out \ n_subset=2000 split_dataset=True
After refinement, use cxi.display_metrology, dev.dials.plot_detector_shifts, and/or cspad.detector_shifts to evaluate how much the tiles moved:
cspad.detector_shifts \ t002_1k_1_refined_experiments_level2.json t002_1k_1_refined_reflections_level2.pickle \ t003_2k_1_refined_experiments_level2.json t003_2k_1_refined_reflections_level2.pickle
The Detector XY column shows the detector moved 5.8 microns in the XY plane, the quadrants on averaged moved 7.9+/- 4.0 microns and the 2x1 sensors moved 10.4+/-10.8 microns. The tile positions moved substantially less between v2 and v3 then they did between v1 and v2.
Evaluate metrology
Iterative hierarchical joint refinement should proceed until convergence, meaning until rounds of indexing and refinement do not improve the model. cxi.display_metrology, dev.dials.plot_detector_shfits, and cspad.detector_shifts measure the magnitude of changes during refinement. An additional tool is available, cspad.detector_statistics, that can be used to evaluate the precision of refinement, and the state of the cspad after refinement.
cspad.detector_statistics tag=t003_2k
Warning, this dumps a lot of output. The program examines the current directory for files from cspad.cbf_metrology. For each hierarchy level 0-3, the program creates three tables of statistics:
1) Detector congruence. The two half datasets from each level of refinement are compared to each other, and agreement between the two independent refinement runs is reported.
2) Detector statistics. Using the two half datasets as independent measurements, statistics about the cspad are reported, such as normal vector tilts and XYZ offsets. Weighted means and standard deviations are reported as well, and can be used to estimate the overall precision of refinement.
3) RMSDs by detector number. For each of the half datasets, overall, radial and transverse RMSDs are shown.
For this walkthrough, we will extract a few statistics only. Look for the third set of tables, delineated with
******************************************************************************** Showing statistics for detector at level 2 (sensors, I.E. 2x1s) ********************************************************************************
In the second table, under Detector statistics, a few of the columns are reproduced here (click expand to show the table):
| Panel group ID | Radial Dist (mm) | Fast Offset sigma (microns) | Slow offset sigma (microns) | N reflections |
|---|---|---|---|---|
| 1 | 21.8 | 2.3 | 0.7 | 7585 |
| 9 | 23.4 | 2 | 1.9 | 7411 |
| 25 | 24.5 | 1.6 | 1.7 | 7211 |
| 17 | 26 | 0.3 | 1.1 | 7182 |
| 0 | 40.7 | 4.4 | 3.4 | 6429 |
| 8 | 41.1 | 0.3 | 1.5 | 6522 |
| 24 | 43.7 | 2.1 | 3.4 | 6733 |
| 16 | 44.3 | 3.1 | 2.5 | 6576 |
| 7 | 55.3 | 0.7 | 2.4 | 6048 |
| 31 | 57.5 | 0.7 | 0.4 | 5774 |
| 15 | 57.8 | 5 | 0.9 | 5763 |
| 23 | 59.7 | 2.6 | 1.9 | 5924 |
| 11 | 71.9 | 3.3 | 12.6 | 3750 |
| 3 | 73 | 20.7 | 4.5 | 3968 |
| 19 | 75.1 | 19.2 | 2.4 | 3614 |
| 6 | 76.3 | 1.3 | 8.3 | 3440 |
| 27 | 76.3 | 1.2 | 3.2 | 3813 |
| 2 | 77.6 | 10.7 | 5.2 | 3323 |
| 10 | 78 | 5.4 | 10.6 | 2964 |
| 30 | 78.4 | 11.3 | 6.4 | 2925 |
| 14 | 78.7 | 9.6 | 7.2 | 2863 |
| 22 | 80.4 | 0.8 | 0.3 | 3039 |
| 18 | 80.9 | 7.8 | 2.2 | 2970 |
| 26 | 81.2 | 4.5 | 12.5 | 3264 |
| 4 | 87.5 | 4.5 | 1.8 | 2193 |
| 12 | 89.6 | 3.4 | 2.7 | 2016 |
| 28 | 90.4 | 6.4 | 7.3 | 2011 |
| 20 | 91.9 | 34.4 | 31.1 | 1897 |
| 5 | 104.8 | 11.6 | 6 | 983 |
| 13 | 106.1 | 6.8 | 4.9 | 750 |
| 29 | 107.7 | 14 | 16.7 | 1028 |
| 21 | 108.5 | 6.2 | 4.5 | 874 |
| All | 4.7 | 3.9 | ||
| Mean | 4088.8 |
Here are the meaning of the columns:
- Panel G ID: here we are looking at the sensors, of which there are 32.
- Dist (mm): distance from the sensor center to the beam center. The table is sorted by this number.
- F Offset sigma (microns): weighted standard deviation of the two measurements of the sensor's fast coordinate.
- S Offset sigma (microns): weighted standard deviation of the two measurements of the sensor's slow coordinate.
- N refls: sum of the number of reflections recorded on the sensor between the two half dataset. Used as a weighting term.
The weighted mean of the fast and slow offset sigmas is 4.7 and 3.9 microns, respectively. This measure of the precision of this refinement indicates it quite precise compared to the pixel size of the detector (110 microns). However, there are still several panels with high fast or slow offset sigmas. That, plus the fact that the number of indexed images increased while using v3 metrology implies that refinement has not converged.
Further indexing and refinement (v4)
Reindexing the data using v3 and re-refining the data to create v4 metrology proceeds as described in the above steps. After generating v4, we evaluate it as described above and see this table:
| PanelG ID | (mm) | F Offset Sigma (microns) | S Offset Sigma (microns) | N refls |
|---|---|---|---|---|
| 1 | 21.8 | 1.6 | 1 | 6892 |
| 9 | 23.4 | 0 | 0.2 | 6878 |
| 25 | 24.5 | 0.5 | 1.1 | 6609 |
| 17 | 26 | 1.4 | 2.6 | 6637 |
| 0 | 40.7 | 4 | 2.1 | 6848 |
| 8 | 41.1 | 0.6 | 2.8 | 6624 |
| 24 | 43.7 | 0.3 | 0.6 | 6652 |
| 16 | 44.3 | 3.5 | 2.4 | 6655 |
| 7 | 55.3 | 0.7 | 4.1 | 6444 |
| 31 | 57.5 | 1.9 | 1.8 | 6538 |
| 15 | 57.8 | 2.5 | 3 | 6552 |
| 23 | 59.7 | 2.5 | 2.8 | 6443 |
| 11 | 71.9 | 1.2 | 13.2 | 6139 |
| 3 | 73 | 3.5 | 0.9 | 6070 |
| 19 | 75.1 | 1.8 | 1.4 | 6065 |
| 6 | 76.3 | 1.5 | 1.1 | 6118 |
| 27 | 76.3 | 0.8 | 2.9 | 6010 |
| 2 | 77.6 | 17.5 | 4.5 | 5917 |
| 10 | 78 | 2 | 1 | 5823 |
| 30 | 78.5 | 3.4 | 1.2 | 5885 |
| 14 | 78.7 | 10.4 | 0.5 | 5974 |
| 22 | 80.4 | 2.1 | 8.8 | 5867 |
| 18 | 80.9 | 3.6 | 6.5 | 5943 |
| 26 | 81.2 | 1.7 | 5.2 | 5891 |
| 4 | 87.5 | 17.1 | 19.6 | 5570 |
| 12 | 89.6 | 0.1 | 3.4 | 5514 |
| 28 | 90.4 | 4.4 | 3.7 | 5494 |
| 20 | 91.9 | 1.9 | 1 | 5417 |
| 5 | 104.7 | 8.4 | 0.9 | 5016 |
| 13 | 106.1 | 17.4 | 27.7 | 4957 |
| 29 | 107.7 | 10.8 | 14.6 | 4980 |
| 21 | 108.5 | 10.7 | 6.9 | 4855 |
| All | 4.1 | 4.4 | ||
| Mean | 6039.9 |
The fast and slow offset sigmas are lower generally, but not substantially. 5509 images were indexed, a small increase of 0.5% over v3 metrology. Further, the change in tile position is minimal, as shown by cspad.detector_shifts (delta XY movements (microns): detector: 4.7, quadrants: 2.4+/1.3, sensors: 7.8+/-7.2). The metrology has likely converged.
Deploy metrology
v4 metrology is now ready for use. Deploy it or use it directly for indexing as described in indexing using v2 metrology.