Difference between revisions of "Cxi02416 calibration"

From cctbx_xfel
Jump to: navigation, search
m (Initial indexing)
(Refine metrology (v3))
 
(One intermediate revision by the same user not shown)
Line 143: Line 143:
 
     mp.nproc=36 mp.queue=psanaq output.split_logs=True \
 
     mp.nproc=36 mp.queue=psanaq output.split_logs=True \
 
     input.dispatcher=cctbx.xfel.xtc_process \
 
     input.dispatcher=cctbx.xfel.xtc_process \
     input.target=cxi02416-lyso-t000.phil input.trial=0 \
+
     input.target=[[cxi02416-lyso-t000.phil]] input.trial=0 \
 
     input.run_num=$i; done
 
     input.run_num=$i; done
  
Line 155: Line 155:
 
Let's call the metrology deployed by the beamline operator version 0 (v0).  After quadrant alignment, the operators updated the metrology to version 1 (v1). The following command will do an iterative joint hierarchical refinement of the components of the CSPAD detector.  The new tile positions we call version 2 (v2).
 
Let's call the metrology deployed by the beamline operator version 0 (v0).  After quadrant alignment, the operators updated the metrology to version 1 (v1). The following command will do an iterative joint hierarchical refinement of the components of the CSPAD detector.  The new tile positions we call version 2 (v2).
  
   bsub -q psanaq -o t002_1k.out '''cspad.cbf_metrology''' tag=t002_1k \
+
   bsub -q psanaq -o t000_1k.out '''cspad.cbf_metrology''' tag=t000_1k \
     cxi02416-refine.phil reflections=indexed ../r0*/002/out \
+
     reflections=indexed ../r0*/000/out refine_distance=True \
 
     n_subset=1000 split_dataset=True
 
     n_subset=1000 split_dataset=True
  
Line 163: Line 163:
 
Details about the parameters used:
 
Details about the parameters used:
  
* tag=t002_1k: the output files will be named t002_1k*
+
* tag=t000_1k: the output files will be named t000_1k*
* cxi02416-refine.phil: [[cxi02416-refine.phil|this phil file]] contains parameters for joint hierarchical refinement.
+
 
* reflections=indexed: the indexing process produces two sets of reflection files, bright indexed reflections, and final integrated reflections (can include weak intensities).  Here, I've chosen to refine only against the bright, indexed reflections.
 
* reflections=indexed: the indexing process produces two sets of reflection files, bright indexed reflections, and final integrated reflections (can include weak intensities).  Here, I've chosen to refine only against the bright, indexed reflections.
 +
* refine_distance=True: as the detector distance is not fully known yet, allow the detector distance to refine. In subsequent rounds of metrology refinement, we will fix the distance to a constant value.
 
* n_subset=1000: pick 1000 images at random to refine
 
* n_subset=1000: pick 1000 images at random to refine
 
* split_dataset=True: the refinement is done twice independently, using odd numbered or even numbered images, each time using n_subset images.  This will be useful later for evaluating the accuracy of the metrology.
 
* split_dataset=True: the refinement is done twice independently, using odd numbered or even numbered images, each time using n_subset images.  This will be useful later for evaluating the accuracy of the metrology.
Line 176: Line 176:
  
 
Compare it to the refined geometry:
 
Compare it to the refined geometry:
   '''cxi.display_metrology''' 0-end.data.t002_1k_1
+
   '''cxi.display_metrology''' 0-end.data.t000_1k_1
  
 
You will see a small change in the origin (center arrow) and obvious changes in the quadrant positions.  The relative positions of the sensors to each other will not change a large amount, though if you inspect the files themselves you will see changes.
 
You will see a small change in the origin (center arrow) and obvious changes in the quadrant positions.  The relative positions of the sensors to each other will not change a large amount, though if you inspect the files themselves you will see changes.
Line 182: Line 182:
 
Another tool that is useful for evaluating changes in metrology is '''dev.dials.plot_detector_shifts'''.  This program will plot detector shifts in the X and Y directions (also known as the fast and slow directions, a convention referring to how data is read from the byte arrays stored on disk).  It will also plot shifts along the Z axis, I.E., the detector's normal axis.  Let's use it to compare the unrefined geometry to the geometry refined at level 2 (I.E. at the level of the 2x1 sensors):
 
Another tool that is useful for evaluating changes in metrology is '''dev.dials.plot_detector_shifts'''.  This program will plot detector shifts in the X and Y directions (also known as the fast and slow directions, a convention referring to how data is read from the byte arrays stored on disk).  It will also plot shifts along the Z axis, I.E., the detector's normal axis.  Let's use it to compare the unrefined geometry to the geometry refined at level 2 (I.E. at the level of the 2x1 sensors):
  
   '''dev.dials.plot_detector_shifts''' t002_1k_1_combined_experiments.json \
+
   '''dev.dials.plot_detector_shifts''' t000_1k_1_combined_experiments.json \
     t002_1k_1_refined_experiments_level2.json \
+
     t000_1k_1_refined_experiments_level2.json \
 
     plot_type=spherical_polar tag=v1v2level2
 
     plot_type=spherical_polar tag=v1v2level2
 
   display v1v2level2*.png
 
   display v1v2level2*.png
Line 189: Line 189:
 
Use the spacebar to cycle between the images.  The pixels are mapped to the Ewald sphere and then displayed as a function of azimuthal and elevation angles along the Ewald sphere, which distorts the detector appearance but is independent of any detector layout.  The colorbar scales will show the magnitude of panel shifts.
 
Use the spacebar to cycle between the images.  The pixels are mapped to the Ewald sphere and then displayed as a function of azimuthal and elevation angles along the Ewald sphere, which distorts the detector appearance but is independent of any detector layout.  The colorbar scales will show the magnitude of panel shifts.
  
Try running the program using t002_1k_1_refined_experiments_level0.json to see how much the detector as a whole shifted, or t002_1k_1_refined_experiments_level1.json to see shifts in the quadrants.
+
Try running the program using t000_1k_1_refined_experiments_level0.json to see how much the detector as a whole shifted, or t000_1k_1_refined_experiments_level1.json to see shifts in the quadrants.
  
 
Finally, the program '''cspad.detector_shifts''' shows the magnitude of shifts between a reference and a moving detector in tabular form:
 
Finally, the program '''cspad.detector_shifts''' shows the magnitude of shifts between a reference and a moving detector in tabular form:
  
 
   '''cspad.detector_shifts''' \
 
   '''cspad.detector_shifts''' \
     t002_1k_1_combined_experiments.json t002_1k_1_combined_reflections.pickle \
+
     t000_1k_1_combined_experiments.json t000_1k_1_combined_reflections.pickle \
     t002_1k_1_refined_experiments_level2.json t002_1k_1_refined_reflections_level2.pickle
+
     t000_1k_1_refined_experiments_level2.json t000_1k_1_refined_reflections_level2.pickle
  
Looking at the Delta XY columns, we see the detector moved 2.6 microns in the XY plane, the quadrants moved on average 223.5+/-52.0 microns and the 2x1 sensors moved 31.7+/-25.1 microns.  Note how most of the change is in the quadrant locations, which is expected as these were done by eye.  The 2x1 sensor positions, determined using optical microscopy, changed by an order of magnitude less than the quadrants.  Also, the same program ran against the other half of the split dataset (t002_1k_2_*) will reveal very similar shifts, indicating these shifts are not due to random chance.
+
Looking at the Delta XY columns, we see the detector moved 4.6 microns in the XY plane, the quadrants moved on average 219.8+/-57.9 microns and the 2x1 sensors moved 17.6+/-16.1 microns.  Note how most of the change is in the quadrant locations, which is expected as these were done by eye.  The 2x1 sensor positions, determined using optical microscopy, changed by an order of magnitude less than the quadrants.  Also, the same program ran against the other half of the split dataset (t000_1k_2_*) will reveal very similar shifts, indicating these shifts are not due to random chance.
  
 
Full output for '''cspad.detector_shifts''':
 
Full output for '''cspad.detector_shifts''':
Line 209: Line 209:
 
       ID        (mm)  (microns) (microns) (microns) (microns)  (deg)  (deg)    (deg)    (deg)
 
       ID        (mm)  (microns) (microns) (microns) (microns)  (deg)  (deg)    (deg)    (deg)
 
-----------------------------------------------------------------------------------------------------------
 
-----------------------------------------------------------------------------------------------------------
         0          2.4      2.6      23.9       0.0      0.0  -0.0262 -0.0045   0.0266   0.0000   35366
+
         0          2.4      4.6       3.5     -2.9     -74.0  -0.0003 -0.0213   0.0213   0.0003   29527
  Weighted mean              2.6      23.9       0.0     -5.3 -0.0262 -0.0045   0.0266   0.0000
+
  Weighted mean              4.6       3.5     -2.9     -74.0  -0.0003 -0.0213   0.0213   0.0003
 
Weighted stddev              0.0      0.0      0.0      0.0  0.0000  0.0000    0.0000  0.0000
 
Weighted stddev              0.0      0.0      0.0      0.0  0.0000  0.0000    0.0000  0.0000
 
-----------------------------------------------------------------------------------------------------------
 
-----------------------------------------------------------------------------------------------------------
Line 218: Line 218:
 
       ID        (mm)  (microns) (microns) (microns) (microns)  (deg)  (deg)    (deg)    (deg)
 
       ID        (mm)  (microns) (microns) (microns) (microns)  (deg)  (deg)    (deg)    (deg)
 
-----------------------------------------------------------------------------------------------------------
 
-----------------------------------------------------------------------------------------------------------
         0          62.9    166.8   -179.3    -91.5    -91.5 -0.0796  0.0429   0.0729   0.0000     9500
+
         0          62.9    157.7   -153.5    -36.0    1236.4 -0.0308 -0.0658   0.0550   0.0111     7766
         1          64.3    244.0   -238.1      98.3      98.3 -0.1394  0.0577   0.1368   0.0000     8388
+
         1          64.3    262.5   -172.9    -197.5    1532.9 -0.0042 -0.1422   0.1411   0.0260     7112
         3          66.0    205.4    -187.9     55.9      55.9 -0.0969 -0.0916   0.1224   0.0000     9244
+
         3          66.0    190.2     22.6    188.8    1347.7 -0.0945 -0.0391   0.1228   0.1139     7811
         2          67.0    288.3    -222.8    -142.5   -142.5   0.0529 0.0676   0.0669   0.0000     8234
+
         2          67.0    279.6    277.7      32.1   1374.9   0.0216 0.0397   0.0305   0.0169     6838
  Weighted mean            223.5    -205.6     -19.9   1288.1 -0.0675  0.0170   0.0996   0.0000
+
  Weighted mean            219.8     -11.7      0.3   1369.3 -0.0291 -0.0527   0.0880   0.0432
Weighted stddev            52.0      28.0     113.8     119.3   0.0806 0.0753   0.0348   0.0000
+
Weighted stddev            57.9    204.3     161.0     122.4   0.0500 0.0731   0.0524   0.0494
 
-----------------------------------------------------------------------------------------------------------
 
-----------------------------------------------------------------------------------------------------------
 
Hierarchy level 2 Detector shifts
 
Hierarchy level 2 Detector shifts
Line 230: Line 230:
 
       ID        (mm)  (microns) (microns) (microns) (microns)  (deg)  (deg)    (deg)    (deg)
 
       ID        (mm)  (microns) (microns) (microns) (microns)  (deg)  (deg)    (deg)    (deg)
 
-----------------------------------------------------------------------------------------------------------
 
-----------------------------------------------------------------------------------------------------------
         1          21.9      3.1      16.1      -6.7      -6.7 -0.1707 -0.1239   0.2046   0.0001     2051
+
         1          21.9      2.7      0.1      -2.7      67.0 -0.2345 -0.0794   0.1925   0.0227     1817
         9          23.6     28.4      7.0     18.7     18.7   0.0587 0.3254   0.3081   0.0001     1752
+
         9          23.6      2.5     -0.3      2.4    -34.8   0.0229 0.1911   0.2632   0.0290     1578
       25          24.7     37.7     -36.9     -58.3    -58.3 -0.2511 -0.6087   0.5523   0.0006     1897
+
       25          24.7     7.9     -2.6      7.4      34.8 -0.0817 -0.1288   0.2195   0.1031     1644
       17          26.3     15.3     -6.3     20.9     20.9   0.0621 0.2271   0.1694   0.0000     1645
+
       17          26.3     5.8     -3.8     -4.4    -72.0   0.0402 0.0412   0.0588   0.0107     1357
         0          40.8    18.2      1.9      3.9      3.9   0.0204 -0.2756   0.3172   0.0006     1790
+
         0          40.8    23.0    -23.0      -0.3     44.5   0.0064 -0.3016   0.3031   0.0169     1626
         8          41.3     11.5     -1.0      2.7      2.7   0.1900  0.1839   0.3758   0.0011     1926
+
         8          41.3     5.8     -4.7      3.6    -44.8   0.2528 -0.0850   0.3773   0.0512     1687
       24          43.9    16.8      5.3    -47.4    -47.4 -0.2186 -0.4251   0.3455   0.0006     2024
+
       24          43.9    10.2      -0.4    -10.3      36.9 -0.1491 -0.2116   0.2813   0.0628     1665
       16          44.5    26.1     18.6      14.3      14.3   0.1773 0.2270   0.2060   0.0000     1733
+
       16          44.5    17.8     17.1      4.9    -13.0   0.0640 0.1521   0.1525   0.0178     1499
         7          55.5    15.4      -0.6      -6.6      -6.6 -0.1542 0.1246   0.1117   0.0002     1928
+
         7          55.5    12.6      0.4    -12.6      52.8 -0.1670 0.0284   0.1042   0.0097     1382
       31          57.7    23.0    -16.4      -2.7     -2.0.0358 -0.2582   0.2656   0.0004     1598
+
       31          57.7    28.9      -7.5    -27.9     81.9  -0.0803  0.0442   0.0742   0.1336     1406
       15          58.0    26.5      20.6       0.9      0.-0.0135 0.1436   0.0989   0.0001     1491
+
       15          58.0    13.5      -6.2      12.0    -52.4  0.0155 0.1185   0.1853   0.0004     1323
       23          60.0    19.6      -7.8     25.2      25.2  -0.0851  0.0755   0.1639   0.0002     1838
+
       23          60.0    16.8    -16.1      4.8     -17.2  -0.0080 -0.0531   0.0518   0.0357     1560
       11          72.2     68.7    -64.6     -18.8     -18.8 -0.6625  0.1031   0.5312   0.0001    1017
+
       11          72.2     4.4      -1.6     -4.2      8.5 -0.5439 -0.0891   0.4503   0.0258      843
         3          73.1    66.6    -47.0      -9.2     -9.2 -0.3160 0.1041   0.2524   0.0005     917
+
         3          73.1    22.5      17.6    -13.9      22.6 -0.2173 0.0156   0.1600   0.0165     856
       19          75.3     62.9    -67.9    -12.9     -12.9  -0.0452 0.1673   0.0967   0.0001     672
+
       19          75.3     2.1      -0.5      2.0     -17.0.0061 0.1337   0.1232   0.0245     584
         6          76.5    13.2     23.5     -9.4     -9.4   0.0184 -0.1715   0.2574   0.0006    1022
+
         6          76.5    11.2      -4.4     -10.3      14.9   0.0371 -0.1290   0.1975   0.0252      730
       27          76.5    41.8       7.0      11.9     11.9 -0.0461 -0.2238   0.0953   0.0002    1114
+
       27          76.5    59.0       6.4    -58.6     93.7 -0.0486  0.0859   0.0744   0.0859      851
         2          77.7    40.9     -21.9     -6.7      -6.7 -0.2080 -0.0272   0.1273   0.0000     914
+
         2          77.7    45.1      1.6     -45.2     71.3 -0.1845  0.0036   0.1143   0.0259     772
       10          78.3    30.7    -26.7     -21.8    -21.8 -0.3839  0.1773   0.2850   0.0001     908
+
       10          78.3    19.5      -3.5     -19.1      24.1 -0.4560 -0.0550   0.3362   0.0117     651
       30          78.6    61.4      1.4      34.1     34.1   0.1073 -0.2332   0.3137   0.0006     737
+
       30          78.6    17.4      -3.3     17.2    -53.8   0.0098 -0.1055   0.2080   0.0894     651
       14          78.9     27.4     20.4     -15.4    -15.4   0.1525 0.2232   0.2635   0.0001     605
+
       14          78.9     4.1     -4.1      -0.5      14.5   0.0433 0.1315   0.2032   0.0036     504
       22          80.6    28.1       4.5     -20.9    -20.9   0.0905 0.0922   0.0623   0.0001     923
+
       22          80.6    22.0      21.9       2.1     -17.6   0.0394 0.0027   0.0347   0.0234     696
       18          81.2    39.7    -40.6    -16.3     -16.3  -0.0614  0.1521   0.1167   0.0002     739
+
       18          81.2    31.8      9.1      30.5     -80.0.0141 -0.0178   0.0374   0.0487     620
       26          81.3   126.2     117.0     19.3     19.0.1011 -0.1083   0.1667   0.0003    1027
+
       26          81.3    25.6     -8.1    -24.2     68.4  -0.0185  0.0909   0.0750   0.1275      819
         4          87.6    26.3      -9.7     -4.4     -4.4 -0.2709 0.0775   0.1987   0.0003     608
+
         4          87.6    32.3      31.6     -5.7     53.6 -0.2149 0.0391   0.1460   0.0154     431
       12          89.8    25.2      22.6     -4.7      -4.0.0539 0.0448   0.1875   0.0007     527
+
       12          89.8    25.3      6.9    -24.2    -21.1  -0.0419 0.0008   0.1011   0.0173     406
       28          90.7    63.4    -17.3     34.0     34.0 -0.3379 0.0477   0.2630   0.0006      577
+
       28          90.7    37.7      36.9      3.7     79.9 -0.3678 0.0389   0.3173   0.0006      529
       20          92.1    67.6     52.5    -31.5    -31.5   0.0800  0.0600   0.0192   0.0000     496
+
       20          92.1    24.0      2.7     23.9      16.6   0.0213 -0.1148   0.1014   0.0777     374
         5        104.9    14.6     31.0      -9.8      -9.8 -0.5151 0.0433   0.4343   0.0026     270
+
         5        104.9    92.9    -91.3      6.3    -133.6 -0.6185 0.2555   0.6003   0.0674     152
       13        106.3     36.9     26.6    -33.0     -33.0 -0.0042  0.0441   0.1372   0.0003     162
+
       13        106.3     9.9       9.9      0.4      4.2 -0.1141 -0.0277   0.0259   0.0031     120
       29        108.0    105.5    -46.3     67.5      67.5 -0.2877 0.0256   0.2276   0.0001     270
+
       29        108.0    101.5    101.1     -1.7    122.2 -0.3588 0.1173   0.2946   0.0482     246
       21        108.7    59.5      42.9     -30.9    -30.9   0.1146 0.0449   0.0680   0.0000     188
+
       21        108.7    16.4     -13.1      9.9    -17.3   0.1983 0.0814   0.1786   0.0392     148
  Weighted mean              31.7     -0.9     -3.9     18.3 -0.0710 -0.0185   0.2433   0.0003
+
  Weighted mean              17.6     -0.1     -4.7     12.6 -0.0687 -0.0164   0.1883   0.0415
Weighted stddev            25.1      32.1     24.0     84.4  0.1932 0.2486   0.1307   0.0004
+
Weighted stddev            16.1      16.5     16.8     51.4  0.1702 0.1286   0.1140   0.0381
 
-----------------------------------------------------------------------------------------------------------
 
-----------------------------------------------------------------------------------------------------------
 
Hierarchy level 3 Detector shifts
 
Hierarchy level 3 Detector shifts
Line 270: Line 270:
 
       ID        (mm)  (microns) (microns) (microns) (microns)  (deg)  (deg)    (deg)    (deg)
 
       ID        (mm)  (microns) (microns) (microns) (microns)  (deg)  (deg)    (deg)    (deg)
 
-----------------------------------------------------------------------------------------------------------
 
-----------------------------------------------------------------------------------------------------------
         2          15.8     0.0      0.0       0.1       0.1 -0.1048 -0.1830   0.0000  0.0001    1046
+
         2          15.9     0.0      0.0     -0.0       0.0 -0.0669 -0.2384   0.0000  0.0227      941
       18          16.2      0.0       0.0      0.1       0.1 -0.0812 0.3206   0.0000  0.0001     929
+
       18          16.2      0.0     -0.0      0.0       0.0 -0.1455 0.1261   0.0000  0.0290     885
       50          19.2      0.0       0.3       0.4       0.4   0.0254 -0.6580   0.0000  0.0006     974
+
       50          19.2      0.0     -0.0       0.0       0.0   0.0606 -0.1400   0.0000  0.1031     865
       34          19.3      0.0       0.0       0.1       0.1 -0.0389 0.2322   0.0000  0.0000     740
+
       34          19.3      0.0     -0.0     -0.0       0.0 -0.0121 0.0563   0.0000  0.0107     578
         3          30.7      0.0     -0.1      -0.0     -0.0  -0.1936 -0.0835   0.0000  0.0001    1005
+
         3          30.7      0.0       0.0      0.0       0.0  -0.2095 -0.1320   0.0000  0.0227      876
       19          33.0      0.0     -0.1      -0.0     -0.0   0.1283 0.3048   0.0000  0.0001     823
+
       19          33.0      0.0       0.0      0.0       0.0 -0.0226 0.1912   0.0000  0.0290     693
       51          33.0      0.0     -0.5      -0.1      -0.1 -0.3796 -0.5381   0.0000  0.0006     923
+
       51          33.0      0.0       0.0      0.0      0.0 -0.0494 -0.1443   0.0000  0.1031     779
       35          35.2      0.0     -0.1      -0.1      -0.1   0.1108 0.2077   0.0000  0.0000     905
+
       35          35.2      0.0       0.0      0.0      0.0   0.0293 0.0496   0.0000  0.0107     779
       16          37.6      0.0      -0.2       0.1       0.1   0.1378 0.2257   0.0000  0.0011     947
+
       16          37.6      0.0      -0.0       0.0       0.0   0.2624 0.0480   0.0000  0.0512     866
         0          37.9      0.0      0.1      0.0      0.0  0.0885 -0.2618   0.0000  0.0006     853
+
         0          37.9      0.0      0.0      -0.0      0.0  0.1518 -0.2606   0.0000  0.0169     775
       32          40.8      0.0      0.0       0.2       0.0.1146 0.2643   0.0000  0.0000     731
+
       32          40.8      0.0      0.0     -0.0       0.0  -0.0180 0.1640   0.0000  0.0178     583
       48          41.1      0.0      -0.1       0.3       0.3 -0.1043 -0.4665   0.0000  0.0006     977
+
       48          41.1      0.0      -0.0       0.0       0.0 -0.0272 -0.2575   0.0000  0.0628     863
         1          46.2      0.0      -0.1       0.0      0.0 -0.0361 -0.2740   0.0000  0.0006     937
+
         1          46.2      0.0      -0.0       0.0      0.0   0.0304 -0.3001   0.0000  0.0169     851
       17          47.3      0.0      0.1      -0.2      -0.2   0.2236  0.1412   0.0000  0.0011     979
+
       17          47.3      0.0      0.0      0.0      0.0   0.2587 -0.0648   0.0000  0.0512     821
       49          49.1      0.0     -0.0     -0.3      -0.3 -0.3012 -0.3712   0.0000  0.0006    1047
+
       49          49.1      0.0       0.0       0.0      0.0 -0.1319 -0.2228   0.0000  0.0628      802
       33          50.4      0.0      -0.1     -0.1      -0.1   0.2201 0.1859   0.0000  0.0000    1002
+
       33          50.4      0.0      -0.0     -0.0      0.0   0.0517 0.1567   0.0000  0.0178      916
       15          51.3      0.0      0.1      -0.1      -0.1 -0.1295 0.1501   0.0000  0.0002    1044
+
       15          51.3      0.0      -0.0      0.0      0.0  -0.1644 0.0407   0.0000  0.0097      740
       63          52.9      0.0      -0.0     -0.1      -0.1 -0.0109 -0.2605   0.0000  0.0004     814
+
       63          52.9      0.0      -0.0       0.0      0.0 -0.0768  0.0501   0.0000  0.1336     732
       31          54.2      0.0      0.0     -0.0     -0.0  0.0123 0.1437   0.0000  0.0001     728
+
       31          54.2      0.0      0.0       0.0       0.0  0.0243 0.1170   0.0000  0.0004     667
       47          55.7      0.0     -0.0      -0.1      -0.1 -0.0701  0.0896   0.0000  0.0002     966
+
       47          55.7      0.0       0.0      -0.0      0.0 -0.0119 -0.0523   0.0000  0.0357     885
       14          61.4      0.0      -0.1       0.1       0.1 -0.1712  0.1001   0.0000  0.0002     884
+
       14          61.4      0.0      -0.0       0.0       0.0 -0.1694 -0.0022   0.0000  0.0097     642
       23          61.5      0.0       0.1     -0.0      -0.0  -0.6607  0.1139   0.0000  0.0001     790
+
       23          61.5      0.0      -0.0      -0.0      0.0  -0.5509 -0.0155   0.0000  0.0258     690
         7          62.3      0.0       0.1       0.1       0.1 -0.3144 0.1086   0.0000  0.0005     686
+
         7          62.3      0.0     -0.0       0.0       0.0 -0.2132 0.0445   0.0000  0.0165     626
       30          63.5      0.0      -0.0      0.0      0.0  -0.0349 0.1399   0.0000  0.0001     763
+
       30          63.5      0.0      -0.0      0.0      0.0  -0.0061 0.1193   0.0000  0.0004     656
       62          64.0      0.0       0.0       0.1       0.0.0742 -0.2499   0.0000  0.0004     784
+
       62          64.0      0.0     -0.0     -0.0       0.0 -0.0869  0.0291   0.0000  0.1336     674
       39          64.6      0.0       0.0      -0.0     -0.0 -0.0428 0.1679   0.0000  0.0001     494
+
       39          64.6      0.0     -0.0      -0.0       0.0   0.0240 0.1317   0.0000  0.0245     435
       55          65.7      0.0       0.0      0.0      0.0  -0.0493 -0.2231   0.0000  0.0002     779
+
       55          65.7      0.0     -0.0      0.0      0.0  -0.0367  0.0916   0.0000  0.0859     631
       46          65.8      0.0      -0.0      0.1      0.-0.0955  0.0618   0.0000  0.0002     872
+
       46          65.8      0.0      0.0      -0.0      0.0.0017 -0.0537   0.0000  0.0357     675
         5          67.7      0.0       0.1     -0.0     -0.0  -0.2092 -0.0153   0.0000  0.0000     674
+
         5          67.7      0.0      -0.0       0.0      0.0  -0.1829  0.0246   0.0000  0.0259     591
       21          68.5      0.0      -0.0     -0.0     -0.0  -0.3729  0.1994   0.0000  0.0001     712
+
       21          68.5      0.0      -0.0       0.0       0.0  -0.4593 -0.0032   0.0000  0.0117     519
       53          71.2      0.0      0.0      -0.1      -0.0.0948 -0.1139   0.0000  0.0003     726
+
       53          71.2      0.0      -0.0      0.0      0.-0.0081  0.0924   0.0000  0.1275     605
       37          71.4      0.0      0.0      -0.1      -0.1  -0.0525  0.1554   0.0000  0.0002     545
+
       37          71.4      0.0      0.0      -0.0      0.0.0120 -0.0193   0.0000  0.0487     447
       13          73.5      0.0     -0.1     -0.0     -0.0 -0.0048 -0.1724   0.0000  0.0006     595
+
       13          73.5      0.0       0.0     -0.0       0.0   0.0317 -0.1305   0.0000  0.0252     447
       61          75.2      0.0     -0.1      -0.1      -0.1   0.0751 -0.2455   0.0000  0.0006     439
+
       61          75.2      0.0       0.0      0.0      0.0   0.0054 -0.1058   0.0000  0.0894     360
       29          76.1      0.0       0.0     -0.0     -0.0  0.1812 0.2006   0.0000  0.0001     326
+
       29          76.1      0.0     -0.0       0.0       0.0  0.0488 0.1296   0.0000  0.0036     264
       45          77.5      0.0     -0.0     -0.0     -0.0  0.1022 0.0791   0.0000  0.0001     567
+
       45          77.5      0.0       0.0       0.0       0.0  0.0395 0.0011   0.0000  0.0234     440
         9          80.3      0.0     -0.0      0.1       0.1 -0.2769 0.0524   0.0000  0.0003     403
+
         9          80.3      0.0       0.0      0.0       0.0 -0.2107 0.0576   0.0000  0.0154     296
       12          80.8      0.0      0.1       0.0      0.0  0.0392 -0.1679   0.0000  0.0006     427
+
       12          80.8      0.0      0.0       0.0      0.0  0.0586 -0.1208   0.0000  0.0252     283
       25          82.2      0.0      -0.1      0.1       0.0.0496 0.0496   0.0000  0.0007     374
+
       25          82.2      0.0      0.0      -0.0      0.0  -0.0416 0.0044   0.0000  0.0173     270
       28          82.9      0.0      -0.0       0.0      0.0  0.1240 0.2403   0.0000  0.0001     279
+
       28          82.9      0.0      -0.0     -0.0      0.0  0.0201 0.1370   0.0000  0.0036     240
       22          83.0      0.0     -0.1       0.0      0.0  -0.6637  0.0950   0.0000  0.0001     227
+
       22          83.0      0.0       0.0       0.0      0.0  -0.5494 -0.0436   0.0000  0.0258     153
       60          83.3      0.0      0.1       0.1       0.1   0.1350 -0.2184   0.0000  0.0006     298
+
       60          83.3      0.0      0.0       0.0       0.0   0.0276 -0.1023   0.0000  0.0894     291
       57          83.5      0.0      -0.2      0.0      0.0  -0.3409 0.0165   0.0000  0.0006      406
+
       57          83.5      0.0      0.0      -0.0      0.0  -0.3631 0.0704   0.0000  0.0006      365
         6          83.9      0.0      -0.1     -0.1      -0.1 -0.3171 0.1006   0.0000  0.0005     231
+
         6          83.9      0.0      -0.0     -0.0      0.0 -0.2152 0.0340   0.0000  0.0165     230
       41          84.6      0.0      0.0      0.0      0.0  0.0742  0.0671   0.0000  0.0000     352
+
       41          84.6      0.0      0.0      0.0      0.0  0.0112 -0.1162   0.0000  0.0777     259
       44          85.0      0.0       0.0       0.1       0.1   0.0784 0.1027   0.0000  0.0001     356
+
       44          85.0      0.0     -0.0     -0.0       0.0   0.0383 0.0095   0.0000  0.0234     256
       38          86.1      0.0      -0.0       0.0      0.0 -0.0471 0.1668   0.0000  0.0001     178
+
       38          86.1      0.0      -0.0     -0.0      0.0   0.0174 0.1327   0.0000  0.0245     149
       54          87.3      0.0      -0.0      -0.0     -0.0  -0.0437 -0.2243   0.0000  0.0002     335
+
       54          87.3      0.0      -0.0      -0.0       0.0  -0.0411  0.0897   0.0000  0.0859     220
         4          88.0      0.0      -0.1      0.0      0.0  -0.2065 -0.0364   0.0000  0.0000     240
+
         4          88.0      0.0      -0.0      -0.0      0.0  -0.1844  0.0064   0.0000  0.0259     181
       20          88.3      0.0       0.0       0.0      0.0  -0.3915  0.1598   0.0000  0.0001     196
+
       20          88.3      0.0     -0.0     -0.0      0.0  -0.4567 -0.0489   0.0000  0.0117     132
       36          91.3      0.0     -0.0      0.0      0.0 -0.0681  0.1492   0.0000  0.0002     194
+
       36          91.3      0.0       0.0      0.0      0.0   0.0139 -0.0180   0.0000  0.0487     173
       52          91.5      0.0      -0.0       0.1       0.0.1058 -0.1037   0.0000  0.0003     301
+
       52          91.5      0.0      -0.0     -0.0       0.0 -0.0171  0.0912   0.0000  0.1275     214
         8          95.7      0.0      0.0     -0.1      -0.1 -0.2641 0.0981   0.0000  0.0003     205
+
         8          95.7      0.0      0.0       0.0      0.0 -0.2010 0.0856   0.0000  0.0154     135
       24          98.1      0.0       0.0     -0.1      -0.0.0572 0.0406   0.0000  0.0007     153
+
       24          98.1      0.0     -0.0       0.0      0.0  -0.0407 0.0099   0.0000  0.0173     136
       56          98.5      0.0      0.1      -0.1      -0.1 -0.3333 0.0736   0.0000  0.0006      171
+
       56          98.5      0.0      0.0      0.0      0.0 -0.3501 0.1192   0.0000  0.0006      164
       11          98.8      0.0      -0.5       0.2       0.2 -0.5170 -0.0010   0.0000  0.0026     161
+
       11          98.8      0.0      -0.0       0.0       0.0 -0.6019  0.2924   0.0000  0.0674     105
       27        100.0      0.0      -0.0      0.1      0.1 -0.0079  0.0436   0.0000  0.0003      111
+
       27        100.0      0.0      0.0      -0.0      0.0  -0.1156 -0.0207   0.0000  0.0031      77
       40        100.2      0.0      -0.0     -0.0     -0.0   0.0844  0.0537   0.0000  0.0000     144
+
       40        100.2      0.0      -0.0       0.0       0.0 -0.0046 -0.1166   0.0000  0.0777     115
       59        102.0      0.0      -0.0       0.0      0.0  -0.2889 0.0009   0.0000  0.0001     182
+
       59        102.0      0.0      -0.0     -0.0      0.0  -0.3510 0.1389   0.0000  0.0482     164
       43        102.4      0.0      0.0       0.0      0.0  0.1103 0.0546   0.0000  0.0000      117
+
       43        102.4      0.0      0.0     -0.0      0.0  0.2029 0.0691   0.0000  0.0392      98
       10        111.6      0.0      0.4      -0.3      -0.3 -0.5104 0.0823   0.0000  0.0026      109
+
       10        111.6      0.0      0.0      0.0      0.0 -0.5535 0.3760   0.0000  0.0674      47
       26        113.4      0.0       0.0     -0.1      -0.1 -0.0008  0.0443   0.0000  0.0003       51
+
       26        113.4      0.0     -0.0       0.0      0.0 -0.1173 -0.0040   0.0000  0.0031       43
       58        114.7      0.0      0.0      -0.0     -0.0  -0.2850 0.0473   0.0000  0.0001       88
+
       58        114.7      0.0      0.0      -0.0       0.0  -0.3274 0.1879   0.0000  0.0482       82
       42        115.7      0.0     -0.0      -0.0     -0.0  0.1176 0.0361   0.0000  0.0000       71
+
       42        115.7      0.0       0.0      -0.0       0.0  0.2108 0.0392   0.0000  0.0392       50
  Weighted mean              0.0      -0.0      0.0      0.0  -0.0678 -0.0189   0.0000  0.0003
+
  Weighted mean              0.0      -0.0      0.0      0.0  -0.0571 -0.0152   0.0000  0.0415
Weighted stddev              0.0      0.1       0.1       0.0  0.1950 0.2441   0.0000  0.0004
+
Weighted stddev              0.0      0.0       0.0       0.0  0.1674 0.1345   0.0000  0.0377
 
-----------------------------------------------------------------------------------------------------------
 
-----------------------------------------------------------------------------------------------------------
 
Detector shifts summary
 
Detector shifts summary
Line 343: Line 343:
 
           (microns) (microns) (microns) (microns) (microns) (microns) (microns) (microns)  (deg)  (deg)  (deg)  (deg)    (deg)      (deg)    (deg)  (deg)
 
           (microns) (microns) (microns) (microns) (microns) (microns) (microns) (microns)  (deg)  (deg)  (deg)  (deg)    (deg)      (deg)    (deg)  (deg)
 
---------------------------------------------------------------------------------------------------------------------------------------------------------------
 
---------------------------------------------------------------------------------------------------------------------------------------------------------------
     0          2.6      0.0     23.9       0.0       0.0      0.0     -5.3       0.0  -0.0262 0.0000 -0.0045 0.0000    0.0266     0.0000  0.0000 0.0000
+
     0          4.6      0.0      3.5       0.0     -2.9       0.0     -74.0       0.0  -0.0003 0.0000 -0.0213 0.0000    0.0213     0.0000  0.0003 0.0000
     1        223.5     52.0    -205.6      28.0     -19.9     113.8   1288.1     119.3 -0.0675 0.0806  0.0170 0.0753   0.0996     0.0348   0.0000 0.0000
+
     1        219.8     57.9    -11.7    204.3      0.3     161.0   1369.3     122.4 -0.0291 0.0500 -0.0527 0.0731   0.0880     0.0524   0.0432 0.0494
     2        31.7     25.1      -0.9     32.1     -3.9     24.0     18.3     84.4  -0.0710 0.1932 -0.0185 0.2486   0.2433     0.1307   0.0003 0.0004
+
     2        17.6     16.1      -0.1     16.5     -4.7     16.8     12.6     51.4  -0.0687 0.1702 -0.0164 0.1286   0.1883     0.1140   0.0415 0.0381
     3          0.0      0.0      -0.0      0.1       0.0      0.1       0.0      0.0  -0.0678 0.1950 -0.0189 0.2441   0.0000      0.0000  0.0003 0.0004
+
     3          0.0      0.0      -0.0      0.0       0.0      0.0       0.0      0.0  -0.0571 0.1674 -0.0152 0.1345   0.0000      0.0000  0.0415 0.0377
 
---------------------------------------------------------------------------------------------------------------------------------------------------------------
 
---------------------------------------------------------------------------------------------------------------------------------------------------------------
  
Line 384: Line 384:
 
   cd CsPad::CalibV1/CxiDs2.0:Cspad.0/geometry
 
   cd CsPad::CalibV1/CxiDs2.0:Cspad.0/geometry
 
   mv 0-end.data 0-end.data.v1
 
   mv 0-end.data 0-end.data.v1
   ln -fns <path to 0-end.data.t002_1k_1> 0-end.data
+
   ln -fns <path to 0-end.data.t000_1k_1> 0-end.data
  
We optionally use softlinks here to avoid duplicating data.  We can now reprocess the data as trial 3:
+
We optionally use softlinks here to avoid duplicating data.  We can now reprocess the data as trial 1:
  
 
   for i in `seq 2 16`; do '''cxi.mpi_submit''' input.experiment=cxi02416 \
 
   for i in `seq 2 16`; do '''cxi.mpi_submit''' input.experiment=cxi02416 \
Line 392: Line 392:
 
     mp.nproc=36 mp.queue=psanaq output.split_logs=True \
 
     mp.nproc=36 mp.queue=psanaq output.split_logs=True \
 
     input.dispatcher=cctbx.xfel.xtc_process \
 
     input.dispatcher=cctbx.xfel.xtc_process \
     input.target=cxi02416/lyso-t003.phil input.trial=3 input.run_num=$i \
+
     input.target=[[cxi02416-lyso-t001.phil]] input.trial=1 input.run_num=$i \
 
     input.cfg=cxi02416-calibdir.cfg; done
 
     input.cfg=cxi02416-calibdir.cfg; done
  
The input phil file is the same as before.  However, to instruct psana to use the modified calibration directory, we add a psana config file named cxi02416-calibdir.cfg with these lines:
+
To instruct psana to use the modified calibration directory, we add a psana config file named cxi02416-calibdir.cfg with these lines:
  
 
   [psana]
 
   [psana]
Line 402: Line 402:
 
If you have write permissions to your geometry folder, you don't need to use this config file or make a copy of your calibration directory.  We recommend saving the original metrology like we did in the above example, by renaming it to 0-end.data.v1, so you can return to it as needed.
 
If you have write permissions to your geometry folder, you don't need to use this config file or make a copy of your calibration directory.  We recommend saving the original metrology like we did in the above example, by renaming it to 0-end.data.v1, so you can return to it as needed.
  
Indexing with the new metrology yielded 5484 indexed images.
+
The [[cxi02416-lyso-t001.phil | input phil file]] is the same as before, with a slight change to the unit cell dimensions.  The program '''cspad.detector_statistics''' (described below) reports an average unit cell after joint refinement, which we use to update our indexing cell target. 
 +
 
 +
Indexing with the new metrology yielded 5305 indexed images.
  
 
== Refine metrology (v3) ==
 
== Refine metrology (v3) ==
  
We now refine the metrology generated from the indexed images from trial 3 and call it metrology version 3 (v3).  The command is otherwise the same, with an updated tag. We also increase the number images used to 2000.
+
We now refine the metrology generated from the indexed images from trial 3 and call it metrology version 3 (v3).  Now that we have a better estimate of the unit cell, we [[cxi02416-refine.phil | use a phil file]] that adds a restraint to the joint refinement such the refined unit cells for each crystal have a similar set of dimensions. This assumes the crystals are isomorphous.  We also increase the number images used to 2000.
  
   bsub -q psanaq -o t003_1k.out '''cspad.cbf_metrology''' tag=t003_1k \
+
   bsub -q psanaq -o t001_2k.out '''cspad.cbf_metrology''' tag=t001_2k \
     cxi02416-refine.phil reflections=indexed ../r0*/003/out \
+
     [[cxi02416-refine.phil]] reflections=indexed ../r0*/001/out \
 
     n_subset=2000 split_dataset=True
 
     n_subset=2000 split_dataset=True
  
Line 415: Line 417:
  
 
   '''cspad.detector_shifts''' \
 
   '''cspad.detector_shifts''' \
     t002_1k_1_refined_experiments_level2.json t002_1k_1_refined_reflections_level2.pickle \
+
     t000_1k_1_refined_experiments_level2.json t000_1k_1_refined_reflections_level2.pickle \
     t003_2k_1_refined_experiments_level2.json t003_2k_1_refined_reflections_level2.pickle
+
     t001_2k_1_refined_experiments_level2.json t001_2k_1_refined_reflections_level2.pickle
  
The Detector XY column shows the detector moved 5.8 microns in the XY plane, the quadrants on averaged moved 7.9+/- 4.0 microns and the 2x1 sensors moved 10.4+/-10.8 microns.  The tile positions moved substantially less between v2 and v3 then they did between v1 and v2.
+
The Detector XY column shows the detector moved 4.3 microns in the XY plane, the quadrants on averaged moved 10.2 +/- 5.8 microns and the 2x1 sensors moved 11.7 +/- 11.5 microns.  The tile positions moved substantially less between v2 and v3 then they did between v1 and v2.
  
 
== Evaluate metrology ==
 
== Evaluate metrology ==
Line 424: Line 426:
 
Iterative hierarchical joint refinement should proceed until convergence, meaning until rounds of indexing and refinement do not improve the model.  '''cxi.display_metrology''',  '''dev.dials.plot_detector_shfits''', and '''cspad.detector_shifts''' measure the magnitude of changes during refinement.  An additional tool is available, '''cspad.detector_statistics''', that can be used to evaluate the precision of refinement, and the state of the cspad after refinement.
 
Iterative hierarchical joint refinement should proceed until convergence, meaning until rounds of indexing and refinement do not improve the model.  '''cxi.display_metrology''',  '''dev.dials.plot_detector_shfits''', and '''cspad.detector_shifts''' measure the magnitude of changes during refinement.  An additional tool is available, '''cspad.detector_statistics''', that can be used to evaluate the precision of refinement, and the state of the cspad after refinement.
  
   '''cspad.detector_statistics''' tag=t003_2k
+
   '''cspad.detector_statistics''' tag=t001_2k
  
 
Warning, this dumps a lot of output.  The program examines the current directory for files from cspad.cbf_metrology.  For each hierarchy level 0-3, the program creates three tables of statistics:
 
Warning, this dumps a lot of output.  The program examines the current directory for files from cspad.cbf_metrology.  For each hierarchy level 0-3, the program creates three tables of statistics:
Line 443: Line 445:
 
{| class="wikitable"
 
{| class="wikitable"
 
|-
 
|-
!Panel group ID
+
|Panel G ID
!Radial Dist (mm)
+
|Dist (mm)
!Fast Offset sigma (microns)
+
|F Offset Sigma (microns)
!Slow offset sigma (microns)
+
|S Offset Sigma (microns)
!N reflections
+
|N refls
 
|-
 
|-
 
|1
 
|1
 
|21.8
 
|21.8
|2.3
+
|1.457
|0.7
+
|3.795
|7585
+
|6916
 
|-
 
|-
 
|9
 
|9
 
|23.4
 
|23.4
|2
+
|3.055
|1.9
+
|1.747
|7411
+
|6498
 
|-
 
|-
 
|25
 
|25
 
|24.5
 
|24.5
|1.6
+
|0.614
|1.7
+
|1.891
|7211
+
|6428
 
|-
 
|-
 
|17
 
|17
 
|26
 
|26
|0.3
+
|1.007
|1.1
+
|6.704
|7182
+
|5772
 
|-
 
|-
 
|0
 
|0
 
|40.7
 
|40.7
|4.4
+
|2.003
|3.4
+
|1.707
|6429
+
|5712
 
|-
 
|-
 
|8
 
|8
 
|41.1
 
|41.1
|0.3
+
|2.507
|1.5
+
|7.437
|6522
+
|5882
 
|-
 
|-
 
|24
 
|24
 
|43.7
 
|43.7
|2.1
+
|0.139
|3.4
+
|4.121
|6733
+
|5934
 
|-
 
|-
 
|16
 
|16
 
|44.3
 
|44.3
|3.1
+
|3.088
|2.5
+
|5.562
|6576
+
|5470
 
|-
 
|-
 
|7
 
|7
 
|55.3
 
|55.3
|0.7
+
|1.048
|2.4
+
|5.097
|6048
+
|5221
 
|-
 
|-
 
|31
 
|31
 
|57.5
 
|57.5
|0.7
+
|4.456
|0.4
+
|2.402
|5774
+
|4837
 
|-
 
|-
 
|15
 
|15
|57.8
+
|57.8
|5
+
|1.447
|0.9
+
|0.87
|5763
+
|4768
 
|-
 
|-
 
|23
 
|23
 
|59.7
 
|59.7
|2.6
+
|1.39
|1.9
+
|1.814
|5924
+
|5165
 
|-
 
|-
 
|11
 
|11
 
|71.9
 
|71.9
|3.3
+
|0.38
|12.6
+
|10.784
|3750
+
|3203
 
|-
 
|-
 
|3
 
|3
 
|73
 
|73
|20.7
+
|9.648
|4.5
+
|1.552
|3968
+
|3238
 
|-
 
|-
 
|19
 
|19
|75.1
+
|75.2
|19.2
+
|9.258
|2.4
+
|3.055
|3614
+
|2610
 
|-
 
|-
 
|6
 
|6
 
|76.3
 
|76.3
|1.3
+
|1.85
|8.3
+
|9.05
|3440
+
|2921
 
|-
 
|-
 
|27
 
|27
 
|76.3
 
|76.3
|1.2
+
|3.867
|3.2
+
|12.665
|3813
+
|3329
 
|-
 
|-
 
|2
 
|2
 
|77.6
 
|77.6
|10.7
+
|11.894
|5.2
+
|1.966
|3323
+
|2781
 
|-
 
|-
 
|10
 
|10
 
|78
 
|78
|5.4
+
|1.809
|10.6
+
|5.188
|2964
+
|2537
 
|-
 
|-
 
|30
 
|30
 
|78.4
 
|78.4
|11.3
+
|1.374
|6.4
+
|4.348
|2925
+
|2265
 
|-
 
|-
 
|14
 
|14
 
|78.7
 
|78.7
|9.6
+
|1.845
|7.2
+
|1.796
|2863
+
|2187
 
|-
 
|-
 
|22
 
|22
 
|80.4
 
|80.4
|0.8
+
|0.119
|0.3
+
|2.188
|3039
+
|2574
 
|-
 
|-
 
|18
 
|18
|80.9
+
|81
|7.8
+
|19.347
|2.2
+
|9.268
|2970
+
|2256
 
|-
 
|-
 
|26
 
|26
|81.2
+
|81.1
|4.5
+
|3.51
|12.5
+
|15.827
|3264
+
|2767
 
|-
 
|-
 
|4
 
|4
 
|87.5
 
|87.5
|4.5
+
|2.449
|1.8
+
|8.722
|2193
+
|1844
 
|-
 
|-
 
|12
 
|12
 
|89.6
 
|89.6
|3.4
+
|23.074
|2.7
+
|14.689
|2016
+
|1636
 
|-
 
|-
 
|28
 
|28
|90.4
+
|90.5
|6.4
+
|12.99
|7.3
+
|19.289
|2011
+
|1615
 
|-
 
|-
 
|20
 
|20
|91.9
+
|91.8
|34.4
+
|2.404
|31.1
+
|0.28
|1897
+
|1548
 
|-
 
|-
 
|5
 
|5
 
|104.8
 
|104.8
|11.6
+
|32.112
|6
+
|16.115
|983
+
|797
 
|-
 
|-
 
|13
 
|13
|106.1
+
|106
|6.8
+
|10.282
|4.9
+
|19.482
|750
+
|559
 
|-
 
|-
 
|29
 
|29
|107.7
+
|107.8
|14
+
|8.739
|16.7
+
|12.337
|1028
+
|762
 
|-
 
|-
 
|21
 
|21
 
|108.5
 
|108.5
|6.2
+
|23.591
|4.5
+
|18.247
|874
+
|682
 
|-
 
|-
 
|All
 
|All
 
|
 
|
|4.7
+
|3.786
|3.9
+
|5.281
 
|
 
|
 
|-
 
|-
Line 651: Line 653:
 
|
 
|
 
|
 
|
|4088.8
+
|3459.8
|-
+
 
|}
 
|}
 
</div>
 
</div>
Line 663: Line 664:
 
* N refls: sum of the number of reflections recorded on the sensor between the two half dataset.  Used as a weighting term.
 
* N refls: sum of the number of reflections recorded on the sensor between the two half dataset.  Used as a weighting term.
  
The weighted mean of the fast and slow offset sigmas is 4.7 and 3.9 microns, respectively.  This measure of the precision of this refinement indicates it quite precise compared to the pixel size of the detector (110 microns).  However, there are still several panels with high fast or slow offset sigmas.  That, plus the fact that the number of indexed images increased while using v3 metrology implies that refinement has not converged.
+
The weighted mean of the fast and slow offset sigmas is 3.8 and 5.3 microns, respectively.  This measure of the precision of this refinement indicates it quite precise compared to the pixel size of the detector (110 microns).  However, there are still several panels with high fast or slow offset sigmas.  That, plus the fact that the number of indexed images increased while using v3 metrology implies that refinement has not converged.
  
 
== Further indexing and refinement (v4) ==
 
== Further indexing and refinement (v4) ==
Line 672: Line 673:
 
|-
 
|-
 
!PanelG ID
 
!PanelG ID
!Dist |(mm)
+
!Dist (mm)
 
!F Offset Sigma (microns)
 
!F Offset Sigma (microns)
 
!S Offset Sigma (microns)
 
!S Offset Sigma (microns)
Line 679: Line 680:
 
|1
 
|1
 
|21.8
 
|21.8
|1.6
+
|1.1
|1
+
|1.7
|6892
+
|6731
 
|-
 
|-
 
|9
 
|9
 
|23.4
 
|23.4
|0
 
 
|0.2
 
|0.2
|6878
+
|0.1
 +
|6681
 
|-
 
|-
 
|25
 
|25
 
|24.5
 
|24.5
|0.5
+
|0.9
|1.1
+
|1.2
|6609
+
|6321
 
|-
 
|-
 
|17
 
|17
 
|26
 
|26
|1.4
+
|1.1
|2.6
+
|2.5
|6637
+
|6144
 
|-
 
|-
 
|0
 
|0
 
|40.7
 
|40.7
|4
+
|0.1
|2.1
+
|1.9
|6848
+
|5533
 
|-
 
|-
 
|8
 
|8
|41.1
+
|41
|0.6
+
|1.7
|2.8
+
|4.7
|6624
+
|5501
 
|-
 
|-
 
|24
 
|24
 
|43.7
 
|43.7
|0.3
+
|1.0
|0.6
+
|1.5
|6652
+
|5576
 
|-
 
|-
 
|16
 
|16
 
|44.3
 
|44.3
|3.5
+
|4.7
|2.4
+
|1.3
|6655
+
|5707
 
|-
 
|-
 
|7
 
|7
 
|55.3
 
|55.3
|0.7
+
|0.5
|4.1
+
|0.5
|6444
+
|4764
 
|-
 
|-
 
|31
 
|31
 
|57.5
 
|57.5
|1.9
+
|2.0
|1.8
+
|1.6
|6538
+
|4722
 
|-
 
|-
 
|15
 
|15
 
|57.8
 
|57.8
|2.5
+
|2.2
|3
+
|1.8
|6552
+
|4804
 
|-
 
|-
 
|23
 
|23
 
|59.7
 
|59.7
|2.5
+
|3.7
|2.8
+
|4.1
|6443
+
|4681
 
|-
 
|-
 
|11
 
|11
 
|71.9
 
|71.9
|1.2
+
|1.7
|13.2
+
|8.7
|6139
+
|2887
 
|-
 
|-
 
|3
 
|3
 
|73
 
|73
|3.5
+
|16.1
|0.9
+
|0.1
|6070
+
|3234
 
|-
 
|-
 
|19
 
|19
 
|75.1
 
|75.1
|1.8
+
|7.3
|1.4
+
|0.1
|6065
+
|2984
 
|-
 
|-
 
|6
 
|6
 
|76.3
 
|76.3
|1.5
+
|2.4
 
|1.1
 
|1.1
|6118
+
|2664
 
|-
 
|-
 
|27
 
|27
 
|76.3
 
|76.3
|0.8
+
|3.7
|2.9
+
|16.8
|6010
+
|2903
 
|-
 
|-
 
|2
 
|2
 
|77.6
 
|77.6
|17.5
+
|3.1
|4.5
+
|4.1
|5917
+
|2866
 
|-
 
|-
 
|10
 
|10
 
|78
 
|78
|2
+
|2.5
|1
+
|5.2
|5823
+
|2388
 
|-
 
|-
 
|30
 
|30
|78.5
+
|78.4
|3.4
+
|1.1
|1.2
+
|0.6
|5885
+
|2413
 
|-
 
|-
 
|14
 
|14
 
|78.7
 
|78.7
|10.4
+
|14.2
|0.5
+
|1.7
|5974
+
|2460
 
|-
 
|-
 
|22
 
|22
 
|80.4
 
|80.4
|2.1
+
|2.3
|8.8
+
|2.0
|5867
+
|2255
 
|-
 
|-
 
|18
 
|18
|80.9
+
|81
|3.6
+
|10.2
|6.5
+
|4.6
|5943
+
|2348
 
|-
 
|-
 
|26
 
|26
|81.2
+
|81.1
|1.7
+
|4.8
|5.2
+
|21.8
|5891
+
|2545
 
|-
 
|-
 
|4
 
|4
 
|87.5
 
|87.5
|17.1
+
|8.0
|19.6
+
|4.9
|5570
+
|1832
 
|-
 
|-
 
|12
 
|12
 
|89.6
 
|89.6
|0.1
+
|11.8
|3.4
+
|2.3
|5514
+
|1559
 
|-
 
|-
 
|28
 
|28
 
|90.4
 
|90.4
|4.4
+
|9.7
|3.7
+
|14.8
|5494
+
|1497
 
|-
 
|-
 
|20
 
|20
|91.9
+
|91.8
|1.9
+
|19.3
|1
+
|17.7
|5417
+
|1434
 
|-
 
|-
 
|5
 
|5
|104.7
+
|104.8
|8.4
+
|47.9
|0.9
+
|35.9
|5016
+
|875
 
|-
 
|-
 
|13
 
|13
 
|106.1
 
|106.1
|17.4
+
|6.2
|27.7
+
|9.9
|4957
+
|550
 
|-
 
|-
 
|29
 
|29
|107.7
+
|107.8
|10.8
+
|15.4
|14.6
+
|24.2
|4980
+
|748
 
|-
 
|-
 
|21
 
|21
 
|108.5
 
|108.5
|10.7
+
|17.7
|6.9
+
|7.2
|4855
+
|698
 
|-
 
|-
 
|All
 
|All
 
|
 
|
|4.1
+
|4.0
|4.4
+
|3.9
 
|
 
|
 
|-
 
|-
Line 879: Line 880:
 
|
 
|
 
|
 
|
|6039.9
+
|3384.5
 
|}
 
|}
 
</div>
 
</div>
  
The fast and slow offset sigmas are lower generally, but not substantially.  5509 images were indexed, a small increase of 0.5% over v3 metrology.  Further, the change in tile position is minimal, as shown by '''cspad.detector_shifts''' (delta XY movements (microns): detector: 4.7, quadrants: 2.4+/1.3, sensors: 7.8+/-7.2).  The metrology has likely converged.
+
The fast and slow offset sigmas are lower generally, but not substantially.  5475 images were indexed, an increase of 1% over v3 metrology.  The change in tile position is minimal, as shown by '''cspad.detector_shifts''' (delta XY movements (microns): detector: 1.5, quadrants: 4.0+/2.3, sensors: 5.5+/-5.9).  The metrology has likely converged.
  
 
== Deploy metrology ==
 
== Deploy metrology ==
  
 
v4 metrology is now ready for use.  Deploy it or use it directly for indexing as described in [[Cxi0216 calibration#Index using v2 metrology | indexing using v2 metrology]].
 
v4 metrology is now ready for use.  Deploy it or use it directly for indexing as described in [[Cxi0216 calibration#Index using v2 metrology | indexing using v2 metrology]].

Latest revision as of 16:55, 18 May 2016

Calibration of cspad using cxi02416, cctbx.xfel and DIALS

This page is not intended as a manual for processing XFEL data using cctbx.xfel and DIALS. Rather, this is a documentation of steps taken to calibrate the CSPAD using data collected at CXI in February 2016, on detector Ds2.

Program names are highlighted in bold. Generally, help can be obtained using -h or -c options to the program at the command line.

In this walkthrough, we will be calibrating the tile positions of the CSPAD detector at the CXI endstation at LCLS. We will be using a calibration dataset collected by CXI staff in February 2016, during calibration experiment cxi02614. The procedure is to index the images, refine the tile positions minimizing the difference between observed and calculated Bragg reflections, re-index the images, re-refine the tile positions, and repeat until convergence. At each step of the way, we will label our current metrology with a version number, starting at version 0:

Metrology version Description
Version 0 (v0) Initial metrology deployed by beamline operators. The tile positions are measured using an optical microscope, but as the quadrants can move independently, they are not correctly aligned in relation to each other or to the beam center.
Version 1 (v1) After collecting some data, powder rings can be seen after averaging the events in a run. Several tools are available for aligning the quadrants by eye or automatically using powder rings.
Version 2 (v2) After indexing the images using v1, we will refine the tile positions to produce metrology v2.
Version 3 (v3) After re-indexing the images using v2, we will re-refine the tile positions to produce metrology v3.
... And so forth until convergence

We start with this known information provided by the beamline operators:

  • Data is contained in runs 2-16
  • Detector address: CxiDs2.0:Cspad.0. This string identifies the front CSPAD detector in the XTC streams.
  • detz_offset (IE the distance from the sample position to the back of the detector rail): 568 mm.

Dark pedestal, common mode correction, and untrusted pixels

Subtracting the dark pedestal, applying common mode correction and masking out untrusted pixels is critical for spot picking and integration. cctbx.xfel uses masks and pedestals deployed into the experiment’s calibration store using the LCLS program calibman [1] . Dark pedestal correction, hot pixel determination, and user defined mask generation for regions of interest or for masking out shadows is all available through that program.

Common mode correction accounts for flat, per-panel fluctuations that occur non-uniformly throughout a single exposure. LCLS has several algorithms available [2]. We recommend the non-bonded pixels algorithm; the parameter files below specifies this algorithm during indexing and integration.

Aligning quadrant positions (v1)

Alignment of quadrant positions using powder rings can be done manually using cctbx.xfel or LCLS's calibman tool [3]. ctbx.xfel also provides an automatic algorithm for quadrant alignment, provided powder rings of sufficient quality. For cxi02416, the beamline operators already calibrated quadrant positions to a point where indexing can proceed. Regardless, if needed, here is how to accomplish the same task using cctbx.xfel

Averaging diffraction data to create powder patterns

The cctbx.xfel program cxi.mpi_average is used to create averages:

 for i in `seq 2 16`; do bsub -n 12 -q psanaq -o avg_r$i.log \
   cxi.mpi_average -x cxi02416 -r $i -a CxiDs2.0:Cspad.0 -d 568 -o . -v; done

For each of the runs with data (2-16), this command submits an averaging job using 12 processors, providing information about the experiment such as detector address and detz_offset in the form of command line arguments.

Progress can be monitored with the command bjobs.

When the averages are complete, they will consist of files named cxi02416_avg-r0002.cbf, cxi02416_stddev-r0002.cbf, and cxi02416_max-r0002.cbf for each run, representing the average, standard deviation, and maximum of all the pixel data in each of the runs. Display the data using cctbx.image_viewer:

 cctbx.image_viewer *.cbf

Manual calibration using cctbx.xfel

Typically the best powder rings come from the maximum projection (example: cxi02416_max-r0013.cbf). To manually align the quadrant positions, use cctbx.image_viewer cxi02416_max-r0013.cbf. Under actions, click on 'Show quadrant calibration', then use the spinners to align the powder rings. You may find the ring tool or the unit cell tool, also under the Actions menu, to be useful visual aids during this process. When done, click 'Save current metrology' to save the changes to a .def file, which is a cbf header.

This walkthrough uses pre-aligned quadrant locations. However, if it is desired to use the metrology from manual quadrant re-alignment for indexing, it first needs to be converted to SLAC's metrology file format. Use this command:

 cxi.cbfheader2slaccalib cbf_header=quadrants.cbf

This command will create a 0-end.data file. See the instructions under indexing using v2 metrology for deploying it for use.

Automatic calibration using cctbx.xfel

If a quadrant is properly placed, the pixel values for a strong powder pattern will be highly correlated after rotating it 45 degrees around the beam center. cspad.quadrants_cbf performs a grid search of XY offsets for each quadrant, searching for the position with the highest rotational autocorrelation. It then writes out a new cbf file with the adjusted header:

 cspad.quadrants_cbf cxi02416_max-r0013.cbf

Specify the '-p' parameter to enable plots of the grid search results for each quadrant. Here is the output:

Doing cross-correlation on panel ARRAY_D0Q0S1A0
Searching a grid with dimensions (41, 41)
max cc  0.1592 is at (0, 0)
Doing cross-correlation on panel ARRAY_D0Q1S1A0
Searching a grid with dimensions (41, 41)
max cc  0.1209 is at (0, 5)
Doing cross-correlation on panel ARRAY_D0Q2S1A0
Searching a grid with dimensions (41, 41)
max cc  0.1686 is at (0, -2)
Doing cross-correlation on panel ARRAY_D0Q3S1A0
Searching a grid with dimensions (41, 41)
max cc  0.1144 is at (1, 2)

The CC values are all less that 0.2 which indicates that even though run 13 shows the strongest powder diffraction, the rings are not contiguous or strong enough compared to the background to get a good rotational autocorrelation. Use the image viewer to verify the new quadrant positions are not ideal:

 cctbx.image_viewer cxi02416_max-r0013_cc.cbf

It is possible that using a maximum projection of all the runs would make the rings more contiguous and brighter, leading to higher CC values. This can be done quickly using the maximum projections already made:

 cxi.cspad_average *_max*.cbf -m all_max.cbf

Use cctbx.image_viewer to compare all_max.cbf to cxi02416_max-r0013.cbf. The rings are noticeably better. Now, do the grid search:

 cspad.quadrants_cbf all_max.cbf

Results:

Doing cross-correlation on panel ARRAY_D0Q0S1A0
Searching a grid with dimensions (41, 41)
max cc  0.2568 is at (-1, 0)
Doing cross-correlation on panel ARRAY_D0Q1S1A0
Searching a grid with dimensions (41, 41)
max cc  0.2022 is at (0, 4)
Doing cross-correlation on panel ARRAY_D0Q2S1A0
Searching a grid with dimensions (41, 41)
max cc  0.2328 is at (-4, 0)
Doing cross-correlation on panel ARRAY_D0Q3S1A0
Searching a grid with dimensions (41, 41)
max cc  0.1853 is at (0, 2)

The CC values are higher, and likely high enough for at least two of the quadrants to get some initial indexing results. Regardless, we recommend silver behenate powder for this automatic procedure as it gives very smooth, contiguous rings.

Again, for this walkthrough we use quadrant positions aligned by the beamline operator. However, if after automatic alignment it is desired to use the results for indexing, they first need to be converted to SLAC's metrology file format. Use this command:

 cxi.cbfheader2slaccalib cbf_header=all_max_cc.cbf

This command will create a 0-end.data file. See the instructions under indexing using v2 metrology for deploying it for use.

Initial indexing

The initial metrology deployed by the beamline operator is sufficient to get initial indexing results. Indexing in cctbx.xfel typically is done in a series of trials. Our first trial will be trial 0, using metrology v1 (initial metrology from beamline operators, with quadrants corrected by eye by the beamline operators).

With this information, and this phil file, we can index the data:

 for i in `seq 2 16`; do cxi.mpi_submit input.experiment=cxi02416 \
   output.output_dir=/reg/d/psdm/cxi/cxi02416/ftc/brewster/dials \
   mp.nproc=36 mp.queue=psanaq output.split_logs=True \
   input.dispatcher=cctbx.xfel.xtc_process \
   input.target=cxi02416-lyso-t000.phil input.trial=0 \
   input.run_num=$i; done

This command submits jobs for runs 2 through 16, using the DIALS backend of cctbx.xfel. To save time during initial indexing and metrology refinement, we use dispatch.integrate=False to skip the integration step. After indexing is completed, we got 4986 indexed images, as can be shown by this command:

 cd /reg/d/psdm/cxi/cxi02416/ftc/brewster/dials
 ls r0*/000/out/*.json | wc -l

Refine metrology (v2)

Let's call the metrology deployed by the beamline operator version 0 (v0). After quadrant alignment, the operators updated the metrology to version 1 (v1). The following command will do an iterative joint hierarchical refinement of the components of the CSPAD detector. The new tile positions we call version 2 (v2).

 bsub -q psanaq -o t000_1k.out cspad.cbf_metrology tag=t000_1k \
   reflections=indexed ../r0*/000/out refine_distance=True \
   n_subset=1000 split_dataset=True

The program first aggregates the requested number of images into a single dataset. Then, it refines the detector as a whole (including Z position and tilt). Using the new detector position, it refines the quadrants independently from each other, and then the 2x1 sensors, and then the individual panels. Finally, it converts the DIALS format metrology into the SLAC file format (0-end.data). The refinement is a 'joint refinement' because the information from many crystals is used to refine a single detector model.

Details about the parameters used:

  • tag=t000_1k: the output files will be named t000_1k*
  • reflections=indexed: the indexing process produces two sets of reflection files, bright indexed reflections, and final integrated reflections (can include weak intensities). Here, I've chosen to refine only against the bright, indexed reflections.
  • refine_distance=True: as the detector distance is not fully known yet, allow the detector distance to refine. In subsequent rounds of metrology refinement, we will fix the distance to a constant value.
  • n_subset=1000: pick 1000 images at random to refine
  • split_dataset=True: the refinement is done twice independently, using odd numbered or even numbered images, each time using n_subset images. This will be useful later for evaluating the accuracy of the metrology.

Visualize tile shifts

In order to get a sense of the magnitude of the shifts in panel position after refinement, use the program cxi.display_metrology. For example, the original detector geometry can be displayed thusly:

 cxi.display_metrology \
   /reg/d/psdm/cxi/cxi02416/calib/CsPad::CalibV1/CxiDs2.0:Cspad.0/geometry/0-end.data

Compare it to the refined geometry:

 cxi.display_metrology 0-end.data.t000_1k_1

You will see a small change in the origin (center arrow) and obvious changes in the quadrant positions. The relative positions of the sensors to each other will not change a large amount, though if you inspect the files themselves you will see changes.

Another tool that is useful for evaluating changes in metrology is dev.dials.plot_detector_shifts. This program will plot detector shifts in the X and Y directions (also known as the fast and slow directions, a convention referring to how data is read from the byte arrays stored on disk). It will also plot shifts along the Z axis, I.E., the detector's normal axis. Let's use it to compare the unrefined geometry to the geometry refined at level 2 (I.E. at the level of the 2x1 sensors):

 dev.dials.plot_detector_shifts t000_1k_1_combined_experiments.json \
   t000_1k_1_refined_experiments_level2.json \
   plot_type=spherical_polar tag=v1v2level2
 display v1v2level2*.png

Use the spacebar to cycle between the images. The pixels are mapped to the Ewald sphere and then displayed as a function of azimuthal and elevation angles along the Ewald sphere, which distorts the detector appearance but is independent of any detector layout. The colorbar scales will show the magnitude of panel shifts.

Try running the program using t000_1k_1_refined_experiments_level0.json to see how much the detector as a whole shifted, or t000_1k_1_refined_experiments_level1.json to see shifts in the quadrants.

Finally, the program cspad.detector_shifts shows the magnitude of shifts between a reference and a moving detector in tabular form:

 cspad.detector_shifts \
   t000_1k_1_combined_experiments.json t000_1k_1_combined_reflections.pickle \
   t000_1k_1_refined_experiments_level2.json t000_1k_1_refined_reflections_level2.pickle

Looking at the Delta XY columns, we see the detector moved 4.6 microns in the XY plane, the quadrants moved on average 219.8+/-57.9 microns and the 2x1 sensors moved 17.6+/-16.1 microns. Note how most of the change is in the quadrant locations, which is expected as these were done by eye. The 2x1 sensor positions, determined using optical microscopy, changed by an order of magnitude less than the quadrants. Also, the same program ran against the other half of the split dataset (t000_1k_2_*) will reveal very similar shifts, indicating these shifts are not due to random chance.

Full output for cspad.detector_shifts:

Found 4 hierarchy levels
Hierarchy level 0 Detector shifts
-----------------------------------------------------------------------------------------------------------
     PanelG     BC dist  Delta XY R Offsets T Offsets Z Offsets dR Norm dT Norm Local dNorm Rot Z  N Refls
       ID         (mm)  (microns) (microns) (microns) (microns)  (deg)   (deg)     (deg)    (deg)
-----------------------------------------------------------------------------------------------------------
        0           2.4      4.6       3.5      -2.9     -74.0  -0.0003 -0.0213    0.0213   0.0003    29527
 Weighted mean               4.6       3.5      -2.9     -74.0  -0.0003 -0.0213    0.0213   0.0003
Weighted stddev              0.0       0.0       0.0       0.0   0.0000  0.0000    0.0000   0.0000
-----------------------------------------------------------------------------------------------------------
Hierarchy level 1 Detector shifts
-----------------------------------------------------------------------------------------------------------
     PanelG     BC dist  Delta XY R Offsets T Offsets Z Offsets dR Norm dT Norm Local dNorm Rot Z  N Refls
       ID         (mm)  (microns) (microns) (microns) (microns)  (deg)   (deg)     (deg)    (deg)
-----------------------------------------------------------------------------------------------------------
        0          62.9    157.7    -153.5     -36.0    1236.4  -0.0308 -0.0658    0.0550   0.0111     7766
        1          64.3    262.5    -172.9    -197.5    1532.9  -0.0042 -0.1422    0.1411   0.0260     7112
        3          66.0    190.2      22.6     188.8    1347.7  -0.0945 -0.0391    0.1228   0.1139     7811
        2          67.0    279.6     277.7      32.1    1374.9   0.0216  0.0397    0.0305   0.0169     6838
 Weighted mean             219.8     -11.7       0.3    1369.3  -0.0291 -0.0527    0.0880   0.0432
Weighted stddev             57.9     204.3     161.0     122.4   0.0500  0.0731    0.0524   0.0494
-----------------------------------------------------------------------------------------------------------
Hierarchy level 2 Detector shifts
-----------------------------------------------------------------------------------------------------------
     PanelG     BC dist  Delta XY R Offsets T Offsets Z Offsets dR Norm dT Norm Local dNorm Rot Z  N Refls
       ID         (mm)  (microns) (microns) (microns) (microns)  (deg)   (deg)     (deg)    (deg)
-----------------------------------------------------------------------------------------------------------
        1          21.9      2.7       0.1      -2.7      67.0  -0.2345 -0.0794    0.1925   0.0227     1817
        9          23.6      2.5      -0.3       2.4     -34.8   0.0229  0.1911    0.2632   0.0290     1578
       25          24.7      7.9      -2.6       7.4      34.8  -0.0817 -0.1288    0.2195   0.1031     1644
       17          26.3      5.8      -3.8      -4.4     -72.0   0.0402  0.0412    0.0588   0.0107     1357
        0          40.8     23.0     -23.0      -0.3      44.5   0.0064 -0.3016    0.3031   0.0169     1626
        8          41.3      5.8      -4.7       3.6     -44.8   0.2528 -0.0850    0.3773   0.0512     1687
       24          43.9     10.2      -0.4     -10.3      36.9  -0.1491 -0.2116    0.2813   0.0628     1665
       16          44.5     17.8      17.1       4.9     -13.0   0.0640  0.1521    0.1525   0.0178     1499
        7          55.5     12.6       0.4     -12.6      52.8  -0.1670  0.0284    0.1042   0.0097     1382
       31          57.7     28.9      -7.5     -27.9      81.9  -0.0803  0.0442    0.0742   0.1336     1406
       15          58.0     13.5      -6.2      12.0     -52.4   0.0155  0.1185    0.1853   0.0004     1323
       23          60.0     16.8     -16.1       4.8     -17.2  -0.0080 -0.0531    0.0518   0.0357     1560
       11          72.2      4.4      -1.6      -4.2       8.5  -0.5439 -0.0891    0.4503   0.0258      843
        3          73.1     22.5      17.6     -13.9      22.6  -0.2173  0.0156    0.1600   0.0165      856
       19          75.3      2.1      -0.5       2.0     -17.4   0.0061  0.1337    0.1232   0.0245      584
        6          76.5     11.2      -4.4     -10.3      14.9   0.0371 -0.1290    0.1975   0.0252      730
       27          76.5     59.0       6.4     -58.6      93.7  -0.0486  0.0859    0.0744   0.0859      851
        2          77.7     45.1       1.6     -45.2      71.3  -0.1845  0.0036    0.1143   0.0259      772
       10          78.3     19.5      -3.5     -19.1      24.1  -0.4560 -0.0550    0.3362   0.0117      651
       30          78.6     17.4      -3.3      17.2     -53.8   0.0098 -0.1055    0.2080   0.0894      651
       14          78.9      4.1      -4.1      -0.5      14.5   0.0433  0.1315    0.2032   0.0036      504
       22          80.6     22.0      21.9       2.1     -17.6   0.0394  0.0027    0.0347   0.0234      696
       18          81.2     31.8       9.1      30.5     -80.5   0.0141 -0.0178    0.0374   0.0487      620
       26          81.3     25.6      -8.1     -24.2      68.4  -0.0185  0.0909    0.0750   0.1275      819
        4          87.6     32.3      31.6      -5.7      53.6  -0.2149  0.0391    0.1460   0.0154      431
       12          89.8     25.3       6.9     -24.2     -21.1  -0.0419  0.0008    0.1011   0.0173      406
       28          90.7     37.7      36.9       3.7      79.9  -0.3678  0.0389    0.3173   0.0006      529
       20          92.1     24.0       2.7      23.9      16.6   0.0213 -0.1148    0.1014   0.0777      374
        5         104.9     92.9     -91.3       6.3    -133.6  -0.6185  0.2555    0.6003   0.0674      152
       13         106.3      9.9       9.9       0.4       4.2  -0.1141 -0.0277    0.0259   0.0031      120
       29         108.0    101.5     101.1      -1.7     122.2  -0.3588  0.1173    0.2946   0.0482      246
       21         108.7     16.4     -13.1       9.9     -17.3   0.1983  0.0814    0.1786   0.0392      148
 Weighted mean              17.6      -0.1      -4.7      12.6  -0.0687 -0.0164    0.1883   0.0415
Weighted stddev             16.1      16.5      16.8      51.4   0.1702  0.1286    0.1140   0.0381
-----------------------------------------------------------------------------------------------------------
Hierarchy level 3 Detector shifts
-----------------------------------------------------------------------------------------------------------
     PanelG     BC dist  Delta XY R Offsets T Offsets Z Offsets dR Norm dT Norm Local dNorm Rot Z  N Refls
       ID         (mm)  (microns) (microns) (microns) (microns)  (deg)   (deg)     (deg)    (deg)
-----------------------------------------------------------------------------------------------------------
        2          15.9      0.0       0.0      -0.0       0.0  -0.0669 -0.2384    0.0000   0.0227      941
       18          16.2      0.0      -0.0       0.0       0.0  -0.1455  0.1261    0.0000   0.0290      885
       50          19.2      0.0      -0.0       0.0       0.0   0.0606 -0.1400    0.0000   0.1031      865
       34          19.3      0.0      -0.0      -0.0       0.0  -0.0121  0.0563    0.0000   0.0107      578
        3          30.7      0.0       0.0       0.0       0.0  -0.2095 -0.1320    0.0000   0.0227      876
       19          33.0      0.0       0.0       0.0       0.0  -0.0226  0.1912    0.0000   0.0290      693
       51          33.0      0.0       0.0       0.0       0.0  -0.0494 -0.1443    0.0000   0.1031      779
       35          35.2      0.0       0.0       0.0       0.0   0.0293  0.0496    0.0000   0.0107      779
       16          37.6      0.0      -0.0       0.0       0.0   0.2624  0.0480    0.0000   0.0512      866
        0          37.9      0.0       0.0      -0.0       0.0   0.1518 -0.2606    0.0000   0.0169      775
       32          40.8      0.0       0.0      -0.0       0.0  -0.0180  0.1640    0.0000   0.0178      583
       48          41.1      0.0      -0.0       0.0       0.0  -0.0272 -0.2575    0.0000   0.0628      863
        1          46.2      0.0      -0.0       0.0       0.0   0.0304 -0.3001    0.0000   0.0169      851
       17          47.3      0.0       0.0       0.0       0.0   0.2587 -0.0648    0.0000   0.0512      821
       49          49.1      0.0       0.0       0.0       0.0  -0.1319 -0.2228    0.0000   0.0628      802
       33          50.4      0.0      -0.0      -0.0       0.0   0.0517  0.1567    0.0000   0.0178      916
       15          51.3      0.0      -0.0       0.0       0.0  -0.1644  0.0407    0.0000   0.0097      740
       63          52.9      0.0      -0.0       0.0       0.0  -0.0768  0.0501    0.0000   0.1336      732
       31          54.2      0.0       0.0       0.0       0.0   0.0243  0.1170    0.0000   0.0004      667
       47          55.7      0.0       0.0      -0.0       0.0  -0.0119 -0.0523    0.0000   0.0357      885
       14          61.4      0.0      -0.0       0.0       0.0  -0.1694 -0.0022    0.0000   0.0097      642
       23          61.5      0.0      -0.0      -0.0       0.0  -0.5509 -0.0155    0.0000   0.0258      690
        7          62.3      0.0      -0.0       0.0       0.0  -0.2132  0.0445    0.0000   0.0165      626
       30          63.5      0.0      -0.0       0.0       0.0  -0.0061  0.1193    0.0000   0.0004      656
       62          64.0      0.0      -0.0      -0.0       0.0  -0.0869  0.0291    0.0000   0.1336      674
       39          64.6      0.0      -0.0      -0.0       0.0   0.0240  0.1317    0.0000   0.0245      435
       55          65.7      0.0      -0.0       0.0       0.0  -0.0367  0.0916    0.0000   0.0859      631
       46          65.8      0.0       0.0      -0.0       0.0   0.0017 -0.0537    0.0000   0.0357      675
        5          67.7      0.0      -0.0       0.0       0.0  -0.1829  0.0246    0.0000   0.0259      591
       21          68.5      0.0      -0.0       0.0       0.0  -0.4593 -0.0032    0.0000   0.0117      519
       53          71.2      0.0      -0.0       0.0       0.0  -0.0081  0.0924    0.0000   0.1275      605
       37          71.4      0.0       0.0      -0.0       0.0   0.0120 -0.0193    0.0000   0.0487      447
       13          73.5      0.0       0.0      -0.0       0.0   0.0317 -0.1305    0.0000   0.0252      447
       61          75.2      0.0       0.0       0.0       0.0   0.0054 -0.1058    0.0000   0.0894      360
       29          76.1      0.0      -0.0       0.0       0.0   0.0488  0.1296    0.0000   0.0036      264
       45          77.5      0.0       0.0       0.0       0.0   0.0395  0.0011    0.0000   0.0234      440
        9          80.3      0.0       0.0       0.0       0.0  -0.2107  0.0576    0.0000   0.0154      296
       12          80.8      0.0       0.0       0.0       0.0   0.0586 -0.1208    0.0000   0.0252      283
       25          82.2      0.0       0.0      -0.0       0.0  -0.0416  0.0044    0.0000   0.0173      270
       28          82.9      0.0      -0.0      -0.0       0.0   0.0201  0.1370    0.0000   0.0036      240
       22          83.0      0.0       0.0       0.0       0.0  -0.5494 -0.0436    0.0000   0.0258      153
       60          83.3      0.0       0.0       0.0       0.0   0.0276 -0.1023    0.0000   0.0894      291
       57          83.5      0.0       0.0      -0.0       0.0  -0.3631  0.0704    0.0000   0.0006      365
        6          83.9      0.0      -0.0      -0.0       0.0  -0.2152  0.0340    0.0000   0.0165      230
       41          84.6      0.0       0.0       0.0       0.0   0.0112 -0.1162    0.0000   0.0777      259
       44          85.0      0.0      -0.0      -0.0       0.0   0.0383  0.0095    0.0000   0.0234      256
       38          86.1      0.0      -0.0      -0.0       0.0   0.0174  0.1327    0.0000   0.0245      149
       54          87.3      0.0      -0.0      -0.0       0.0  -0.0411  0.0897    0.0000   0.0859      220
        4          88.0      0.0      -0.0      -0.0       0.0  -0.1844  0.0064    0.0000   0.0259      181
       20          88.3      0.0      -0.0      -0.0       0.0  -0.4567 -0.0489    0.0000   0.0117      132
       36          91.3      0.0       0.0       0.0       0.0   0.0139 -0.0180    0.0000   0.0487      173
       52          91.5      0.0      -0.0      -0.0       0.0  -0.0171  0.0912    0.0000   0.1275      214
        8          95.7      0.0       0.0       0.0       0.0  -0.2010  0.0856    0.0000   0.0154      135
       24          98.1      0.0      -0.0       0.0       0.0  -0.0407  0.0099    0.0000   0.0173      136
       56          98.5      0.0       0.0       0.0       0.0  -0.3501  0.1192    0.0000   0.0006      164
       11          98.8      0.0      -0.0       0.0       0.0  -0.6019  0.2924    0.0000   0.0674      105
       27         100.0      0.0       0.0      -0.0       0.0  -0.1156 -0.0207    0.0000   0.0031       77
       40         100.2      0.0      -0.0       0.0       0.0  -0.0046 -0.1166    0.0000   0.0777      115
       59         102.0      0.0      -0.0      -0.0       0.0  -0.3510  0.1389    0.0000   0.0482      164
       43         102.4      0.0       0.0      -0.0       0.0   0.2029  0.0691    0.0000   0.0392       98
       10         111.6      0.0       0.0       0.0       0.0  -0.5535  0.3760    0.0000   0.0674       47
       26         113.4      0.0      -0.0       0.0       0.0  -0.1173 -0.0040    0.0000   0.0031       43
       58         114.7      0.0       0.0      -0.0       0.0  -0.3274  0.1879    0.0000   0.0482       82
       42         115.7      0.0       0.0      -0.0       0.0   0.2108  0.0392    0.0000   0.0392       50
 Weighted mean               0.0      -0.0       0.0       0.0  -0.0571 -0.0152    0.0000   0.0415
Weighted stddev              0.0       0.0       0.0       0.0   0.1674  0.1345    0.0000   0.0377
-----------------------------------------------------------------------------------------------------------
Detector shifts summary
---------------------------------------------------------------------------------------------------------------------------------------------------------------
Hierarchy  Delta XY  Delta XY R Offsets R Offsets T Offsets T Offsets Z Offsets Z Offsets dR Norm dR Norm dT Norm dT Norm Local dNorm Local dNorm Rot Z  Rot Z
  Level               Sigma               Sigma               Sigma               Sigma            Sigma           Sigma                 Sigma           Sigma
          (microns) (microns) (microns) (microns) (microns) (microns) (microns) (microns)  (deg)   (deg)   (deg)   (deg)     (deg)       (deg)    (deg)  (deg)
---------------------------------------------------------------------------------------------------------------------------------------------------------------
    0          4.6       0.0       3.5       0.0      -2.9       0.0     -74.0       0.0  -0.0003  0.0000 -0.0213  0.0000    0.0213      0.0000   0.0003 0.0000
    1        219.8      57.9     -11.7     204.3       0.3     161.0    1369.3     122.4  -0.0291  0.0500 -0.0527  0.0731    0.0880      0.0524   0.0432 0.0494
    2         17.6      16.1      -0.1      16.5      -4.7      16.8      12.6      51.4  -0.0687  0.1702 -0.0164  0.1286    0.1883      0.1140   0.0415 0.0381
    3          0.0       0.0      -0.0       0.0       0.0       0.0       0.0       0.0  -0.0571  0.1674 -0.0152  0.1345    0.0000      0.0000   0.0415 0.0377
---------------------------------------------------------------------------------------------------------------------------------------------------------------


For each hierarchy level, the average shifts in are computed among objects at that level, weighted by the number of reflections recorded on each object. For example, for a four quadrant detector, the average Z shift will be the average of the four quadrant Z values, each weighted by the number of reflections on that quadrant.

-------------------
Column descriptions
-------------------

Individual hierarchy level tables only:
PanelG id: ID of the panel group.
BC dist: distance of the panel group from the beam center.
N Refls: number of reflections on this panel group

All tables:
Delta XY: magnitude of the shift in the local XY frame.
R, T offsets: shifts relative to the parent object's location in the radial and transverse directions (relative to the detector center).
Z offsets: relative shifts in the local frame in the local Z direction.
R, T Norm: angle between normal vectors in lab space, projected onto the radial or transverse plane.
Local dNorm: local relative angle between normal vectors.
Rot Z: rotation around detector normal in lab space

Index using v2 metrology

In order to improve our metrology we will re-index using the new tile positions. We assume we don't have write access to the geometry file for this detector, namely /reg/d/psdm/cxi/cxi02416/calib/CsPad::CalibV1/CxiDs2.0:Cspad.0/geometry/0-end.data. To that end, we need a copy of the calibration directory for the experiment so we can modify it:

 cd <a subfolder in your home directory>
 mkdir -p cxi02416/calib
 cd cxi02416/calib
 cp -r /reg/d/psdm/cxi/cxi02416/calib/* .

Now, we can link in v2, backing up v1 first:

 cd CsPad::CalibV1/CxiDs2.0:Cspad.0/geometry
 mv 0-end.data 0-end.data.v1
 ln -fns <path to 0-end.data.t000_1k_1> 0-end.data

We optionally use softlinks here to avoid duplicating data. We can now reprocess the data as trial 1:

 for i in `seq 2 16`; do cxi.mpi_submit input.experiment=cxi02416 \
   output.output_dir=/reg/d/psdm/cxi/cxi02416/ftc/brewster/dials \
   mp.nproc=36 mp.queue=psanaq output.split_logs=True \
   input.dispatcher=cctbx.xfel.xtc_process \
   input.target=cxi02416-lyso-t001.phil input.trial=1 input.run_num=$i \
   input.cfg=cxi02416-calibdir.cfg; done

To instruct psana to use the modified calibration directory, we add a psana config file named cxi02416-calibdir.cfg with these lines:

 [psana]
 calib-dir = <a subfolder in your home directory>/cxi02416/calib

If you have write permissions to your geometry folder, you don't need to use this config file or make a copy of your calibration directory. We recommend saving the original metrology like we did in the above example, by renaming it to 0-end.data.v1, so you can return to it as needed.

The input phil file is the same as before, with a slight change to the unit cell dimensions. The program cspad.detector_statistics (described below) reports an average unit cell after joint refinement, which we use to update our indexing cell target.

Indexing with the new metrology yielded 5305 indexed images.

Refine metrology (v3)

We now refine the metrology generated from the indexed images from trial 3 and call it metrology version 3 (v3). Now that we have a better estimate of the unit cell, we use a phil file that adds a restraint to the joint refinement such the refined unit cells for each crystal have a similar set of dimensions. This assumes the crystals are isomorphous. We also increase the number images used to 2000.

 bsub -q psanaq -o t001_2k.out cspad.cbf_metrology tag=t001_2k \
   cxi02416-refine.phil reflections=indexed ../r0*/001/out \
   n_subset=2000 split_dataset=True

After refinement, use cxi.display_metrology, dev.dials.plot_detector_shifts, and/or cspad.detector_shifts to evaluate how much the tiles moved:

 cspad.detector_shifts \
   t000_1k_1_refined_experiments_level2.json t000_1k_1_refined_reflections_level2.pickle \
   t001_2k_1_refined_experiments_level2.json t001_2k_1_refined_reflections_level2.pickle

The Detector XY column shows the detector moved 4.3 microns in the XY plane, the quadrants on averaged moved 10.2 +/- 5.8 microns and the 2x1 sensors moved 11.7 +/- 11.5 microns. The tile positions moved substantially less between v2 and v3 then they did between v1 and v2.

Evaluate metrology

Iterative hierarchical joint refinement should proceed until convergence, meaning until rounds of indexing and refinement do not improve the model. cxi.display_metrology, dev.dials.plot_detector_shfits, and cspad.detector_shifts measure the magnitude of changes during refinement. An additional tool is available, cspad.detector_statistics, that can be used to evaluate the precision of refinement, and the state of the cspad after refinement.

 cspad.detector_statistics tag=t001_2k

Warning, this dumps a lot of output. The program examines the current directory for files from cspad.cbf_metrology. For each hierarchy level 0-3, the program creates three tables of statistics:

1) Detector congruence. The two half datasets from each level of refinement are compared to each other, and agreement between the two independent refinement runs is reported.

2) Detector statistics. Using the two half datasets as independent measurements, statistics about the cspad are reported, such as normal vector tilts and XYZ offsets. Weighted means and standard deviations are reported as well, and can be used to estimate the overall precision of refinement.

3) RMSDs by detector number. For each of the half datasets, overall, radial and transverse RMSDs are shown.

For this walkthrough, we will extract a few statistics only. Look for the third set of tables, delineated with

 ********************************************************************************
 Showing statistics for detector at level 2 (sensors, I.E. 2x1s)
 ********************************************************************************

In the second table, under Detector statistics, a few of the columns are reproduced here (click expand to show the table):

Panel G ID Dist (mm) F Offset Sigma (microns) S Offset Sigma (microns) N refls
1 21.8 1.457 3.795 6916
9 23.4 3.055 1.747 6498
25 24.5 0.614 1.891 6428
17 26 1.007 6.704 5772
0 40.7 2.003 1.707 5712
8 41.1 2.507 7.437 5882
24 43.7 0.139 4.121 5934
16 44.3 3.088 5.562 5470
7 55.3 1.048 5.097 5221
31 57.5 4.456 2.402 4837
15 57.8 1.447 0.87 4768
23 59.7 1.39 1.814 5165
11 71.9 0.38 10.784 3203
3 73 9.648 1.552 3238
19 75.2 9.258 3.055 2610
6 76.3 1.85 9.05 2921
27 76.3 3.867 12.665 3329
2 77.6 11.894 1.966 2781
10 78 1.809 5.188 2537
30 78.4 1.374 4.348 2265
14 78.7 1.845 1.796 2187
22 80.4 0.119 2.188 2574
18 81 19.347 9.268 2256
26 81.1 3.51 15.827 2767
4 87.5 2.449 8.722 1844
12 89.6 23.074 14.689 1636
28 90.5 12.99 19.289 1615
20 91.8 2.404 0.28 1548
5 104.8 32.112 16.115 797
13 106 10.282 19.482 559
29 107.8 8.739 12.337 762
21 108.5 23.591 18.247 682
All 3.786 5.281
Mean 3459.8

Here are the meaning of the columns:

  • Panel G ID: here we are looking at the sensors, of which there are 32.
  • Dist (mm): distance from the sensor center to the beam center. The table is sorted by this number.
  • F Offset sigma (microns): weighted standard deviation of the two measurements of the sensor's fast coordinate.
  • S Offset sigma (microns): weighted standard deviation of the two measurements of the sensor's slow coordinate.
  • N refls: sum of the number of reflections recorded on the sensor between the two half dataset. Used as a weighting term.

The weighted mean of the fast and slow offset sigmas is 3.8 and 5.3 microns, respectively. This measure of the precision of this refinement indicates it quite precise compared to the pixel size of the detector (110 microns). However, there are still several panels with high fast or slow offset sigmas. That, plus the fact that the number of indexed images increased while using v3 metrology implies that refinement has not converged.

Further indexing and refinement (v4)

Reindexing the data using v3 and re-refining the data to create v4 metrology proceeds as described in the above steps. After generating v4, we evaluate it as described above and see this table:

PanelG ID Dist (mm) F Offset Sigma (microns) S Offset Sigma (microns) N refls
1 21.8 1.1 1.7 6731
9 23.4 0.2 0.1 6681
25 24.5 0.9 1.2 6321
17 26 1.1 2.5 6144
0 40.7 0.1 1.9 5533
8 41 1.7 4.7 5501
24 43.7 1.0 1.5 5576
16 44.3 4.7 1.3 5707
7 55.3 0.5 0.5 4764
31 57.5 2.0 1.6 4722
15 57.8 2.2 1.8 4804
23 59.7 3.7 4.1 4681
11 71.9 1.7 8.7 2887
3 73 16.1 0.1 3234
19 75.1 7.3 0.1 2984
6 76.3 2.4 1.1 2664
27 76.3 3.7 16.8 2903
2 77.6 3.1 4.1 2866
10 78 2.5 5.2 2388
30 78.4 1.1 0.6 2413
14 78.7 14.2 1.7 2460
22 80.4 2.3 2.0 2255
18 81 10.2 4.6 2348
26 81.1 4.8 21.8 2545
4 87.5 8.0 4.9 1832
12 89.6 11.8 2.3 1559
28 90.4 9.7 14.8 1497
20 91.8 19.3 17.7 1434
5 104.8 47.9 35.9 875
13 106.1 6.2 9.9 550
29 107.8 15.4 24.2 748
21 108.5 17.7 7.2 698
All 4.0 3.9
Mean 3384.5

The fast and slow offset sigmas are lower generally, but not substantially. 5475 images were indexed, an increase of 1% over v3 metrology. The change in tile position is minimal, as shown by cspad.detector_shifts (delta XY movements (microns): detector: 1.5, quadrants: 4.0+/2.3, sensors: 5.5+/-5.9). The metrology has likely converged.

Deploy metrology

v4 metrology is now ready for use. Deploy it or use it directly for indexing as described in indexing using v2 metrology.