Cppxfel Initial Orientation Matrix Refinement: Difference between revisions
Jump to navigation
Jump to search
No edit summary |
No edit summary |
||
| (6 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
Initial orientation matrix refinement covers the first stage of the dedicated C++ code in ''cppxfel'' and uses an input file which is called <code>integrate.txt</code> by default. This should have been generated during the [[Cppxfel Indexing | previous stage]]. | Initial orientation matrix refinement covers the first stage of the dedicated C++ code in [[Cppxfel|''cppxfel'']] and uses an input file which is called <code>integrate.txt</code> by default. This should have been generated during the [[Cppxfel Indexing | previous stage]]. | ||
== The contents of integrate.txt == | == The contents of integrate.txt == | ||
| Line 9: | Line 9: | ||
</pre> | </pre> | ||
This shows an input script along the following lines | This shows an input script along the following lines. This input script is divided into a number of parameters, followed by the line <code>COMMANDS</code>, after which the task(s) for ''cppxfel'' to complete are listed. In this case, ''cppxfel'' should <code>INTEGRATE</code> the data. | ||
<pre> | <pre> | ||
| Line 27: | Line 27: | ||
METROLOGY_SEARCH_SIZE 1 | METROLOGY_SEARCH_SIZE 1 | ||
SHOEBOX_FOREGROUND_PADDING 1 | |||
SHOEBOX_NEITHER_PADDING 2 | |||
SHOEBOX_BACKGROUND_PADDING 3 | |||
INTENSITY_THRESHOLD 12 | INTENSITY_THRESHOLD 12 | ||
| Line 45: | Line 45: | ||
<pre> | <pre> | ||
head - | head -21 matrices.dat | ||
</pre> | </pre> | ||
| Line 72: | Line 72: | ||
</pre> | </pre> | ||
The image tag has the name of the image file minus the extension. The values for the detector distance, beam centre and wavelength are | The image tag has the name of the image file minus the extension. The values for the detector distance, beam centre and wavelength are overridden by the values in the <code>integrate.txt</code> file if they are present, but the overriding values can be removed if the values vary from one image to another. The orientation matrix is constructed from the unit cell matrix and the rotation matrix listed. The unit cell matrix is the inverse of the real space unit cell matrix, and the unit cell matrix is multiplied by the rotation matrix before use (rotation × unit cell matrix). | ||
== Executing integrate.txt == | == Executing integrate.txt == | ||
| Line 81: | Line 81: | ||
cppxfel.run -i integrate.txt | cppxfel.run -i integrate.txt | ||
</pre> | </pre> | ||
== Wavelength histograms == | |||
The output tracks individual images which execute on different threads as their orientation matrices are refined. An example of a wavelength histogram created is shown below. These wavelength histograms were first described in Ginn et al, Nat Comms 2015. | |||
<pre> | |||
Wavelength histogram for shot-s00-20130316165134721.img | |||
1.41322 . | |||
1.41759 | |||
1.42196 | |||
1.42633 . | |||
1.4307 . | |||
1.43508 .. | |||
1.43945 | |||
1.44382 .. | |||
1.44819 ..... | |||
1.45256 ......... | |||
1.45693 .................. | |||
1.4613 ...................................................................... | |||
1.46567 ............................................... | |||
1.47004 ..... | |||
1.47441 ....... | |||
1.47878 ... | |||
1.48315 . | |||
1.48752 . | |||
1.4919 .. | |||
1.49627 | |||
</pre> | |||
These integration peaks should be as long and narrow as possible, depending on the extent of over-prediction as determined by the parameter <code>OVER_PRED_BANDWIDTH</code>. | |||
== Final integration summary table == | |||
The initial orientation matrix refinement commands finish with a summary table detailing the main results for each crystal and each image. It will start with something like this: | |||
<pre> | |||
Filename Refl num Score Horiz rot Vert rot Beam rot Stdev Wavelength Distance UnitCellA UnitCellB UnitCellC | |||
shot-s00-20130316165156302.img 111 8.91892 0 0 0 0.0260573 1.45691 91.0018 105.918 105.918 105.918 | |||
shot-s00-20130316165043053.img 80 6.33333 -1 1 0 0.0240737 1.45647 91.002 105.918 105.918 105.918 | |||
shot-s01-20130316165128672.img 14 4.28571 -1.11914 -1.11523 0 0.0170082 1.45962 91.0018 105.918 105.918 105.918 | |||
shot-s01-20130316165047878.img 87 1.2069 -0.625 0.320312 0 0.0105548 1.45852 91.0018 105.918 105.918 105.918 | |||
shot-s01-20130316165035212.img 21 9.47368 -1.06836 0.908203 0 0.0209798 1.45496 91.0018 105.918 105.918 105.918 | |||
shot-s01-20130316165033837.img 37 6.77419 -0.00390625 0.533203 0 0.0239777 1.45621 91.0018 105.918 105.918 105.918 | |||
shot-s00-20130316165336174.img 188 1.45161 -0.365234 0.371094 0 0.0104029 1.45705 91.0018 105.918 105.918 105.918 | |||
shot-s00-20130316165131305.img 171 1.06509 0.212891 0.621094 0 0.0107561 1.4579 91.0018 105.918 105.918 105.918 | |||
shot-s00-20130316165041011.img 26 6.11111 -0.445312 -0.298828 0 0.0223847 1.45643 91.0018 105.918 105.918 105.918 | |||
shot-s00-20130316165126972.img 211 0.972222 -0.00585938 0.609375 0 0.00877267 1.45685 91.0018 105.918 105.918 105.918 | |||
shot-s00-20130316165126972.img 47 7.875 0.183594 1.97266 0 0.0240756 1.45685 91.0018 105.918 105.918 105.918 | |||
shot-s00-20130316165138553.img 41 9.28571 -0.365234 0.625 0 0.0253017 1.45673 91.0018 105.918 105.918 105.918 | |||
shot-s01-20130316165050712.img 19 6.75 0.0214844 2.17578 0 0.0230937 1.45894 91.0018 105.918 105.918 105.918 | |||
shot-s00-20130316165215513.img 146 2.02128 -0.498047 0.240234 0 0.010419 1.45674 91.0018 105.918 105.918 105.918 | |||
shot-s00-20130316165133179.img 143 2.36301 -0.376953 0.283203 0 0.0129344 1.45528 91.0018 105.918 105.918 105.918 | |||
shot-s00-20130316165143844.img 142 1.99301 -0.132812 0.554688 0 0.0117068 1.45789 91.0018 105.918 105.918 105.918 | |||
shot-s00-20130316165231472.img 6 12.5 1 1.875 0 0.013229 1.45868 91.0018 105.918 105.918 105.918 | |||
shot-s00-20130316165052512.img 67 8.23944 -0.365234 0.482422 0 0.0251099 1.45578 91.0018 105.918 105.918 105.918 | |||
shot-s00-20130316165052512.img 168 1.86335 -0.458984 0.542969 0 0.01226 1.45578 91.0018 105.918 105.918 105.918 | |||
shot-s00-20130316165125972.img 200 1.6129 -0.388672 0.638672 0 0.010696 1.456 91.0018 105.918 105.918 105.918 | |||
shot-s00-20130316165052178.img 217 1.70616 -0.1875 0.6875 0 0.0116042 1.45657 91.0018 105.918 105.918 105.918 | |||
</pre> | |||
The "score" column is the result of the minimisation function used to refine the horizontal and vertical rotations, which are listed in the following two columns. This is a weighted combination of the total number of reflections and the standard deviation of the Ewald sphere wavelengths. In general, a score of 3 or less is good, though will be higher for low resolution data, using the standard setting for <code>ORIENTATION_SCORE</code>. The horizontal and vertical rotations show the rotations in degrees required to find the optimal solution. | |||
== Output files == | |||
The important files created by running <code>integrate.txt</code> are: | |||
* A new set of orientation matrices which have been refined under three file names: | |||
* * <code>integrate-orientations.dat</code> - this file is to be provided to input files which use the <code>INTEGRATE</code> command, such as <code>integrate.txt</code> generated by <code>cppxfel.input_gen</code> | |||
* * <code>refine-orientations.dat</code> - this file is to be provided to input files which use the <code>REFINE_PARTIALITY</code> command, such as <code>refine.txt</code> generated by <code>cppxfel.input_gen</code> | |||
* * <code>merge-orientations.dat</code> - this file is to be provided to input files which use the <code>MERGE</code> command, such as <code>merge.txt</code> generated by <code>cppxfel.input_gen</code>. | |||
* MTZ files following the format <code>img*.mtz</code>, which contain over-predicted reflection intensities. The first crystal will end in <code>_0.mtz</code>, the second with <code>_1.mtz</code> and so on for the number of lattices indexed in one image. | |||
* <code>shot*.dat</code> files, containing tab-delimited information on the coordinates and intensities of each reflection. | |||
* <code>integration.csv</code>, which contains the information in the final integration summary table which can be imported into other graph-drawing software suites. | |||
Once these files are generated, it is time to do [[Cppxfel Post-refinement]]. | |||
Latest revision as of 23:12, 25 November 2015
Initial orientation matrix refinement covers the first stage of the dedicated C++ code in cppxfel and uses an input file which is called integrate.txt by default. This should have been generated during the previous stage.
The contents of integrate.txt
One can look at the contents of integrate.txt by executing:
cat integrate.txt
This shows an input script along the following lines. This input script is divided into a number of parameters, followed by the line COMMANDS, after which the task(s) for cppxfel to complete are listed. In this case, cppxfel should INTEGRATE the data.
ORIENTATION_MATRIX_LIST matrices.dat NEW_MATRIX_LIST orientations.dat SPACE_GROUP 197 UNIT_CELL 105.917757854 105.917757854 105.917757854 90.0 90.0 90.0 FIX_UNIT_CELL ON MM_PER_PIXEL 0.11 BEAM_CENTRE 881.755 881.5075 DETECTOR_DISTANCE 91.0018 INTEGRATION_WAVELENGTH 1.45738121124 PANEL_LIST panels.txt METROLOGY_SEARCH_SIZE 1 SHOEBOX_FOREGROUND_PADDING 1 SHOEBOX_NEITHER_PADDING 2 SHOEBOX_BACKGROUND_PADDING 3 INTENSITY_THRESHOLD 12 ABSOLUTE_INTENSITY OFF REFINE_ORIENTATIONS ON VERBOSITY_LEVEL 1 COMMANDS INTEGRATE
The matrix information from indexing is passed through using the generated file
matrices.dat
under the keyword ORIENTATION_MATRIX_LIST which was produced in the previous stage. One can see the contents of matrices.dat:
head -21 matrices.dat
image shot-s00-20130316165156302 wavelength 1.45690748941 distance 91.0018 centre 881.755 881.5075 unitcell 0.00945122403519 0.0 0.0 -5.78701450245e-19 0.00945122403519 0.0 -5.78701450245e-19 -1.51989185671e-18 0.00945122403519 rotation 0.209524357745 -0.976621953811 0.0480531252429 -0.360236467579 -0.122785692599 -0.924745024923 0.909026529282 0.176446119331 -0.377541436183 image shot-s01-20130316165035212 wavelength 1.45496194705 distance 91.0018 centre 881.755 881.5075 unitcell 0.00944171966033 0.0 0.0 7.80781215281e-18 0.00944171966033 0.0 3.61484632923e-18 2.67460240614e-18 0.00944171966033 rotation -0.351867835054 0.0121912864248 0.935970298241 -0.852741351745 -0.416546104403 -0.315153184867 0.386032658825 -0.909032846184 0.156965190022 image shot-s00-20130316165041011 wavelength 1.45643207192 distance 91.0018 centre 881.755 881.5075 unitcell 0.00943218298265 0.0 0.0 1.51682978341e-18 0.00943218298265 0.0 3.61119512738e-18 5.77535560549e-19 0.00943218298265 rotation -0.21213292526 0.320476950034 0.92319778299 -0.956128815937 -0.263369266933 -0.128274380022 0.202032941263 -0.909907222596 0.362286539789 image shot-s00-20130316165215513 wavelength 1.45673961831 distance 91.0018
The image tag has the name of the image file minus the extension. The values for the detector distance, beam centre and wavelength are overridden by the values in the integrate.txt file if they are present, but the overriding values can be removed if the values vary from one image to another. The orientation matrix is constructed from the unit cell matrix and the rotation matrix listed. The unit cell matrix is the inverse of the real space unit cell matrix, and the unit cell matrix is multiplied by the rotation matrix before use (rotation × unit cell matrix).
Executing integrate.txt
This integration input file can be executed through cppxfel using the following command:
cppxfel.run -i integrate.txt
Wavelength histograms
The output tracks individual images which execute on different threads as their orientation matrices are refined. An example of a wavelength histogram created is shown below. These wavelength histograms were first described in Ginn et al, Nat Comms 2015.
Wavelength histogram for shot-s00-20130316165134721.img 1.41322 . 1.41759 1.42196 1.42633 . 1.4307 . 1.43508 .. 1.43945 1.44382 .. 1.44819 ..... 1.45256 ......... 1.45693 .................. 1.4613 ...................................................................... 1.46567 ............................................... 1.47004 ..... 1.47441 ....... 1.47878 ... 1.48315 . 1.48752 . 1.4919 .. 1.49627
These integration peaks should be as long and narrow as possible, depending on the extent of over-prediction as determined by the parameter OVER_PRED_BANDWIDTH.
Final integration summary table
The initial orientation matrix refinement commands finish with a summary table detailing the main results for each crystal and each image. It will start with something like this:
Filename Refl num Score Horiz rot Vert rot Beam rot Stdev Wavelength Distance UnitCellA UnitCellB UnitCellC shot-s00-20130316165156302.img 111 8.91892 0 0 0 0.0260573 1.45691 91.0018 105.918 105.918 105.918 shot-s00-20130316165043053.img 80 6.33333 -1 1 0 0.0240737 1.45647 91.002 105.918 105.918 105.918 shot-s01-20130316165128672.img 14 4.28571 -1.11914 -1.11523 0 0.0170082 1.45962 91.0018 105.918 105.918 105.918 shot-s01-20130316165047878.img 87 1.2069 -0.625 0.320312 0 0.0105548 1.45852 91.0018 105.918 105.918 105.918 shot-s01-20130316165035212.img 21 9.47368 -1.06836 0.908203 0 0.0209798 1.45496 91.0018 105.918 105.918 105.918 shot-s01-20130316165033837.img 37 6.77419 -0.00390625 0.533203 0 0.0239777 1.45621 91.0018 105.918 105.918 105.918 shot-s00-20130316165336174.img 188 1.45161 -0.365234 0.371094 0 0.0104029 1.45705 91.0018 105.918 105.918 105.918 shot-s00-20130316165131305.img 171 1.06509 0.212891 0.621094 0 0.0107561 1.4579 91.0018 105.918 105.918 105.918 shot-s00-20130316165041011.img 26 6.11111 -0.445312 -0.298828 0 0.0223847 1.45643 91.0018 105.918 105.918 105.918 shot-s00-20130316165126972.img 211 0.972222 -0.00585938 0.609375 0 0.00877267 1.45685 91.0018 105.918 105.918 105.918 shot-s00-20130316165126972.img 47 7.875 0.183594 1.97266 0 0.0240756 1.45685 91.0018 105.918 105.918 105.918 shot-s00-20130316165138553.img 41 9.28571 -0.365234 0.625 0 0.0253017 1.45673 91.0018 105.918 105.918 105.918 shot-s01-20130316165050712.img 19 6.75 0.0214844 2.17578 0 0.0230937 1.45894 91.0018 105.918 105.918 105.918 shot-s00-20130316165215513.img 146 2.02128 -0.498047 0.240234 0 0.010419 1.45674 91.0018 105.918 105.918 105.918 shot-s00-20130316165133179.img 143 2.36301 -0.376953 0.283203 0 0.0129344 1.45528 91.0018 105.918 105.918 105.918 shot-s00-20130316165143844.img 142 1.99301 -0.132812 0.554688 0 0.0117068 1.45789 91.0018 105.918 105.918 105.918 shot-s00-20130316165231472.img 6 12.5 1 1.875 0 0.013229 1.45868 91.0018 105.918 105.918 105.918 shot-s00-20130316165052512.img 67 8.23944 -0.365234 0.482422 0 0.0251099 1.45578 91.0018 105.918 105.918 105.918 shot-s00-20130316165052512.img 168 1.86335 -0.458984 0.542969 0 0.01226 1.45578 91.0018 105.918 105.918 105.918 shot-s00-20130316165125972.img 200 1.6129 -0.388672 0.638672 0 0.010696 1.456 91.0018 105.918 105.918 105.918 shot-s00-20130316165052178.img 217 1.70616 -0.1875 0.6875 0 0.0116042 1.45657 91.0018 105.918 105.918 105.918
The "score" column is the result of the minimisation function used to refine the horizontal and vertical rotations, which are listed in the following two columns. This is a weighted combination of the total number of reflections and the standard deviation of the Ewald sphere wavelengths. In general, a score of 3 or less is good, though will be higher for low resolution data, using the standard setting for ORIENTATION_SCORE. The horizontal and vertical rotations show the rotations in degrees required to find the optimal solution.
Output files
The important files created by running integrate.txt are:
- A new set of orientation matrices which have been refined under three file names:
- *
integrate-orientations.dat- this file is to be provided to input files which use theINTEGRATEcommand, such asintegrate.txtgenerated bycppxfel.input_gen - *
refine-orientations.dat- this file is to be provided to input files which use theREFINE_PARTIALITYcommand, such asrefine.txtgenerated bycppxfel.input_gen - *
merge-orientations.dat- this file is to be provided to input files which use theMERGEcommand, such asmerge.txtgenerated bycppxfel.input_gen. - MTZ files following the format
img*.mtz, which contain over-predicted reflection intensities. The first crystal will end in_0.mtz, the second with_1.mtzand so on for the number of lattices indexed in one image. shot*.datfiles, containing tab-delimited information on the coordinates and intensities of each reflection.integration.csv, which contains the information in the final integration summary table which can be imported into other graph-drawing software suites.
Once these files are generated, it is time to do Cppxfel Post-refinement.