http://viper.lbl.gov:8080/cctbx.xfel/api.php?action=feedcontributions&user=Aaron&feedformat=atomcctbx_xfel - User contributions [en]2024-03-29T11:28:12ZUser contributionsMediaWiki 1.23.1http://viper.lbl.gov:8080/cctbx.xfel/index.php/Main_PageMain Page2020-07-17T16:53:57Z<p>Aaron: </p>
<hr />
<div>News! See our [[https://doi.org/10.1107/S2059798318009191 new paper]]! This paper shows current best practices using ''cctbx.xfel'' with ''DIALS'' for serial crystallographic data. Also see the [https://github.com/phyy-nx/dials_refinement_brewster2018/wiki companion set of tutorials].<br />
<br />
News! See our [[http://cci.lbl.gov/publications/download/CCN_2019_p22_Brewster.pdf newsletter article]] that describes the XFEL GUI, an interactive program for handling large XFEL datasets.<br />
<br />
= Open-source tools for serial crystallographic data processing =<br />
<br />
''cctbx.xfel'' is a suite of software tools designed to process diffraction data from serial femtosecond crystallography (SFX) measurements at an X-ray free-electron laser (XFEL) or a synchrotron. Built on the Computational Crystallographic Toolbox ([http://cctbx.sourceforge.net ''cctbx'']), the same toolbox on which [http://www.phenix-online.org ''PHENIX''], [http://dials.github.io DIALS], [http://adder.lbl.gov/labelit ''LABELIT''], and post-refinement and merging program, [http://viper.lbl.gov/cctbx.xfel/index.php/Cctbx.prime ''PRIME''] are built, it enables the user to solve difficult problems relating to processing serial crystallographic data. The programs and modules provided by ''cctbx.xfel'' can reduce a large set of still diffraction images recorded at Stanford’s Linac Coherent Light Source ([http://lcls.slac.stanford.edu LCLS]), [http://sacla.xfel.jp SACLA], the [https://www.xfel.eu/ European XFEL], or a synchrotron, to a single MTZ file containing merged reflection intensities suitable for structure solution.<br />
<br />
== ''cctbx.xfel'' resources ==<br />
<br />
The tutorials on this wiki provide detailed instructions for indexing and integrating still diffraction images extracted from the raw data streams recorded at the LCLS, including pre-processing steps such as dark pedestal generation and refinement of the detector geometry of the Cornell–SLAC pixel array detectors (CSPAD) in use at the CXI and XPP end stations. The tutorials also cover tools to efficiently leverage the LCLS computing cluster to process the thousands to millions of diffraction images that can be recorded in a short time. Finally, processing approaches for data collection at non-LCLS sources are also provided.<br />
<br />
* [[Overview]] to the system architecture at LCLS and real-time progress monitoring of data processing<br />
* Installation: there are two ways to get ''cctbx.xfel'':<br />
** ''cctbx.xfel'' is installed for general use at SLAC. To start using an existing installation, [[Using the pre-built cctbx at LCLS|simply source the current build]]. <br />
** [https://dials.github.io/installation.html Download] a binary bundle of DIALS. Use this option if you will not be processing LCLS data because, for example, if you have serial crystallographic data from a synchrotron.<br />
** Build a psana/cctbx.xfel build [https://exafel.github.io/docs/psana-cctbx-install from scratch]. Use this option if you want to process LCLS data outside of LCLS or want your own build.<br />
* [[Tutorials]] on pre-processing, data reduction, and merging. <br />
* ''[[cctbx.prime]]'': tutorials on post-refinement using PRIME. <br />
* ''[[IOTA]]'': tutorial on spotfinding optimization using ''IOTA''.<br />
<br />
Other related information:<br />
<br />
* [[Serial XFEL Crystallography References]]<br />
* [[Data Processing at SACLA]]<br />
* [[2017 Tutorials]]. Berkeley Lab Serial Crystallography Workshop, 16-17 Feb 2017.<br />
* [[Experiment-day guidelines]]: a short, very hands-on set of notes for online monitoring and data-processing using cctbx.xfel.<br />
* [[File formats]]: for developers, information on file formats ''cctbx.xfel'' uses<br />
* [[Cppxfel]]: XFEL data processing with the CPPXFEL package from Diamond/Wellcome Trust.<br />
* [[Ha14 documentation]]: tutorials and support for processing using the [https://doi.org/10.1038/nmeth.2887 Hattne 2014] methods.<br />
<br />
This project is under active development. For any assistance, please contact the authors.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/Using_the_pre-built_cctbx_at_LCLSUsing the pre-built cctbx at LCLS2019-08-09T19:05:54Z<p>Aaron: </p>
<hr />
<div>''cctbx.xfel'' is installed for general use at LCLS. To use it, source the setup.sh file provided:<br />
<br />
<code><br />
source /reg/g/cctbx/conda_build/setup.sh<br />
</code><br />
<br />
Alternatively, add these lines to your .bashrc file (bash users):<br />
<br />
<code><br />
source /reg/g/cctbx/conda_build/miniconda2/etc/profile.d/conda.sh<br />
conda activate myEnv<br />
source /reg/g/cctbx/conda_build/build/setpaths.sh<br />
</code><br />
<br />
Likewise for csh users:<br />
<br />
<code><br />
source /reg/g/cctbx/conda_build/miniconda2/etc/profile.d/conda.csh<br />
conda activate myEnv<br />
source /reg/g/cctbx/conda_build/build/setpaths.csh<br />
</code><br />
<br />
Then log in as normal. All ''DIALS'' and ''cctbx.xfel'' commands will be available.<br />
<br />
----<br />
<br />
/reg/g/cctbx was last updated Aug 9, 2019</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/Serial_XFEL_Crystallography_ReferencesSerial XFEL Crystallography References2018-11-06T23:47:30Z<p>Aaron: </p>
<hr />
<div>Following is a list of references useful for learning about serial XFEL crystallography in general, and ''cctbx.xfel'' in particular.<br />
<br />
* Uervirojnangkoorn M, ''et. al.'' (2015): [http://elifesciences.org/content/4/e05421 Enabling X-ray free electron laser crystallography for challenging biological systems from a limited number of crystals.] ''eLife'' '''4(March)''', e05421.<br />
<br />
* Sauter NK, Hattne J, Brewster AS, Echols N, Zwart PH, Adams PD (2014): [http://dx.doi.org/10.1107/S1399004714024134 Improved crystal orientation and physical properties from single-shot XFEL stills.] ''Acta Crystallographica Section D: Biological. Crystallography D'' '''70''', 3299-309. [PMID:25478847] [PMC reprint: [http://ncbi.nlm.nih.gov/pmc/articles/PMC4257623/ PMC4257623] [http://cci.lbl.gov/publications/download/Mosaicity_wa5077.pdf (Reprint)]<br />
<br />
* Sauter NK, ''et. al.'' (2013): [http://dx.doi.org/10.1107/S0907444913000863 New Python-based methods for data processing.] ''Acta Crystallographica Section D: Biological Crystallography D'' '''69''', 1274-1282. [http://cci.lbl.gov/publications/download/ba5193_reprint.pdf (Reprint)]<br />
<br />
* Kern J, ''et. al.'' (2013): [http://www.sciencemag.org/content/early/2013/02/13/science.1234273 Simultaneous Femtosecond X-ray Spectroscopy and Diffraction of Photosystem II at Room Temperature.] ''Science'' '''340''', 491-495.<br />
<br />
* Kern J, ''et. al.'' (2012): [http://www.pnas.org/content/early/2012/05/31/1204598109.abstract Room temperature femtosecond X-ray diffraction of photosytem II microcrystals]. ''Proc. Natl. Acad. Sci.'' USA '''109''', 9721-9726. [http://cci.lbl.gov/publications/download/PNAS-2012-Kern-9721-6.pdf (Reprint)]<br />
<br />
* Sierra RG ''et. al.'' (2012): [http://dx.doi.org/10.1107/S0907444912038152 Nanoflow electrospinning serial femtosecond crystallography]. ''Acta Cryst. D'' '''68''', 1584-1587. [http://cci.lbl.gov/publications/download/lv5021.pdf (Reprint)]<br />
<br />
* Alonso-Mori R, ''et. al.'' (2012): [http://dx.doi.org/10.1073/pnas.1211384109Energy-dispersive X-ray emission spectroscopy using an X-ray free electron laser in a shot-by-shot mode]. ''Proc. Natl. Acad. Sci.'' USA '''109''', 19103-19107. [http://cci.lbl.gov/publications/download/PNAS-2012-Alonso-Mori-19103-7.pdf (Reprint)]<br />
<br />
* H. N. Chapmanet ''et. al.'' (2011): [http://www.ncbi.nlm.nih.gov/pubmed/21293373 Femtosecond x-ray protein nanocrystallography]. ''Nature'' '''470''', 73.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/Main_PageMain Page2018-11-06T23:43:46Z<p>Aaron: </p>
<hr />
<div>News! See our [[https://doi.org/10.1107/S2059798318009191 new paper]]! This paper shows current best practices using ''cctbx.xfel'' with ''DIALS'' for serial crystallographic data. Also see the [https://github.com/phyy-nx/dials_refinement_brewster2018/wiki companion set of tutorials].<br />
<br />
= Open-source tools for serial crystallographic data processing =<br />
<br />
''cctbx.xfel'' is a suite of software tools designed to process diffraction data from serial femtosecond crystallography (SFX) measurements at an X-ray free-electron laser (XFEL) or a synchrotron. Built on the Computational Crystallographic Toolbox ([http://cctbx.sourceforge.net ''cctbx'']), the same toolbox on which [http://www.phenix-online.org ''PHENIX''], [http://dials.github.io DIALS], [http://adder.lbl.gov/labelit ''LABELIT''], and post-refinement and merging program, [http://viper.lbl.gov/cctbx.xfel/index.php/Cctbx.prime ''PRIME''] are built, it enables the user to solve difficult problems relating to processing serial crystallographic data. The programs and modules provided by ''cctbx.xfel'' can reduce a large set of still diffraction images recorded at Stanford’s Linac Coherent Light Source ([http://lcls.slac.stanford.edu LCLS]), [http://sacla.xfel.jp SACLA], the [https://www.xfel.eu/ European XFEL], or a synchrotron, to a single MTZ file containing merged reflection intensities suitable for structure solution.<br />
<br />
== ''cctbx.xfel'' resources ==<br />
<br />
The tutorials on this wiki provide detailed instructions for indexing and integrating still diffraction images extracted from the raw data streams recorded at the LCLS, including pre-processing steps such as dark pedestal generation and refinement of the detector geometry of the Cornell–SLAC pixel array detectors (CSPAD) in use at the CXI and XPP end stations. The tutorials also cover tools to efficiently leverage the LCLS computing cluster to process the thousands to millions of diffraction images that can be recorded in a short time. Finally, processing approaches for data collection at non-LCLS sources are also provided.<br />
<br />
* [[Overview]] to the system architecture at LCLS and real-time progress monitoring of data processing<br />
* Installation: there are two ways to get ''cctbx.xfel'':<br />
** ''cctbx.xfel'' is installed for general use at SLAC. To start using an existing installation, [[Using the pre-built cctbx at LCLS|simply source the current build]]. <br />
** [https://dials.github.io/installation.html Download] a binary bundle of DIALS. Use this option if you will not be processing LCLS data because, for example, if you have serial crystallographic data from a synchrotron.<br />
** Build a psana/cctbx.xfel build [https://exafel.github.io/docs/psana-cctbx-install from scratch]. Use this option if you want to process LCLS data outside of LCLS or want your own build.<br />
* [[Tutorials]] on pre-processing, data reduction, and merging. <br />
* ''[[cctbx.prime]]'': tutorials on post-refinement using PRIME. <br />
* ''[[IOTA]]'': tutorial on spotfinding optimization using ''IOTA''.<br />
<br />
Other related information:<br />
<br />
* [[Serial XFEL Crystallography References]]<br />
* [[Data Processing at SACLA]]<br />
* [[2017 Tutorials]]. Berkeley Lab Serial Crystallography Workshop, 16-17 Feb 2017.<br />
* [[Experiment-day guidelines]]: a short, very hands-on set of notes for online monitoring and data-processing using cctbx.xfel.<br />
* [[File formats]]: for developers, information on file formats ''cctbx.xfel'' uses<br />
* [[Cppxfel]]: XFEL data processing with the CPPXFEL package from Diamond/Wellcome Trust.<br />
* [[Ha14 documentation]]: tutorials and support for processing using the [https://doi.org/10.1038/nmeth.2887 Hattne 2014] methods.<br />
<br />
This project is under active development. For any assistance, please contact the authors.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/Main_PageMain Page2018-11-06T23:40:46Z<p>Aaron: </p>
<hr />
<div>= Open-source tools for serial crystallographic data processing =<br />
<br />
''cctbx.xfel'' is a suite of software tools designed to process diffraction data from serial femtosecond crystallography (SFX) measurements at an X-ray free-electron laser (XFEL) or a synchrotron. Built on the Computational Crystallographic Toolbox ([http://cctbx.sourceforge.net ''cctbx'']), the same toolbox on which [http://www.phenix-online.org ''PHENIX''], [http://dials.github.io DIALS], [http://adder.lbl.gov/labelit ''LABELIT''], and post-refinement and merging program, [http://viper.lbl.gov/cctbx.xfel/index.php/Cctbx.prime ''PRIME''] are built, it enables the user to solve difficult problems relating to processing serial crystallographic data. The programs and modules provided by ''cctbx.xfel'' can reduce a large set of still diffraction images recorded at Stanford’s Linac Coherent Light Source ([http://lcls.slac.stanford.edu LCLS]), [http://sacla.xfel.jp SACLA], the [https://www.xfel.eu/ European XFEL], or a synchrotron, to a single MTZ file containing merged reflection intensities suitable for structure solution.<br />
<br />
== ''cctbx.xfel'' resources ==<br />
<br />
The tutorials on this wiki provide detailed instructions for indexing and integrating still diffraction images extracted from the raw data streams recorded at the LCLS, including pre-processing steps such as dark pedestal generation and refinement of the detector geometry of the Cornell–SLAC pixel array detectors (CSPAD) in use at the CXI and XPP end stations. The tutorials also cover tools to efficiently leverage the LCLS computing cluster to process the thousands to millions of diffraction images that can be recorded in a short time. Finally, processing approaches for data collection at non-LCLS sources are also provided.<br />
<br />
* [[Overview]] to the system architecture at LCLS and real-time progress monitoring of data processing<br />
* Installation: there are two ways to get ''cctbx.xfel'':<br />
** ''cctbx.xfel'' is installed for general use at SLAC. To start using an existing installation, [[Using the pre-built cctbx at LCLS|simply source the current build]]. <br />
** [https://dials.github.io/installation.html Download] a binary bundle of DIALS. Use this option if you will not be processing LCLS data because, for example, if you have serial crystallographic data from a synchrotron.<br />
** Build a psana/cctbx.xfel build [https://exafel.github.io/docs/psana-cctbx-install from scratch]. Use this option if you want to process LCLS data outside of LCLS or want your own build.<br />
* [[Tutorials]] on pre-processing, data reduction, and merging. <br />
* ''[[cctbx.prime]]'': tutorials on post-refinement using PRIME. <br />
* ''[[IOTA]]'': tutorial on spotfinding optimization using ''IOTA''.<br />
<br />
Other related information:<br />
<br />
* [[Serial XFEL Crystallography References]]<br />
* [[Data Processing at SACLA]]<br />
* [[2017 Tutorials]]. Berkeley Lab Serial Crystallography Workshop, 16-17 Feb 2017.<br />
* [[Experiment-day guidelines]]: a short, very hands-on set of notes for online monitoring and data-processing using cctbx.xfel.<br />
* [[File formats]]: for developers, information on file formats ''cctbx.xfel'' uses<br />
* [[Cppxfel]]: XFEL data processing with the CPPXFEL package from Diamond/Wellcome Trust.<br />
* [[Ha14 documentation]]: tutorials and support for processing using the [https://doi.org/10.1038/nmeth.2887 Hattne 2014] methods.<br />
<br />
This project is under active development. For any assistance, please contact the authors.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/Experiment-day_guidelinesExperiment-day guidelines2018-11-06T23:39:57Z<p>Aaron: /* Programs worth having up online */</p>
<hr />
<div><br />
== Before getting to the beamline ==<br />
<br />
* If spacegroup and unit cells of the samples are known, these can be used for indexing. Put these in your indexing phil file.<br />
* Make sure than an up to date meteorology phil file is available, as well as an accurate detz value.<br />
<br />
== As soon as you get to the beamline ==<br />
<br />
* Collect and average a dark run of at least 1000 images. Repeat this if anything happens that may affect the detector, e.g. ice crystals forming that may destroy pixels.<br />
* Set up a sample spreadsheet, minimum columns are run number and sample description. Other useful fields may include number of images, comments, 'hit' rate, number of indexed images. Some people like to have additional fields such as energy/wavelength, attenuation, sample injection or mounting details (e.g. consumption rate and volume). This is really a question of personal preference, but it is important to consider what is duplicated from the metadata, and that sometimes fewer fields can be better.<br />
<br />
== Quick file system overview ==<br />
<br />
When data are acquired, they are first passed to the DAQ (Data AQuisition) systems. These data are multiplexed into five or six streams in order to allow the high-speed readoff. This can be seen in the _sXX_ part of XTC filenames (s for stream). These streams are written to the FFB filesystem located at /reg/d/ffb, which is a fast flash drive system to be used only while online. Files are first written as XXX.inprogress during data collection, then the extension is changed to XXX.xtc once data acquisition is complete. Do not attempt to process data which is in progress, since pyana will crash when it hits the end of the file. <br />
<br />
Data are copied to the PSDM file format for offline processing, but this is not so fast. Therefore, when online, data should be read from the FFB filesystem, and written to the PSDM system (/reg/d/psdm). When offline, read and write from PSDM. Do not read from ffb when not at the beamline.<br />
<br />
== Programs worth having up online ==<br />
<br />
* Use dials.image_viewer to have immediate visual inspection of data quality.<br />
* Light average/max to see aggregate of whole run. The max is particularly useful as a virtual powder pattern to test for diffraction.<br />
* Either process the whole dataset, or just run a hitfinder with real-time logging to see the data quality semi-quantiatvely. This will be highly sensitive to the choices made in the hitfinding and indexing parameter files. If doing a quick hitfinder, it may be worth also submitting jobs to the queue for initial hitfinding and indexing. It is important to note that each sample will have its own optimal hitfinding and indexing parameters.<br />
<br />
Because indexing requires some iteration of the parameters, a sensible approach to generating on-line metrics is to focus on the image viewer, average/max images, and reasonably permissive hitfinding. A good first guess at a suitable hitfinding threshold can be obtained by inspection of the high resolution edges of light average/max images. Integration and merging on successful data collections can then be performed during any downtime, or while online metrics are working appropriately.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/Main_PageMain Page2018-11-06T23:38:08Z<p>Aaron: </p>
<hr />
<div>= Open-source tools for serial crystallographic data processing =<br />
<br />
''cctbx.xfel'' is a suite of software tools designed to process diffraction data from serial femtosecond crystallography (SFX) measurements at an X-ray free-electron laser (XFEL) or a synchrotron. Built on the Computational Crystallographic Toolbox ([http://cctbx.sourceforge.net ''cctbx'']), the same toolbox on which [http://www.phenix-online.org ''PHENIX''], [http://dials.github.io DIALS], [http://adder.lbl.gov/labelit ''LABELIT''], and post-refinement and merging program, [http://viper.lbl.gov/cctbx.xfel/index.php/Cctbx.prime ''PRIME''] are built, it enables the user to solve difficult problems relating to processing serial crystallographic data. The programs and modules provided by ''cctbx.xfel'' can reduce a large set of still diffraction images recorded at Stanford’s Linac Coherent Light Source ([http://lcls.slac.stanford.edu LCLS]), [http://sacla.xfel.jp SACLA], the [https://www.xfel.eu/ European XFEL], or a synchrotron, to a single MTZ file containing merged reflection intensities suitable for structure solution.<br />
<br />
== ''cctbx.xfel'' resources ==<br />
<br />
The tutorials on this wiki provide detailed instructions for indexing and integrating still diffraction images extracted from the raw data streams recorded at the LCLS, including pre-processing steps such as dark pedestal generation and refinement of the detector geometry of the Cornell–SLAC pixel array detectors (CSPAD) in use at the CXI and XPP end stations. The tutorials also cover tools to efficiently leverage the LCLS computing cluster to process the thousands to millions of diffraction images that can be recorded in a short time. Finally, processing approaches for data collection at non-LCLS sources are also provided.<br />
<br />
* [[Overview]] to the system architecture at LCLS and real-time progress monitoring of data processing<br />
* Installation: there are two ways to get ''cctbx.xfel'':<br />
** ''cctbx.xfel'' is installed for general use at SLAC. To start using an existing installation, [[Using the pre-built cctbx at LCLS|simply source the current build]]. <br />
** [https://dials.github.io/installation.html Download] a binary bundle of DIALS. Use this option if you will not be processing LCLS data because, for example, if you have serial crystallographic data from a synchrotron.<br />
** Build a psana/cctbx.xfel build [https://exafel.github.io/docs/psana-cctbx-install from scratch]. Use this option if you want to process LCLS data outside of LCLS or want your own build.<br />
* [[Tutorials]] on pre-processing, data reduction, and merging. <br />
* ''[[cctbx.prime]]'': tutorials on post-refinement using PRIME. <br />
* ''[[IOTA]]'': tutorial on spotfinding optimization using ''IOTA''.<br />
<br />
Other related information:<br />
<br />
* [[Serial XFEL Crystallography References]]<br />
* [[Data Processing at SACLA]]<br />
* [[2017 Tutorials]]. Berkeley Lab Serial Crystallography Workshop, 16-17 Feb 2017.<br />
* [[Experiment-day guidelines]]: a short, very hands-on set of notes for online monitoring and data-processing using cctbx.xfel. This page assumes that the setup and tutorial have been completed.<br />
* [[File formats]]: for developers, information on file formats ''cctbx.xfel'' uses<br />
* [[Cppxfel]]: XFEL data processing with the CPPXFEL package from Diamond/Wellcome Trust.<br />
* [[Ha14 documentation]]: tutorials and support for processing using the [https://doi.org/10.1038/nmeth.2887 Hattne 2014] methods.<br />
<br />
This project is under active development. For any assistance, please contact the authors.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/TutorialsTutorials2018-11-06T23:22:11Z<p>Aaron: </p>
<hr />
<div>Available tutorials:<br />
<br />
* [https://github.com/phyy-nx/dials_refinement_brewster2018/wiki List of tutorials] as part of our recent paper, [https://doi.org/10.1107/S2059798318009191 Brewster 2018].<br />
* [[2017_dials.stills_process | dials.stills_process]] for processing synchrotron data, including detector position refinement</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/Ha14_documentationHa14 documentation2018-11-06T23:17:51Z<p>Aaron: </p>
<hr />
<div>THIS PAGE IS DEPRECATED. This page details processing procedures described in [https://doi.org/10.1038/nmeth.2887 Hattne 2014] (Ha14), using LABELIT for initial indexing and stills-specific refinement and integration code implemented in the package ''cctbx.rstbx''<br />
<br />
* [[Ha14 installation]] setting up using the old psana<br />
* [[Ha14 Tutorials]] general tutorials for processing LCLS data<br />
* [[Processing Blog]] describing how we solved typical problems. <br />
* [[Processing L498 thermolysin]] details instructions to reproduce the results published in the forthcoming ''cctbx.xfel'' paper.<br />
* [[2014 Tutorials | 2014 Workshop Tutorial: ''cctbx.xfel'' ]]. Note, due to recent changes at LCLS, many of these tutorials are out of date for data collected late 2014 and onwards. For more up to date information, see above [[Ha14 Tutorials|tutorials]]. For data collected before late 2014, these workshop tutorials are useful.<br />
* [[2014_workshop | Powerpoint presentations]]</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/Ha14_documentationHa14 documentation2018-11-06T23:17:01Z<p>Aaron: </p>
<hr />
<div>THIS PAGE IS DEPRECATED. This page details processing procedures described in [https://doi.org/10.1038/nmeth.2887 Hattne 2014] (Ha14), using LABELIT for initial indexing and stills-specific refinement and integration code implemented in the package ''cctbx.rstbx''<br />
<br />
* [[Ha14 installation]] setting up using the old psana<br />
* [[Ha14 Tutorials]] general tutorials for processing LCLS data<br />
* [[Processing Blog]] describing how we solved typical problems. <br />
* [[Processing L498 thermolysin]] details instructions to reproduce the results published in the forthcoming ''cctbx.xfel'' paper.<br />
* [[2014 Tutorials | 2014 Workshop Tutorial: ''cctbx.xfel'' ]]. Note, due to recent changes at LCLS, many of these tutorials are out of date for data collected late 2014 and onwards. For more up to date information, see above [[tutorials]]. For data collected before late 2014, these workshop tutorials are useful.<br />
* [[2014_workshop | Powerpoint presentations]]</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/Ha14_documentationHa14 documentation2018-11-06T23:16:24Z<p>Aaron: </p>
<hr />
<div>THIS PAGE IS DEPRECATED. This page details processing procedures described in [https://doi.org/10.1038/nmeth.2887 Hattne 2014] (Ha14), using LABELIT for initial indexing and stills-specific refinement and integration code implemented in the package ''cctbx.rstbx''<br />
<br />
* [[Ha14 Installation]] setting up using the old psana<br />
* [[Ha14 Tutorials]] general tutorials for processing LCLS data<br />
* [[Processing Blog]] describing how we solved typical problems. <br />
* [[Processing L498 thermolysin]] details instructions to reproduce the results published in the forthcoming ''cctbx.xfel'' paper.<br />
* [[2014 Tutorials | 2014 Workshop Tutorial: ''cctbx.xfel'' ]]. Note, due to recent changes at LCLS, many of these tutorials are out of date for data collected late 2014 and onwards. For more up to date information, see above [[tutorials]]. For data collected before late 2014, these workshop tutorials are useful.<br />
* [[2014_workshop | Powerpoint presentations]]</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/Main_PageMain Page2018-11-06T23:16:07Z<p>Aaron: </p>
<hr />
<div>= Open-source tools for serial crystallographic data processing =<br />
<br />
''cctbx.xfel'' is a suite of software tools designed to process diffraction data from serial femtosecond crystallography (SFX) measurements at an X-ray free-electron laser (XFEL) or a synchrotron. Built on the Computational Crystallographic Toolbox ([http://cctbx.sourceforge.net ''cctbx'']), the same toolbox on which [http://www.phenix-online.org ''PHENIX''], [http://dials.github.io DIALS], [http://adder.lbl.gov/labelit ''LABELIT''], and post-refinement and merging program, [http://viper.lbl.gov/cctbx.xfel/index.php/Cctbx.prime ''PRIME''] are built, it enables the user to solve difficult problems relating to processing serial crystallographic data. The programs and modules provided by ''cctbx.xfel'' can reduce a large set of still diffraction images recorded at Stanford’s Linac Coherent Light Source ([http://lcls.slac.stanford.edu LCLS]), [http://sacla.xfel.jp SACLA], the [https://www.xfel.eu/ European XFEL], or a synchrotron, to a single MTZ file containing merged reflection intensities suitable for structure solution.<br />
<br />
== ''cctbx.xfel'' resources ==<br />
<br />
The tutorials on this wiki provide detailed instructions for indexing and integrating still diffraction images extracted from the raw data streams recorded at the LCLS, including pre-processing steps such as dark pedestal generation and refinement of the detector geometry of the Cornell–SLAC pixel array detectors (CSPAD) in use at the CXI and XPP end stations. The tutorials also cover tools to efficiently leverage the LCLS computing cluster to process the thousands to millions of diffraction images that can be recorded in a short time. Finally, processing approaches for data collection at non-LCLS sources are also provided.<br />
<br />
* [[Overview]] to the system architecture at LCLS and real-time progress monitoring of data processing<br />
* Installation: there are two ways to get ''cctbx.xfel'':<br />
** ''cctbx.xfel'' is installed for general use at SLAC. To start using an existing installation, [[Using the pre-built cctbx at LCLS|simply source the current build]]. <br />
** [https://dials.github.io/installation.html Download] a binary bundle of DIALS. Use this option if you will not be processing LCLS data because, for example, if you have serial crystallographic data from a synchrotron.<br />
** Build a psana/cctbx.xfel build [https://exafel.github.io/docs/psana-cctbx-install from scratch]. Use this option if you want to process LCLS data outside of LCLS or want your own build.<br />
* [[Tutorials]] on pre-processing, data reduction, and merging. <br />
* ''[[cctbx.prime]]'': tutorials on post-refinement using PRIME. <br />
* ''[[IOTA]]'': tutorial on spotfinding optimization using ''IOTA''.<br />
<br />
Other related information:<br />
<br />
* [[Serial XFEL Crystallography References]]<br />
* [[Data Processing at SACLA]]<br />
* [[2017 Tutorials]]. Berkeley Lab Serial Crystallography Workshop, 16-17 Feb 2017.<br />
* [[Experiment-day guidelines]]: a short, very hands-on set of notes for online monitoring and data-processing using cctbx.xfel. This page assumes that the setup and tutorial have been completed.<br />
* [[File formats]]: for developers, information on file formats ''cctbx.xfel'' uses<br />
* [[Cppxfel]]: XFEL data processing with the CPPXFEL package from Diamond/Wellcome Trust.<br />
* For developers only: [[Installation]] instructions for cctbx developmental build.<br />
* [[Ha14 documentation]]: tutorials and support for processing using the [https://doi.org/10.1038/nmeth.2887 Hattne 2014] methods.<br />
<br />
This project is under active development. For any assistance, please contact the authors.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/Ha14_documentationHa14 documentation2018-11-06T23:15:02Z<p>Aaron: Created page with "THIS PAGE IS DEPRECATED. This page details processing procedures described in [https://doi.org/10.1038/nmeth.2887 Hattne 2014] (Ha14), using LABELIT for initial indexing and..."</p>
<hr />
<div>THIS PAGE IS DEPRECATED. This page details processing procedures described in [https://doi.org/10.1038/nmeth.2887 Hattne 2014] (Ha14), using LABELIT for initial indexing and stills-specific refinement and integration code implemented in the package ''cctbx.rstbx''<br />
<br />
* [[Ha14 Tutorials]] general tutorials for processing LCLS data<br />
* [[Processing Blog]] describing how we solved typical problems. <br />
* [[Processing L498 thermolysin]] details instructions to reproduce the results published in the forthcoming ''cctbx.xfel'' paper.<br />
* [[2014 Tutorials | 2014 Workshop Tutorial: ''cctbx.xfel'' ]]. Note, due to recent changes at LCLS, many of these tutorials are out of date for data collected late 2014 and onwards. For more up to date information, see above [[tutorials]]. For data collected before late 2014, these workshop tutorials are useful.<br />
* [[2014_workshop | Powerpoint presentations]]</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/InstallationInstallation2018-11-06T23:14:22Z<p>Aaron: </p>
<hr />
<div>Incoming!</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/Ha14_installationHa14 installation2018-11-06T23:13:49Z<p>Aaron: Aaron moved page Installation to Ha14 installation: deprecated installation instructions</p>
<hr />
<div>== Prerequisites ==<br />
<br />
While these tutorials assume you wish to process XTC streams at SLAC, some users have stills collected from other sources and do not need the full PSDM suite. If this is the case, see the installation directions below for installing on a non PSDM system. Otherwise, it is assumed that the [[Set up PSDM software | PSDM software distribution has been set up]]. Note again that several sites already have ''cctbx.xfel'' installed, and so regular users not involved in the development of the software will not need the instructions here.<br />
<br />
Once ''cctbx.xfel'' has been installed it must be [[Setup | set up]] before it can be used. Developers may additionally want to [[Set up ssh-agent | set up an ssh-agent]]. <br />
<br />
Finally, you must have a user account on cci.lbl.gov in order to proceed, or else you will not be able to download the sources. A sourceforge account is only needed if you wish to commit changes back to the repository. If you don't have or don't want to use a sourceforge account, you can leave it off in the below commands where it is specified.<br />
<br />
== Standard installation using PSDM ==<br />
===Quick start===<br />
This step has to be performed on a host with Internet access. Not all hosts at SLAC have that, but the members of <i>e.g.</i> pslogin pool do. Make and change to a working directory to contain the new source code and build (this directory should be accessible from any computing nodes ''cctbx.xfel'' will be run. Then download these bootstrap modules:<br />
<br />
wget https://raw.githubusercontent.com/cctbx/cctbx_project/master/libtbx/auto_build/bootstrap.py<br />
<br />
Then:<br />
<br />
python bootstrap.py hot update --builder=xfel --cciuser=<cciusername> --sfuser=<githubusername><br />
<br />
This command instructs bootstrap.py to download static packages required for ''cctbx.xfel'' (hot), and then to checkout the rest of the sources from cci.lbl.gov and github (update), using the user accounts specified by the cciuser and sfuser parameters, respectively.<br />
<br />
===Alternate instructions for non-LBNL developers===<br />
Our non-LBNL colleagues will now realize that the "python bootstrap" step does not work without cci username credentials, needed to obtain source code for the program LABELIT. Developers are instructed to use the following alternate procedure:<br />
<br />
python bootstrap.py hot --builder=xfel --sfuser=<githubusername> # download the static packages<br />
python bootstrap.py update --builder=dials --sfuser=<githubusername> # download the github code for dials & cctbx<br />
mkdir modules/cxi_xdr_xes # create a stub directory for experiment-specific python code<br />
<br />
Then download phenix from http://phenix-online.org and untar the package. Locate the modules directory in the phenix directory tree, and copy-paste the labelit subdirectory into the modules directory of your current working area.<br />
<br />
===Finish up the build===<br />
After downloading the sources, you need to be sure you have the appropriate compilers before executing the next command. At SLAC, that means you need to ssh to one of the psana nodes, as the pslogin nodes do not have the requisite compilers. When ready, configure and compile thusly:<br />
<br />
python bootstrap.py build --builder=xfel --with-python=`which python` --nproc=<# cores available for compile><br />
<br />
On SLAC's interactive nodes, this takes just over 6 minutes. To avoid problems with run-time dynamic linking of Python extensions, the Python interpreter required for the above command must be the one provided by the PSDM software distribution. That interpreter can be located using find, <i>e.g.</i><br />
$ find $SIT_ROOT/sw/external/python -perm /0111 -type f -wholename "*/$SIT_ARCH/*/python" 2> /dev/null<br />
At SLAC this interpreter is located somewhere under <code>/reg/g/psdm/sw/external/python</code>. Here we use `which python` to get this path automatically. <br />
<br />
Initialise the running shell using the newly created configuration files. bash-users should<br />
$ . build/setpaths.sh<br />
while csh-users will instead need to run<br />
% source build/setpaths.csh<br />
<br />
To finalize the installation, see [[Setup]].<br />
<br />
Note, you may also follow this procedure if you are on a machine where you have your own python that you want to use, instead of the one provided by PSDM.<br />
<br />
== Installation on a non-PSDM system ==<br />
<br />
If you find yourself on a machine without PSDM and won't be dealing with XTC directly, for example if you have your own data collected as stills from a non-XFEL source, you can use the below commands in a new directory (again assuming you have a cci user account):<br />
<br />
# On Linux:<br />
wget https://raw.githubusercontent.com/cctbx/cctbx_project/master/libtbx/auto_build/bootstrap.py<br />
# On MacOSX:<br />
curl https://raw.githubusercontent.com/cctbx/cctbx_project/master/libtbx/auto_build/bootstrap.py > bootstrap.py<br />
<br />
python bootstrap.py hot update base build --builder=xfel --cciuser=<cciusername> --sfuser=<sourceforgeusername><br />
<br />
Here, in addition to hot, update and build, there's a command called 'base', which downloads a python and needed packages and compiles it. After this process completes, source the installation (for details see above), and then add scipy to it:<br />
<br />
. build/setpaths.sh<br />
libtbx.python -m easy_install scipy<br />
<br />
Please contact the authors if with any issues. Some gotchas:<br />
<br />
1) On a Mac, you will need to make sure you have the latest version of xcode installed. Consider the command xcode-select --install. [NOTE: Xcode v. 8.2.1. for Mac OS 10.12 (Sierra) does not include a gFortran compiler. You will need to install gFortran separately if you need scipy.]<br />
<br />
2) On linux you will likely need to install some dependencies, including gFortran. [Note: gFortran is needed only for the scipy installation, not for the core build based on cctbx. Therefore, only install gFortran if you need scipy.]<br />
<br />
== External links ==<br />
<br />
* [http://cctbx.sourceforge.net/current_cvs/installation.html#manually-building-from-sources-under-unix Manually building from sources under Unix]<br />
<br />
<br />
<!-- POSSBILE ADDITIONS:<br />
<br />
COMMON PITFALLS:<br />
<br />
forget to source the shell configuration files<br />
forget to run sit_setup<br />
running cxi.pyana from outside the myrelease directory<br />
<br />
DEBUGGING:<br />
<br />
And a few more things to check: output of "ls -lR arch/*", output of "which python2.7", and the "scons.out" file that results from "scons TRACE=7 > scons.out 2>&1".<br />
<br />
What Linux distribution are you running this on (could you past the output of "uname -a" and "cat /etc/issue")? Could you also tell me which scons you are using (output of "which scons"). Last, could you attach a dump of the environment (env > env.out)?<br />
<br />
Andy's information request on debugging: full output from these commands:<br />
<br />
% rm -rf arch build<br />
% printenv<br />
% which scons<br />
% which python<br />
% which python2.7<br />
% scons TRACE=9<br />
% ls -lR arch<br />
% ls -lR my_ana_pkg<br />
<br />
--></div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/InstallationInstallation2018-11-06T23:13:49Z<p>Aaron: Aaron moved page Installation to Ha14 installation: deprecated installation instructions</p>
<hr />
<div>#REDIRECT [[Ha14 installation]]</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/File_formatsFile formats2018-11-06T23:10:13Z<p>Aaron: </p>
<hr />
<div>== Introduction ==<br />
<br />
<br />
''cctbx.xfel'''s current native file formats, image pickles and integration pickles, are mainly intermediate file formats useful for debugging and software development. These binary pickle files are serialized python dictionaries optimized for machine readability. The rationale here is that the raw data from an experiment at LCLS is stored in the xtc streams. We process these in memory without needing to write images to disk at all, but provide image pickles if requested for use with the ''cctbx'' image viewer for diagnostic purposes. Integration pickles contain integrated intensities and crystal cell and orientation parameters and are read by the merging programs cxi.merge and prime.postrefine.<br />
<br />
In an effort to provide more human-readable data files and conform to international standards, we have worked with representatives of ImageCIF to create 64 tile segmented data in CBF format, the same format used in a wide variety of detectors. These CSPAD CBFs are designed to be used with other crystallographic software packages.<br />
<br />
Format specifications for these data files are provided below.<br />
<br />
== Image pickles ==<br />
<br />
''cctbx.xfel'' image pickles are a binary file format containing pixel data and image metadata. Under the hood these are serialized python dictionaries, but for general users it's sufficient to know that the blob of data consists of name/value pairs. The contents of an image pickle can be inspected with this command:<br />
<br />
'''cxi.print_pickle''' image.pickle<br />
<br />
For example, here's the output from an image with a thermolysin diffraction pattern collected recently:<br />
<br />
<div class="toccolours mw-collapsible mw-collapsed"><br />
<pre><br />
Printing contents of idx-20141201073601249.pickle<br />
Detector format version: CXI 10.1<br />
DISTANCE 119.002<br />
DETECTOR_ADDRESS CxiDs1-0|Cspad-0<br />
SIZE1 1765<br />
SIZE2 1765<br />
TIMESTAMP 2014-12-01T07:36Z01.249<br />
CCD_IMAGE_SATURATION 90000<br />
SATURATED_VALUE 90000<br />
64 active areas, first one: [715, 439, 909, 624]<br />
PIXEL_SIZE 0.11<br />
BEAM_CENTER_Y 97.075<br />
MIN_TRUSTED_VALUE -2000<br />
WAVELENGTH 1.75124427107 , converted to eV: 7079.77687909<br />
xtal_target None<br />
BEAM_CENTER_X 97.075<br />
SEQUENCE_NUMBER 0<br />
DATA len=3115225 max=106065.000000 min=-104526.000000 dimensions=(1765, 1765)<br />
cxi_versioned_extract()::cxi_version: CXI 10.1<br />
64 translated active areas, first one: [725, 449, 917, 632]<br />
</pre><br />
</div><br />
<br />
Each of the lines here is a name/value pair of data or metadata for the image. LABELIT, DIALS, cctbx.image_viewer, are all programs capable of processing these files directly. Here's a description of each of the name/value pairs:<br />
* Detector format version: CXI 10.1. Technically this value is not stored in the image pickle. The detector format version is used internally by cctbx.xfel for data collected up to LCLS run 11, and is a way of looking up tile corrections stored in the software. After run 11, this information should be stored in the phil file used to processes the data. The detector format version is looked up based on the detector address and event timestamp. All known detector format versions can be displayed with the command '''cxi.detector_format_versions'''.<br />
* DISTANCE: detector distance (mm)<br />
* DETECTOR_ADDRESS: string to identify the detector in the XTC stream. Recorded here as a unique identifier for the data when coupled with the event timestamp.<br />
* SIZE1 and SIZE2: image dimensions (pixels)<br />
* TIMESTAMP: unique time stamp for this event in the form year-month-day'''T'''hour:minute'''Z'''second.microsecond<br />
* CCD_IMAGE_SATURATION and SATURATED_VALUE: overload value<br />
* PIXEL_SIZE: pixel size in mm<br />
* BEAM_CENTER_X, BEAM_CENTER_Y: beam center, specified as positive mm offsets from the origin of the detector (upper-left corner)<br />
* MIN_TRUSTED_VALUE: underload<br />
* WAVELENGTH: wavelength as recorded in the XTC stream (angstroms)<br />
* DATA: pixel data. Some vital statistics are shown.<br />
* xtal_target and SEQUENCE_NUMBER: unused<br />
* 64 active areas: the active areas are coordinates that specify where the 64 tiles are in the image. They are in the form of quartets of numbers [x1,y1,x2,y2], one quartet for each of the 64 tiles. The first active area is shown. Note, for non-CSPAD image pickles, there will only be one active area and it will be the size of the image entire.<br />
* 64 translated active areas: if corrections were available for this image, the first corrected active area is shown here.<br />
<br />
=== A note on coordinate systems ===<br />
<br />
Internally, our software uses the ImageCIF coordinate system with the exception that no goniometer is typically available at XFELs, therefore Z is along the beam (sample to source), Y points against gravity and X completes a right-handed coordinate system. In the image pickles, the origin of the detector pixel readout is assumed to be in the upper left of the image, therefore the beam center is specified as positive offsets from that point. In example above, assuming the sample is at the laboratory origin, the vector pointing to the beam center on the detector in the ImageCIF coordinate system would be (0, 0, -119.002). A vector from the laboratory origin to the pixel readout origin (upper-left of the detector) would be -97.075, 97.075, -119.002). dxtbx.print_header is a useful tool to get these vectors from any file format understood by ''cctbx.xfel''. For more information see [http://cci.lbl.gov/publications/download/jo5001_reprint.pdf Parkhurst et al, 2014].<br />
<br />
=== Creating image pickles from xtc streams ===<br />
<br />
The program cctbx.xfel.xtc_dump can be used to convert xtc streams to image pickle format. Here we provide an example for the Rayonix detector that is often found at XPP and MFX. The psana module mod_image_dict is used to read the data and generate the images (example: [[mod_image_dict|mod_image_dict.cfg]]). After setting up a release directory as described in [[Setup]]:<br />
<br />
cd ~/myrelease # or wherever your release is<br />
sit_setup<br />
cctbx.xfel.xtc_dump experiment=mfxm9316 run_num=14 address="MfxEndstation-0|Rayonix-0" file_format=pickle cfg=mod_image_dict.cfg <br />
<br />
This will fill your working folder with image pickles. You can change your output directory with output.output_dir and you can limit the number of images saved with dispatch.max_events.<br />
<br />
This approach will also work for CSPAD images. Here, you must add calib_dir to your config file, as described in the indexing tutorials.<br />
<br />
== Integration pickles ==<br />
<br />
Integration pickles are the output of cctbx.xfel indexing and integration. They are like image pickles in that they are serialized python dictionaries and can be inspected with '''cxi.print_pickle'''. They are the primary input to '''cxi.merge''' and '''prime.run'''.<br />
<br />
== CSPAD CBFs ==<br />
<br />
CSPAD CBFs are used while processing CSPAD images using the DIALS backend of ''cctbx.xfel''. The complete specification for CSPAD CBFs is laid out in an article in the Computational Crystallographic Newsletter: "XFEL Detectors and ImageCIF", Computational Crystallography Newsletter 5, 19-24. ([http://cci.lbl.gov/publications/download/CCN_2014_p19.pdf Reprint]). Documentation for using DIALS to index and integrate CSPAD data can be found in an article in the Computational Crystallographic Newsletter: "Processing XFEL data with cctbx.xfel and DIALS", Computational Crystallography Newsletter 7, 32-53 ([http://www.phenix-online.org/newsletter/CCN_2016_07.pdf#page=18 Reprint]).<br />
<br />
=== Creating CSPAD CBFs from XTC streams ===<br />
<br />
cctbx.xfel.xtc_dump is useful for converting XTC streams to CBF files as needed. Use -c to get a listing of all options, and -c -a 2 to get a full listing with help strings. Example command:<br />
<br />
cd ~/myrelease # or wherever your release is<br />
sit_setup<br />
cctbx.xfel.xtc_dump dispatch.max_events=1 input.experiment=xpptut15 input.address=cspad \<br />
input.run_num=54 format.file_format=cbf output.output_dir=xpptut15/out \<br />
format.cbf.detz_offset=100 input.override_energy=7000<br />
<br />
Here, one image is created from experiment xpptut15, run 54. You can display the image with cctbx.image_viewer xpptut15/out/*.cbf. Note the pink gaps between the tiles. This results from the segmented nature of the CSPAD, preserved in the CBF file.<br />
<br />
xpptut15 is data from XPP's CSPAD and was collected without xray's on. Hence format.cbf.detz_offset=100 input.override_energy=7000 are set to fake values in this command.<br />
<br />
=== Converting from SLAC's metrology to CBF ===<br />
<br />
The tile positions of the hierarchical CSPAD detector are specified by SLAC in a geometry file in the calib folder of each experiment. Typically this file is named something like 0-end.data. If desired, this file can be converted to just the human readable header portion of a CSPAD cbf using the ''cctbx.xfel'' command cxi.slaccalib2cbfheader. Example:<br />
<br />
cxi.slaccalib2cbfheader metrology_file=0-end.data out=tmp.cbf<br />
<br />
This cbf header can be converted back to SLAC format using cxi.cbfheader2slaccalib:<br />
<br />
cxi.cbfheader2slaccalib cbf_header=tmp.cbf out_metrology_file=tmp.data<br />
<br />
=== Displaying metrology files ===<br />
<br />
cxi.display_metrology <filename> can be used to show a plot of tile positions. It accepts SLAC metrology files, CSPAD CBFs and image pickles.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/Main_PageMain Page2018-11-06T23:06:27Z<p>Aaron: </p>
<hr />
<div>= Open-source tools for serial crystallographic data processing =<br />
<br />
''cctbx.xfel'' is a suite of software tools designed to process diffraction data from serial femtosecond crystallography (SFX) measurements at an X-ray free-electron laser (XFEL) or a synchrotron. Built on the Computational Crystallographic Toolbox ([http://cctbx.sourceforge.net ''cctbx'']), the same toolbox on which [http://www.phenix-online.org ''PHENIX''], [http://dials.github.io DIALS], [http://adder.lbl.gov/labelit ''LABELIT''], and post-refinement and merging program, [http://viper.lbl.gov/cctbx.xfel/index.php/Cctbx.prime ''PRIME''] are built, it enables the user to solve difficult problems relating to processing serial crystallographic data. The programs and modules provided by ''cctbx.xfel'' can reduce a large set of still diffraction images recorded at Stanford’s Linac Coherent Light Source ([http://lcls.slac.stanford.edu LCLS]), [http://sacla.xfel.jp SACLA], the [https://www.xfel.eu/ European XFEL], or a synchrotron, to a single MTZ file containing merged reflection intensities suitable for structure solution.<br />
<br />
== ''cctbx.xfel'' resources ==<br />
<br />
The tutorials on this wiki provide detailed instructions for indexing and integrating still diffraction images extracted from the raw data streams recorded at the LCLS, including pre-processing steps such as dark pedestal generation and refinement of the detector geometry of the Cornell–SLAC pixel array detectors (CSPAD) in use at the CXI and XPP end stations. The tutorials also cover tools to efficiently leverage the LCLS computing cluster to process the thousands to millions of diffraction images that can be recorded in a short time. Finally, processing approaches for data collection at non-LCLS sources are also provided.<br />
<br />
* [[Overview]] to the system architecture at LCLS and real-time progress monitoring of data processing<br />
* Installation: there are two ways to get ''cctbx.xfel'':<br />
** ''cctbx.xfel'' is installed for general use at SLAC. To start using an existing installation, [[Using the pre-built cctbx at LCLS|simply source the current build]]. <br />
** [https://dials.github.io/installation.html Download] a binary bundle of DIALS. Use this option if you will not be processing LCLS data because, for example, if you have serial crystallographic data from a synchrotron.<br />
** Build a psana/cctbx.xfel build [https://exafel.github.io/docs/psana-cctbx-install from scratch]. Use this option if you want to process LCLS data outside of LCLS or want your own build.<br />
* [[Tutorials]] on pre-processing, data reduction, and merging. <br />
* ''[[cctbx.prime]]'': tutorials on post-refinement using PRIME. <br />
* ''[[IOTA]]'': tutorial on spotfinding optimization using ''IOTA''.<br />
<br />
Other related information:<br />
<br />
* [[Serial XFEL Crystallography References]]<br />
* [[Data Processing at SACLA]]<br />
* [[2017 Tutorials]]. Berkeley Lab Serial Crystallography Workshop, 16-17 Feb 2017.<br />
* [[2014 Tutorials | 2014 Workshop Tutorial: ''cctbx.xfel'' ]]. Note, due to recent changes at LCLS, many of these tutorials are out of date for data collected late 2014 and onwards. For more up to date information, see above [[tutorials]]. For data collected before late 2014, these workshop tutorials are useful.<br />
* [[2014_workshop | Powerpoint presentations]]<br />
* [[Experiment-day guidelines]]: a short, very hands-on set of notes for online monitoring and data-processing using cctbx.xfel. This page assumes that the setup and tutorial have been completed.<br />
* [[File formats]]: for developers, information on file formats ''cctbx.xfel'' uses<br />
* [[Cppxfel]]: XFEL data processing with the CPPXFEL package from Diamond/Wellcome Trust.<br />
* For developers only: [[Installation]] instructions for cctbx developmental build.<br />
* [[Ha14 documentation]]: tutorials and support for processing using the [https://doi.org/10.1038/nmeth.2887 Hattne 2014] methods.<br />
<br />
This project is under active development. For any assistance, please contact the authors.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/Ha14_TutorialsHa14 Tutorials2018-11-06T23:05:20Z<p>Aaron: </p>
<hr />
<div>THIS PAGE IS DEPRECATED. This page details processing procedures described in [https://doi.org/10.1038/nmeth.2887 Hattne 2014] (Ha14), using LABELIT for initial indexing and stills-specific refinement and integration code implemented in the package ''cctbx.rstbx''<br />
<br />
Processing xfel data at LCLS consists of four steps: <br />
<br />
* Averaging and other [[Preparatory steps]]<br />
* [[Metrology refinement]]<br />
* [[Indexing and integration]]. Optional steps:<br />
** Real-time [[progress monitoring]].<br />
** Streaming [[data viewer]].<br />
** Configuring specific spotfinder, indexing and integration parameters using ''[[phil]]''.<br />
* [[Merging]]<br />
* [[Advanced Merging]]: Use cases for isomorphous replacement and <em>de novo</em> structure<br />
* ''[[cctbx.prime]]'': Merging using the ''prime.postrefine'' program.<br />
* [[Resolving an Indexing Ambiguity]]<br />
<br />
Processing outside of LCLS can be done by [[Indexing individual stills]]<br />
<br />
Additionally, ''cctbx.xfel'' utilizes ''[[phil]]'' files to manage its configuration settings.<br />
<br />
[[Processing L498 thermolysin]] details instructions to reproduce the results published in the forthcoming ''cctbx.xfel'' paper.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/Indexing_individual_stillsIndexing individual stills2018-11-06T23:04:28Z<p>Aaron: </p>
<hr />
<div>THIS PAGE IS DEPRECATED. This page details processing procedures described in Hattne 2014 (Ha14), using LABELIT for initial indexing and stills-specific refinement and integration code implemented in the package ''cctbx.rstbx''<br />
<br />
== Background ==<br />
<br />
Recent advances at synchrotron and XFEL sources, such as brighter light and new injection technologies, are enabling the collection of crystal diffraction data by non-traditional means, specifically, without the use of a goniometer. Without rotational information, still photographs of single crystals present new challenges for indexing software. cctbx.xfel was designed to address these issues for still images collected from X-ray free electron lasers, specifically when processing with the significant computing and storage facilities available at the Stanford Linear Accelerator. However, as still images become more common for other sources, we have adapted our first public release of cctbx.xfel to be able to process still images from non-XFEL sources. A complete guide is presented here.<br />
It is worth noting that a significant revision of cctbx.xfel is underway, which will make much of this tutorial obsolete. However, until then we hope the information presented here is useful.<br />
<br />
== Converting image files to the cctbx.xfel pickle image format ==<br />
<br />
cctbx.xfel has certain requirements on the images it indexes. While some crystal images are directly compatible with the software, many are not, due to two limitations:<br />
1) cctbx.xfel requires a complete description of the detector geometry and experimental parameters to be present in the image’s header. Generally this consists of detector distance, wavelength, pixel size (square pixels are assumed), saturation limits and beam center. Initial estimates of these values are required to a certain accuracy, and are often missing from the image header.<br />
2) For legacy reasons, cctbx.xfel currently requires the beam center to match the image center (though this will not be true in the near future). <br />
In order to use images that don't satisfy both of these criteria, use cxi.image2pickle to convert them to cctbx.xfel image pickles:<br />
<br />
$ cxi.image2pickle --help<br />
Usage: cxi.image2pickle [-v] [-c] [-s] [-w wavelength] [-d distance] [-p pixel_size] [-x beam_x] [-y beam_y] [-o overload] files<br />
Options:<br />
-h, --help show this help message and exit<br />
-v, --verbose Print more information about progress<br />
-c, --crop Crop the image such that the beam center is in the<br />
middle<br />
-s, --skip-converted Skip converting if an image already exists that matches<br />
the destination file name<br />
-w WAVELENGTH, --wavelength=WAVELENGTH<br />
Override the image's wavelength (angstroms)<br />
-d DISTANCE, --distance=DISTANCE<br />
Override the detector distance (mm)<br />
-p PIXEL_SIZE, --pixel-size=PIXEL_SIZE<br />
Override the detector pixel size (mm)<br />
-x BEAM_CENTER_X, --beam-x=BEAM_CENTER_X<br />
Override the beam x position (pixels)<br />
-y BEAM_CENTER_Y, --beam-y=BEAM_CENTER_Y<br />
Override the beam y position (pixels)<br />
-o OVERLOAD, --overload=OVERLOAD<br />
Override the detector overload value (ADU)<br />
Note, if the image already has elements that don't need to be overridden, such as wavelength or detector distance, they can be omitted as the program will try to determine them automatically. The program will not run if all of the information is not present.<br />
The -c option (crop) will crop the image such that the image center matches the beam center, and adjust the beam center location accordingly. This option is recommended for most data. However, if this leads to significant loss of pixels at the edges, please contact the authors. A padding option can be made available instead.<br />
<br />
== Verifying the experimental geometry with virtual powder rings ==<br />
<br />
Optionally, the detector distance, wavelength, and beam center can be verified using a maximum projection of the images availalible. A maximum projection is a composite image where each pixel represents the brightest pixel in a given set of images. After converting the images in question to pickle, use cxi.image_average:<br />
$ cxi.image_average --help<br />
Usage: cxi.image_average [-v] [-a PATH] [-m PATH] [-s PATH] image1 image2 [image3 ...]<br />
Options:<br />
-h, --help show this help message and exit<br />
-a PATH, --average-path=PATH<br />
Write average image to PATH<br />
-m PATH, --maximum-path=PATH<br />
Write maximum projection image to PATH<br />
-s PATH, --stddev-path=PATH<br />
Write standard deviation image to PATH<br />
-v, --verbose Print more information about progress<br />
The resultant maximum projection can be inspected with cctbx.image_viewer. Once loaded, the ring tool in the Actions menu can be used to verify the beam center is aligned to the center of the virtual powder rings from the maximum projection. If it is not, use cxi.image2pixel with further adjustments.<br />
<br />
To verify the distance and wavelength, assuming the unit cell of the crystal is already known, use cxi.view thusly (given a maximum projection named max.pickle):<br />
$ cxi.view max.pickle viewer.calibrate_unitcell.d_min=4 viewer.calibrate_unitcell.unitcell=78,78,38,90,90,90 viewer.calibrate_unitcell.spacegroup=P43212<br />
Here, powder rings for lysozyme will be displayed. The distance can be changed until the rings align. Then, regenerate your image pickles using cxi.image2pickle and the new parameters.<br />
<br />
== Determine the best spotfinder parameters ==<br />
<br />
Every crystal preparation is different, from the size of the crystals to the composition of the mother liquor that contributes to the background. Because of this, it is important to optimize your spotfinder parameters for your specific data of interest. We use Labelit’s spotfinder, whose parameters are described in detail here [http://adder.lbl.gov/labelit/html/spotfinder.html]. The defaults are fairly well tuned for synchrotron data in general, but can easily need further adjustment. Two programs are of interest, distl.image_viewer and distl.signal_strength . The former displays the image with spotfinder spots highlighted and the latter simply prints a summary of spot finder spot counts in various categories. We recommended viewing the image with the default settings, then tweaking distl.minimum_spot_area or distl.minimum_signal_height and distl.minimum_spot height (the latter two should usually be the same). Pass the new parameters using this syntax:<br />
$ distl.image_viewer <image file> distl.minimum_spot_area=2<br />
The goal here is to adjust the parameters until only real spots are found by the program and noise is not. It is also feasible to optimize these parameters using a grid search and cxi.index (see below).<br />
<br />
== Using cxi.index to process the images ==<br />
<br />
(If using the CSPAD detector with cxi.index, see important note below)<br />
<br />
cxi.index is the command line interface to our indexing software. It uses the Rossmand DPS indexing algorithm as implemented by Labelit to get initial basis vectors, then uses still-specific algorithms for reflection prediction, as documented here [http://dx.doi.org/10.1107/S0907444913000863] and here [http://dx.doi.org/10.1038/nmeth.2887]. Notably, we have recently added multiprocessing to cxi.index (see the -n command). Here is the full description:<br />
$ cxi.index --help<br />
Usage: cxi.index [-d] [-n num_procs] [-o output_dir] [-b output_basename] target=targetpath files<br />
Options:<br />
-h, --help show this help message and exit<br />
-d, --no-display Do not show indexing graphics<br />
-n NUM_PROCS, --num-procs=NUM_PROCS<br />
Number of processors to use<br />
-o OUTPUT_DIR, --output-dir=OUTPUT_DIR<br />
Directory for integration pickles<br />
-b OUTPUT_BASENAME, --output-basename=OUTPUT_BASENAME<br />
String to append to the front of output integration<br />
pickles<br />
Target: the phil file containing further indexing/integration parameters<br />
The simplest way to index a single image is as follows:<br />
$ cxi.index <imagefile> target=<phil file><br />
This saves no output and displays three views of the image (assuming indexing was successful. The first is the image indexed in the P1 setting. Red pixels are spotfinder spots, blue are integration mask pixels and yellow are background pixels. The second display is as the first, but in the highest symmetry setting found by indexing, and with an automatically determined resolution cutoff applied. A third is also displayed showing only the spotfinder spots. To hide these displays, use -d. The target is a phil file with sample-specific indexing settings as described in our [[Phil]] section (though note the discussion of metrology is only relevant for the CSPAD detector).<br />
A typical indexing run of a large set of images in a single directory, assuming they have been converted with cxi.image2pickle and that there are 12 cores available, that the directory results/000 exists, using a phil file named target.phil, is as follows:<br />
$ cxi.index -d -n 12 -o results/000 target=target.phil *.pickle | tee results/trial_000.log<br />
After indexing, results/000 will contain files of the form int_<original file name>.pickle. These are not image pickles, but they are integration pickles, containing the integrated intensities and miller indices for each image. They are the direct input for cxi.merge, as described in [Merging] and [Advanced Merging]<br />
<br />
=== Using a grid search with cxi.index ===<br />
<br />
It is usually pretty straightforward to find spotfinder parameters that work well for your data. However, if you'd like to be more thorough below is an example command for testing a variety of spotfinding parameters in a grid-based fashion. Comment out the overridden spotfinder parameters from this command in your phil file first. Assume a directory named results exists in the current folder.<br />
<br />
$ for i in `seq 2 6`; do for j in `seq 2 6`; do mkdir -p results/trial_a${i}_s${j}; cxi.index <pickle files> target=<path to phil file> \<br />
-d -o results/trial_a${i}_s${j} distl.minimum_spot_area=$i distl.minimum_signal_height=${j} distl.minimum_spot_height=${j} \<br />
| tee results/trial_a${i}_s${j}.log; done; done<br />
<br />
Using -n here for cxi.index will be useful! If, say, 100 images are tested, whichever combination gives the most number of indexed images is likely best for the entire dataset.<br />
<br />
== Important note for the CSPAD detector ==<br />
<br />
If you are using cxi.index with image pickles made from ''cctbx.xfel'' from an LCLS CSPAD datastream, you need to add distl.detector_format_version to your cxi.index command, or add the metrology directly to your phil file. If the data was collected during run 9 or before (Summer of 2014), use cxi.print_pickle on the image and use the value displayed for detector format version. For example, it could say CXI 7.1, so you would add detector_format_version="CXI 7.1" to your cxi.index command. After run 9, we stopped updating ''cctbx.xfel'' to include metrology natively so the user needs to use ''cspad.metrology'' to generate the pixel corrections. In that case, if the corrections are in your target phil file, distl.detector_format_version should not be needed. For more information, see [[Metrology refinement]].</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/TutorialsTutorials2018-11-06T23:01:22Z<p>Aaron: </p>
<hr />
<div>See the list of tutorials as part of our recent paper, Brewster 2018, [https://github.com/phyy-nx/dials_refinement_brewster2018/wiki here].</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/Main_PageMain Page2018-11-06T22:48:34Z<p>Aaron: /* cctbx.xfel resources */</p>
<hr />
<div>= Open-source tools for serial crystallographic data processing =<br />
<br />
''cctbx.xfel'' is a suite of software tools designed to process diffraction data from serial femtosecond crystallography (SFX) measurements at an X-ray free-electron laser (XFEL) or a synchrotron. Built on the Computational Crystallographic Toolbox ([http://cctbx.sourceforge.net ''cctbx'']), the same toolbox on which [http://www.phenix-online.org ''PHENIX''], [http://dials.github.io DIALS], [http://adder.lbl.gov/labelit ''LABELIT''], and post-refinement and merging program, [http://viper.lbl.gov/cctbx.xfel/index.php/Cctbx.prime ''PRIME''] are built, it enables the user to solve difficult problems relating to processing serial crystallographic data. The programs and modules provided by ''cctbx.xfel'' can reduce a large set of still diffraction images recorded at Stanford’s Linac Coherent Light Source ([http://lcls.slac.stanford.edu LCLS]), [http://sacla.xfel.jp SACLA], the [https://www.xfel.eu/ European XFEL], or a synchrotron, to a single MTZ file containing merged reflection intensities suitable for structure solution.<br />
<br />
== ''cctbx.xfel'' resources ==<br />
<br />
The tutorials on this wiki provide detailed instructions for indexing and integrating still diffraction images extracted from the raw data streams recorded at the LCLS, including pre-processing steps such as dark pedestal generation and refinement of the detector geometry of the Cornell–SLAC pixel array detectors (CSPAD) in use at the CXI and XPP end stations. The tutorials also cover tools to efficiently leverage the LCLS computing cluster to process the thousands to millions of diffraction images that can be recorded in a short time. Finally, processing approaches for data collection at non-LCLS sources are also provided.<br />
<br />
* [[Overview]] to the system architecture at LCLS and real-time progress monitoring of data processing<br />
* Installation: there are two ways to get ''cctbx.xfel'':<br />
** ''cctbx.xfel'' is installed for general use at SLAC. To start using an existing installation, [[Using the pre-built cctbx at LCLS|simply source the current build]]. <br />
** [https://dials.github.io/installation.html Download] a binary bundle of DIALS. Use this option if you will not be processing LCLS data because, for example, if you have serial crystallographic data from a synchrotron.<br />
** Build a psana/cctbx.xfel build [https://exafel.github.io/docs/psana-cctbx-install from scratch]. Use this option if you want to process LCLS data outside of LCLS or want your own build.<br />
* [[Tutorials]] on pre-processing, data reduction, and merging. <br />
* ''[[cctbx.prime]]'': tutorials on post-refinement using PRIME. <br />
* ''[[IOTA]]'': tutorial on spotfinding optimization using ''IOTA''.<br />
<br />
Other related information:<br />
<br />
* [[Serial XFEL Crystallography References]]<br />
* [[Data Processing at SACLA]]<br />
* [[Indexing individual stills]]: how to index stills not necessarily from an XFEL source.<br />
* [[Processing Blog]] describing how we solved typical problems. <br />
* [[Processing L498 thermolysin]] details instructions to reproduce the results published in the forthcoming ''cctbx.xfel'' paper.<br />
* [[2017 Tutorials]]. Berkeley Lab Serial Crystallography Workshop, 16-17 Feb 2017.<br />
* [[2014 Tutorials | 2014 Workshop Tutorial: ''cctbx.xfel'' ]]. Note, due to recent changes at LCLS, many of these tutorials are out of date for data collected late 2014 and onwards. For more up to date information, see above [[tutorials]]. For data collected before late 2014, these workshop tutorials are useful.<br />
* [[2014_workshop | Powerpoint presentations]]<br />
* [[Experiment-day guidelines]]: a short, very hands-on set of notes for online monitoring and data-processing using cctbx.xfel. This page assumes that the setup and tutorial have been completed.<br />
* [[File formats]]: for developers, information on file formats ''cctbx.xfel'' uses<br />
* [[Cppxfel]]: XFEL data processing with the CPPXFEL package from Diamond/Wellcome Trust.<br />
* For developers only: [[Installation]] instructions for cctbx developmental build.<br />
* [[Ha14 Tutorials]]: tutorials for processing using the [https://doi.org/10.1038/nmeth.2887 Hattne 2014] methods.<br />
<br />
This project is under active development. For any assistance, please contact the authors.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/TutorialsTutorials2018-11-06T22:45:55Z<p>Aaron: </p>
<hr />
<div>INCOMING</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/TutorialsTutorials2018-11-06T22:45:31Z<p>Aaron: Aaron moved page Tutorials to Ha14 Tutorials: deprecated tutorials</p>
<hr />
<div>#REDIRECT [[Ha14 Tutorials]]</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/Ha14_TutorialsHa14 Tutorials2018-11-06T22:45:30Z<p>Aaron: Aaron moved page Tutorials to Ha14 Tutorials: deprecated tutorials</p>
<hr />
<div>THIS PAGE IS DEPRECATED. This page details processing procedures described in [https://doi.org/10.1038/nmeth.2887 Hattne 2014] (Ha14), using LABELIT for initial indexing and stills-specific refinement and integration code implemented in the package ''cctbx.rstbx''<br />
<br />
Processing xfel data at LCLS consists of four steps: <br />
<br />
* Averaging and other [[Preparatory steps]]<br />
* [[Metrology refinement]]<br />
* [[Indexing and integration]]. Optional steps:<br />
** Real-time [[progress monitoring]].<br />
** Streaming [[data viewer]].<br />
** Configuring specific spotfinder, indexing and integration parameters using ''[[phil]]''.<br />
* [[Merging]]<br />
* [[Advanced Merging]]: Use cases for isomorphous replacement and <em>de novo</em> structure<br />
* ''[[cctbx.prime]]'': Merging using the ''prime.postrefine'' program.<br />
* [[Resolving an Indexing Ambiguity]]<br />
<br />
Additionally, ''cctbx.xfel'' utilizes ''[[phil]]'' files to manage its configuration settings.<br />
<br />
[[Processing L498 thermolysin]] details instructions to reproduce the results published in the forthcoming ''cctbx.xfel'' paper.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/Ha14_TutorialsHa14 Tutorials2018-11-06T22:34:46Z<p>Aaron: </p>
<hr />
<div>THIS PAGE IS DEPRECATED. This page details processing procedures described in [https://doi.org/10.1038/nmeth.2887 Hattne 2014] (Ha14), using LABELIT for initial indexing and stills-specific refinement and integration code implemented in the package ''cctbx.rstbx''<br />
<br />
Processing xfel data at LCLS consists of four steps: <br />
<br />
* Averaging and other [[Preparatory steps]]<br />
* [[Metrology refinement]]<br />
* [[Indexing and integration]]. Optional steps:<br />
** Real-time [[progress monitoring]].<br />
** Streaming [[data viewer]].<br />
** Configuring specific spotfinder, indexing and integration parameters using ''[[phil]]''.<br />
* [[Merging]]<br />
* [[Advanced Merging]]: Use cases for isomorphous replacement and <em>de novo</em> structure<br />
* ''[[cctbx.prime]]'': Merging using the ''prime.postrefine'' program.<br />
* [[Resolving an Indexing Ambiguity]]<br />
<br />
Additionally, ''cctbx.xfel'' utilizes ''[[phil]]'' files to manage its configuration settings.<br />
<br />
[[Processing L498 thermolysin]] details instructions to reproduce the results published in the forthcoming ''cctbx.xfel'' paper.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/Using_the_pre-built_cctbx_at_LCLSUsing the pre-built cctbx at LCLS2018-11-06T22:14:29Z<p>Aaron: Created page with "''cctbx.xfel'' is installed for general use at LCLS. To use it, add these lines to your .bashrc file (bash users): <code> source /reg/g/cctbx/conda_build/miniconda2/etc/pr..."</p>
<hr />
<div>''cctbx.xfel'' is installed for general use at LCLS. To use it, add these lines to your .bashrc file (bash users):<br />
<br />
<code><br />
source /reg/g/cctbx/conda_build/miniconda2/etc/profile.d/conda.sh<br />
conda activate myEnv<br />
source /reg/g/cctbx/conda_build/build/setpaths.sh<br />
</code><br />
<br />
Likewise for csh users:<br />
<br />
<code><br />
source /reg/g/cctbx/conda_build/miniconda2/etc/profile.d/conda.csh<br />
conda activate myEnv<br />
source /reg/g/cctbx/conda_build/build/setpaths.csh<br />
</code><br />
<br />
Then log in as normal. All ''DIALS'' and ''cctbx.xfel'' commands will be available.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/Main_PageMain Page2018-11-06T22:10:03Z<p>Aaron: /* Open-source tools for free-electron laser data processing */</p>
<hr />
<div>= Open-source tools for serial crystallographic data processing =<br />
<br />
''cctbx.xfel'' is a suite of software tools designed to process diffraction data from serial femtosecond crystallography (SFX) measurements at an X-ray free-electron laser (XFEL) or a synchrotron. Built on the Computational Crystallographic Toolbox ([http://cctbx.sourceforge.net ''cctbx'']), the same toolbox on which [http://www.phenix-online.org ''PHENIX''], [http://dials.github.io DIALS], [http://adder.lbl.gov/labelit ''LABELIT''], and post-refinement and merging program, [http://viper.lbl.gov/cctbx.xfel/index.php/Cctbx.prime ''PRIME''] are built, it enables the user to solve difficult problems relating to processing serial crystallographic data. The programs and modules provided by ''cctbx.xfel'' can reduce a large set of still diffraction images recorded at Stanford’s Linac Coherent Light Source ([http://lcls.slac.stanford.edu LCLS]), [http://sacla.xfel.jp SACLA], the [https://www.xfel.eu/ European XFEL], or a synchrotron, to a single MTZ file containing merged reflection intensities suitable for structure solution.<br />
<br />
== ''cctbx.xfel'' resources ==<br />
<br />
The tutorials on this wiki provide detailed instructions for indexing and integrating still diffraction images extracted from the raw data streams recorded at the LCLS, including pre-processing steps such as dark pedestal generation and refinement of the detector geometry of the Cornell–SLAC pixel array detectors (CSPAD) in use at the CXI and XPP end stations. The tutorials also cover tools to efficiently leverage the LCLS computing cluster to process the thousands to millions of diffraction images that can be recorded in a short time. Finally, processing approaches for data collection at non-LCLS sources are also provided.<br />
<br />
* [[Overview]] to the system architecture at LCLS and real-time progress monitoring of data processing<br />
* Installation: there are two ways to get ''cctbx.xfel'':<br />
** ''cctbx.xfel'' is installed for general use at SLAC. To start using an existing installation, [[Using the pre-built cctbx at LCLS|simply source the current build]]. <br />
** [https://dials.github.io/installation.html Download] a binary bundle of DIALS. Use this option if you will not be processing LCLS data because, for example, if you have serial crystallographic data from a synchrotron.<br />
** Build a psana/cctbx.xfel build [https://exafel.github.io/docs/psana-cctbx-install from scratch]. Use this option if you want to process LCLS data outside of LCLS or want your own build.<br />
* [[Tutorials]] on pre-processing, data reduction, and merging. <br />
* ''[[cctbx.prime]]'': tutorials on post-refinement using PRIME. <br />
* ''[[IOTA]]'': tutorial on spotfinding optimization using ''IOTA''.<br />
<br />
Other related information:<br />
<br />
* [[Serial XFEL Crystallography References]]<br />
* [[Data Processing at SACLA]]<br />
* [[Indexing individual stills]]: how to index stills not necessarily from an XFEL source.<br />
* [[Processing Blog]] describing how we solved typical problems. <br />
* [[Processing L498 thermolysin]] details instructions to reproduce the results published in the forthcoming ''cctbx.xfel'' paper.<br />
* [[2017 Tutorials]]. Berkeley Lab Serial Crystallography Workshop, 16-17 Feb 2017.<br />
* [[2014 Tutorials | 2014 Workshop Tutorial: ''cctbx.xfel'' ]]. Note, due to recent changes at LCLS, many of these tutorials are out of date for data collected late 2014 and onwards. For more up to date information, see above [[tutorials]]. For data collected before late 2014, these workshop tutorials are useful.<br />
* [[2014_workshop | Powerpoint presentations]]<br />
* [[Experiment-day guidelines]]: a short, very hands-on set of notes for online monitoring and data-processing using cctbx.xfel. This page assumes that the setup and tutorial have been completed.<br />
* [[File formats]]: for developers, information on file formats ''cctbx.xfel'' uses<br />
* [[Cppxfel]]: XFEL data processing with the CPPXFEL package from Diamond/Wellcome Trust.<br />
* For developers only: [[Installation]] instructions for cctbx developmental build.<br />
<br />
This project is under active development. For any assistance, please contact the authors.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/Configure_cctbx.xfel_from_existing_Phenix_installConfigure cctbx.xfel from existing Phenix install2018-11-06T22:02:55Z<p>Aaron: </p>
<hr />
<div>THIS PAGE IS DEPRECATED AND WILL BE DELETED<br />
<br />
Much of the code to run ''cctbx.xfel'' is already bundled in the latest phenix installers. The only missing package is DIALS. Here's how to get ''cctbx.xfel'' configured from an existing Phenix installation.<br />
<br />
Note: this only applies to processing individual stills. Functionality to read XTC streams from LCLS is not included when using this procedure.<br />
<br />
Once Phenix is installed and sourced correctly:<br />
<br />
cd <path to phenix installation>/modules<br />
git clone https://github.com/dials/dials.git<br />
cd ../build<br />
libtbx.configure xfel<br />
make<br />
make<br />
<br />
All cctbx.xfel modules are now scipy free. The installation method described here should work without any installation of other third-party software.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/Set_up_PSDM_softwareSet up PSDM software2018-11-06T21:31:37Z<p>Aaron: </p>
<hr />
<div>THIS IS DEPRECATED. psana is now provided using conda. [https://exafel.github.io/docs/psana-cctbx-instal Directions].<br />
<br />
Note that throughout the instructions <code>$</code> denotes the prompt of any unprivileged user on the system. Certain shells may be configured to use <code>%</code> or <code>&gt;</code> for that purpose. The prompt should <em>not</em> be included in the commands typed to the shell.<br />
<br />
== Prerequisites ==<br />
<br />
The [https://confluence.slac.stanford.edu/display/PSDM/Software+Distribution Photon Science Data Management (PSDM)] software distribution must be installed. The PSDM software distribution provides ''psana'', which is a critical dependency of ''cctbx.xfel''. Its installation root directory affects the absolute paths used in the following instructions and differs from site to site. Some common installation root directories are listed below.<br />
{| class="wikitable sortable"<br />
|-<br />
! Site<br />
! class="unsortable"| Installation root directory<br />
! class="unsortable"| Notes<br />
|-<br />
| BNL<br />
| <code>/usr/local/crys/psdm</code><br />
|<br />
|-<br />
| CCI, LBNL<br />
| <code>/net/viper/raid1/psdm</code><br />
| Only available on viper.lbl.gov<br />
|-<br />
| NERSC<br />
| <code>/global/project/projectdirs/lcls/psdm-carver</code><br />
| Only to be used on the carver system<br />
|-<br />
| SLAC<br />
| <code>/reg/g/psdm</code><br />
|<br />
|}<br />
The PSDM software distribution is supported on the most popular flavors of Linux and is installed at several sites worldwide. Separate installation and maintenance instructions are provided by the PSDM project at [https://confluence.slac.stanford.edu/display/PSDM/Software+Distribution PSDM software distribution]. The package can be installed anywhere on the filesystem, and it is generally possible to install it without superuser privileges.<br />
<br />
== Create a test release and an analysis package ==<br />
<br />
A <i>test release</i>, which refers to a particular release of the PSDM distribution, is represented by a directory in the file system, often called <code>analysis-rel</code> or <code>myrelease</code>. An analysis package within the test release in turn refers to the files implementing the analysis modules of the package. Details about test releases and analysis packages are covered in [https://confluence.slac.stanford.edu/display/PCDS/Analysis+Workbook.+Quick+Tour Analysis Workbook. Quick Tour]. To prepare the environment for using the PSDM software distribution, bash-users should source <code>ana_env.sh</code>,<br />
$ . <b><i>/path/to/psdm</i></b>/etc/ana_env.sh<br />
where <code><b><i>/path/to/psdm</i></b></code> should be replaced with the path to the installation root directory. csh-users should source <code>ana_env.csh</code> instead,<br />
% source <b><i>/path/to/psdm</i></b>/etc/ana_env.csh<br />
Sourcing <code>ana_env.sh</code> or <code>ana_env.csh</code> only modifies the environment of the <em>current shell</em>, and would have to be repeated every time a new shell is started. To make the changes persistent, add<br />
test -r <b><i>/path/to/psdm</i></b>/etc/ana_env.sh && . <b><i>/path/to/psdm</i></b>/etc/ana_env.sh<br />
to <code>~/.bashrc</code>, or<br />
test -r <b><i>/path/to/psdm</i></b>/etc/ana_env.csh && source <b><i>/path/to/psdm</i></b>/etc/ana_env.csh<br />
to <code>~/.cshrc</code> as appropriate. This ensures the environment is properly prepared for non-interactive shells, <i>e.g.</i> the shells that are used to run jobs submitted to the cluster. To also be able to use the PSDM software from interactive shells, it is recommended that bash-users put<br />
test -r ~/.bashrc && . ~/.bashrc<br />
near the top of <code>~/.bash_profile</code>. There is no need to edit any additional files for csh-users, because csh always reads <code>~/.cshrc</code>.<br />
<br />
A test released based on the most current PSDM distribution is then set up in the current working directory using<br />
$ newrel ana-current <i>myrelease</i><br />
$ cd <i>myrelease</i><br />
$ sit_setup<br />
$ newpkg <i>my_ana_pkg</i><br />
Note that the names for the test release and the analysis package, <i>myrelease</i> and <i>my_ana_pkg</i> above, are common, but nevertheless arbitrary, choices.<br />
<br />
These steps only need to be performed once.<br />
<br />
== External links ==<br />
<br />
* [https://confluence.slac.stanford.edu/display/PSDM/psana+-+User+Manual Psana User Manual]<br />
* [https://confluence.slac.stanford.edu/display/PSDM/Packages+and+Releases Packages and Releases]</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/File_formatsFile formats2017-07-21T19:22:48Z<p>Aaron: /* A note on coordinate systems */</p>
<hr />
<div>This page is under construction<br />
<br />
== Introduction ==<br />
<br />
<br />
''cctbx.xfel'''s current native file formats, image pickles and integration pickles, are mainly intermediate file formats useful for debugging and software development. These binary pickle files are serialized python dictionaries optimized for machine readability. The rationale here is that the raw data from an experiment at LCLS is stored in the xtc streams. We process these in memory without needing to write images to disk at all, but provide image pickles if requested for use with the ''cctbx'' image viewer for diagnostic purposes. Integration pickles contain integrated intensities and crystal cell and orientation parameters and are read by the merging programs cxi.merge and prime.postrefine.<br />
<br />
In an effort to provide more human-readable data files and conform to international standards, we have worked with representatives of ImageCIF to create 64 tile segmented data in CBF format, the same format used in a wide variety of detectors. These CSPAD CBFs are designed to be used with other crystallographic software packages.<br />
<br />
Format specifications for these data files are provided below.<br />
<br />
== Image pickles ==<br />
<br />
''cctbx.xfel'' image pickles are a binary file format containing pixel data and image metadata. Under the hood these are serialized python dictionaries, but for general users it's sufficient to know that the blob of data consists of name/value pairs. The contents of an image pickle can be inspected with this command:<br />
<br />
'''cxi.print_pickle''' image.pickle<br />
<br />
For example, here's the output from an image with a thermolysin diffraction pattern collected recently:<br />
<br />
<div class="toccolours mw-collapsible mw-collapsed"><br />
<pre><br />
Printing contents of idx-20141201073601249.pickle<br />
Detector format version: CXI 10.1<br />
DISTANCE 119.002<br />
DETECTOR_ADDRESS CxiDs1-0|Cspad-0<br />
SIZE1 1765<br />
SIZE2 1765<br />
TIMESTAMP 2014-12-01T07:36Z01.249<br />
CCD_IMAGE_SATURATION 90000<br />
SATURATED_VALUE 90000<br />
64 active areas, first one: [715, 439, 909, 624]<br />
PIXEL_SIZE 0.11<br />
BEAM_CENTER_Y 97.075<br />
MIN_TRUSTED_VALUE -2000<br />
WAVELENGTH 1.75124427107 , converted to eV: 7079.77687909<br />
xtal_target None<br />
BEAM_CENTER_X 97.075<br />
SEQUENCE_NUMBER 0<br />
DATA len=3115225 max=106065.000000 min=-104526.000000 dimensions=(1765, 1765)<br />
cxi_versioned_extract()::cxi_version: CXI 10.1<br />
64 translated active areas, first one: [725, 449, 917, 632]<br />
</pre><br />
</div><br />
<br />
Each of the lines here is a name/value pair of data or metadata for the image. LABELIT, DIALS, cctbx.image_viewer, are all programs capable of processing these files directly. Here's a description of each of the name/value pairs:<br />
* Detector format version: CXI 10.1. Technically this value is not stored in the image pickle. The detector format version is used internally by cctbx.xfel for data collected up to LCLS run 11, and is a way of looking up tile corrections stored in the software. After run 11, this information should be stored in the phil file used to processes the data. The detector format version is looked up based on the detector address and event timestamp. All known detector format versions can be displayed with the command '''cxi.detector_format_versions'''.<br />
* DISTANCE: detector distance (mm)<br />
* DETECTOR_ADDRESS: string to identify the detector in the XTC stream. Recorded here as a unique identifier for the data when coupled with the event timestamp.<br />
* SIZE1 and SIZE2: image dimensions (pixels)<br />
* TIMESTAMP: unique time stamp for this event in the form year-month-day'''T'''hour:minute'''Z'''second.microsecond<br />
* CCD_IMAGE_SATURATION and SATURATED_VALUE: overload value<br />
* PIXEL_SIZE: pixel size in mm<br />
* BEAM_CENTER_X, BEAM_CENTER_Y: beam center, specified as positive mm offsets from the origin of the detector (upper-left corner)<br />
* MIN_TRUSTED_VALUE: underload<br />
* WAVELENGTH: wavelength as recorded in the XTC stream (angstroms)<br />
* DATA: pixel data. Some vital statistics are shown.<br />
* xtal_target and SEQUENCE_NUMBER: unused<br />
* 64 active areas: the active areas are coordinates that specify where the 64 tiles are in the image. They are in the form of quartets of numbers [x1,y1,x2,y2], one quartet for each of the 64 tiles. The first active area is shown. Note, for non-CSPAD image pickles, there will only be one active area and it will be the size of the image entire.<br />
* 64 translated active areas: if corrections were available for this image, the first corrected active area is shown here.<br />
<br />
=== A note on coordinate systems ===<br />
<br />
Internally, our software uses the ImageCIF coordinate system with the exception that no goniometer is typically available at XFELs, therefore Z is along the beam (sample to source), Y points against gravity and X completes a right-handed coordinate system. In the image pickles, the origin of the detector pixel readout is assumed to be in the upper left of the image, therefore the beam center is specified as positive offsets from that point. In example above, assuming the sample is at the laboratory origin, the vector pointing to the beam center on the detector in the ImageCIF coordinate system would be (0, 0, -119.002). A vector from the laboratory origin to the pixel readout origin (upper-left of the detector) would be -97.075, 97.075, -119.002). dxtbx.print_header is a useful tool to get these vectors from any file format understood by ''cctbx.xfel''. For more information see [http://cci.lbl.gov/publications/download/jo5001_reprint.pdf Parkhurst et al, 2014].<br />
<br />
=== Creating image pickles from xtc streams ===<br />
<br />
The program cctbx.xfel.xtc_dump can be used to convert xtc streams to image pickle format. Here we provide an example for the Rayonix detector that is often found at XPP and MFX. The psana module mod_image_dict is used to read the data and generate the images (example: [[mod_image_dict|mod_image_dict.cfg]]). After setting up a release directory as described in [[Setup]]:<br />
<br />
cd ~/myrelease # or wherever your release is<br />
sit_setup<br />
cctbx.xfel.xtc_dump experiment=mfxm9316 run_num=14 address="MfxEndstation-0|Rayonix-0" file_format=pickle cfg=mod_image_dict.cfg <br />
<br />
This will fill your working folder with image pickles. You can change your output directory with output.output_dir and you can limit the number of images saved with dispatch.max_events.<br />
<br />
This approach will also work for CSPAD images. Here, you must add calib_dir to your config file, as described in the indexing tutorials.<br />
<br />
== Integration pickles ==<br />
<br />
Integration pickles are the output of cctbx.xfel indexing and integration. They are like image pickles in that they are serialized python dictionaries and can be inspected with '''cxi.print_pickle'''. They are the primary input to '''cxi.merge''' and '''prime.run'''.<br />
<br />
== CSPAD CBFs ==<br />
<br />
CSPAD CBFs are used while processing CSPAD images using the DIALS backend of ''cctbx.xfel''. The complete specification for CSPAD CBFs is laid out in an article in the Computational Crystallographic Newsletter: "XFEL Detectors and ImageCIF", Computational Crystallography Newsletter 5, 19-24. ([http://cci.lbl.gov/publications/download/CCN_2014_p19.pdf Reprint]). Documentation for using DIALS to index and integrate CSPAD data can be found in an article in the Computational Crystallographic Newsletter: "Processing XFEL data with cctbx.xfel and DIALS", Computational Crystallography Newsletter 7, 32-53 ([http://www.phenix-online.org/newsletter/CCN_2016_07.pdf#page=18 Reprint]).<br />
<br />
=== Creating CSPAD CBFs from XTC streams ===<br />
<br />
cctbx.xfel.xtc_dump is useful for converting XTC streams to CBF files as needed. Use -c to get a listing of all options, and -c -a 2 to get a full listing with help strings. Example command:<br />
<br />
cd ~/myrelease # or wherever your release is<br />
sit_setup<br />
cctbx.xfel.xtc_dump dispatch.max_events=1 input.experiment=xpptut15 input.address=cspad \<br />
input.run_num=54 format.file_format=cbf output.output_dir=xpptut15/out \<br />
format.cbf.detz_offset=100 input.override_energy=7000<br />
<br />
Here, one image is created from experiment xpptut15, run 54. You can display the image with cctbx.image_viewer xpptut15/out/*.cbf. Note the pink gaps between the tiles. This results from the segmented nature of the CSPAD, preserved in the CBF file.<br />
<br />
xpptut15 is data from XPP's CSPAD and was collected without xray's on. Hence format.cbf.detz_offset=100 input.override_energy=7000 are set to fake values in this command.<br />
<br />
=== Converting from SLAC's metrology to CBF ===<br />
<br />
The tile positions of the hierarchical CSPAD detector are specified by SLAC in a geometry file in the calib folder of each experiment. Typically this file is named something like 0-end.data. If desired, this file can be converted to just the human readable header portion of a CSPAD cbf using the ''cctbx.xfel'' command cxi.slaccalib2cbfheader. Example:<br />
<br />
cxi.slaccalib2cbfheader metrology_file=0-end.data out=tmp.cbf<br />
<br />
This cbf header can be converted back to SLAC format using cxi.cbfheader2slaccalib:<br />
<br />
cxi.cbfheader2slaccalib cbf_header=tmp.cbf out_metrology_file=tmp.data<br />
<br />
=== Displaying metrology files ===<br />
<br />
cxi.display_metrology <filename> can be used to show a plot of tile positions. It accepts SLAC metrology files, CSPAD CBFs and image pickles.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/Ha14_installationHa14 installation2017-03-31T22:45:56Z<p>Aaron: /* Finish up the build */</p>
<hr />
<div>== Prerequisites ==<br />
<br />
While these tutorials assume you wish to process XTC streams at SLAC, some users have stills collected from other sources and do not need the full PSDM suite. If this is the case, see the installation directions below for installing on a non PSDM system. Otherwise, it is assumed that the [[Set up PSDM software | PSDM software distribution has been set up]]. Note again that several sites already have ''cctbx.xfel'' installed, and so regular users not involved in the development of the software will not need the instructions here.<br />
<br />
Once ''cctbx.xfel'' has been installed it must be [[Setup | set up]] before it can be used. Developers may additionally want to [[Set up ssh-agent | set up an ssh-agent]]. <br />
<br />
Finally, you must have a user account on cci.lbl.gov in order to proceed, or else you will not be able to download the sources. A sourceforge account is only needed if you wish to commit changes back to the repository. If you don't have or don't want to use a sourceforge account, you can leave it off in the below commands where it is specified.<br />
<br />
== Standard installation using PSDM ==<br />
===Quick start===<br />
This step has to be performed on a host with Internet access. Not all hosts at SLAC have that, but the members of <i>e.g.</i> pslogin pool do. Make and change to a working directory to contain the new source code and build (this directory should be accessible from any computing nodes ''cctbx.xfel'' will be run. Then download these bootstrap modules:<br />
<br />
wget https://raw.githubusercontent.com/cctbx/cctbx_project/master/libtbx/auto_build/bootstrap.py<br />
<br />
Then:<br />
<br />
python bootstrap.py hot update --builder=xfel --cciuser=<cciusername> --sfuser=<githubusername><br />
<br />
This command instructs bootstrap.py to download static packages required for ''cctbx.xfel'' (hot), and then to checkout the rest of the sources from cci.lbl.gov and github (update), using the user accounts specified by the cciuser and sfuser parameters, respectively.<br />
<br />
===Alternate instructions for non-LBNL developers===<br />
Our non-LBNL colleagues will now realize that the "python bootstrap" step does not work without cci username credentials, needed to obtain source code for the program LABELIT. Developers are instructed to use the following alternate procedure:<br />
<br />
python bootstrap.py hot --builder=xfel --sfuser=<githubusername> # download the static packages<br />
python bootstrap.py update --builder=dials --sfuser=<githubusername> # download the github code for dials & cctbx<br />
mkdir modules/cxi_xdr_xes # create a stub directory for experiment-specific python code<br />
<br />
Then download phenix from http://phenix-online.org and untar the package. Locate the modules directory in the phenix directory tree, and copy-paste the labelit subdirectory into the modules directory of your current working area.<br />
<br />
===Finish up the build===<br />
After downloading the sources, you need to be sure you have the appropriate compilers before executing the next command. At SLAC, that means you need to ssh to one of the psana nodes, as the pslogin nodes do not have the requisite compilers. When ready, configure and compile thusly:<br />
<br />
python bootstrap.py build --builder=xfel --with-python=`which python` --nproc=<# cores available for compile><br />
<br />
On SLAC's interactive nodes, this takes just over 6 minutes. To avoid problems with run-time dynamic linking of Python extensions, the Python interpreter required for the above command must be the one provided by the PSDM software distribution. That interpreter can be located using find, <i>e.g.</i><br />
$ find $SIT_ROOT/sw/external/python -perm /0111 -type f -wholename "*/$SIT_ARCH/*/python" 2> /dev/null<br />
At SLAC this interpreter is located somewhere under <code>/reg/g/psdm/sw/external/python</code>. Here we use `which python` to get this path automatically. <br />
<br />
Initialise the running shell using the newly created configuration files. bash-users should<br />
$ . build/setpaths.sh<br />
while csh-users will instead need to run<br />
% source build/setpaths.csh<br />
<br />
To finalize the installation, see [[Setup]].<br />
<br />
Note, you may also follow this procedure if you are on a machine where you have your own python that you want to use, instead of the one provided by PSDM.<br />
<br />
== Installation on a non-PSDM system ==<br />
<br />
If you find yourself on a machine without PSDM and won't be dealing with XTC directly, for example if you have your own data collected as stills from a non-XFEL source, you can use the below commands in a new directory (again assuming you have a cci user account):<br />
<br />
# On Linux:<br />
wget https://raw.githubusercontent.com/cctbx/cctbx_project/master/libtbx/auto_build/bootstrap.py<br />
# On MacOSX:<br />
curl https://raw.githubusercontent.com/cctbx/cctbx_project/master/libtbx/auto_build/bootstrap.py > bootstrap.py<br />
<br />
python bootstrap.py hot update base build --builder=xfel --cciuser=<cciusername> --sfuser=<sourceforgeusername><br />
<br />
Here, in addition to hot, update and build, there's a command called 'base', which downloads a python and needed packages and compiles it. After this process completes, source the installation (for details see above), and then add scipy to it:<br />
<br />
. build/setpaths.sh<br />
libtbx.python -m easy_install scipy<br />
<br />
Please contact the authors if with any issues. Some gotchas:<br />
<br />
1) On a Mac, you will need to make sure you have the latest version of xcode installed. Consider the command xcode-select --install. [NOTE: Xcode v. 8.2.1. for Mac OS 10.12 (Sierra) does not include a gFortran compiler. You will need to install gFortran separately if you need scipy.]<br />
<br />
2) On linux you will likely need to install some dependencies, including gFortran. [Note: gFortran is needed only for the scipy installation, not for the core build based on cctbx. Therefore, only install gFortran if you need scipy.]<br />
<br />
== External links ==<br />
<br />
* [http://cctbx.sourceforge.net/current_cvs/installation.html#manually-building-from-sources-under-unix Manually building from sources under Unix]<br />
<br />
<br />
<!-- POSSBILE ADDITIONS:<br />
<br />
COMMON PITFALLS:<br />
<br />
forget to source the shell configuration files<br />
forget to run sit_setup<br />
running cxi.pyana from outside the myrelease directory<br />
<br />
DEBUGGING:<br />
<br />
And a few more things to check: output of "ls -lR arch/*", output of "which python2.7", and the "scons.out" file that results from "scons TRACE=7 > scons.out 2>&1".<br />
<br />
What Linux distribution are you running this on (could you past the output of "uname -a" and "cat /etc/issue")? Could you also tell me which scons you are using (output of "which scons"). Last, could you attach a dump of the environment (env > env.out)?<br />
<br />
Andy's information request on debugging: full output from these commands:<br />
<br />
% rm -rf arch build<br />
% printenv<br />
% which scons<br />
% which python<br />
% which python2.7<br />
% scons TRACE=9<br />
% ls -lR arch<br />
% ls -lR my_ana_pkg<br />
<br />
--></div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/2017_Tutorials2017 Tutorials2017-03-02T00:17:30Z<p>Aaron: /* Feb 16th */</p>
<hr />
<div>= Feb 16th =<br />
<br />
9:00am: Session 1<br />
*Nick Sauter: Welcome and "Trials and tribulations merging still image data"<br />
*Aaron Brewster: "Metrology and non-isomorphism: hidden challenges in still image data reduction"<br />
*Axel Brunger: "Data processing of XFEL data from a limited number of crystals"<br />
<br />
10:20am: Break<br />
<br />
10:40am: Session 2<br />
*Art Lyubimov: "IOTA: Integration optimization, triage and analysis tool for XFEL data processing"<br />
*Monarin Uervirojnangkoorn: "Up and Running with Prime"<br />
*Jacques-Philippe Colletier: "Mosquito larvicide BinAB revealed by de novo phasing with an X-ray laser"<br />
*James Holton: "What if? Using at-scale image simulations to optimize data processing algorithms"<br />
<br />
Noon: Working Lunch - Roundtable discussion of data processing challenges<br />
<br />
1:00pm: Session 3<br />
*Graeme Winter and Richard Gildea: "DIALS - new methods for processing X-ray diffraction data"<br />
*James Parkhurst: "Robust background modelling in the presence of outliers in DIALS"<br />
*Jan Kern: "Challenges and Promises for serial crystallography of metalloenzymes at XFELs"<br />
*Franklin Fuller: "Drop-by-Drop Transient Serial Crystallography of Metalloenzymes at an X-ray free electron laser"<br />
*Iris Young: "Room temperature studies of the oxygen-evolving complex of photosystem II using an X-ray free electron laser (XFEL)"<br />
<br />
2:40pm: Break<br />
<br />
3:00pm: Session 4<br />
*Aina Cohen: "Serial crystallography approaches at LCLS-MFX and the SSRL micro-focus beamlines 12-1 and 12-2"<br />
*Rahel Woldeyes: "Using X-ray Free Electron lasers to visualize solvent in the M2 proton channel"<br />
*Danny Axford: "Highly efficient serial data collection from high-density fixed targets"<br />
*Christoph Mueller-Dieckmann: "Serial Synchrotron Crystallography at the ESRF using a high viscosity extruder"<br />
*Allen Orville: "Time-resolved SFX with less sample, over longer reactions, and including complementary methods"<br />
<br />
= Feb 17th =<br />
<br />
== 9am: Tutorials 1 ==<br />
Aaron Brewster and Iris Young: cctbx.xfel<br />
Break: 10:00 am<br />
== 10:15 am: Tutorials 2 ==<br />
* Aaron Brewster and James Parkhurst: [[2017_dials.stills_process | dials.stills_process]]<br />
* Art Lyubimov: IOTA<br />
* Monarin Uervirojnangkoorn: [[2017_prime_tutorial | PRIME]]<br />
* Nick Sauter: [[2017_cxi_merge_tutorial | cxi.merge]]<br />
<br />
== 12:15pm-2:30pm: Round table discussion. == <br />
Topics will include:<br />
Is the current software meeting needs?<br />
What are essential/timely avenues most useful for near-term development (first half of 2017)?<br />
Where should we be going next (longer term future)?<br />
Time resolved experiments?<br />
Synchrotron serial crystallography?<br />
== 2:30-4pm: Breakout sessions: ==<br />
users work with developers and instructors on their own data. Hands-on walkthroughs and data analysis.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/2017_dials.stills_process2017 dials.stills process2017-03-02T00:11:23Z<p>Aaron: </p>
<hr />
<div>Here we give the commands for processing the example data from DLS (Credit: Danny Axford). These data are Pilatus data, 11000 images, but these commands are generally applicable for processing stills using dials.stills_process. For the sake of this example, we will be in a folder named $work, and the images are in a folder named $data<br />
<br />
Remember, all these commands have help strings, use -h, or to see details for the parameters, use -c. If you want to see all the parameters, use -e 10. And if you want to see the descriptions of each parameters, use -a 2. Example:<br />
dials.import -c -a 2 -e 10 | less<br />
<br />
== Initial discovery ==<br />
<br />
First, use the dials.image_viewer on 10 images to get a sense of the data<br />
dials.image_viewer $data/*.cbf.bz2<br />
<br />
Next, verify beam center and diffraction extent using an average image.<br />
mkdir $work/averages<br />
cd $work/averages<br />
dxtbx.image_average -v -a avg.pickle -m max.pickle -s stddev.pickle -n 64 $data/*.cbf.bz2<br />
<br />
Here, we specify output for the average, maximum and standard deviation images using -a, -m and -s. -v means verbose output, and we specify 64 cores for averaging using -n. The output images are in image pickle format, and can be displayed using the image viewer:<br />
dials.image_viewer *.pickle<br />
<br />
Use the Actions->Show ring tool with the maximum projection image to verify the rings are circular around the beam center.<br />
<br />
Also, in the average image, note the defects around the beam center and the 8 silicon reflections around the edges of the detector. These artifacts will need to be masked out. Hold down shift and left click to drag boxes onto the image. Click save mask to generate a mask.pickle file we will use during processing. If you want to verify the mask is good, restart the image viewer using mask=mask.pickle, then click the show mask checkbox in the settings panel. The mask will display in red.<br />
<br />
== Trial processing ==<br />
<br />
We will use the following phil file to process the data. Note the use of $work to specify mask paths. Also note, this part I am currently editing, removing values here that are just defaults.<br />
<br />
<pre><br />
spotfinder.lookup.mask=$work/averages/mask.pickle<br />
integration.lookup.mask=$work/averages/mask.pickle<br />
spotfinder.filter.min_spot_size=2<br />
verbosity=10<br />
indexing {<br />
known_symmetry {<br />
space_group = P6<br />
unit_cell = 91.7 91.7 46 90 90 120<br />
}<br />
refinement_protocol.n_macro_cycles = 1<br />
refinement_protocol.d_min_start=2.0<br />
basis_vector_combinations.max_try=10<br />
stills.indexer=stills<br />
stills.method_list=fft1d real_space_grid_search<br />
}<br />
verbosity=10<br />
integration {<br />
integrator=stills<br />
profile.fitting=False<br />
background {<br />
simple {<br />
outlier {<br />
algorithm = null<br />
}<br />
}<br />
}<br />
}<br />
profile {<br />
gaussian_rs {<br />
min_spots.overall = 0<br />
}<br />
}<br />
<br />
refinement {<br />
parameterisation {<br />
beam.fix=all<br />
detector.fix=all<br />
detector.hierarchy_level=0<br />
auto_reduction {<br />
action=fix<br />
min_nref_per_parameter=1<br />
}<br />
treat_single_image_as_still=True<br />
}<br />
reflections {<br />
outlier.algorithm=null<br />
weighting_strategy.override=stills<br />
weighting_strategy.delpsi_constant=1000000<br />
}<br />
}<br />
</pre><br />
<br />
With this phil file we can process the data:<br />
cd $work; mkdir -p results/000; cd results/000<br />
dials.stills_process $data/*.cbf.bz2 $work/process.phil mp.nproc=64<br />
<br />
Here, we put the processed results in a results folder, in a subfolder labeled 000 that represents trial 0.<br />
<br />
When complete, we can look at integration results:<br />
dials.image_viewer idx-zurich0038_00036.cbf_refined_experiments.json idx-zurich0038_00036.cbf_integrated.pickle<br />
<br />
We can look at a graph showing hitfinding and indexing rates:<br />
cctbx.xfel.plot_run_stats_from_experiments . hit_cutoff=40<br />
The hit_cutoff is adjusted up a bit to be above the background (see the top plot, how the images that are essentially blank and didn't index (gray dots) still show up with ~30 or so reflections).<br />
<br />
We can also look at unit cell histograms to see if the results are isomorphous:<br />
cctbx.xfel.plot_uc_cloud_from_experiments *_refined_experiments.json<br />
<br />
Here we see a slight asymmetry in the c axis which indicates the detector position might not be accurate.<br />
<br />
== Ensemble refinement of detector position ==<br />
<br />
To get a more isomorphous distribution we will refine the detector model using many crystal models as input.<br />
<br />
cd $work; mkdir metrology; cd metrology<br />
dials.combine_experiments $work/results/000/*indexed.pickle $work/results/000/*_refined_experiments.json n_subset=2000 reference_from_experiment.detector=0<br />
<br />
Here, since we fixed the detector position in place during indexing, we use reference_from_experiment.detector=0 so that each experiment in the output combined.json file points to the same detector. If you allow the detector to refine separately for each image, you will need to average the detector positions when running dials.combine_experiments, using the parameter reference_from_experiment.average_detector=True instead.<br />
<br />
Now, we refine the detector and all 2000 crystals at once:<br />
dials.refine combined_experiments.json combined_reflections.pickle refine.phil<br />
<br />
The phil file is here:<br />
<pre><br />
refinement {<br />
parameterisation {<br />
auto_reduction {<br />
min_nref_per_parameter = 3<br />
action = fail fix *remove<br />
}<br />
beam {<br />
fix = *all in_spindle_plane out_spindle_plane wavelength<br />
}<br />
detector {<br />
fix_list = Tau1<br />
}<br />
}<br />
refinery {<br />
engine = SimpleLBFGS LBFGScurvs GaussNewton LevMar *SparseLevMar<br />
}<br />
reflections {<br />
outlier {<br />
algorithm = null auto mcd tukey *sauter_poon<br />
separate_experiments = False<br />
separate_panels = True<br />
}<br />
}<br />
}<br />
</pre><br />
<br />
To compare the old to the new metrology: <br />
dxtbx.plot_detector_models combined_experiments.json refined_experiments.json<br />
<br />
We can reprocess all the data using this updated geometry, adding this to the process.phil<br />
input.reference_geometry=$work/metrology/refined_experiments.json<br />
<br />
Then create a new trial and reprocess the data.<br />
cd $work/results; mkdir 001; cd 001<br />
dials.stills_process $data/*.cbf.bz2 $work/process.phil mp.nproc=64<br />
<br />
The results should be more isomorphous:<br />
cctbx.xfel.plot_uc_cloud_from_experiments *_refined_experiments.json<br />
<br />
From here, proceed to merging:<br />
<br />
[[2017_prime_tutorial | PRIME]]</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/2017_dials.stills_process2017 dials.stills process2017-03-02T00:07:23Z<p>Aaron: </p>
<hr />
<div>Processing the example data from DLS (Credit: Danny Axford). Pilatus data, 11000 images. For the sake of this example, we will be in a folder named $work, and the images are in a folder named $data<br />
<br />
Remember, all these commands have great help strings, use -h, or to see details for the parameters, use -c. If you want to see all the parameters, use -e 10. And if you want to see the descriptions of each parameters, use -a 2. Example:<br />
dials.import -c -a 2 -e 10 | less<br />
<br />
== Initial discovery ==<br />
<br />
First, use the dials.image_viewer on 10 images to get a sense of the data<br />
dials.image_viewer $data/*.cbf.bz2<br />
<br />
Next, verify beam center and diffraction extent using an average image.<br />
mkdir $work/averages<br />
cd $work/averages<br />
dxtbx.image_average -v -a avg.pickle -m max.pickle -s stddev.pickle -n 64 $data/*.cbf.bz2<br />
<br />
Here, we specify output for the average, maximum and standard deviation images using -a, -m and -s. -v means verbose output, and we specify 64 cores for averaging using -n. The output images are in image pickle format, and can be displayed using the image viewer:<br />
dials.image_viewer *.pickle<br />
<br />
Use the Actions->Show ring tool with the maximum projection image to verify the rings are circular around the beam center.<br />
<br />
Also, in the average image, note the defects around the beam center and the 8 silicon reflections around the edges of the detector. These artifacts will need to be masked out. Hold down shift and left click to drag boxes onto the image. Click save mask to generate a mask.pickle file we will use during processing. If you want to verify the mask is good, restart the image viewer using mask=mask.pickle, then click the show mask checkbox in the settings panel. The mask will display in red.<br />
<br />
== Trial processing ==<br />
<br />
We will use the following phil file to process the data. Note the use of $work to specify mask paths. Also note, this part I am currently editing, removing values here that are just defaults.<br />
<br />
<pre><br />
spotfinder.lookup.mask=$work/averages/mask.pickle<br />
integration.lookup.mask=$work/averages/mask.pickle<br />
spotfinder.filter.min_spot_size=2<br />
verbosity=10<br />
indexing {<br />
known_symmetry {<br />
space_group = P6<br />
unit_cell = 91.7 91.7 46 90 90 120<br />
}<br />
refinement_protocol.n_macro_cycles = 1<br />
refinement_protocol.d_min_start=2.0<br />
basis_vector_combinations.max_try=10<br />
stills.indexer=stills<br />
stills.method_list=fft1d real_space_grid_search<br />
}<br />
verbosity=10<br />
integration {<br />
integrator=stills<br />
profile.fitting=False<br />
background {<br />
simple {<br />
outlier {<br />
algorithm = null<br />
}<br />
}<br />
}<br />
}<br />
profile {<br />
gaussian_rs {<br />
min_spots.overall = 0<br />
}<br />
}<br />
<br />
refinement {<br />
parameterisation {<br />
beam.fix=all<br />
detector.fix=all<br />
detector.hierarchy_level=0<br />
auto_reduction {<br />
action=fix<br />
min_nref_per_parameter=1<br />
}<br />
treat_single_image_as_still=True<br />
}<br />
reflections {<br />
outlier.algorithm=null<br />
weighting_strategy.override=stills<br />
weighting_strategy.delpsi_constant=1000000<br />
}<br />
}<br />
</pre><br />
<br />
With this phil file we can process the data:<br />
cd $work; mkdir -p results/000; cd results/000<br />
dials.stills_process $data/*.cbf.bz2 $work/process.phil mp.nproc=64<br />
<br />
Here, we put the processed results in a results folder, in a subfolder labeled 000 that represents trial 0.<br />
<br />
When complete, we can look at integration results:<br />
dials.image_viewer idx-zurich0038_00036.cbf_refined_experiments.json idx-zurich0038_00036.cbf_integrated.pickle<br />
<br />
We can look at a graph showing hitfinding and indexing rates:<br />
cctbx.xfel.plot_run_stats_from_experiments . hit_cutoff=40<br />
The hit_cutoff is adjusted up a bit to be above the background (see the top plot, how the images that are essentially blank and didn't index (gray dots) still show up with ~30 or so reflections).<br />
<br />
We can also look at unit cell histograms to see if the results are isomorphous:<br />
cctbx.xfel.plot_uc_cloud_from_experiments *_refined_experiments.json<br />
<br />
Here we see a slight asymmetry in the c axis which indicates the detector position might not be accurate.<br />
<br />
== Ensemble refinement of detector position ==<br />
<br />
To get a more isomorphous distribution we will refine the detector model using many crystal models as input.<br />
<br />
cd $work; mkdir metrology; cd metrology<br />
dials.combine_experiments $work/results/000/*indexed.pickle $work/results/000/*_refined_experiments.json n_subset=2000 reference_from_experiment.detector=0<br />
<br />
Here, since we fixed the detector position in place during indexing, we use reference_from_experiment.detector=0 so that each experiment in the output combined.json file points to the same detector. If you allow the detector to refine separately for each image, you will need to average the detector positions when running dials.combine_experiments, using the parameter reference_from_experiment.average_detector=True instead.<br />
<br />
Now, we refine the detector and all 2000 crystals at once:<br />
dials.refine combined_experiments.json combined_reflections.pickle refine.phil<br />
<br />
The phil file is here:<br />
<pre><br />
refinement {<br />
parameterisation {<br />
auto_reduction {<br />
min_nref_per_parameter = 3<br />
action = fail fix *remove<br />
}<br />
beam {<br />
fix = *all in_spindle_plane out_spindle_plane wavelength<br />
}<br />
detector {<br />
fix_list = Tau1<br />
}<br />
}<br />
refinery {<br />
engine = SimpleLBFGS LBFGScurvs GaussNewton LevMar *SparseLevMar<br />
}<br />
reflections {<br />
outlier {<br />
algorithm = null auto mcd tukey *sauter_poon<br />
separate_experiments = False<br />
separate_panels = True<br />
}<br />
}<br />
}<br />
</pre><br />
<br />
To compare the old to the new metrology: <br />
dxtbx.plot_detector_models combined_experiments.json refined_experiments.json<br />
<br />
We can reprocess all the data using this updated geometry, adding this to the process.phil<br />
input.reference_geometry=$work/metrology/refined_experiments.json<br />
<br />
Then create a new trial and reprocess the data.<br />
cd $work/results; mkdir 001; cd 001<br />
dials.stills_process $data/*.cbf.bz2 $work/process.phil mp.nproc=64<br />
<br />
The results should be more isomorphous:<br />
cctbx.xfel.plot_uc_cloud_from_experiments *_refined_experiments.json<br />
<br />
From here, proceed to merging:<br />
<br />
[[2017_prime_tutorial | PRIME]]</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/2017_dials.stills_process2017 dials.stills process2017-03-02T00:02:35Z<p>Aaron: </p>
<hr />
<div>Processing the example data from DLS (Credit: Danny Axford). Pilatus data, 11000 images. For the sake of this example, we will be in a folder named $work, and the images are in a folder named $data<br />
<br />
Remember, all these commands have great help strings, use -h, or to see details for the parameters, use -c. If you want to see all the parameters, use -e 10. And if you want to see the descriptions of each parameters, use -a 2. Example:<br />
dials.import -c -a 2 -e 10 | less<br />
<br />
== Initial discovery ==<br />
<br />
First, use the dials.image_viewer on 10 images to get a sense of the data<br />
dials.image_viewer $data/*.cbf.bz2<br />
<br />
Next, verify beam center and diffraction extent using an average image.<br />
mkdir $work/averages<br />
cd $work/averages<br />
dxtbx.image_average -v -a avg.pickle -m max.pickle -s stddev.pickle -n 64 $data/*.cbf.bz2<br />
<br />
Here, we specify output for the average, maximum and standard deviation images using -a, -m and -s. -v means verbose output, and we specify 64 cores for averaging using -n. The output images are in image pickle format, and can be displayed using the image viewer:<br />
dials.image_viewer *.pickle<br />
<br />
Use the Actions->Show ring tool with the maximum projection image to verify the rings are circular around the beam center.<br />
<br />
Also, in the average image, note the defects around the beam center and the 8 silicon reflections around the edges of the detector. These artifacts will need to be masked out. Hold down shift and left click to drag boxes onto the image. Click save mask to generate a mask.pickle file we will use during processing. If you want to verify the mask is good, restart the image viewer using mask=mask.pickle, then click the show mask checkbox in the settings panel. The mask will display in red.<br />
<br />
== Trial processing ==<br />
<br />
We will use the following phil file to process the data. Note the use of $work to specify mask paths. Also note, this part I am currently editing, removing values here that are just defaults.<br />
<br />
<pre><br />
spotfinder.lookup.mask=$work/averages/mask.pickle<br />
integration.lookup.mask=$work/averages/mask.pickle<br />
spotfinder.filter.min_spot_size=2<br />
verbosity=10<br />
indexing {<br />
known_symmetry {<br />
space_group = P6<br />
unit_cell = 91.7 91.7 46 90 90 120<br />
}<br />
refinement_protocol.n_macro_cycles = 1<br />
refinement_protocol.d_min_start=2.0<br />
basis_vector_combinations.max_try=10<br />
stills.indexer=stills<br />
stills.method_list=fft1d real_space_grid_search<br />
}<br />
verbosity=10<br />
integration {<br />
integrator=stills<br />
profile.fitting=False<br />
background {<br />
simple {<br />
outlier {<br />
algorithm = null<br />
}<br />
}<br />
}<br />
}<br />
profile {<br />
gaussian_rs {<br />
min_spots.overall = 0<br />
}<br />
}<br />
<br />
refinement {<br />
parameterisation {<br />
beam.fix=all<br />
detector.fix=all<br />
detector.hierarchy_level=0<br />
auto_reduction {<br />
action=fix<br />
min_nref_per_parameter=1<br />
}<br />
treat_single_image_as_still=True<br />
}<br />
reflections {<br />
outlier.algorithm=null<br />
weighting_strategy.override=stills<br />
weighting_strategy.delpsi_constant=1000000<br />
}<br />
}<br />
</pre><br />
<br />
With this phil file we can process the data:<br />
cd $work; mkdir -p results/000; cd results/000<br />
dials.stills_process $data/*.cbf.bz2 $work/process.phil mp.nproc=64<br />
<br />
Here, we put the processed results in a results folder, in a subfolder labeled 000 that represents trial 0.<br />
<br />
When complete, we can look at integration results:<br />
dials.image_viewer idx-zurich0038_00036.cbf_refined_experiments.json idx-zurich0038_00036.cbf_integrated.pickle<br />
<br />
We can look at a graph showing hitfinding and indexing rates:<br />
cctbx.xfel.plot_run_stats_from_experiments . hit_cutoff=40<br />
The hit_cutoff is adjusted up a bit to be above the background (see the top plot, how the images that are essentially blank and didn't index (gray dots) still show up with ~30 or so reflections).<br />
<br />
We can also look at unit cell histograms to see if the results are isomorphous:<br />
cctbx.xfel.plot_uc_cloud_from_experiments *_refined_experiments.json<br />
<br />
Here we see a slight asymmetry in the c axis which indicates the detector position might not be accurate.<br />
<br />
== Ensemble refinement of detector position ==<br />
<br />
To get a more isomorphous distribution we will refine the detector model using many crystal models as input.<br />
<br />
cd $work; mkdir metrology; cd metrology<br />
dials.combine_experiments $work/results/000/*indexed.pickle $work/results/000/*_refined_experiments.json n_subset=2000 reference_from_experiment.detector=0<br />
<br />
Here, since we fixed the detector position in place during indexing, we use reference_from_experiment.detector=0 so that each experiment in the output combined.json file points to the same detector. If you allow the detector to refine separately for each image, you will need to average the detector positions when running dials.combine_experiments, using the parameter reference_from_experiment.average_detector=True instead.<br />
<br />
Now, we refine the detector and all 2000 crystals at once:<br />
dials.refine combined_experiments.json combined_reflections.pickle refine.phil<br />
<br />
The phil file is here:<br />
<pre><br />
refinement {<br />
parameterisation {<br />
auto_reduction {<br />
min_nref_per_parameter = 3<br />
action = fail fix *remove<br />
}<br />
beam {<br />
fix = *all in_spindle_plane out_spindle_plane wavelength<br />
}<br />
detector {<br />
fix_list = Tau1<br />
}<br />
}<br />
refinery {<br />
engine = SimpleLBFGS LBFGScurvs GaussNewton LevMar *SparseLevMar<br />
}<br />
reflections {<br />
outlier {<br />
algorithm = null auto mcd tukey *sauter_poon<br />
separate_experiments = False<br />
separate_panels = True<br />
}<br />
}<br />
}<br />
</pre><br />
<br />
To compare the old to the new metrology: <br />
dxtbx.plot_detector_models combined_experiments.json refined_experiments.json<br />
<br />
We can reprocess all the data using this updated geometry, y adding this to the process.phil<br />
input.reference_geometry=$work/metrology/refined_experiments.json<br />
<br />
Then create a new trial and reprocess the data.<br />
cd $work/results; mkdir 001; cd 001<br />
dials.stills_process $data/*.cbf.bz2 $work/process.phil mp.nproc=64<br />
<br />
The results should be more isomorphous:<br />
cctbx.xfel.plot_uc_cloud_from_experiments *_refined_experiments.json<br />
<br />
From here, proceed to merging:<br />
<br />
(link to mona’s merging on wiki)</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/2017_dials.stills_process2017 dials.stills process2017-02-17T18:08:38Z<p>Aaron: Created page with "Phil file for today: <pre> spotfinder.lookup.mask=/net/dials/raid1/aaron/zurich0038/mask2.pickle spotfinder.filter.min_spot_size=2 integration.lookup.mask=/net/dials/raid1/a..."</p>
<hr />
<div>Phil file for today: <br />
<br />
<pre><br />
spotfinder.lookup.mask=/net/dials/raid1/aaron/zurich0038/mask2.pickle<br />
spotfinder.filter.min_spot_size=2<br />
integration.lookup.mask=/net/dials/raid1/aaron/zurich0038/mask2.pickle<br />
verbosity=10<br />
indexing {<br />
known_symmetry {<br />
space_group = P6<br />
unit_cell = 91.7 91.7 46 90 90 120<br />
}<br />
#method=fft1d<br />
#method=real_space_grid_search<br />
refinement_protocol.n_macro_cycles = 1<br />
refinement_protocol.d_min_start=2.0<br />
basis_vector_combinations.max_try=10<br />
stills.indexer=stills<br />
stills.method_list=fft1d real_space_grid_search<br />
}<br />
<br />
verbosity=10<br />
integration {<br />
integrator=stills<br />
profile.fitting=False<br />
background {<br />
simple {<br />
outlier {<br />
algorithm = null<br />
}<br />
}<br />
}<br />
}<br />
profile {<br />
gaussian_rs {<br />
min_spots.overall = 0<br />
}<br />
}<br />
<br />
refinement {<br />
parameterisation {<br />
beam.fix=all<br />
detector.fix=all<br />
detector.hierarchy_level=0<br />
auto_reduction {<br />
action=fix<br />
min_nref_per_parameter=1<br />
}<br />
treat_single_image_as_still=True<br />
}<br />
reflections {<br />
outlier.algorithm=null<br />
weighting_strategy.override=stills<br />
weighting_strategy.delpsi_constant=1000000<br />
}<br />
}<br />
</pre></div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/2017_Tutorials2017 Tutorials2017-02-17T18:05:57Z<p>Aaron: /* 10:15 am: Tutorials 2 */</p>
<hr />
<div>= Feb 16th =<br />
<br />
9:00am: Session 1<br />
<br />
*Nick Sauter: Welcome and "Trials and tribulations merging still image data"<br />
<br />
*Aaron Brewster: "Metrology and non-isomorphism: hidden challenges in still image data reduction"<br />
<br />
*Axel Brunger: "Data processing of XFEL data from a limited number of crystals"<br />
<br />
10:20am: Break<br />
<br />
10:40am: Session 2<br />
<br />
*Art Lyubimov: "IOTA: Integration optimization, triage and analysis tool for XFEL data processing"<br />
<br />
*Monarin Uervirojnangkoorn: "Up and Running with Prime"<br />
<br />
*Jacques-Philippe Colletier: "Mosquito larvicide BinAB revealed by de novo phasing with an X-ray laser"<br />
<br />
*James Holton: "What if? Using at-scale image simulations to optimize data processing algorithms"<br />
<br />
Noon: Working Lunch - Roundtable discussion of data processing challenges<br />
<br />
1:00pm: Session 3<br />
<br />
*Graeme Winter and Richard Gildea: "DIALS - new methods for processing X-ray diffraction data"<br />
<br />
*James Parkhurst: "Robust background modelling in the presence of outliers in DIALS"<br />
<br />
*Jan Kern: TBD<br />
<br />
*Franklin Fuller: "Drop-by-Drop Transient Serial Crystallography of Metalloenzymes at an X-ray free electron laser"<br />
<br />
*Iris Young: "Room temperature studies of the oxygen-evolving complex of photosystem II using an X-ray free electron laser (XFEL)"<br />
<br />
2:40pm: Break<br />
<br />
3:00pm: Session 4<br />
<br />
*Aina Cohen: TBD<br />
<br />
*Rahel Woldeyes: "Using X-ray Free Electron lasers to visualize solvent in the M2 proton channel"<br />
<br />
*Danny Axford: "Highly efficient serial data collection from high-density fixed targets"<br />
<br />
*Christoph Mueller-Dieckmann: "Serial Synchrotron Crystallography at the ESRF using a high viscosity extruder"<br />
<br />
*Allen Orville: TBD<br />
<br />
= Feb 17th =<br />
<br />
== 9am: Tutorials 1 ==<br />
Aaron Brewster and Iris Young: cctbx.xfel<br />
Break: 10:00 am<br />
== 10:15 am: Tutorials 2 ==<br />
* Aaron Brewster and James Parkhurst: [[2017_dials.stills_process | dials.stills_process]]<br />
* Art Lyubimov: IOTA<br />
* Monarin Uervirojnangkoorn: [[2017_prime_tutorial | PRIME]]<br />
* Nick Sauter: [[2017_cxi_merge_tutorial | cxi.merge]]<br />
<br />
== 12:15pm-2:30pm: Round table discussion. == <br />
Topics will include:<br />
Is the current software meeting needs?<br />
What are essential/timely avenues most useful for near-term development (first half of 2017)?<br />
Where should we be going next (longer term future)?<br />
Time resolved experiments?<br />
Synchrotron serial crystallography?<br />
== 2:30-4pm: Breakout sessions: ==<br />
users work with developers and instructors on their own data. Hands-on walkthroughs and data analysis.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/Mod_image_dictMod image dict2016-11-03T00:26:57Z<p>Aaron: </p>
<hr />
<div><pre><br />
[psana]<br />
modules = my_ana_pkg.mod_image_dict<br />
[my_ana_pkg.mod_image_dict]<br />
address = MfxEndstation-0|Rayonix-0<br />
dark_path = None<br />
dark_stddev = None<br />
mask_path = None<br />
detz_offset = 89.0<br />
gain_map_path = None<br />
gain_map_level = None<br />
override_beam_x = 962.0<br />
override_beam_y = 963.0<br />
override_energy = None<br />
bin_size = 2<br />
crop_rayonix = False<br />
</pre><br />
<br />
These parameters are for a Rayonix image. Use crop_rayonix to trim the image such that it is square and centered on the beam center. Useful for the LABELIT backend of ''cctbx.xfel'' but not needed for DIALS.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/Mod_image_dictMod image dict2016-11-03T00:20:50Z<p>Aaron: </p>
<hr />
<div><pre><br />
[psana]<br />
modules = my_ana_pkg.mod_image_dict<br />
[my_ana_pkg.mod_image_dict]<br />
address = MfxEndstation-0|Rayonix-0<br />
dark_path = None<br />
dark_stddev = None<br />
mask_path = None<br />
detz_offset = 89.0<br />
gain_map_path = None<br />
gain_map_level = None<br />
override_beam_x = 962.0<br />
override_beam_y = 963.0<br />
override_energy = None<br />
bin_size = 2<br />
crop_rayonix = False<br />
</pre></div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/File_formatsFile formats2016-11-03T00:17:39Z<p>Aaron: /* Image pickles */</p>
<hr />
<div>This page is under construction<br />
<br />
== Introduction ==<br />
<br />
<br />
''cctbx.xfel'''s current native file formats, image pickles and integration pickles, are mainly intermediate file formats useful for debugging and software development. These binary pickle files are serialized python dictionaries optimized for machine readability. The rationale here is that the raw data from an experiment at LCLS is stored in the xtc streams. We process these in memory without needing to write images to disk at all, but provide image pickles if requested for use with the ''cctbx'' image viewer for diagnostic purposes. Integration pickles contain integrated intensities and crystal cell and orientation parameters and are read by the merging programs cxi.merge and prime.postrefine.<br />
<br />
In an effort to provide more human-readable data files and conform to international standards, we have worked with representatives of ImageCIF to create 64 tile segmented data in CBF format, the same format used in a wide variety of detectors. These CSPAD CBFs are designed to be used with other crystallographic software packages.<br />
<br />
Format specifications for these data files are provided below.<br />
<br />
== Image pickles ==<br />
<br />
''cctbx.xfel'' image pickles are a binary file format containing pixel data and image metadata. Under the hood these are serialized python dictionaries, but for general users it's sufficient to know that the blob of data consists of name/value pairs. The contents of an image pickle can be inspected with this command:<br />
<br />
'''cxi.print_pickle''' image.pickle<br />
<br />
For example, here's the output from an image with a thermolysin diffraction pattern collected recently:<br />
<br />
<div class="toccolours mw-collapsible mw-collapsed"><br />
<pre><br />
Printing contents of idx-20141201073601249.pickle<br />
Detector format version: CXI 10.1<br />
DISTANCE 119.002<br />
DETECTOR_ADDRESS CxiDs1-0|Cspad-0<br />
SIZE1 1765<br />
SIZE2 1765<br />
TIMESTAMP 2014-12-01T07:36Z01.249<br />
CCD_IMAGE_SATURATION 90000<br />
SATURATED_VALUE 90000<br />
64 active areas, first one: [715, 439, 909, 624]<br />
PIXEL_SIZE 0.11<br />
BEAM_CENTER_Y 97.075<br />
MIN_TRUSTED_VALUE -2000<br />
WAVELENGTH 1.75124427107 , converted to eV: 7079.77687909<br />
xtal_target None<br />
BEAM_CENTER_X 97.075<br />
SEQUENCE_NUMBER 0<br />
DATA len=3115225 max=106065.000000 min=-104526.000000 dimensions=(1765, 1765)<br />
cxi_versioned_extract()::cxi_version: CXI 10.1<br />
64 translated active areas, first one: [725, 449, 917, 632]<br />
</pre><br />
</div><br />
<br />
Each of the lines here is a name/value pair of data or metadata for the image. LABELIT, DIALS, cctbx.image_viewer, are all programs capable of processing these files directly. Here's a description of each of the name/value pairs:<br />
* Detector format version: CXI 10.1. Technically this value is not stored in the image pickle. The detector format version is used internally by cctbx.xfel for data collected up to LCLS run 11, and is a way of looking up tile corrections stored in the software. After run 11, this information should be stored in the phil file used to processes the data. The detector format version is looked up based on the detector address and event timestamp. All known detector format versions can be displayed with the command '''cxi.detector_format_versions'''.<br />
* DISTANCE: detector distance (mm)<br />
* DETECTOR_ADDRESS: string to identify the detector in the XTC stream. Recorded here as a unique identifier for the data when coupled with the event timestamp.<br />
* SIZE1 and SIZE2: image dimensions (pixels)<br />
* TIMESTAMP: unique time stamp for this event in the form year-month-day'''T'''hour:minute'''Z'''second.microsecond<br />
* CCD_IMAGE_SATURATION and SATURATED_VALUE: overload value<br />
* PIXEL_SIZE: pixel size in mm<br />
* BEAM_CENTER_X, BEAM_CENTER_Y: beam center, specified as positive mm offsets from the origin of the detector (upper-left corner)<br />
* MIN_TRUSTED_VALUE: underload<br />
* WAVELENGTH: wavelength as recorded in the XTC stream (angstroms)<br />
* DATA: pixel data. Some vital statistics are shown.<br />
* xtal_target and SEQUENCE_NUMBER: unused<br />
* 64 active areas: the active areas are coordinates that specify where the 64 tiles are in the image. They are in the form of quartets of numbers [x1,y1,x2,y2], one quartet for each of the 64 tiles. The first active area is shown. Note, for non-CSPAD image pickles, there will only be one active area and it will be the size of the image entire.<br />
* 64 translated active areas: if corrections were available for this image, the first corrected active area is shown here.<br />
<br />
=== A note on coordinate systems ===<br />
<br />
Internally, our software uses the ImageCIF coordinate system with the exception that no goniometer is typically available at XFELs, therefore Z is along the beam (source to sample), Y points against gravity and X completes a right-handed coordinate system. In the image pickles, the origin of the detector pixel readout is assumed to be in the upper left of the image, therefore the beam center is specified as positive offsets from that point. In example above, assuming the sample is at the laboratory origin, the vector pointing to the beam center on the detector in the ImageCIF coordinate system would be (0, 0, -119.002). A vector from the laboratory origin to the pixel readout origin (upper-left of the detector) would be -97.075, 97.075, -119.002). dxtbx.print_header is a useful tool to get these vectors from any file format understood by ''cctbx.xfel''. For more information see [http://cci.lbl.gov/publications/download/jo5001_reprint.pdf Parkhurst et al, 2014].<br />
<br />
=== Creating image pickles from xtc streams ===<br />
<br />
The program cctbx.xfel.xtc_dump can be used to convert xtc streams to image pickle format. Here we provide an example for the Rayonix detector that is often found at XPP and MFX. The psana module mod_image_dict is used to read the data and generate the images (example: [[mod_image_dict|mod_image_dict.cfg]]). After setting up a release directory as described in [[Setup]]:<br />
<br />
cd ~/myrelease # or wherever your release is<br />
sit_setup<br />
cctbx.xfel.xtc_dump experiment=mfxm9316 run_num=14 address="MfxEndstation-0|Rayonix-0" file_format=pickle cfg=mod_image_dict.cfg <br />
<br />
This will fill your working folder with image pickles. You can change your output directory with output.output_dir and you can limit the number of images saved with dispatch.max_events.<br />
<br />
This approach will also work for CSPAD images. Here, you must add calib_dir to your config file, as described in the indexing tutorials.<br />
<br />
== Integration pickles ==<br />
<br />
Integration pickles are the output of cctbx.xfel indexing and integration. They are like image pickles in that they are serialized python dictionaries and can be inspected with '''cxi.print_pickle'''. They are the primary input to '''cxi.merge''' and '''prime.run'''.<br />
<br />
== CSPAD CBFs ==<br />
<br />
CSPAD CBFs are used while processing CSPAD images using the DIALS backend of ''cctbx.xfel''. The complete specification for CSPAD CBFs is laid out in an article in the Computational Crystallographic Newsletter: "XFEL Detectors and ImageCIF", Computational Crystallography Newsletter 5, 19-24. ([http://cci.lbl.gov/publications/download/CCN_2014_p19.pdf Reprint]). Documentation for using DIALS to index and integrate CSPAD data can be found in an article in the Computational Crystallographic Newsletter: "Processing XFEL data with cctbx.xfel and DIALS", Computational Crystallography Newsletter 7, 32-53 ([http://www.phenix-online.org/newsletter/CCN_2016_07.pdf#page=18 Reprint]).<br />
<br />
=== Creating CSPAD CBFs from XTC streams ===<br />
<br />
cctbx.xfel.xtc_dump is useful for converting XTC streams to CBF files as needed. Use -c to get a listing of all options, and -c -a 2 to get a full listing with help strings. Example command:<br />
<br />
cd ~/myrelease # or wherever your release is<br />
sit_setup<br />
cctbx.xfel.xtc_dump dispatch.max_events=1 input.experiment=xpptut15 input.address=cspad \<br />
input.run_num=54 format.file_format=cbf output.output_dir=xpptut15/out \<br />
format.cbf.detz_offset=100 input.override_energy=7000<br />
<br />
Here, one image is created from experiment xpptut15, run 54. You can display the image with cctbx.image_viewer xpptut15/out/*.cbf. Note the pink gaps between the tiles. This results from the segmented nature of the CSPAD, preserved in the CBF file.<br />
<br />
xpptut15 is data from XPP's CSPAD and was collected without xray's on. Hence format.cbf.detz_offset=100 input.override_energy=7000 are set to fake values in this command.<br />
<br />
=== Converting from SLAC's metrology to CBF ===<br />
<br />
The tile positions of the hierarchical CSPAD detector are specified by SLAC in a geometry file in the calib folder of each experiment. Typically this file is named something like 0-end.data. If desired, this file can be converted to just the human readable header portion of a CSPAD cbf using the ''cctbx.xfel'' command cxi.slaccalib2cbfheader. Example:<br />
<br />
cxi.slaccalib2cbfheader metrology_file=0-end.data out=tmp.cbf<br />
<br />
This cbf header can be converted back to SLAC format using cxi.cbfheader2slaccalib:<br />
<br />
cxi.cbfheader2slaccalib cbf_header=tmp.cbf out_metrology_file=tmp.data<br />
<br />
=== Displaying metrology files ===<br />
<br />
cxi.display_metrology <filename> can be used to show a plot of tile positions. It accepts SLAC metrology files, CSPAD CBFs and image pickles.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/File_formatsFile formats2016-11-01T22:40:08Z<p>Aaron: /* Creating image pickles from xtc streams */</p>
<hr />
<div>This page is under construction<br />
<br />
== Introduction ==<br />
<br />
<br />
''cctbx.xfel'''s current native file formats, image pickles and integration pickles, are mainly intermediate file formats useful for debugging and software development. These binary pickle files are serialized python dictionaries optimized for machine readability. The rationale here is that the raw data from an experiment at LCLS is stored in the xtc streams. We process these in memory without needing to write images to disk at all, but provide image pickles if requested for use with the ''cctbx'' image viewer for diagnostic purposes. Integration pickles contain integrated intensities and crystal cell and orientation parameters and are read by the merging programs cxi.merge and prime.postrefine.<br />
<br />
In an effort to provide more human-readable data files and conform to international standards, we have worked with representatives of ImageCIF to create 64 tile segmented data in CBF format, the same format used in a wide variety of detectors. These CSPAD CBFs are designed to be used with other crystallographic software packages.<br />
<br />
Format specifications for these data files are provided below.<br />
<br />
== Image pickles ==<br />
<br />
''cctbx.xfel'' image pickles are a binary file format containing pixel data and image metadata. Under the hood these are serialized python dictionaries, but for general users it's sufficient to know that the blob of data consists of name/value pairs. The contents of an image pickle can be inspected with this command:<br />
<br />
'''cxi.print_pickle''' image.pickle<br />
<br />
For example, here's the output from an image with a thermolysin diffraction pattern collected recently:<br />
<br />
<div class="toccolours mw-collapsible mw-collapsed"><br />
<pre><br />
Printing contents of idx-20141201073601249.pickle<br />
Detector format version: CXI 10.1<br />
DISTANCE 119.002<br />
DETECTOR_ADDRESS CxiDs1-0|Cspad-0<br />
SIZE1 1765<br />
SIZE2 1765<br />
TIMESTAMP 2014-12-01T07:36Z01.249<br />
CCD_IMAGE_SATURATION 90000<br />
SATURATED_VALUE 90000<br />
64 active areas, first one: [715, 439, 909, 624]<br />
PIXEL_SIZE 0.11<br />
BEAM_CENTER_Y 97.075<br />
MIN_TRUSTED_VALUE -2000<br />
WAVELENGTH 1.75124427107 , converted to eV: 7079.77687909<br />
xtal_target None<br />
BEAM_CENTER_X 97.075<br />
SEQUENCE_NUMBER 0<br />
DATA len=3115225 max=106065.000000 min=-104526.000000 dimensions=(1765, 1765)<br />
cxi_versioned_extract()::cxi_version: CXI 10.1<br />
64 translated active areas, first one: [725, 449, 917, 632]<br />
</pre><br />
</div><br />
<br />
Each of the lines here is a name/value pair of data or metadata for the image. LABELIT, DIALS, cctbx.image_viewer, are all programs capable of processing these files directly. Here's a description of each of the name/value pairs:<br />
* Detector format version: CXI 10.1. Technically this value is not stored in the image pickle. The detector format version is used internally by cctbx.xfel for data collected up to LCLS run 11, and is a way of looking up tile corrections stored in the software. After run 11, this information should be stored in the phil file used to processes the data. The detector format version is looked up based on the detector address and event timestamp. All known detector format versions can be displayed with the command '''cxi.detector_format_versions'''.<br />
* DISTANCE: detector distance<br />
* DETECTOR_ADDRESS: string to identify the detector in the XTC stream. Recorded here as a unique identifier for the data when coupled with the event timestamp.<br />
* SIZE1 and SIZE2: image dimensions<br />
* TIMESTAMP: unique time stamp for this event in the form year-month-day'''T'''hour:minute'''Z'''second.microsecond<br />
* CCD_IMAGE_SATURATION and SATURATED_VALUE: overload value<br />
* PIXEL_SIZE: pixel size in mm<br />
* BEAM_CENTER_X, BEAM_CENTER_Y: beam center<br />
* MIN_TRUSTED_VALUE: underload<br />
* WAVELENGTH: wavelength as recorded in the XTC stream<br />
* DATA: pixel data. Some vital statistics are shown.<br />
* xtal_target and SEQUENCE_NUMBER: unused<br />
* 64 active areas: the active areas are coordinates that specify where the 64 tiles are in the image. They are in the form of quartets of numbers [x1,y1,x2,y2], one quartet for each of the 64 tiles. The first active area is shown. Note, for non-CSPAD image pickles, there will only be one active area and it will be the size of the image entire.<br />
* 64 translated active areas: if corrections were available for this image, the first corrected active area is shown here.<br />
<br />
=== Creating image pickles from xtc streams ===<br />
<br />
The program cctbx.xfel.xtc_dump can be used to convert xtc streams to image pickle format. Here we provide an example for the Rayonix detector that is often found at XPP and MFX. The psana module mod_image_dict is used to read the data and generate the images (example: [[mod_image_dict|mod_image_dict.cfg]]). After setting up a release directory as described in [[Setup]]:<br />
<br />
cd ~/myrelease # or wherever your release is<br />
sit_setup<br />
cctbx.xfel.xtc_dump experiment=mfxm9316 run_num=14 address="MfxEndstation-0|Rayonix-0" file_format=pickle cfg=mod_image_dict.cfg <br />
<br />
This will fill your working folder with image pickles. You can change your output directory with output.output_dir and you can limit the number of images saved with dispatch.max_events.<br />
<br />
This approach will also work for CSPAD images. Here, you must add calib_dir to your config file, as described in the indexing tutorials.<br />
<br />
== Integration pickles ==<br />
<br />
Integration pickles are the output of cctbx.xfel indexing and integration. They are like image pickles in that they are serialized python dictionaries and can be inspected with '''cxi.print_pickle'''. They are the primary input to '''cxi.merge''' and '''prime.run'''.<br />
<br />
== CSPAD CBFs ==<br />
<br />
CSPAD CBFs are used while processing CSPAD images using the DIALS backend of ''cctbx.xfel''. The complete specification for CSPAD CBFs is laid out in an article in the Computational Crystallographic Newsletter: "XFEL Detectors and ImageCIF", Computational Crystallography Newsletter 5, 19-24. ([http://cci.lbl.gov/publications/download/CCN_2014_p19.pdf Reprint]). Documentation for using DIALS to index and integrate CSPAD data can be found in an article in the Computational Crystallographic Newsletter: "Processing XFEL data with cctbx.xfel and DIALS", Computational Crystallography Newsletter 7, 32-53 ([http://www.phenix-online.org/newsletter/CCN_2016_07.pdf#page=18 Reprint]).<br />
<br />
=== Creating CSPAD CBFs from XTC streams ===<br />
<br />
cctbx.xfel.xtc_dump is useful for converting XTC streams to CBF files as needed. Use -c to get a listing of all options, and -c -a 2 to get a full listing with help strings. Example command:<br />
<br />
cd ~/myrelease # or wherever your release is<br />
sit_setup<br />
cctbx.xfel.xtc_dump dispatch.max_events=1 input.experiment=xpptut15 input.address=cspad \<br />
input.run_num=54 format.file_format=cbf output.output_dir=xpptut15/out \<br />
format.cbf.detz_offset=100 input.override_energy=7000<br />
<br />
Here, one image is created from experiment xpptut15, run 54. You can display the image with cctbx.image_viewer xpptut15/out/*.cbf. Note the pink gaps between the tiles. This results from the segmented nature of the CSPAD, preserved in the CBF file.<br />
<br />
xpptut15 is data from XPP's CSPAD and was collected without xray's on. Hence format.cbf.detz_offset=100 input.override_energy=7000 are set to fake values in this command.<br />
<br />
=== Converting from SLAC's metrology to CBF ===<br />
<br />
The tile positions of the hierarchical CSPAD detector are specified by SLAC in a geometry file in the calib folder of each experiment. Typically this file is named something like 0-end.data. If desired, this file can be converted to just the human readable header portion of a CSPAD cbf using the ''cctbx.xfel'' command cxi.slaccalib2cbfheader. Example:<br />
<br />
cxi.slaccalib2cbfheader metrology_file=0-end.data out=tmp.cbf<br />
<br />
This cbf header can be converted back to SLAC format using cxi.cbfheader2slaccalib:<br />
<br />
cxi.cbfheader2slaccalib cbf_header=tmp.cbf out_metrology_file=tmp.data<br />
<br />
=== Displaying metrology files ===<br />
<br />
cxi.display_metrology <filename> can be used to show a plot of tile positions. It accepts SLAC metrology files, CSPAD CBFs and image pickles.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/File_formatsFile formats2016-11-01T22:34:32Z<p>Aaron: /* CSPAD CBFs */</p>
<hr />
<div>This page is under construction<br />
<br />
== Introduction ==<br />
<br />
<br />
''cctbx.xfel'''s current native file formats, image pickles and integration pickles, are mainly intermediate file formats useful for debugging and software development. These binary pickle files are serialized python dictionaries optimized for machine readability. The rationale here is that the raw data from an experiment at LCLS is stored in the xtc streams. We process these in memory without needing to write images to disk at all, but provide image pickles if requested for use with the ''cctbx'' image viewer for diagnostic purposes. Integration pickles contain integrated intensities and crystal cell and orientation parameters and are read by the merging programs cxi.merge and prime.postrefine.<br />
<br />
In an effort to provide more human-readable data files and conform to international standards, we have worked with representatives of ImageCIF to create 64 tile segmented data in CBF format, the same format used in a wide variety of detectors. These CSPAD CBFs are designed to be used with other crystallographic software packages.<br />
<br />
Format specifications for these data files are provided below.<br />
<br />
== Image pickles ==<br />
<br />
''cctbx.xfel'' image pickles are a binary file format containing pixel data and image metadata. Under the hood these are serialized python dictionaries, but for general users it's sufficient to know that the blob of data consists of name/value pairs. The contents of an image pickle can be inspected with this command:<br />
<br />
'''cxi.print_pickle''' image.pickle<br />
<br />
For example, here's the output from an image with a thermolysin diffraction pattern collected recently:<br />
<br />
<div class="toccolours mw-collapsible mw-collapsed"><br />
<pre><br />
Printing contents of idx-20141201073601249.pickle<br />
Detector format version: CXI 10.1<br />
DISTANCE 119.002<br />
DETECTOR_ADDRESS CxiDs1-0|Cspad-0<br />
SIZE1 1765<br />
SIZE2 1765<br />
TIMESTAMP 2014-12-01T07:36Z01.249<br />
CCD_IMAGE_SATURATION 90000<br />
SATURATED_VALUE 90000<br />
64 active areas, first one: [715, 439, 909, 624]<br />
PIXEL_SIZE 0.11<br />
BEAM_CENTER_Y 97.075<br />
MIN_TRUSTED_VALUE -2000<br />
WAVELENGTH 1.75124427107 , converted to eV: 7079.77687909<br />
xtal_target None<br />
BEAM_CENTER_X 97.075<br />
SEQUENCE_NUMBER 0<br />
DATA len=3115225 max=106065.000000 min=-104526.000000 dimensions=(1765, 1765)<br />
cxi_versioned_extract()::cxi_version: CXI 10.1<br />
64 translated active areas, first one: [725, 449, 917, 632]<br />
</pre><br />
</div><br />
<br />
Each of the lines here is a name/value pair of data or metadata for the image. LABELIT, DIALS, cctbx.image_viewer, are all programs capable of processing these files directly. Here's a description of each of the name/value pairs:<br />
* Detector format version: CXI 10.1. Technically this value is not stored in the image pickle. The detector format version is used internally by cctbx.xfel for data collected up to LCLS run 11, and is a way of looking up tile corrections stored in the software. After run 11, this information should be stored in the phil file used to processes the data. The detector format version is looked up based on the detector address and event timestamp. All known detector format versions can be displayed with the command '''cxi.detector_format_versions'''.<br />
* DISTANCE: detector distance<br />
* DETECTOR_ADDRESS: string to identify the detector in the XTC stream. Recorded here as a unique identifier for the data when coupled with the event timestamp.<br />
* SIZE1 and SIZE2: image dimensions<br />
* TIMESTAMP: unique time stamp for this event in the form year-month-day'''T'''hour:minute'''Z'''second.microsecond<br />
* CCD_IMAGE_SATURATION and SATURATED_VALUE: overload value<br />
* PIXEL_SIZE: pixel size in mm<br />
* BEAM_CENTER_X, BEAM_CENTER_Y: beam center<br />
* MIN_TRUSTED_VALUE: underload<br />
* WAVELENGTH: wavelength as recorded in the XTC stream<br />
* DATA: pixel data. Some vital statistics are shown.<br />
* xtal_target and SEQUENCE_NUMBER: unused<br />
* 64 active areas: the active areas are coordinates that specify where the 64 tiles are in the image. They are in the form of quartets of numbers [x1,y1,x2,y2], one quartet for each of the 64 tiles. The first active area is shown. Note, for non-CSPAD image pickles, there will only be one active area and it will be the size of the image entire.<br />
* 64 translated active areas: if corrections were available for this image, the first corrected active area is shown here.<br />
<br />
=== Creating image pickles from xtc streams ===<br />
<br />
The program cctbx.xfel.xtc_dump can be used to convert xtc streams to image pickle format. Here we provide an example for the Rayonix detector that is often found at XPP and MFX.<br />
<ol><br />
<li>Set up a release directory as described in [[Setup]]<br />
<li>The psana module mod_image_dict is used to read the data and generate the images. An example: [[mod_image_dict|mod_image_dict.cfg]]<br />
<li>cd /path/to/test/release<br />
<li>sit_setup<br />
<li>cctbx.xfel.xtc_dump experiment=mfxm9316 run_num=14 address="MfxEndstation-0|Rayonix-0" file_format=pickle cfg=mod_image_dict.cfg <br />
</ol><br />
This will fill your working folder with image pickles. You can change your output directory with output.output_dir and you can limit the number of images saved with dispatch.max_events.<br />
<br />
This approach will also work for CSPAD images. Here, you must add calib_dir to your config file, as described in the indexing tutorials.<br />
<br />
== Integration pickles ==<br />
<br />
Integration pickles are the output of cctbx.xfel indexing and integration. They are like image pickles in that they are serialized python dictionaries and can be inspected with '''cxi.print_pickle'''. They are the primary input to '''cxi.merge''' and '''prime.run'''.<br />
<br />
== CSPAD CBFs ==<br />
<br />
CSPAD CBFs are used while processing CSPAD images using the DIALS backend of ''cctbx.xfel''. The complete specification for CSPAD CBFs is laid out in an article in the Computational Crystallographic Newsletter: "XFEL Detectors and ImageCIF", Computational Crystallography Newsletter 5, 19-24. ([http://cci.lbl.gov/publications/download/CCN_2014_p19.pdf Reprint]). Documentation for using DIALS to index and integrate CSPAD data can be found in an article in the Computational Crystallographic Newsletter: "Processing XFEL data with cctbx.xfel and DIALS", Computational Crystallography Newsletter 7, 32-53 ([http://www.phenix-online.org/newsletter/CCN_2016_07.pdf#page=18 Reprint]).<br />
<br />
=== Creating CSPAD CBFs from XTC streams ===<br />
<br />
cctbx.xfel.xtc_dump is useful for converting XTC streams to CBF files as needed. Use -c to get a listing of all options, and -c -a 2 to get a full listing with help strings. Example command:<br />
<br />
cd ~/myrelease # or wherever your release is<br />
sit_setup<br />
cctbx.xfel.xtc_dump dispatch.max_events=1 input.experiment=xpptut15 input.address=cspad \<br />
input.run_num=54 format.file_format=cbf output.output_dir=xpptut15/out \<br />
format.cbf.detz_offset=100 input.override_energy=7000<br />
<br />
Here, one image is created from experiment xpptut15, run 54. You can display the image with cctbx.image_viewer xpptut15/out/*.cbf. Note the pink gaps between the tiles. This results from the segmented nature of the CSPAD, preserved in the CBF file.<br />
<br />
xpptut15 is data from XPP's CSPAD and was collected without xray's on. Hence format.cbf.detz_offset=100 input.override_energy=7000 are set to fake values in this command.<br />
<br />
=== Converting from SLAC's metrology to CBF ===<br />
<br />
The tile positions of the hierarchical CSPAD detector are specified by SLAC in a geometry file in the calib folder of each experiment. Typically this file is named something like 0-end.data. If desired, this file can be converted to just the human readable header portion of a CSPAD cbf using the ''cctbx.xfel'' command cxi.slaccalib2cbfheader. Example:<br />
<br />
cxi.slaccalib2cbfheader metrology_file=0-end.data out=tmp.cbf<br />
<br />
This cbf header can be converted back to SLAC format using cxi.cbfheader2slaccalib:<br />
<br />
cxi.cbfheader2slaccalib cbf_header=tmp.cbf out_metrology_file=tmp.data<br />
<br />
=== Displaying metrology files ===<br />
<br />
cxi.display_metrology <filename> can be used to show a plot of tile positions. It accepts SLAC metrology files, CSPAD CBFs and image pickles.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/Mod_image_dictMod image dict2016-11-01T22:28:08Z<p>Aaron: Created page with "<pre> [psana] modules = my_ana_pkg.mod_image_dict [my_ana_pkg.mod_image_dict] address = MfxEndstation-0|Rayonix-0 dark_path = None dark_stddev = None mask_pa..."</p>
<hr />
<div><pre><br />
[psana]<br />
modules = my_ana_pkg.mod_image_dict<br />
[my_ana_pkg.mod_image_dict]<br />
address = MfxEndstation-0|Rayonix-0<br />
dark_path = None<br />
dark_stddev = None<br />
mask_path = None<br />
detz_offset = 89.0<br />
gain_map_path = None<br />
gain_map_level = None<br />
override_beam_x = 962.0<br />
override_beam_y = 963.0<br />
override_energy = None<br />
bin_size = 2<br />
crop_rayonix = True<br />
</pre></div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/File_formatsFile formats2016-11-01T22:27:00Z<p>Aaron: /* Creating image pickles from xtc streams */</p>
<hr />
<div>This page is under construction<br />
<br />
== Introduction ==<br />
<br />
<br />
''cctbx.xfel'''s current native file formats, image pickles and integration pickles, are mainly intermediate file formats useful for debugging and software development. These binary pickle files are serialized python dictionaries optimized for machine readability. The rationale here is that the raw data from an experiment at LCLS is stored in the xtc streams. We process these in memory without needing to write images to disk at all, but provide image pickles if requested for use with the ''cctbx'' image viewer for diagnostic purposes. Integration pickles contain integrated intensities and crystal cell and orientation parameters and are read by the merging programs cxi.merge and prime.postrefine.<br />
<br />
In an effort to provide more human-readable data files and conform to international standards, we have worked with representatives of ImageCIF to create 64 tile segmented data in CBF format, the same format used in a wide variety of detectors. These CSPAD CBFs are designed to be used with other crystallographic software packages.<br />
<br />
Format specifications for these data files are provided below.<br />
<br />
== Image pickles ==<br />
<br />
''cctbx.xfel'' image pickles are a binary file format containing pixel data and image metadata. Under the hood these are serialized python dictionaries, but for general users it's sufficient to know that the blob of data consists of name/value pairs. The contents of an image pickle can be inspected with this command:<br />
<br />
'''cxi.print_pickle''' image.pickle<br />
<br />
For example, here's the output from an image with a thermolysin diffraction pattern collected recently:<br />
<br />
<div class="toccolours mw-collapsible mw-collapsed"><br />
<pre><br />
Printing contents of idx-20141201073601249.pickle<br />
Detector format version: CXI 10.1<br />
DISTANCE 119.002<br />
DETECTOR_ADDRESS CxiDs1-0|Cspad-0<br />
SIZE1 1765<br />
SIZE2 1765<br />
TIMESTAMP 2014-12-01T07:36Z01.249<br />
CCD_IMAGE_SATURATION 90000<br />
SATURATED_VALUE 90000<br />
64 active areas, first one: [715, 439, 909, 624]<br />
PIXEL_SIZE 0.11<br />
BEAM_CENTER_Y 97.075<br />
MIN_TRUSTED_VALUE -2000<br />
WAVELENGTH 1.75124427107 , converted to eV: 7079.77687909<br />
xtal_target None<br />
BEAM_CENTER_X 97.075<br />
SEQUENCE_NUMBER 0<br />
DATA len=3115225 max=106065.000000 min=-104526.000000 dimensions=(1765, 1765)<br />
cxi_versioned_extract()::cxi_version: CXI 10.1<br />
64 translated active areas, first one: [725, 449, 917, 632]<br />
</pre><br />
</div><br />
<br />
Each of the lines here is a name/value pair of data or metadata for the image. LABELIT, DIALS, cctbx.image_viewer, are all programs capable of processing these files directly. Here's a description of each of the name/value pairs:<br />
* Detector format version: CXI 10.1. Technically this value is not stored in the image pickle. The detector format version is used internally by cctbx.xfel for data collected up to LCLS run 11, and is a way of looking up tile corrections stored in the software. After run 11, this information should be stored in the phil file used to processes the data. The detector format version is looked up based on the detector address and event timestamp. All known detector format versions can be displayed with the command '''cxi.detector_format_versions'''.<br />
* DISTANCE: detector distance<br />
* DETECTOR_ADDRESS: string to identify the detector in the XTC stream. Recorded here as a unique identifier for the data when coupled with the event timestamp.<br />
* SIZE1 and SIZE2: image dimensions<br />
* TIMESTAMP: unique time stamp for this event in the form year-month-day'''T'''hour:minute'''Z'''second.microsecond<br />
* CCD_IMAGE_SATURATION and SATURATED_VALUE: overload value<br />
* PIXEL_SIZE: pixel size in mm<br />
* BEAM_CENTER_X, BEAM_CENTER_Y: beam center<br />
* MIN_TRUSTED_VALUE: underload<br />
* WAVELENGTH: wavelength as recorded in the XTC stream<br />
* DATA: pixel data. Some vital statistics are shown.<br />
* xtal_target and SEQUENCE_NUMBER: unused<br />
* 64 active areas: the active areas are coordinates that specify where the 64 tiles are in the image. They are in the form of quartets of numbers [x1,y1,x2,y2], one quartet for each of the 64 tiles. The first active area is shown. Note, for non-CSPAD image pickles, there will only be one active area and it will be the size of the image entire.<br />
* 64 translated active areas: if corrections were available for this image, the first corrected active area is shown here.<br />
<br />
=== Creating image pickles from xtc streams ===<br />
<br />
The program cctbx.xfel.xtc_dump can be used to convert xtc streams to image pickle format. Here we provide an example for the Rayonix detector that is often found at XPP and MFX.<br />
<ol><br />
<li>Set up a release directory as described in [[Setup]]<br />
<li>The psana module mod_image_dict is used to read the data and generate the images. An example: [[mod_image_dict|mod_image_dict.cfg]]<br />
<li>cd /path/to/test/release<br />
<li>sit_setup<br />
<li>cctbx.xfel.xtc_dump experiment=mfxm9316 run_num=14 address="MfxEndstation-0|Rayonix-0" file_format=pickle cfg=mod_image_dict.cfg <br />
</ol><br />
This will fill your working folder with image pickles. You can change your output directory with output.output_dir and you can limit the number of images saved with dispatch.max_events.<br />
<br />
This approach will also work for CSPAD images. Here, you must add calib_dir to your config file, as described in the indexing tutorials.<br />
<br />
== Integration pickles ==<br />
<br />
Integration pickles are the output of cctbx.xfel indexing and integration. They are like image pickles in that they are serialized python dictionaries and can be inspected with '''cxi.print_pickle'''. They are the primary input to '''cxi.merge''' and '''prime.run'''.<br />
<br />
== CSPAD CBFs ==<br />
<br />
CSPAD CBFs are not used directly by ''cctbx.xfel'' at this time. This aspect of the project is under development. The complete specification for CSPAD CBFs is laid out in an article in the Computational Crystallographic Newsletter: "XFEL Detectors and ImageCIF", Computational Crystallography Newsletter 5, 19-24. ([http://cci.lbl.gov/publications/download/CCN_2014_p19.pdf Reprint]). CSPAD CBFs<br />
<br />
=== Creating CSPAD CBFs from XTC streams ===<br />
<br />
cctbx.xfel.xtc_dump is useful for converting XTC streams to CBF files as needed. Use -c to get a listing of all options, and -c -a 2 to get a full listing with help strings. Example command:<br />
<br />
cd ~/myrelease # or wherever your release is<br />
sit_setup<br />
cctbx.xfel.xtc_dump dispatch.max_events=1 input.experiment=xpptut15 input.address=cspad \<br />
input.run_num=54 format.file_format=cbf output.output_dir=xpptut15/out \<br />
format.cbf.detz_offset=100 input.override_energy=7000<br />
<br />
Here, one image is created from experiment xpptut15, run 54. You can display the image with cctbx.image_viewer xpptut15/out/*.cbf. Note the pink gaps between the tiles. This results from the segmented nature of the CSPAD, preserved in the CBF file.<br />
<br />
xpptut15 is data from XPP's CSPAD and was collected without xray's on. Hence format.cbf.detz_offset=100 input.override_energy=7000 are set to fake values in this command.<br />
<br />
=== Converting from SLAC's metrology to CBF ===<br />
<br />
The tile positions of the hierarchical CSPAD detector are specified by SLAC in a geometry file in the calib folder of each experiment. Typically this file is named something like 0-end.data. If desired, this file can be converted to just the human readable header portion of a CSPAD cbf using the ''cctbx.xfel'' command cxi.slaccalib2cbfheader. Example:<br />
<br />
cxi.slaccalib2cbfheader metrology_file=0-end.data out=tmp.cbf<br />
<br />
This cbf header can be converted back to SLAC format using cxi.cbfheader2slaccalib:<br />
<br />
cxi.cbfheader2slaccalib cbf_header=tmp.cbf out_metrology_file=tmp.data<br />
<br />
=== Displaying metrology files ===<br />
<br />
cxi.display_metrology <filename> can be used to show a plot of tile positions. It accepts SLAC metrology files, CSPAD CBFs and image pickles.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/File_formatsFile formats2016-11-01T22:25:26Z<p>Aaron: /* Integration pickles */</p>
<hr />
<div>This page is under construction<br />
<br />
== Introduction ==<br />
<br />
<br />
''cctbx.xfel'''s current native file formats, image pickles and integration pickles, are mainly intermediate file formats useful for debugging and software development. These binary pickle files are serialized python dictionaries optimized for machine readability. The rationale here is that the raw data from an experiment at LCLS is stored in the xtc streams. We process these in memory without needing to write images to disk at all, but provide image pickles if requested for use with the ''cctbx'' image viewer for diagnostic purposes. Integration pickles contain integrated intensities and crystal cell and orientation parameters and are read by the merging programs cxi.merge and prime.postrefine.<br />
<br />
In an effort to provide more human-readable data files and conform to international standards, we have worked with representatives of ImageCIF to create 64 tile segmented data in CBF format, the same format used in a wide variety of detectors. These CSPAD CBFs are designed to be used with other crystallographic software packages.<br />
<br />
Format specifications for these data files are provided below.<br />
<br />
== Image pickles ==<br />
<br />
''cctbx.xfel'' image pickles are a binary file format containing pixel data and image metadata. Under the hood these are serialized python dictionaries, but for general users it's sufficient to know that the blob of data consists of name/value pairs. The contents of an image pickle can be inspected with this command:<br />
<br />
'''cxi.print_pickle''' image.pickle<br />
<br />
For example, here's the output from an image with a thermolysin diffraction pattern collected recently:<br />
<br />
<div class="toccolours mw-collapsible mw-collapsed"><br />
<pre><br />
Printing contents of idx-20141201073601249.pickle<br />
Detector format version: CXI 10.1<br />
DISTANCE 119.002<br />
DETECTOR_ADDRESS CxiDs1-0|Cspad-0<br />
SIZE1 1765<br />
SIZE2 1765<br />
TIMESTAMP 2014-12-01T07:36Z01.249<br />
CCD_IMAGE_SATURATION 90000<br />
SATURATED_VALUE 90000<br />
64 active areas, first one: [715, 439, 909, 624]<br />
PIXEL_SIZE 0.11<br />
BEAM_CENTER_Y 97.075<br />
MIN_TRUSTED_VALUE -2000<br />
WAVELENGTH 1.75124427107 , converted to eV: 7079.77687909<br />
xtal_target None<br />
BEAM_CENTER_X 97.075<br />
SEQUENCE_NUMBER 0<br />
DATA len=3115225 max=106065.000000 min=-104526.000000 dimensions=(1765, 1765)<br />
cxi_versioned_extract()::cxi_version: CXI 10.1<br />
64 translated active areas, first one: [725, 449, 917, 632]<br />
</pre><br />
</div><br />
<br />
Each of the lines here is a name/value pair of data or metadata for the image. LABELIT, DIALS, cctbx.image_viewer, are all programs capable of processing these files directly. Here's a description of each of the name/value pairs:<br />
* Detector format version: CXI 10.1. Technically this value is not stored in the image pickle. The detector format version is used internally by cctbx.xfel for data collected up to LCLS run 11, and is a way of looking up tile corrections stored in the software. After run 11, this information should be stored in the phil file used to processes the data. The detector format version is looked up based on the detector address and event timestamp. All known detector format versions can be displayed with the command '''cxi.detector_format_versions'''.<br />
* DISTANCE: detector distance<br />
* DETECTOR_ADDRESS: string to identify the detector in the XTC stream. Recorded here as a unique identifier for the data when coupled with the event timestamp.<br />
* SIZE1 and SIZE2: image dimensions<br />
* TIMESTAMP: unique time stamp for this event in the form year-month-day'''T'''hour:minute'''Z'''second.microsecond<br />
* CCD_IMAGE_SATURATION and SATURATED_VALUE: overload value<br />
* PIXEL_SIZE: pixel size in mm<br />
* BEAM_CENTER_X, BEAM_CENTER_Y: beam center<br />
* MIN_TRUSTED_VALUE: underload<br />
* WAVELENGTH: wavelength as recorded in the XTC stream<br />
* DATA: pixel data. Some vital statistics are shown.<br />
* xtal_target and SEQUENCE_NUMBER: unused<br />
* 64 active areas: the active areas are coordinates that specify where the 64 tiles are in the image. They are in the form of quartets of numbers [x1,y1,x2,y2], one quartet for each of the 64 tiles. The first active area is shown. Note, for non-CSPAD image pickles, there will only be one active area and it will be the size of the image entire.<br />
* 64 translated active areas: if corrections were available for this image, the first corrected active area is shown here.<br />
<br />
=== Creating image pickles from xtc streams ===<br />
<br />
The program cctbx.xfel.xtc_dump can be used to convert xtc streams to image pickle format. Here we provide an example for the Rayonix detector that is often found at XPP and MFX.<br />
<ol><br />
<li>Set up a release directory as described in [[Setup]]<br />
<li>The psana module mod_image_dict is used to read the data and generate the images. An example: [[mod_image_dict.cfg]]<br />
<li>cd /path/to/test/release<br />
<li>sit_setup<br />
<li>cctbx.xfel.xtc_dump experiment=mfxm9316 run_num=14 address="MfxEndstation-0|Rayonix-0" file_format=pickle cfg=mod_image_dict.cfg <br />
</ol><br />
This will fill your working folder with image pickles. You can change your output directory with output.output_dir and you can limit the number of images saved with dispatch.max_events.<br />
<br />
This approach will also work for CSPAD images. Here, you must add calib_dir to your config file, as described in the indexing tutorials.<br />
<br />
== Integration pickles ==<br />
<br />
Integration pickles are the output of cctbx.xfel indexing and integration. They are like image pickles in that they are serialized python dictionaries and can be inspected with '''cxi.print_pickle'''. They are the primary input to '''cxi.merge''' and '''prime.run'''.<br />
<br />
== CSPAD CBFs ==<br />
<br />
CSPAD CBFs are not used directly by ''cctbx.xfel'' at this time. This aspect of the project is under development. The complete specification for CSPAD CBFs is laid out in an article in the Computational Crystallographic Newsletter: "XFEL Detectors and ImageCIF", Computational Crystallography Newsletter 5, 19-24. ([http://cci.lbl.gov/publications/download/CCN_2014_p19.pdf Reprint]). CSPAD CBFs<br />
<br />
=== Creating CSPAD CBFs from XTC streams ===<br />
<br />
cctbx.xfel.xtc_dump is useful for converting XTC streams to CBF files as needed. Use -c to get a listing of all options, and -c -a 2 to get a full listing with help strings. Example command:<br />
<br />
cd ~/myrelease # or wherever your release is<br />
sit_setup<br />
cctbx.xfel.xtc_dump dispatch.max_events=1 input.experiment=xpptut15 input.address=cspad \<br />
input.run_num=54 format.file_format=cbf output.output_dir=xpptut15/out \<br />
format.cbf.detz_offset=100 input.override_energy=7000<br />
<br />
Here, one image is created from experiment xpptut15, run 54. You can display the image with cctbx.image_viewer xpptut15/out/*.cbf. Note the pink gaps between the tiles. This results from the segmented nature of the CSPAD, preserved in the CBF file.<br />
<br />
xpptut15 is data from XPP's CSPAD and was collected without xray's on. Hence format.cbf.detz_offset=100 input.override_energy=7000 are set to fake values in this command.<br />
<br />
=== Converting from SLAC's metrology to CBF ===<br />
<br />
The tile positions of the hierarchical CSPAD detector are specified by SLAC in a geometry file in the calib folder of each experiment. Typically this file is named something like 0-end.data. If desired, this file can be converted to just the human readable header portion of a CSPAD cbf using the ''cctbx.xfel'' command cxi.slaccalib2cbfheader. Example:<br />
<br />
cxi.slaccalib2cbfheader metrology_file=0-end.data out=tmp.cbf<br />
<br />
This cbf header can be converted back to SLAC format using cxi.cbfheader2slaccalib:<br />
<br />
cxi.cbfheader2slaccalib cbf_header=tmp.cbf out_metrology_file=tmp.data<br />
<br />
=== Displaying metrology files ===<br />
<br />
cxi.display_metrology <filename> can be used to show a plot of tile positions. It accepts SLAC metrology files, CSPAD CBFs and image pickles.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/File_formatsFile formats2016-11-01T22:25:05Z<p>Aaron: /* Image pickles */</p>
<hr />
<div>This page is under construction<br />
<br />
== Introduction ==<br />
<br />
<br />
''cctbx.xfel'''s current native file formats, image pickles and integration pickles, are mainly intermediate file formats useful for debugging and software development. These binary pickle files are serialized python dictionaries optimized for machine readability. The rationale here is that the raw data from an experiment at LCLS is stored in the xtc streams. We process these in memory without needing to write images to disk at all, but provide image pickles if requested for use with the ''cctbx'' image viewer for diagnostic purposes. Integration pickles contain integrated intensities and crystal cell and orientation parameters and are read by the merging programs cxi.merge and prime.postrefine.<br />
<br />
In an effort to provide more human-readable data files and conform to international standards, we have worked with representatives of ImageCIF to create 64 tile segmented data in CBF format, the same format used in a wide variety of detectors. These CSPAD CBFs are designed to be used with other crystallographic software packages.<br />
<br />
Format specifications for these data files are provided below.<br />
<br />
== Image pickles ==<br />
<br />
''cctbx.xfel'' image pickles are a binary file format containing pixel data and image metadata. Under the hood these are serialized python dictionaries, but for general users it's sufficient to know that the blob of data consists of name/value pairs. The contents of an image pickle can be inspected with this command:<br />
<br />
'''cxi.print_pickle''' image.pickle<br />
<br />
For example, here's the output from an image with a thermolysin diffraction pattern collected recently:<br />
<br />
<div class="toccolours mw-collapsible mw-collapsed"><br />
<pre><br />
Printing contents of idx-20141201073601249.pickle<br />
Detector format version: CXI 10.1<br />
DISTANCE 119.002<br />
DETECTOR_ADDRESS CxiDs1-0|Cspad-0<br />
SIZE1 1765<br />
SIZE2 1765<br />
TIMESTAMP 2014-12-01T07:36Z01.249<br />
CCD_IMAGE_SATURATION 90000<br />
SATURATED_VALUE 90000<br />
64 active areas, first one: [715, 439, 909, 624]<br />
PIXEL_SIZE 0.11<br />
BEAM_CENTER_Y 97.075<br />
MIN_TRUSTED_VALUE -2000<br />
WAVELENGTH 1.75124427107 , converted to eV: 7079.77687909<br />
xtal_target None<br />
BEAM_CENTER_X 97.075<br />
SEQUENCE_NUMBER 0<br />
DATA len=3115225 max=106065.000000 min=-104526.000000 dimensions=(1765, 1765)<br />
cxi_versioned_extract()::cxi_version: CXI 10.1<br />
64 translated active areas, first one: [725, 449, 917, 632]<br />
</pre><br />
</div><br />
<br />
Each of the lines here is a name/value pair of data or metadata for the image. LABELIT, DIALS, cctbx.image_viewer, are all programs capable of processing these files directly. Here's a description of each of the name/value pairs:<br />
* Detector format version: CXI 10.1. Technically this value is not stored in the image pickle. The detector format version is used internally by cctbx.xfel for data collected up to LCLS run 11, and is a way of looking up tile corrections stored in the software. After run 11, this information should be stored in the phil file used to processes the data. The detector format version is looked up based on the detector address and event timestamp. All known detector format versions can be displayed with the command '''cxi.detector_format_versions'''.<br />
* DISTANCE: detector distance<br />
* DETECTOR_ADDRESS: string to identify the detector in the XTC stream. Recorded here as a unique identifier for the data when coupled with the event timestamp.<br />
* SIZE1 and SIZE2: image dimensions<br />
* TIMESTAMP: unique time stamp for this event in the form year-month-day'''T'''hour:minute'''Z'''second.microsecond<br />
* CCD_IMAGE_SATURATION and SATURATED_VALUE: overload value<br />
* PIXEL_SIZE: pixel size in mm<br />
* BEAM_CENTER_X, BEAM_CENTER_Y: beam center<br />
* MIN_TRUSTED_VALUE: underload<br />
* WAVELENGTH: wavelength as recorded in the XTC stream<br />
* DATA: pixel data. Some vital statistics are shown.<br />
* xtal_target and SEQUENCE_NUMBER: unused<br />
* 64 active areas: the active areas are coordinates that specify where the 64 tiles are in the image. They are in the form of quartets of numbers [x1,y1,x2,y2], one quartet for each of the 64 tiles. The first active area is shown. Note, for non-CSPAD image pickles, there will only be one active area and it will be the size of the image entire.<br />
* 64 translated active areas: if corrections were available for this image, the first corrected active area is shown here.<br />
<br />
=== Creating image pickles from xtc streams ===<br />
<br />
The program cctbx.xfel.xtc_dump can be used to convert xtc streams to image pickle format. Here we provide an example for the Rayonix detector that is often found at XPP and MFX.<br />
<ol><br />
<li>Set up a release directory as described in [[Setup]]<br />
<li>The psana module mod_image_dict is used to read the data and generate the images. An example: [[mod_image_dict.cfg]]<br />
<li>cd /path/to/test/release<br />
<li>sit_setup<br />
<li>cctbx.xfel.xtc_dump experiment=mfxm9316 run_num=14 address="MfxEndstation-0|Rayonix-0" file_format=pickle cfg=mod_image_dict.cfg <br />
</ol><br />
This will fill your working folder with image pickles. You can change your output directory with output.output_dir and you can limit the number of images saved with dispatch.max_events.<br />
<br />
This approach will also work for CSPAD images. Here, you must add calib_dir to your config file, as described in the indexing tutorials.<br />
<br />
== Integration pickles ==<br />
<br />
Integration pickles are the output of cctbx.xfel indexing and integration. They are like image pickles in that they are serialized python dictionaries and can be inspected with '''cxi.print_pickle'''. They are the primary input to '''cxi.merge''' and '''prime.postrefine'''.<br />
<br />
== CSPAD CBFs ==<br />
<br />
CSPAD CBFs are not used directly by ''cctbx.xfel'' at this time. This aspect of the project is under development. The complete specification for CSPAD CBFs is laid out in an article in the Computational Crystallographic Newsletter: "XFEL Detectors and ImageCIF", Computational Crystallography Newsletter 5, 19-24. ([http://cci.lbl.gov/publications/download/CCN_2014_p19.pdf Reprint]). CSPAD CBFs<br />
<br />
=== Creating CSPAD CBFs from XTC streams ===<br />
<br />
cctbx.xfel.xtc_dump is useful for converting XTC streams to CBF files as needed. Use -c to get a listing of all options, and -c -a 2 to get a full listing with help strings. Example command:<br />
<br />
cd ~/myrelease # or wherever your release is<br />
sit_setup<br />
cctbx.xfel.xtc_dump dispatch.max_events=1 input.experiment=xpptut15 input.address=cspad \<br />
input.run_num=54 format.file_format=cbf output.output_dir=xpptut15/out \<br />
format.cbf.detz_offset=100 input.override_energy=7000<br />
<br />
Here, one image is created from experiment xpptut15, run 54. You can display the image with cctbx.image_viewer xpptut15/out/*.cbf. Note the pink gaps between the tiles. This results from the segmented nature of the CSPAD, preserved in the CBF file.<br />
<br />
xpptut15 is data from XPP's CSPAD and was collected without xray's on. Hence format.cbf.detz_offset=100 input.override_energy=7000 are set to fake values in this command.<br />
<br />
=== Converting from SLAC's metrology to CBF ===<br />
<br />
The tile positions of the hierarchical CSPAD detector are specified by SLAC in a geometry file in the calib folder of each experiment. Typically this file is named something like 0-end.data. If desired, this file can be converted to just the human readable header portion of a CSPAD cbf using the ''cctbx.xfel'' command cxi.slaccalib2cbfheader. Example:<br />
<br />
cxi.slaccalib2cbfheader metrology_file=0-end.data out=tmp.cbf<br />
<br />
This cbf header can be converted back to SLAC format using cxi.cbfheader2slaccalib:<br />
<br />
cxi.cbfheader2slaccalib cbf_header=tmp.cbf out_metrology_file=tmp.data<br />
<br />
=== Displaying metrology files ===<br />
<br />
cxi.display_metrology <filename> can be used to show a plot of tile positions. It accepts SLAC metrology files, CSPAD CBFs and image pickles.</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/Data_viewerData viewer2016-10-19T16:16:10Z<p>Aaron: </p>
<hr />
<div>The command "cxi.pyana" can be configured to display the diffraction images streamed from an XTC file.<br />
<br />
== Configuring the viewer ==<br />
Add to the mod_view section of your pyana config file as follows:<br />
<pre><br />
[pyana]<br />
modules = my_ana_pkg.mod_view<br />
num-cpu = 1<br />
<br />
[my_ana_pkg.mod_view]<br />
address = CxiDs1-0|Cspad-0<br />
calib_dir = /reg/g/cctbx/sources/cctbx_project/xfel/metrology/CSPad/run4/CxiDs1.0_Cspad.0<br />
dark_path = /reg/d/psdm/cxi/cxib9713/res/darks/r0001-Ds1-avg.pickle<br />
dark_stddev = /reg/d/psdm/cxi/cxib9713/res/darks/r0001-Ds1-stddev.pickle<br />
detz_offset = 555<br />
<br />
# Suitable values depend on the repetition rate of the instrument.<br />
n_update = 120<br />
n_collate = 60<br />
</pre><br />
<br />
The command is run as follows:<br />
<pre><br />
cxi.pyana -c myconfig_file.cfg /path/to/xtc/streams/e346-r0007-s0[0-4]-c00.xtc<br />
</pre><br />
The command will marshal together all events from streams 0-4 in run 7 of experiment 346. Those events with the CXI imaging detector with the given <code>address</code> will be sent to the <code>mod_view</code> module that displays images in a GUI. Dark subtraction is done with the files <code>dark_path</code> and <code>dark_stddev</code> that are described on the [[Preparatory steps]] wiki page.<br />
Important parameters are:<br />
* <code>n_update</code>: GUI will only display every Nth event (every 120th image, in this example).<br />
* <code>n_collate</code>: GUI will display an average image over the last N events (last 60 events, in this example).<br />
'''Critical step:'''<br />
'''Please press the Run/Pause (green) button in the GUI to make the action start.'''</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/Cxi02416_calibrationCxi02416 calibration2016-05-18T23:55:15Z<p>Aaron: /* Refine metrology (v3) */</p>
<hr />
<div><br />
== Calibration of cspad using cxi02416, ''cctbx.xfel'' and DIALS ==<br />
<br />
This page is not intended as a manual for processing XFEL data using cctbx.xfel and DIALS. Rather, this is a documentation of steps taken to calibrate the CSPAD using data collected at CXI in February 2016, on detector Ds2. <br />
<br />
Program names are highlighted in '''bold'''. Generally, help can be obtained using -h or -c options to the program at the command line.<br />
<br />
In this walkthrough, we will be calibrating the tile positions of the CSPAD detector at the CXI endstation at LCLS. We will be using a calibration dataset collected by CXI staff in February 2016, during calibration experiment cxi02614. The procedure is to index the images, refine the tile positions minimizing the difference between observed and calculated Bragg reflections, re-index the images, re-refine the tile positions, and repeat until convergence. At each step of the way, we will label our current metrology with a version number, starting at version 0:<br />
<br />
{| class="wikitable"<br />
|-<br />
| '''Metrology version'''<br />
| '''Description'''<br />
|-<br />
|Version 0 (v0)<br />
|Initial metrology deployed by beamline operators. The tile positions are measured using an optical microscope, but as the quadrants can move independently, they are not correctly aligned in relation to each other or to the beam center.<br />
|-<br />
|Version 1 (v1)<br />
|After collecting some data, powder rings can be seen after averaging the events in a run. Several tools are available for aligning the quadrants by eye or automatically using powder rings.<br />
|-<br />
|Version 2 (v2)<br />
|After indexing the images using v1, we will refine the tile positions to produce metrology v2.<br />
|-<br />
|Version 3 (v3)<br />
|After re-indexing the images using v2, we will re-refine the tile positions to produce metrology v3.<br />
|-<br />
|...<br />
|And so forth until convergence<br />
|}<br />
<br />
We start with this known information provided by the beamline operators:<br />
<br />
* Data is contained in runs 2-16<br />
* Detector address: CxiDs2.0:Cspad.0. This string identifies the front CSPAD detector in the XTC streams.<br />
* detz_offset (IE the distance from the sample position to the back of the detector rail): 568 mm.<br />
<br />
== Dark pedestal, common mode correction, and untrusted pixels ==<br />
<br />
Subtracting the dark pedestal, applying common mode correction and masking out untrusted pixels is critical for spot picking and integration. ''cctbx.xfel'' uses masks and pedestals deployed into the experiment’s calibration store using the LCLS program calibman [https://confluence.slac.stanford.edu/display/PSDM/Calibration+Management+Tool] . Dark pedestal correction, hot pixel determination, and user defined mask generation for regions of interest or for masking out shadows is all available through that program.<br />
<br />
Common mode correction accounts for flat, per-panel fluctuations that occur non-uniformly throughout a single exposure. LCLS has several algorithms available [https://confluence.slac.stanford.edu/display/PSDM/Common+mode+correction+algorithms]. We recommend the non-bonded pixels algorithm; the parameter files below specifies this algorithm during indexing and integration.<br />
<br />
== Aligning quadrant positions (v1) ==<br />
<br />
Alignment of quadrant positions using powder rings can be done manually using ''cctbx.xfel'' or LCLS's calibman tool [https://confluence.slac.stanford.edu/display/PSDM/Calibration+Management+Tool]. ''ctbx.xfel'' also provides an automatic algorithm for quadrant alignment, provided powder rings of sufficient quality. For cxi02416, the beamline operators already calibrated quadrant positions to a point where indexing can proceed. Regardless, if needed, here is how to accomplish the same task using ''cctbx.xfel''<br />
<br />
=== Averaging diffraction data to create powder patterns ===<br />
<br />
The ''cctbx.xfel'' program '''cxi.mpi_average''' is used to create averages:<br />
<br />
for i in `seq 2 16`; do bsub -n 12 -q psanaq -o avg_r$i.log \<br />
'''cxi.mpi_average''' -x cxi02416 -r $i -a CxiDs2.0:Cspad.0 -d 568 -o . -v; done<br />
<br />
For each of the runs with data (2-16), this command submits an averaging job using 12 processors, providing information about the experiment such as detector address and detz_offset in the form of command line arguments.<br />
<br />
Progress can be monitored with the command bjobs.<br />
<br />
When the averages are complete, they will consist of files named cxi02416_avg-r0002.cbf, cxi02416_stddev-r0002.cbf, and cxi02416_max-r0002.cbf for each run, representing the average, standard deviation, and maximum of all the pixel data in each of the runs. Display the data using '''cctbx.image_viewer''':<br />
<br />
'''cctbx.image_viewer''' *.cbf<br />
<br />
=== Manual calibration using ''cctbx.xfel'' ===<br />
<br />
Typically the best powder rings come from the maximum projection (example: cxi02416_max-r0013.cbf). To manually align the quadrant positions, use '''cctbx.image_viewer''' cxi02416_max-r0013.cbf. Under actions, click on 'Show quadrant calibration', then use the spinners to align the powder rings. You may find the ring tool or the unit cell tool, also under the Actions menu, to be useful visual aids during this process. When done, click 'Save current metrology' to save the changes to a .def file, which is a cbf header.<br />
<br />
This walkthrough uses pre-aligned quadrant locations. However, if it is desired to use the metrology from manual quadrant re-alignment for indexing, it first needs to be converted to SLAC's metrology file format. Use this command:<br />
<br />
'''cxi.cbfheader2slaccalib''' cbf_header=quadrants.cbf<br />
<br />
This command will create a 0-end.data file. See the instructions under [[Cxi0216 calibration#Index using v2 metrology | indexing using v2 metrology]] for deploying it for use.<br />
<br />
=== Automatic calibration using ''cctbx.xfel'' ===<br />
<br />
If a quadrant is properly placed, the pixel values for a strong powder pattern will be highly correlated after rotating it 45 degrees around the beam center. '''cspad.quadrants_cbf''' performs a grid search of XY offsets for each quadrant, searching for the position with the highest rotational autocorrelation. It then writes out a new cbf file with the adjusted header:<br />
<br />
'''cspad.quadrants_cbf''' cxi02416_max-r0013.cbf<br />
<br />
Specify the '-p' parameter to enable plots of the grid search results for each quadrant. Here is the output:<br />
<br />
<div class="toccolours mw-collapsible mw-collapsed"><br />
<pre><br />
Doing cross-correlation on panel ARRAY_D0Q0S1A0<br />
Searching a grid with dimensions (41, 41)<br />
max cc 0.1592 is at (0, 0)<br />
Doing cross-correlation on panel ARRAY_D0Q1S1A0<br />
Searching a grid with dimensions (41, 41)<br />
max cc 0.1209 is at (0, 5)<br />
Doing cross-correlation on panel ARRAY_D0Q2S1A0<br />
Searching a grid with dimensions (41, 41)<br />
max cc 0.1686 is at (0, -2)<br />
Doing cross-correlation on panel ARRAY_D0Q3S1A0<br />
Searching a grid with dimensions (41, 41)<br />
max cc 0.1144 is at (1, 2)<br />
</pre><br />
</div><br />
<br />
The CC values are all less that 0.2 which indicates that even though run 13 shows the strongest powder diffraction, the rings are not contiguous or strong enough compared to the background to get a good rotational autocorrelation. Use the image viewer to verify the new quadrant positions are not ideal:<br />
<br />
'''cctbx.image_viewer''' cxi02416_max-r0013_cc.cbf<br />
<br />
It is possible that using a maximum projection of all the runs would make the rings more contiguous and brighter, leading to higher CC values. This can be done quickly using the maximum projections already made:<br />
<br />
'''cxi.cspad_average''' *_max*.cbf -m all_max.cbf<br />
<br />
Use '''cctbx.image_viewer''' to compare all_max.cbf to cxi02416_max-r0013.cbf. The rings are noticeably better. Now, do the grid search:<br />
<br />
'''cspad.quadrants_cbf''' all_max.cbf<br />
<br />
Results:<br />
<div class="toccolours mw-collapsible mw-collapsed"><br />
<pre><br />
Doing cross-correlation on panel ARRAY_D0Q0S1A0<br />
Searching a grid with dimensions (41, 41)<br />
max cc 0.2568 is at (-1, 0)<br />
Doing cross-correlation on panel ARRAY_D0Q1S1A0<br />
Searching a grid with dimensions (41, 41)<br />
max cc 0.2022 is at (0, 4)<br />
Doing cross-correlation on panel ARRAY_D0Q2S1A0<br />
Searching a grid with dimensions (41, 41)<br />
max cc 0.2328 is at (-4, 0)<br />
Doing cross-correlation on panel ARRAY_D0Q3S1A0<br />
Searching a grid with dimensions (41, 41)<br />
max cc 0.1853 is at (0, 2)<br />
</pre><br />
</div><br />
<br />
The CC values are higher, and likely high enough for at least two of the quadrants to get some initial indexing results. Regardless, we recommend silver behenate powder for this automatic procedure as it gives very smooth, contiguous rings.<br />
<br />
Again, for this walkthrough we use quadrant positions aligned by the beamline operator. However, if after automatic alignment it is desired to use the results for indexing, they first need to be converted to SLAC's metrology file format. Use this command:<br />
<br />
'''cxi.cbfheader2slaccalib''' cbf_header=all_max_cc.cbf<br />
<br />
This command will create a 0-end.data file. See the instructions under [[Cxi0216 calibration#Index using v2 metrology | indexing using v2 metrology]] for deploying it for use.<br />
<br />
== Initial indexing ==<br />
<br />
The initial metrology deployed by the beamline operator is sufficient to get initial indexing results. Indexing in ''cctbx.xfel'' typically is done in a series of trials. Our first trial will be trial 0, using metrology v1 (initial metrology from beamline operators, with quadrants corrected by eye by the beamline operators).<br />
<br />
With this information, and [[cxi02416-lyso-t000.phil | this phil file]], we can index the data:<br />
<br />
for i in `seq 2 16`; do '''cxi.mpi_submit''' input.experiment=cxi02416 \<br />
output.output_dir=/reg/d/psdm/cxi/cxi02416/ftc/brewster/dials \<br />
mp.nproc=36 mp.queue=psanaq output.split_logs=True \<br />
input.dispatcher=cctbx.xfel.xtc_process \<br />
input.target=[[cxi02416-lyso-t000.phil]] input.trial=0 \<br />
input.run_num=$i; done<br />
<br />
This command submits jobs for runs 2 through 16, using the DIALS backend of ''cctbx.xfel''. To save time during initial indexing and metrology refinement, we use dispatch.integrate=False to skip the integration step. After indexing is completed, we got 4986 indexed images, as can be shown by this command:<br />
<br />
cd /reg/d/psdm/cxi/cxi02416/ftc/brewster/dials<br />
ls r0*/000/out/*.json | wc -l<br />
<br />
== Refine metrology (v2) ==<br />
<br />
Let's call the metrology deployed by the beamline operator version 0 (v0). After quadrant alignment, the operators updated the metrology to version 1 (v1). The following command will do an iterative joint hierarchical refinement of the components of the CSPAD detector. The new tile positions we call version 2 (v2).<br />
<br />
bsub -q psanaq -o t000_1k.out '''cspad.cbf_metrology''' tag=t000_1k \<br />
reflections=indexed ../r0*/000/out refine_distance=True \<br />
n_subset=1000 split_dataset=True<br />
<br />
The program first aggregates the requested number of images into a single dataset. Then, it refines the detector as a whole (including Z position and tilt). Using the new detector position, it refines the quadrants independently from each other, and then the 2x1 sensors, and then the individual panels. Finally, it converts the DIALS format metrology into the SLAC file format (0-end.data). The refinement is a 'joint refinement' because the information from many crystals is used to refine a single detector model.<br />
<br />
Details about the parameters used:<br />
<br />
* tag=t000_1k: the output files will be named t000_1k*<br />
* reflections=indexed: the indexing process produces two sets of reflection files, bright indexed reflections, and final integrated reflections (can include weak intensities). Here, I've chosen to refine only against the bright, indexed reflections.<br />
* refine_distance=True: as the detector distance is not fully known yet, allow the detector distance to refine. In subsequent rounds of metrology refinement, we will fix the distance to a constant value.<br />
* n_subset=1000: pick 1000 images at random to refine<br />
* split_dataset=True: the refinement is done twice independently, using odd numbered or even numbered images, each time using n_subset images. This will be useful later for evaluating the accuracy of the metrology.<br />
<br />
== Visualize tile shifts ==<br />
<br />
In order to get a sense of the magnitude of the shifts in panel position after refinement, use the program cxi.display_metrology. For example, the original detector geometry can be displayed thusly:<br />
'''cxi.display_metrology''' \<br />
/reg/d/psdm/cxi/cxi02416/calib/CsPad::CalibV1/CxiDs2.0:Cspad.0/geometry/0-end.data<br />
<br />
Compare it to the refined geometry:<br />
'''cxi.display_metrology''' 0-end.data.t000_1k_1<br />
<br />
You will see a small change in the origin (center arrow) and obvious changes in the quadrant positions. The relative positions of the sensors to each other will not change a large amount, though if you inspect the files themselves you will see changes.<br />
<br />
Another tool that is useful for evaluating changes in metrology is '''dev.dials.plot_detector_shifts'''. This program will plot detector shifts in the X and Y directions (also known as the fast and slow directions, a convention referring to how data is read from the byte arrays stored on disk). It will also plot shifts along the Z axis, I.E., the detector's normal axis. Let's use it to compare the unrefined geometry to the geometry refined at level 2 (I.E. at the level of the 2x1 sensors):<br />
<br />
'''dev.dials.plot_detector_shifts''' t000_1k_1_combined_experiments.json \<br />
t000_1k_1_refined_experiments_level2.json \<br />
plot_type=spherical_polar tag=v1v2level2<br />
display v1v2level2*.png<br />
<br />
Use the spacebar to cycle between the images. The pixels are mapped to the Ewald sphere and then displayed as a function of azimuthal and elevation angles along the Ewald sphere, which distorts the detector appearance but is independent of any detector layout. The colorbar scales will show the magnitude of panel shifts.<br />
<br />
Try running the program using t000_1k_1_refined_experiments_level0.json to see how much the detector as a whole shifted, or t000_1k_1_refined_experiments_level1.json to see shifts in the quadrants.<br />
<br />
Finally, the program '''cspad.detector_shifts''' shows the magnitude of shifts between a reference and a moving detector in tabular form:<br />
<br />
'''cspad.detector_shifts''' \<br />
t000_1k_1_combined_experiments.json t000_1k_1_combined_reflections.pickle \<br />
t000_1k_1_refined_experiments_level2.json t000_1k_1_refined_reflections_level2.pickle<br />
<br />
Looking at the Delta XY columns, we see the detector moved 4.6 microns in the XY plane, the quadrants moved on average 219.8+/-57.9 microns and the 2x1 sensors moved 17.6+/-16.1 microns. Note how most of the change is in the quadrant locations, which is expected as these were done by eye. The 2x1 sensor positions, determined using optical microscopy, changed by an order of magnitude less than the quadrants. Also, the same program ran against the other half of the split dataset (t000_1k_2_*) will reveal very similar shifts, indicating these shifts are not due to random chance.<br />
<br />
Full output for '''cspad.detector_shifts''':<br />
<br />
<div class="toccolours mw-collapsible mw-collapsed"><br />
<pre><br />
Found 4 hierarchy levels<br />
Hierarchy level 0 Detector shifts<br />
-----------------------------------------------------------------------------------------------------------<br />
PanelG BC dist Delta XY R Offsets T Offsets Z Offsets dR Norm dT Norm Local dNorm Rot Z N Refls<br />
ID (mm) (microns) (microns) (microns) (microns) (deg) (deg) (deg) (deg)<br />
-----------------------------------------------------------------------------------------------------------<br />
0 2.4 4.6 3.5 -2.9 -74.0 -0.0003 -0.0213 0.0213 0.0003 29527<br />
Weighted mean 4.6 3.5 -2.9 -74.0 -0.0003 -0.0213 0.0213 0.0003<br />
Weighted stddev 0.0 0.0 0.0 0.0 0.0000 0.0000 0.0000 0.0000<br />
-----------------------------------------------------------------------------------------------------------<br />
Hierarchy level 1 Detector shifts<br />
-----------------------------------------------------------------------------------------------------------<br />
PanelG BC dist Delta XY R Offsets T Offsets Z Offsets dR Norm dT Norm Local dNorm Rot Z N Refls<br />
ID (mm) (microns) (microns) (microns) (microns) (deg) (deg) (deg) (deg)<br />
-----------------------------------------------------------------------------------------------------------<br />
0 62.9 157.7 -153.5 -36.0 1236.4 -0.0308 -0.0658 0.0550 0.0111 7766<br />
1 64.3 262.5 -172.9 -197.5 1532.9 -0.0042 -0.1422 0.1411 0.0260 7112<br />
3 66.0 190.2 22.6 188.8 1347.7 -0.0945 -0.0391 0.1228 0.1139 7811<br />
2 67.0 279.6 277.7 32.1 1374.9 0.0216 0.0397 0.0305 0.0169 6838<br />
Weighted mean 219.8 -11.7 0.3 1369.3 -0.0291 -0.0527 0.0880 0.0432<br />
Weighted stddev 57.9 204.3 161.0 122.4 0.0500 0.0731 0.0524 0.0494<br />
-----------------------------------------------------------------------------------------------------------<br />
Hierarchy level 2 Detector shifts<br />
-----------------------------------------------------------------------------------------------------------<br />
PanelG BC dist Delta XY R Offsets T Offsets Z Offsets dR Norm dT Norm Local dNorm Rot Z N Refls<br />
ID (mm) (microns) (microns) (microns) (microns) (deg) (deg) (deg) (deg)<br />
-----------------------------------------------------------------------------------------------------------<br />
1 21.9 2.7 0.1 -2.7 67.0 -0.2345 -0.0794 0.1925 0.0227 1817<br />
9 23.6 2.5 -0.3 2.4 -34.8 0.0229 0.1911 0.2632 0.0290 1578<br />
25 24.7 7.9 -2.6 7.4 34.8 -0.0817 -0.1288 0.2195 0.1031 1644<br />
17 26.3 5.8 -3.8 -4.4 -72.0 0.0402 0.0412 0.0588 0.0107 1357<br />
0 40.8 23.0 -23.0 -0.3 44.5 0.0064 -0.3016 0.3031 0.0169 1626<br />
8 41.3 5.8 -4.7 3.6 -44.8 0.2528 -0.0850 0.3773 0.0512 1687<br />
24 43.9 10.2 -0.4 -10.3 36.9 -0.1491 -0.2116 0.2813 0.0628 1665<br />
16 44.5 17.8 17.1 4.9 -13.0 0.0640 0.1521 0.1525 0.0178 1499<br />
7 55.5 12.6 0.4 -12.6 52.8 -0.1670 0.0284 0.1042 0.0097 1382<br />
31 57.7 28.9 -7.5 -27.9 81.9 -0.0803 0.0442 0.0742 0.1336 1406<br />
15 58.0 13.5 -6.2 12.0 -52.4 0.0155 0.1185 0.1853 0.0004 1323<br />
23 60.0 16.8 -16.1 4.8 -17.2 -0.0080 -0.0531 0.0518 0.0357 1560<br />
11 72.2 4.4 -1.6 -4.2 8.5 -0.5439 -0.0891 0.4503 0.0258 843<br />
3 73.1 22.5 17.6 -13.9 22.6 -0.2173 0.0156 0.1600 0.0165 856<br />
19 75.3 2.1 -0.5 2.0 -17.4 0.0061 0.1337 0.1232 0.0245 584<br />
6 76.5 11.2 -4.4 -10.3 14.9 0.0371 -0.1290 0.1975 0.0252 730<br />
27 76.5 59.0 6.4 -58.6 93.7 -0.0486 0.0859 0.0744 0.0859 851<br />
2 77.7 45.1 1.6 -45.2 71.3 -0.1845 0.0036 0.1143 0.0259 772<br />
10 78.3 19.5 -3.5 -19.1 24.1 -0.4560 -0.0550 0.3362 0.0117 651<br />
30 78.6 17.4 -3.3 17.2 -53.8 0.0098 -0.1055 0.2080 0.0894 651<br />
14 78.9 4.1 -4.1 -0.5 14.5 0.0433 0.1315 0.2032 0.0036 504<br />
22 80.6 22.0 21.9 2.1 -17.6 0.0394 0.0027 0.0347 0.0234 696<br />
18 81.2 31.8 9.1 30.5 -80.5 0.0141 -0.0178 0.0374 0.0487 620<br />
26 81.3 25.6 -8.1 -24.2 68.4 -0.0185 0.0909 0.0750 0.1275 819<br />
4 87.6 32.3 31.6 -5.7 53.6 -0.2149 0.0391 0.1460 0.0154 431<br />
12 89.8 25.3 6.9 -24.2 -21.1 -0.0419 0.0008 0.1011 0.0173 406<br />
28 90.7 37.7 36.9 3.7 79.9 -0.3678 0.0389 0.3173 0.0006 529<br />
20 92.1 24.0 2.7 23.9 16.6 0.0213 -0.1148 0.1014 0.0777 374<br />
5 104.9 92.9 -91.3 6.3 -133.6 -0.6185 0.2555 0.6003 0.0674 152<br />
13 106.3 9.9 9.9 0.4 4.2 -0.1141 -0.0277 0.0259 0.0031 120<br />
29 108.0 101.5 101.1 -1.7 122.2 -0.3588 0.1173 0.2946 0.0482 246<br />
21 108.7 16.4 -13.1 9.9 -17.3 0.1983 0.0814 0.1786 0.0392 148<br />
Weighted mean 17.6 -0.1 -4.7 12.6 -0.0687 -0.0164 0.1883 0.0415<br />
Weighted stddev 16.1 16.5 16.8 51.4 0.1702 0.1286 0.1140 0.0381<br />
-----------------------------------------------------------------------------------------------------------<br />
Hierarchy level 3 Detector shifts<br />
-----------------------------------------------------------------------------------------------------------<br />
PanelG BC dist Delta XY R Offsets T Offsets Z Offsets dR Norm dT Norm Local dNorm Rot Z N Refls<br />
ID (mm) (microns) (microns) (microns) (microns) (deg) (deg) (deg) (deg)<br />
-----------------------------------------------------------------------------------------------------------<br />
2 15.9 0.0 0.0 -0.0 0.0 -0.0669 -0.2384 0.0000 0.0227 941<br />
18 16.2 0.0 -0.0 0.0 0.0 -0.1455 0.1261 0.0000 0.0290 885<br />
50 19.2 0.0 -0.0 0.0 0.0 0.0606 -0.1400 0.0000 0.1031 865<br />
34 19.3 0.0 -0.0 -0.0 0.0 -0.0121 0.0563 0.0000 0.0107 578<br />
3 30.7 0.0 0.0 0.0 0.0 -0.2095 -0.1320 0.0000 0.0227 876<br />
19 33.0 0.0 0.0 0.0 0.0 -0.0226 0.1912 0.0000 0.0290 693<br />
51 33.0 0.0 0.0 0.0 0.0 -0.0494 -0.1443 0.0000 0.1031 779<br />
35 35.2 0.0 0.0 0.0 0.0 0.0293 0.0496 0.0000 0.0107 779<br />
16 37.6 0.0 -0.0 0.0 0.0 0.2624 0.0480 0.0000 0.0512 866<br />
0 37.9 0.0 0.0 -0.0 0.0 0.1518 -0.2606 0.0000 0.0169 775<br />
32 40.8 0.0 0.0 -0.0 0.0 -0.0180 0.1640 0.0000 0.0178 583<br />
48 41.1 0.0 -0.0 0.0 0.0 -0.0272 -0.2575 0.0000 0.0628 863<br />
1 46.2 0.0 -0.0 0.0 0.0 0.0304 -0.3001 0.0000 0.0169 851<br />
17 47.3 0.0 0.0 0.0 0.0 0.2587 -0.0648 0.0000 0.0512 821<br />
49 49.1 0.0 0.0 0.0 0.0 -0.1319 -0.2228 0.0000 0.0628 802<br />
33 50.4 0.0 -0.0 -0.0 0.0 0.0517 0.1567 0.0000 0.0178 916<br />
15 51.3 0.0 -0.0 0.0 0.0 -0.1644 0.0407 0.0000 0.0097 740<br />
63 52.9 0.0 -0.0 0.0 0.0 -0.0768 0.0501 0.0000 0.1336 732<br />
31 54.2 0.0 0.0 0.0 0.0 0.0243 0.1170 0.0000 0.0004 667<br />
47 55.7 0.0 0.0 -0.0 0.0 -0.0119 -0.0523 0.0000 0.0357 885<br />
14 61.4 0.0 -0.0 0.0 0.0 -0.1694 -0.0022 0.0000 0.0097 642<br />
23 61.5 0.0 -0.0 -0.0 0.0 -0.5509 -0.0155 0.0000 0.0258 690<br />
7 62.3 0.0 -0.0 0.0 0.0 -0.2132 0.0445 0.0000 0.0165 626<br />
30 63.5 0.0 -0.0 0.0 0.0 -0.0061 0.1193 0.0000 0.0004 656<br />
62 64.0 0.0 -0.0 -0.0 0.0 -0.0869 0.0291 0.0000 0.1336 674<br />
39 64.6 0.0 -0.0 -0.0 0.0 0.0240 0.1317 0.0000 0.0245 435<br />
55 65.7 0.0 -0.0 0.0 0.0 -0.0367 0.0916 0.0000 0.0859 631<br />
46 65.8 0.0 0.0 -0.0 0.0 0.0017 -0.0537 0.0000 0.0357 675<br />
5 67.7 0.0 -0.0 0.0 0.0 -0.1829 0.0246 0.0000 0.0259 591<br />
21 68.5 0.0 -0.0 0.0 0.0 -0.4593 -0.0032 0.0000 0.0117 519<br />
53 71.2 0.0 -0.0 0.0 0.0 -0.0081 0.0924 0.0000 0.1275 605<br />
37 71.4 0.0 0.0 -0.0 0.0 0.0120 -0.0193 0.0000 0.0487 447<br />
13 73.5 0.0 0.0 -0.0 0.0 0.0317 -0.1305 0.0000 0.0252 447<br />
61 75.2 0.0 0.0 0.0 0.0 0.0054 -0.1058 0.0000 0.0894 360<br />
29 76.1 0.0 -0.0 0.0 0.0 0.0488 0.1296 0.0000 0.0036 264<br />
45 77.5 0.0 0.0 0.0 0.0 0.0395 0.0011 0.0000 0.0234 440<br />
9 80.3 0.0 0.0 0.0 0.0 -0.2107 0.0576 0.0000 0.0154 296<br />
12 80.8 0.0 0.0 0.0 0.0 0.0586 -0.1208 0.0000 0.0252 283<br />
25 82.2 0.0 0.0 -0.0 0.0 -0.0416 0.0044 0.0000 0.0173 270<br />
28 82.9 0.0 -0.0 -0.0 0.0 0.0201 0.1370 0.0000 0.0036 240<br />
22 83.0 0.0 0.0 0.0 0.0 -0.5494 -0.0436 0.0000 0.0258 153<br />
60 83.3 0.0 0.0 0.0 0.0 0.0276 -0.1023 0.0000 0.0894 291<br />
57 83.5 0.0 0.0 -0.0 0.0 -0.3631 0.0704 0.0000 0.0006 365<br />
6 83.9 0.0 -0.0 -0.0 0.0 -0.2152 0.0340 0.0000 0.0165 230<br />
41 84.6 0.0 0.0 0.0 0.0 0.0112 -0.1162 0.0000 0.0777 259<br />
44 85.0 0.0 -0.0 -0.0 0.0 0.0383 0.0095 0.0000 0.0234 256<br />
38 86.1 0.0 -0.0 -0.0 0.0 0.0174 0.1327 0.0000 0.0245 149<br />
54 87.3 0.0 -0.0 -0.0 0.0 -0.0411 0.0897 0.0000 0.0859 220<br />
4 88.0 0.0 -0.0 -0.0 0.0 -0.1844 0.0064 0.0000 0.0259 181<br />
20 88.3 0.0 -0.0 -0.0 0.0 -0.4567 -0.0489 0.0000 0.0117 132<br />
36 91.3 0.0 0.0 0.0 0.0 0.0139 -0.0180 0.0000 0.0487 173<br />
52 91.5 0.0 -0.0 -0.0 0.0 -0.0171 0.0912 0.0000 0.1275 214<br />
8 95.7 0.0 0.0 0.0 0.0 -0.2010 0.0856 0.0000 0.0154 135<br />
24 98.1 0.0 -0.0 0.0 0.0 -0.0407 0.0099 0.0000 0.0173 136<br />
56 98.5 0.0 0.0 0.0 0.0 -0.3501 0.1192 0.0000 0.0006 164<br />
11 98.8 0.0 -0.0 0.0 0.0 -0.6019 0.2924 0.0000 0.0674 105<br />
27 100.0 0.0 0.0 -0.0 0.0 -0.1156 -0.0207 0.0000 0.0031 77<br />
40 100.2 0.0 -0.0 0.0 0.0 -0.0046 -0.1166 0.0000 0.0777 115<br />
59 102.0 0.0 -0.0 -0.0 0.0 -0.3510 0.1389 0.0000 0.0482 164<br />
43 102.4 0.0 0.0 -0.0 0.0 0.2029 0.0691 0.0000 0.0392 98<br />
10 111.6 0.0 0.0 0.0 0.0 -0.5535 0.3760 0.0000 0.0674 47<br />
26 113.4 0.0 -0.0 0.0 0.0 -0.1173 -0.0040 0.0000 0.0031 43<br />
58 114.7 0.0 0.0 -0.0 0.0 -0.3274 0.1879 0.0000 0.0482 82<br />
42 115.7 0.0 0.0 -0.0 0.0 0.2108 0.0392 0.0000 0.0392 50<br />
Weighted mean 0.0 -0.0 0.0 0.0 -0.0571 -0.0152 0.0000 0.0415<br />
Weighted stddev 0.0 0.0 0.0 0.0 0.1674 0.1345 0.0000 0.0377<br />
-----------------------------------------------------------------------------------------------------------<br />
Detector shifts summary<br />
---------------------------------------------------------------------------------------------------------------------------------------------------------------<br />
Hierarchy Delta XY Delta XY R Offsets R Offsets T Offsets T Offsets Z Offsets Z Offsets dR Norm dR Norm dT Norm dT Norm Local dNorm Local dNorm Rot Z Rot Z<br />
Level Sigma Sigma Sigma Sigma Sigma Sigma Sigma Sigma<br />
(microns) (microns) (microns) (microns) (microns) (microns) (microns) (microns) (deg) (deg) (deg) (deg) (deg) (deg) (deg) (deg)<br />
---------------------------------------------------------------------------------------------------------------------------------------------------------------<br />
0 4.6 0.0 3.5 0.0 -2.9 0.0 -74.0 0.0 -0.0003 0.0000 -0.0213 0.0000 0.0213 0.0000 0.0003 0.0000<br />
1 219.8 57.9 -11.7 204.3 0.3 161.0 1369.3 122.4 -0.0291 0.0500 -0.0527 0.0731 0.0880 0.0524 0.0432 0.0494<br />
2 17.6 16.1 -0.1 16.5 -4.7 16.8 12.6 51.4 -0.0687 0.1702 -0.0164 0.1286 0.1883 0.1140 0.0415 0.0381<br />
3 0.0 0.0 -0.0 0.0 0.0 0.0 0.0 0.0 -0.0571 0.1674 -0.0152 0.1345 0.0000 0.0000 0.0415 0.0377<br />
---------------------------------------------------------------------------------------------------------------------------------------------------------------<br />
<br />
<br />
For each hierarchy level, the average shifts in are computed among objects at that level, weighted by the number of reflections recorded on each object. For example, for a four quadrant detector, the average Z shift will be the average of the four quadrant Z values, each weighted by the number of reflections on that quadrant.<br />
<br />
-------------------<br />
Column descriptions<br />
-------------------<br />
<br />
Individual hierarchy level tables only:<br />
PanelG id: ID of the panel group.<br />
BC dist: distance of the panel group from the beam center.<br />
N Refls: number of reflections on this panel group<br />
<br />
All tables:<br />
Delta XY: magnitude of the shift in the local XY frame.<br />
R, T offsets: shifts relative to the parent object's location in the radial and transverse directions (relative to the detector center).<br />
Z offsets: relative shifts in the local frame in the local Z direction.<br />
R, T Norm: angle between normal vectors in lab space, projected onto the radial or transverse plane.<br />
Local dNorm: local relative angle between normal vectors.<br />
Rot Z: rotation around detector normal in lab space<br />
</pre><br />
</div><br />
<br />
== Index using v2 metrology ==<br />
<br />
In order to improve our metrology we will re-index using the new tile positions. We assume we don't have write access to the geometry file for this detector, namely /reg/d/psdm/cxi/cxi02416/calib/CsPad::CalibV1/CxiDs2.0:Cspad.0/geometry/0-end.data. To that end, we need a copy of the calibration directory for the experiment so we can modify it:<br />
<br />
cd <a subfolder in your home directory><br />
mkdir -p cxi02416/calib<br />
cd cxi02416/calib<br />
cp -r /reg/d/psdm/cxi/cxi02416/calib/* .<br />
<br />
Now, we can link in v2, backing up v1 first:<br />
<br />
cd CsPad::CalibV1/CxiDs2.0:Cspad.0/geometry<br />
mv 0-end.data 0-end.data.v1<br />
ln -fns <path to 0-end.data.t000_1k_1> 0-end.data<br />
<br />
We optionally use softlinks here to avoid duplicating data. We can now reprocess the data as trial 1:<br />
<br />
for i in `seq 2 16`; do '''cxi.mpi_submit''' input.experiment=cxi02416 \<br />
output.output_dir=/reg/d/psdm/cxi/cxi02416/ftc/brewster/dials \<br />
mp.nproc=36 mp.queue=psanaq output.split_logs=True \<br />
input.dispatcher=cctbx.xfel.xtc_process \<br />
input.target=[[cxi02416-lyso-t001.phil]] input.trial=1 input.run_num=$i \<br />
input.cfg=cxi02416-calibdir.cfg; done<br />
<br />
To instruct psana to use the modified calibration directory, we add a psana config file named cxi02416-calibdir.cfg with these lines:<br />
<br />
[psana]<br />
calib-dir = <a subfolder in your home directory>/cxi02416/calib<br />
<br />
If you have write permissions to your geometry folder, you don't need to use this config file or make a copy of your calibration directory. We recommend saving the original metrology like we did in the above example, by renaming it to 0-end.data.v1, so you can return to it as needed.<br />
<br />
The [[cxi02416-lyso-t001.phil | input phil file]] is the same as before, with a slight change to the unit cell dimensions. The program '''cspad.detector_statistics''' (described below) reports an average unit cell after joint refinement, which we use to update our indexing cell target. <br />
<br />
Indexing with the new metrology yielded 5305 indexed images.<br />
<br />
== Refine metrology (v3) ==<br />
<br />
We now refine the metrology generated from the indexed images from trial 3 and call it metrology version 3 (v3). Now that we have a better estimate of the unit cell, we [[cxi02416-refine.phil | use a phil file]] that adds a restraint to the joint refinement such the refined unit cells for each crystal have a similar set of dimensions. This assumes the crystals are isomorphous. We also increase the number images used to 2000.<br />
<br />
bsub -q psanaq -o t001_2k.out '''cspad.cbf_metrology''' tag=t001_2k \<br />
[[cxi02416-refine.phil]] reflections=indexed ../r0*/001/out \<br />
n_subset=2000 split_dataset=True<br />
<br />
After refinement, use '''cxi.display_metrology''', '''dev.dials.plot_detector_shifts''', and/or '''cspad.detector_shifts''' to evaluate how much the tiles moved:<br />
<br />
'''cspad.detector_shifts''' \<br />
t000_1k_1_refined_experiments_level2.json t000_1k_1_refined_reflections_level2.pickle \<br />
t001_2k_1_refined_experiments_level2.json t001_2k_1_refined_reflections_level2.pickle<br />
<br />
The Detector XY column shows the detector moved 4.3 microns in the XY plane, the quadrants on averaged moved 10.2 +/- 5.8 microns and the 2x1 sensors moved 11.7 +/- 11.5 microns. The tile positions moved substantially less between v2 and v3 then they did between v1 and v2.<br />
<br />
== Evaluate metrology ==<br />
<br />
Iterative hierarchical joint refinement should proceed until convergence, meaning until rounds of indexing and refinement do not improve the model. '''cxi.display_metrology''', '''dev.dials.plot_detector_shfits''', and '''cspad.detector_shifts''' measure the magnitude of changes during refinement. An additional tool is available, '''cspad.detector_statistics''', that can be used to evaluate the precision of refinement, and the state of the cspad after refinement.<br />
<br />
'''cspad.detector_statistics''' tag=t001_2k<br />
<br />
Warning, this dumps a lot of output. The program examines the current directory for files from cspad.cbf_metrology. For each hierarchy level 0-3, the program creates three tables of statistics:<br />
<br />
1) Detector congruence. The two half datasets from each level of refinement are compared to each other, and agreement between the two independent refinement runs is reported.<br />
<br />
2) Detector statistics. Using the two half datasets as independent measurements, statistics about the cspad are reported, such as normal vector tilts and XYZ offsets. Weighted means and standard deviations are reported as well, and can be used to estimate the overall precision of refinement.<br />
<br />
3) RMSDs by detector number. For each of the half datasets, overall, radial and transverse RMSDs are shown.<br />
<br />
For this walkthrough, we will extract a few statistics only. Look for the third set of tables, delineated with <br />
********************************************************************************<br />
Showing statistics for detector at level 2 (sensors, I.E. 2x1s)<br />
********************************************************************************<br />
<br />
In the second table, under Detector statistics, a few of the columns are reproduced here (click expand to show the table):<br />
<div class="toccolours mw-collapsible mw-collapsed"><br />
{| class="wikitable"<br />
|-<br />
|Panel G ID<br />
|Dist (mm)<br />
|F Offset Sigma (microns)<br />
|S Offset Sigma (microns)<br />
|N refls<br />
|-<br />
|1<br />
|21.8<br />
|1.457<br />
|3.795<br />
|6916<br />
|-<br />
|9<br />
|23.4<br />
|3.055<br />
|1.747<br />
|6498<br />
|-<br />
|25<br />
|24.5<br />
|0.614<br />
|1.891<br />
|6428<br />
|-<br />
|17<br />
|26<br />
|1.007<br />
|6.704<br />
|5772<br />
|-<br />
|0<br />
|40.7<br />
|2.003<br />
|1.707<br />
|5712<br />
|-<br />
|8<br />
|41.1<br />
|2.507<br />
|7.437<br />
|5882<br />
|-<br />
|24<br />
|43.7<br />
|0.139<br />
|4.121<br />
|5934<br />
|-<br />
|16<br />
|44.3<br />
|3.088<br />
|5.562<br />
|5470<br />
|-<br />
|7<br />
|55.3<br />
|1.048<br />
|5.097<br />
|5221<br />
|-<br />
|31<br />
|57.5<br />
|4.456<br />
|2.402<br />
|4837<br />
|-<br />
|15<br />
|57.8<br />
|1.447<br />
|0.87<br />
|4768<br />
|-<br />
|23<br />
|59.7<br />
|1.39<br />
|1.814<br />
|5165<br />
|-<br />
|11<br />
|71.9<br />
|0.38<br />
|10.784<br />
|3203<br />
|-<br />
|3<br />
|73<br />
|9.648<br />
|1.552<br />
|3238<br />
|-<br />
|19<br />
|75.2<br />
|9.258<br />
|3.055<br />
|2610<br />
|-<br />
|6<br />
|76.3<br />
|1.85<br />
|9.05<br />
|2921<br />
|-<br />
|27<br />
|76.3<br />
|3.867<br />
|12.665<br />
|3329<br />
|-<br />
|2<br />
|77.6<br />
|11.894<br />
|1.966<br />
|2781<br />
|-<br />
|10<br />
|78<br />
|1.809<br />
|5.188<br />
|2537<br />
|-<br />
|30<br />
|78.4<br />
|1.374<br />
|4.348<br />
|2265<br />
|-<br />
|14<br />
|78.7<br />
|1.845<br />
|1.796<br />
|2187<br />
|-<br />
|22<br />
|80.4<br />
|0.119<br />
|2.188<br />
|2574<br />
|-<br />
|18<br />
|81<br />
|19.347<br />
|9.268<br />
|2256<br />
|-<br />
|26<br />
|81.1<br />
|3.51<br />
|15.827<br />
|2767<br />
|-<br />
|4<br />
|87.5<br />
|2.449<br />
|8.722<br />
|1844<br />
|-<br />
|12<br />
|89.6<br />
|23.074<br />
|14.689<br />
|1636<br />
|-<br />
|28<br />
|90.5<br />
|12.99<br />
|19.289<br />
|1615<br />
|-<br />
|20<br />
|91.8<br />
|2.404<br />
|0.28<br />
|1548<br />
|-<br />
|5<br />
|104.8<br />
|32.112<br />
|16.115<br />
|797<br />
|-<br />
|13<br />
|106<br />
|10.282<br />
|19.482<br />
|559<br />
|-<br />
|29<br />
|107.8<br />
|8.739<br />
|12.337<br />
|762<br />
|-<br />
|21<br />
|108.5<br />
|23.591<br />
|18.247<br />
|682<br />
|-<br />
|All<br />
|<br />
|3.786<br />
|5.281<br />
|<br />
|-<br />
|Mean<br />
|<br />
|<br />
|<br />
|3459.8<br />
|}<br />
</div><br />
<br />
Here are the meaning of the columns:<br />
* Panel G ID: here we are looking at the sensors, of which there are 32.<br />
* Dist (mm): distance from the sensor center to the beam center. The table is sorted by this number.<br />
* F Offset sigma (microns): weighted standard deviation of the two measurements of the sensor's fast coordinate.<br />
* S Offset sigma (microns): weighted standard deviation of the two measurements of the sensor's slow coordinate.<br />
* N refls: sum of the number of reflections recorded on the sensor between the two half dataset. Used as a weighting term.<br />
<br />
The weighted mean of the fast and slow offset sigmas is 3.8 and 5.3 microns, respectively. This measure of the precision of this refinement indicates it quite precise compared to the pixel size of the detector (110 microns). However, there are still several panels with high fast or slow offset sigmas. That, plus the fact that the number of indexed images increased while using v3 metrology implies that refinement has not converged.<br />
<br />
== Further indexing and refinement (v4) ==<br />
<br />
Reindexing the data using v3 and re-refining the data to create v4 metrology proceeds as described in the above steps. After generating v4, we evaluate it as described above and see this table:<br />
<div class="toccolours mw-collapsible mw-collapsed"><br />
{| class="wikitable"<br />
|-<br />
!PanelG ID<br />
!Dist (mm)<br />
!F Offset Sigma (microns)<br />
!S Offset Sigma (microns)<br />
!N refls<br />
|-<br />
|1<br />
|21.8<br />
|1.1<br />
|1.7<br />
|6731<br />
|-<br />
|9<br />
|23.4<br />
|0.2<br />
|0.1<br />
|6681<br />
|-<br />
|25<br />
|24.5<br />
|0.9<br />
|1.2<br />
|6321<br />
|-<br />
|17<br />
|26<br />
|1.1<br />
|2.5<br />
|6144<br />
|-<br />
|0<br />
|40.7<br />
|0.1<br />
|1.9<br />
|5533<br />
|-<br />
|8<br />
|41<br />
|1.7<br />
|4.7<br />
|5501<br />
|-<br />
|24<br />
|43.7<br />
|1.0<br />
|1.5<br />
|5576<br />
|-<br />
|16<br />
|44.3<br />
|4.7<br />
|1.3<br />
|5707<br />
|-<br />
|7<br />
|55.3<br />
|0.5<br />
|0.5<br />
|4764<br />
|-<br />
|31<br />
|57.5<br />
|2.0<br />
|1.6<br />
|4722<br />
|-<br />
|15<br />
|57.8<br />
|2.2<br />
|1.8<br />
|4804<br />
|-<br />
|23<br />
|59.7<br />
|3.7<br />
|4.1<br />
|4681<br />
|-<br />
|11<br />
|71.9<br />
|1.7<br />
|8.7<br />
|2887<br />
|-<br />
|3<br />
|73<br />
|16.1<br />
|0.1<br />
|3234<br />
|-<br />
|19<br />
|75.1<br />
|7.3<br />
|0.1<br />
|2984<br />
|-<br />
|6<br />
|76.3<br />
|2.4<br />
|1.1<br />
|2664<br />
|-<br />
|27<br />
|76.3<br />
|3.7<br />
|16.8<br />
|2903<br />
|-<br />
|2<br />
|77.6<br />
|3.1<br />
|4.1<br />
|2866<br />
|-<br />
|10<br />
|78<br />
|2.5<br />
|5.2<br />
|2388<br />
|-<br />
|30<br />
|78.4<br />
|1.1<br />
|0.6<br />
|2413<br />
|-<br />
|14<br />
|78.7<br />
|14.2<br />
|1.7<br />
|2460<br />
|-<br />
|22<br />
|80.4<br />
|2.3<br />
|2.0<br />
|2255<br />
|-<br />
|18<br />
|81<br />
|10.2<br />
|4.6<br />
|2348<br />
|-<br />
|26<br />
|81.1<br />
|4.8<br />
|21.8<br />
|2545<br />
|-<br />
|4<br />
|87.5<br />
|8.0<br />
|4.9<br />
|1832<br />
|-<br />
|12<br />
|89.6<br />
|11.8<br />
|2.3<br />
|1559<br />
|-<br />
|28<br />
|90.4<br />
|9.7<br />
|14.8<br />
|1497<br />
|-<br />
|20<br />
|91.8<br />
|19.3<br />
|17.7<br />
|1434<br />
|-<br />
|5<br />
|104.8<br />
|47.9<br />
|35.9<br />
|875<br />
|-<br />
|13<br />
|106.1<br />
|6.2<br />
|9.9<br />
|550<br />
|-<br />
|29<br />
|107.8<br />
|15.4<br />
|24.2<br />
|748<br />
|-<br />
|21<br />
|108.5<br />
|17.7<br />
|7.2<br />
|698<br />
|-<br />
|All<br />
|<br />
|4.0<br />
|3.9<br />
|<br />
|-<br />
|Mean<br />
|<br />
|<br />
|<br />
|3384.5<br />
|}<br />
</div><br />
<br />
The fast and slow offset sigmas are lower generally, but not substantially. 5475 images were indexed, an increase of 1% over v3 metrology. The change in tile position is minimal, as shown by '''cspad.detector_shifts''' (delta XY movements (microns): detector: 1.5, quadrants: 4.0+/2.3, sensors: 5.5+/-5.9). The metrology has likely converged.<br />
<br />
== Deploy metrology ==<br />
<br />
v4 metrology is now ready for use. Deploy it or use it directly for indexing as described in [[Cxi0216 calibration#Index using v2 metrology | indexing using v2 metrology]].</div>Aaronhttp://viper.lbl.gov:8080/cctbx.xfel/index.php/Cxi02416_calibrationCxi02416 calibration2016-05-18T23:52:23Z<p>Aaron: /* Index using v2 metrology */</p>
<hr />
<div><br />
== Calibration of cspad using cxi02416, ''cctbx.xfel'' and DIALS ==<br />
<br />
This page is not intended as a manual for processing XFEL data using cctbx.xfel and DIALS. Rather, this is a documentation of steps taken to calibrate the CSPAD using data collected at CXI in February 2016, on detector Ds2. <br />
<br />
Program names are highlighted in '''bold'''. Generally, help can be obtained using -h or -c options to the program at the command line.<br />
<br />
In this walkthrough, we will be calibrating the tile positions of the CSPAD detector at the CXI endstation at LCLS. We will be using a calibration dataset collected by CXI staff in February 2016, during calibration experiment cxi02614. The procedure is to index the images, refine the tile positions minimizing the difference between observed and calculated Bragg reflections, re-index the images, re-refine the tile positions, and repeat until convergence. At each step of the way, we will label our current metrology with a version number, starting at version 0:<br />
<br />
{| class="wikitable"<br />
|-<br />
| '''Metrology version'''<br />
| '''Description'''<br />
|-<br />
|Version 0 (v0)<br />
|Initial metrology deployed by beamline operators. The tile positions are measured using an optical microscope, but as the quadrants can move independently, they are not correctly aligned in relation to each other or to the beam center.<br />
|-<br />
|Version 1 (v1)<br />
|After collecting some data, powder rings can be seen after averaging the events in a run. Several tools are available for aligning the quadrants by eye or automatically using powder rings.<br />
|-<br />
|Version 2 (v2)<br />
|After indexing the images using v1, we will refine the tile positions to produce metrology v2.<br />
|-<br />
|Version 3 (v3)<br />
|After re-indexing the images using v2, we will re-refine the tile positions to produce metrology v3.<br />
|-<br />
|...<br />
|And so forth until convergence<br />
|}<br />
<br />
We start with this known information provided by the beamline operators:<br />
<br />
* Data is contained in runs 2-16<br />
* Detector address: CxiDs2.0:Cspad.0. This string identifies the front CSPAD detector in the XTC streams.<br />
* detz_offset (IE the distance from the sample position to the back of the detector rail): 568 mm.<br />
<br />
== Dark pedestal, common mode correction, and untrusted pixels ==<br />
<br />
Subtracting the dark pedestal, applying common mode correction and masking out untrusted pixels is critical for spot picking and integration. ''cctbx.xfel'' uses masks and pedestals deployed into the experiment’s calibration store using the LCLS program calibman [https://confluence.slac.stanford.edu/display/PSDM/Calibration+Management+Tool] . Dark pedestal correction, hot pixel determination, and user defined mask generation for regions of interest or for masking out shadows is all available through that program.<br />
<br />
Common mode correction accounts for flat, per-panel fluctuations that occur non-uniformly throughout a single exposure. LCLS has several algorithms available [https://confluence.slac.stanford.edu/display/PSDM/Common+mode+correction+algorithms]. We recommend the non-bonded pixels algorithm; the parameter files below specifies this algorithm during indexing and integration.<br />
<br />
== Aligning quadrant positions (v1) ==<br />
<br />
Alignment of quadrant positions using powder rings can be done manually using ''cctbx.xfel'' or LCLS's calibman tool [https://confluence.slac.stanford.edu/display/PSDM/Calibration+Management+Tool]. ''ctbx.xfel'' also provides an automatic algorithm for quadrant alignment, provided powder rings of sufficient quality. For cxi02416, the beamline operators already calibrated quadrant positions to a point where indexing can proceed. Regardless, if needed, here is how to accomplish the same task using ''cctbx.xfel''<br />
<br />
=== Averaging diffraction data to create powder patterns ===<br />
<br />
The ''cctbx.xfel'' program '''cxi.mpi_average''' is used to create averages:<br />
<br />
for i in `seq 2 16`; do bsub -n 12 -q psanaq -o avg_r$i.log \<br />
'''cxi.mpi_average''' -x cxi02416 -r $i -a CxiDs2.0:Cspad.0 -d 568 -o . -v; done<br />
<br />
For each of the runs with data (2-16), this command submits an averaging job using 12 processors, providing information about the experiment such as detector address and detz_offset in the form of command line arguments.<br />
<br />
Progress can be monitored with the command bjobs.<br />
<br />
When the averages are complete, they will consist of files named cxi02416_avg-r0002.cbf, cxi02416_stddev-r0002.cbf, and cxi02416_max-r0002.cbf for each run, representing the average, standard deviation, and maximum of all the pixel data in each of the runs. Display the data using '''cctbx.image_viewer''':<br />
<br />
'''cctbx.image_viewer''' *.cbf<br />
<br />
=== Manual calibration using ''cctbx.xfel'' ===<br />
<br />
Typically the best powder rings come from the maximum projection (example: cxi02416_max-r0013.cbf). To manually align the quadrant positions, use '''cctbx.image_viewer''' cxi02416_max-r0013.cbf. Under actions, click on 'Show quadrant calibration', then use the spinners to align the powder rings. You may find the ring tool or the unit cell tool, also under the Actions menu, to be useful visual aids during this process. When done, click 'Save current metrology' to save the changes to a .def file, which is a cbf header.<br />
<br />
This walkthrough uses pre-aligned quadrant locations. However, if it is desired to use the metrology from manual quadrant re-alignment for indexing, it first needs to be converted to SLAC's metrology file format. Use this command:<br />
<br />
'''cxi.cbfheader2slaccalib''' cbf_header=quadrants.cbf<br />
<br />
This command will create a 0-end.data file. See the instructions under [[Cxi0216 calibration#Index using v2 metrology | indexing using v2 metrology]] for deploying it for use.<br />
<br />
=== Automatic calibration using ''cctbx.xfel'' ===<br />
<br />
If a quadrant is properly placed, the pixel values for a strong powder pattern will be highly correlated after rotating it 45 degrees around the beam center. '''cspad.quadrants_cbf''' performs a grid search of XY offsets for each quadrant, searching for the position with the highest rotational autocorrelation. It then writes out a new cbf file with the adjusted header:<br />
<br />
'''cspad.quadrants_cbf''' cxi02416_max-r0013.cbf<br />
<br />
Specify the '-p' parameter to enable plots of the grid search results for each quadrant. Here is the output:<br />
<br />
<div class="toccolours mw-collapsible mw-collapsed"><br />
<pre><br />
Doing cross-correlation on panel ARRAY_D0Q0S1A0<br />
Searching a grid with dimensions (41, 41)<br />
max cc 0.1592 is at (0, 0)<br />
Doing cross-correlation on panel ARRAY_D0Q1S1A0<br />
Searching a grid with dimensions (41, 41)<br />
max cc 0.1209 is at (0, 5)<br />
Doing cross-correlation on panel ARRAY_D0Q2S1A0<br />
Searching a grid with dimensions (41, 41)<br />
max cc 0.1686 is at (0, -2)<br />
Doing cross-correlation on panel ARRAY_D0Q3S1A0<br />
Searching a grid with dimensions (41, 41)<br />
max cc 0.1144 is at (1, 2)<br />
</pre><br />
</div><br />
<br />
The CC values are all less that 0.2 which indicates that even though run 13 shows the strongest powder diffraction, the rings are not contiguous or strong enough compared to the background to get a good rotational autocorrelation. Use the image viewer to verify the new quadrant positions are not ideal:<br />
<br />
'''cctbx.image_viewer''' cxi02416_max-r0013_cc.cbf<br />
<br />
It is possible that using a maximum projection of all the runs would make the rings more contiguous and brighter, leading to higher CC values. This can be done quickly using the maximum projections already made:<br />
<br />
'''cxi.cspad_average''' *_max*.cbf -m all_max.cbf<br />
<br />
Use '''cctbx.image_viewer''' to compare all_max.cbf to cxi02416_max-r0013.cbf. The rings are noticeably better. Now, do the grid search:<br />
<br />
'''cspad.quadrants_cbf''' all_max.cbf<br />
<br />
Results:<br />
<div class="toccolours mw-collapsible mw-collapsed"><br />
<pre><br />
Doing cross-correlation on panel ARRAY_D0Q0S1A0<br />
Searching a grid with dimensions (41, 41)<br />
max cc 0.2568 is at (-1, 0)<br />
Doing cross-correlation on panel ARRAY_D0Q1S1A0<br />
Searching a grid with dimensions (41, 41)<br />
max cc 0.2022 is at (0, 4)<br />
Doing cross-correlation on panel ARRAY_D0Q2S1A0<br />
Searching a grid with dimensions (41, 41)<br />
max cc 0.2328 is at (-4, 0)<br />
Doing cross-correlation on panel ARRAY_D0Q3S1A0<br />
Searching a grid with dimensions (41, 41)<br />
max cc 0.1853 is at (0, 2)<br />
</pre><br />
</div><br />
<br />
The CC values are higher, and likely high enough for at least two of the quadrants to get some initial indexing results. Regardless, we recommend silver behenate powder for this automatic procedure as it gives very smooth, contiguous rings.<br />
<br />
Again, for this walkthrough we use quadrant positions aligned by the beamline operator. However, if after automatic alignment it is desired to use the results for indexing, they first need to be converted to SLAC's metrology file format. Use this command:<br />
<br />
'''cxi.cbfheader2slaccalib''' cbf_header=all_max_cc.cbf<br />
<br />
This command will create a 0-end.data file. See the instructions under [[Cxi0216 calibration#Index using v2 metrology | indexing using v2 metrology]] for deploying it for use.<br />
<br />
== Initial indexing ==<br />
<br />
The initial metrology deployed by the beamline operator is sufficient to get initial indexing results. Indexing in ''cctbx.xfel'' typically is done in a series of trials. Our first trial will be trial 0, using metrology v1 (initial metrology from beamline operators, with quadrants corrected by eye by the beamline operators).<br />
<br />
With this information, and [[cxi02416-lyso-t000.phil | this phil file]], we can index the data:<br />
<br />
for i in `seq 2 16`; do '''cxi.mpi_submit''' input.experiment=cxi02416 \<br />
output.output_dir=/reg/d/psdm/cxi/cxi02416/ftc/brewster/dials \<br />
mp.nproc=36 mp.queue=psanaq output.split_logs=True \<br />
input.dispatcher=cctbx.xfel.xtc_process \<br />
input.target=[[cxi02416-lyso-t000.phil]] input.trial=0 \<br />
input.run_num=$i; done<br />
<br />
This command submits jobs for runs 2 through 16, using the DIALS backend of ''cctbx.xfel''. To save time during initial indexing and metrology refinement, we use dispatch.integrate=False to skip the integration step. After indexing is completed, we got 4986 indexed images, as can be shown by this command:<br />
<br />
cd /reg/d/psdm/cxi/cxi02416/ftc/brewster/dials<br />
ls r0*/000/out/*.json | wc -l<br />
<br />
== Refine metrology (v2) ==<br />
<br />
Let's call the metrology deployed by the beamline operator version 0 (v0). After quadrant alignment, the operators updated the metrology to version 1 (v1). The following command will do an iterative joint hierarchical refinement of the components of the CSPAD detector. The new tile positions we call version 2 (v2).<br />
<br />
bsub -q psanaq -o t000_1k.out '''cspad.cbf_metrology''' tag=t000_1k \<br />
reflections=indexed ../r0*/000/out refine_distance=True \<br />
n_subset=1000 split_dataset=True<br />
<br />
The program first aggregates the requested number of images into a single dataset. Then, it refines the detector as a whole (including Z position and tilt). Using the new detector position, it refines the quadrants independently from each other, and then the 2x1 sensors, and then the individual panels. Finally, it converts the DIALS format metrology into the SLAC file format (0-end.data). The refinement is a 'joint refinement' because the information from many crystals is used to refine a single detector model.<br />
<br />
Details about the parameters used:<br />
<br />
* tag=t000_1k: the output files will be named t000_1k*<br />
* reflections=indexed: the indexing process produces two sets of reflection files, bright indexed reflections, and final integrated reflections (can include weak intensities). Here, I've chosen to refine only against the bright, indexed reflections.<br />
* refine_distance=True: as the detector distance is not fully known yet, allow the detector distance to refine. In subsequent rounds of metrology refinement, we will fix the distance to a constant value.<br />
* n_subset=1000: pick 1000 images at random to refine<br />
* split_dataset=True: the refinement is done twice independently, using odd numbered or even numbered images, each time using n_subset images. This will be useful later for evaluating the accuracy of the metrology.<br />
<br />
== Visualize tile shifts ==<br />
<br />
In order to get a sense of the magnitude of the shifts in panel position after refinement, use the program cxi.display_metrology. For example, the original detector geometry can be displayed thusly:<br />
'''cxi.display_metrology''' \<br />
/reg/d/psdm/cxi/cxi02416/calib/CsPad::CalibV1/CxiDs2.0:Cspad.0/geometry/0-end.data<br />
<br />
Compare it to the refined geometry:<br />
'''cxi.display_metrology''' 0-end.data.t000_1k_1<br />
<br />
You will see a small change in the origin (center arrow) and obvious changes in the quadrant positions. The relative positions of the sensors to each other will not change a large amount, though if you inspect the files themselves you will see changes.<br />
<br />
Another tool that is useful for evaluating changes in metrology is '''dev.dials.plot_detector_shifts'''. This program will plot detector shifts in the X and Y directions (also known as the fast and slow directions, a convention referring to how data is read from the byte arrays stored on disk). It will also plot shifts along the Z axis, I.E., the detector's normal axis. Let's use it to compare the unrefined geometry to the geometry refined at level 2 (I.E. at the level of the 2x1 sensors):<br />
<br />
'''dev.dials.plot_detector_shifts''' t000_1k_1_combined_experiments.json \<br />
t000_1k_1_refined_experiments_level2.json \<br />
plot_type=spherical_polar tag=v1v2level2<br />
display v1v2level2*.png<br />
<br />
Use the spacebar to cycle between the images. The pixels are mapped to the Ewald sphere and then displayed as a function of azimuthal and elevation angles along the Ewald sphere, which distorts the detector appearance but is independent of any detector layout. The colorbar scales will show the magnitude of panel shifts.<br />
<br />
Try running the program using t000_1k_1_refined_experiments_level0.json to see how much the detector as a whole shifted, or t000_1k_1_refined_experiments_level1.json to see shifts in the quadrants.<br />
<br />
Finally, the program '''cspad.detector_shifts''' shows the magnitude of shifts between a reference and a moving detector in tabular form:<br />
<br />
'''cspad.detector_shifts''' \<br />
t000_1k_1_combined_experiments.json t000_1k_1_combined_reflections.pickle \<br />
t000_1k_1_refined_experiments_level2.json t000_1k_1_refined_reflections_level2.pickle<br />
<br />
Looking at the Delta XY columns, we see the detector moved 4.6 microns in the XY plane, the quadrants moved on average 219.8+/-57.9 microns and the 2x1 sensors moved 17.6+/-16.1 microns. Note how most of the change is in the quadrant locations, which is expected as these were done by eye. The 2x1 sensor positions, determined using optical microscopy, changed by an order of magnitude less than the quadrants. Also, the same program ran against the other half of the split dataset (t000_1k_2_*) will reveal very similar shifts, indicating these shifts are not due to random chance.<br />
<br />
Full output for '''cspad.detector_shifts''':<br />
<br />
<div class="toccolours mw-collapsible mw-collapsed"><br />
<pre><br />
Found 4 hierarchy levels<br />
Hierarchy level 0 Detector shifts<br />
-----------------------------------------------------------------------------------------------------------<br />
PanelG BC dist Delta XY R Offsets T Offsets Z Offsets dR Norm dT Norm Local dNorm Rot Z N Refls<br />
ID (mm) (microns) (microns) (microns) (microns) (deg) (deg) (deg) (deg)<br />
-----------------------------------------------------------------------------------------------------------<br />
0 2.4 4.6 3.5 -2.9 -74.0 -0.0003 -0.0213 0.0213 0.0003 29527<br />
Weighted mean 4.6 3.5 -2.9 -74.0 -0.0003 -0.0213 0.0213 0.0003<br />
Weighted stddev 0.0 0.0 0.0 0.0 0.0000 0.0000 0.0000 0.0000<br />
-----------------------------------------------------------------------------------------------------------<br />
Hierarchy level 1 Detector shifts<br />
-----------------------------------------------------------------------------------------------------------<br />
PanelG BC dist Delta XY R Offsets T Offsets Z Offsets dR Norm dT Norm Local dNorm Rot Z N Refls<br />
ID (mm) (microns) (microns) (microns) (microns) (deg) (deg) (deg) (deg)<br />
-----------------------------------------------------------------------------------------------------------<br />
0 62.9 157.7 -153.5 -36.0 1236.4 -0.0308 -0.0658 0.0550 0.0111 7766<br />
1 64.3 262.5 -172.9 -197.5 1532.9 -0.0042 -0.1422 0.1411 0.0260 7112<br />
3 66.0 190.2 22.6 188.8 1347.7 -0.0945 -0.0391 0.1228 0.1139 7811<br />
2 67.0 279.6 277.7 32.1 1374.9 0.0216 0.0397 0.0305 0.0169 6838<br />
Weighted mean 219.8 -11.7 0.3 1369.3 -0.0291 -0.0527 0.0880 0.0432<br />
Weighted stddev 57.9 204.3 161.0 122.4 0.0500 0.0731 0.0524 0.0494<br />
-----------------------------------------------------------------------------------------------------------<br />
Hierarchy level 2 Detector shifts<br />
-----------------------------------------------------------------------------------------------------------<br />
PanelG BC dist Delta XY R Offsets T Offsets Z Offsets dR Norm dT Norm Local dNorm Rot Z N Refls<br />
ID (mm) (microns) (microns) (microns) (microns) (deg) (deg) (deg) (deg)<br />
-----------------------------------------------------------------------------------------------------------<br />
1 21.9 2.7 0.1 -2.7 67.0 -0.2345 -0.0794 0.1925 0.0227 1817<br />
9 23.6 2.5 -0.3 2.4 -34.8 0.0229 0.1911 0.2632 0.0290 1578<br />
25 24.7 7.9 -2.6 7.4 34.8 -0.0817 -0.1288 0.2195 0.1031 1644<br />
17 26.3 5.8 -3.8 -4.4 -72.0 0.0402 0.0412 0.0588 0.0107 1357<br />
0 40.8 23.0 -23.0 -0.3 44.5 0.0064 -0.3016 0.3031 0.0169 1626<br />
8 41.3 5.8 -4.7 3.6 -44.8 0.2528 -0.0850 0.3773 0.0512 1687<br />
24 43.9 10.2 -0.4 -10.3 36.9 -0.1491 -0.2116 0.2813 0.0628 1665<br />
16 44.5 17.8 17.1 4.9 -13.0 0.0640 0.1521 0.1525 0.0178 1499<br />
7 55.5 12.6 0.4 -12.6 52.8 -0.1670 0.0284 0.1042 0.0097 1382<br />
31 57.7 28.9 -7.5 -27.9 81.9 -0.0803 0.0442 0.0742 0.1336 1406<br />
15 58.0 13.5 -6.2 12.0 -52.4 0.0155 0.1185 0.1853 0.0004 1323<br />
23 60.0 16.8 -16.1 4.8 -17.2 -0.0080 -0.0531 0.0518 0.0357 1560<br />
11 72.2 4.4 -1.6 -4.2 8.5 -0.5439 -0.0891 0.4503 0.0258 843<br />
3 73.1 22.5 17.6 -13.9 22.6 -0.2173 0.0156 0.1600 0.0165 856<br />
19 75.3 2.1 -0.5 2.0 -17.4 0.0061 0.1337 0.1232 0.0245 584<br />
6 76.5 11.2 -4.4 -10.3 14.9 0.0371 -0.1290 0.1975 0.0252 730<br />
27 76.5 59.0 6.4 -58.6 93.7 -0.0486 0.0859 0.0744 0.0859 851<br />
2 77.7 45.1 1.6 -45.2 71.3 -0.1845 0.0036 0.1143 0.0259 772<br />
10 78.3 19.5 -3.5 -19.1 24.1 -0.4560 -0.0550 0.3362 0.0117 651<br />
30 78.6 17.4 -3.3 17.2 -53.8 0.0098 -0.1055 0.2080 0.0894 651<br />
14 78.9 4.1 -4.1 -0.5 14.5 0.0433 0.1315 0.2032 0.0036 504<br />
22 80.6 22.0 21.9 2.1 -17.6 0.0394 0.0027 0.0347 0.0234 696<br />
18 81.2 31.8 9.1 30.5 -80.5 0.0141 -0.0178 0.0374 0.0487 620<br />
26 81.3 25.6 -8.1 -24.2 68.4 -0.0185 0.0909 0.0750 0.1275 819<br />
4 87.6 32.3 31.6 -5.7 53.6 -0.2149 0.0391 0.1460 0.0154 431<br />
12 89.8 25.3 6.9 -24.2 -21.1 -0.0419 0.0008 0.1011 0.0173 406<br />
28 90.7 37.7 36.9 3.7 79.9 -0.3678 0.0389 0.3173 0.0006 529<br />
20 92.1 24.0 2.7 23.9 16.6 0.0213 -0.1148 0.1014 0.0777 374<br />
5 104.9 92.9 -91.3 6.3 -133.6 -0.6185 0.2555 0.6003 0.0674 152<br />
13 106.3 9.9 9.9 0.4 4.2 -0.1141 -0.0277 0.0259 0.0031 120<br />
29 108.0 101.5 101.1 -1.7 122.2 -0.3588 0.1173 0.2946 0.0482 246<br />
21 108.7 16.4 -13.1 9.9 -17.3 0.1983 0.0814 0.1786 0.0392 148<br />
Weighted mean 17.6 -0.1 -4.7 12.6 -0.0687 -0.0164 0.1883 0.0415<br />
Weighted stddev 16.1 16.5 16.8 51.4 0.1702 0.1286 0.1140 0.0381<br />
-----------------------------------------------------------------------------------------------------------<br />
Hierarchy level 3 Detector shifts<br />
-----------------------------------------------------------------------------------------------------------<br />
PanelG BC dist Delta XY R Offsets T Offsets Z Offsets dR Norm dT Norm Local dNorm Rot Z N Refls<br />
ID (mm) (microns) (microns) (microns) (microns) (deg) (deg) (deg) (deg)<br />
-----------------------------------------------------------------------------------------------------------<br />
2 15.9 0.0 0.0 -0.0 0.0 -0.0669 -0.2384 0.0000 0.0227 941<br />
18 16.2 0.0 -0.0 0.0 0.0 -0.1455 0.1261 0.0000 0.0290 885<br />
50 19.2 0.0 -0.0 0.0 0.0 0.0606 -0.1400 0.0000 0.1031 865<br />
34 19.3 0.0 -0.0 -0.0 0.0 -0.0121 0.0563 0.0000 0.0107 578<br />
3 30.7 0.0 0.0 0.0 0.0 -0.2095 -0.1320 0.0000 0.0227 876<br />
19 33.0 0.0 0.0 0.0 0.0 -0.0226 0.1912 0.0000 0.0290 693<br />
51 33.0 0.0 0.0 0.0 0.0 -0.0494 -0.1443 0.0000 0.1031 779<br />
35 35.2 0.0 0.0 0.0 0.0 0.0293 0.0496 0.0000 0.0107 779<br />
16 37.6 0.0 -0.0 0.0 0.0 0.2624 0.0480 0.0000 0.0512 866<br />
0 37.9 0.0 0.0 -0.0 0.0 0.1518 -0.2606 0.0000 0.0169 775<br />
32 40.8 0.0 0.0 -0.0 0.0 -0.0180 0.1640 0.0000 0.0178 583<br />
48 41.1 0.0 -0.0 0.0 0.0 -0.0272 -0.2575 0.0000 0.0628 863<br />
1 46.2 0.0 -0.0 0.0 0.0 0.0304 -0.3001 0.0000 0.0169 851<br />
17 47.3 0.0 0.0 0.0 0.0 0.2587 -0.0648 0.0000 0.0512 821<br />
49 49.1 0.0 0.0 0.0 0.0 -0.1319 -0.2228 0.0000 0.0628 802<br />
33 50.4 0.0 -0.0 -0.0 0.0 0.0517 0.1567 0.0000 0.0178 916<br />
15 51.3 0.0 -0.0 0.0 0.0 -0.1644 0.0407 0.0000 0.0097 740<br />
63 52.9 0.0 -0.0 0.0 0.0 -0.0768 0.0501 0.0000 0.1336 732<br />
31 54.2 0.0 0.0 0.0 0.0 0.0243 0.1170 0.0000 0.0004 667<br />
47 55.7 0.0 0.0 -0.0 0.0 -0.0119 -0.0523 0.0000 0.0357 885<br />
14 61.4 0.0 -0.0 0.0 0.0 -0.1694 -0.0022 0.0000 0.0097 642<br />
23 61.5 0.0 -0.0 -0.0 0.0 -0.5509 -0.0155 0.0000 0.0258 690<br />
7 62.3 0.0 -0.0 0.0 0.0 -0.2132 0.0445 0.0000 0.0165 626<br />
30 63.5 0.0 -0.0 0.0 0.0 -0.0061 0.1193 0.0000 0.0004 656<br />
62 64.0 0.0 -0.0 -0.0 0.0 -0.0869 0.0291 0.0000 0.1336 674<br />
39 64.6 0.0 -0.0 -0.0 0.0 0.0240 0.1317 0.0000 0.0245 435<br />
55 65.7 0.0 -0.0 0.0 0.0 -0.0367 0.0916 0.0000 0.0859 631<br />
46 65.8 0.0 0.0 -0.0 0.0 0.0017 -0.0537 0.0000 0.0357 675<br />
5 67.7 0.0 -0.0 0.0 0.0 -0.1829 0.0246 0.0000 0.0259 591<br />
21 68.5 0.0 -0.0 0.0 0.0 -0.4593 -0.0032 0.0000 0.0117 519<br />
53 71.2 0.0 -0.0 0.0 0.0 -0.0081 0.0924 0.0000 0.1275 605<br />
37 71.4 0.0 0.0 -0.0 0.0 0.0120 -0.0193 0.0000 0.0487 447<br />
13 73.5 0.0 0.0 -0.0 0.0 0.0317 -0.1305 0.0000 0.0252 447<br />
61 75.2 0.0 0.0 0.0 0.0 0.0054 -0.1058 0.0000 0.0894 360<br />
29 76.1 0.0 -0.0 0.0 0.0 0.0488 0.1296 0.0000 0.0036 264<br />
45 77.5 0.0 0.0 0.0 0.0 0.0395 0.0011 0.0000 0.0234 440<br />
9 80.3 0.0 0.0 0.0 0.0 -0.2107 0.0576 0.0000 0.0154 296<br />
12 80.8 0.0 0.0 0.0 0.0 0.0586 -0.1208 0.0000 0.0252 283<br />
25 82.2 0.0 0.0 -0.0 0.0 -0.0416 0.0044 0.0000 0.0173 270<br />
28 82.9 0.0 -0.0 -0.0 0.0 0.0201 0.1370 0.0000 0.0036 240<br />
22 83.0 0.0 0.0 0.0 0.0 -0.5494 -0.0436 0.0000 0.0258 153<br />
60 83.3 0.0 0.0 0.0 0.0 0.0276 -0.1023 0.0000 0.0894 291<br />
57 83.5 0.0 0.0 -0.0 0.0 -0.3631 0.0704 0.0000 0.0006 365<br />
6 83.9 0.0 -0.0 -0.0 0.0 -0.2152 0.0340 0.0000 0.0165 230<br />
41 84.6 0.0 0.0 0.0 0.0 0.0112 -0.1162 0.0000 0.0777 259<br />
44 85.0 0.0 -0.0 -0.0 0.0 0.0383 0.0095 0.0000 0.0234 256<br />
38 86.1 0.0 -0.0 -0.0 0.0 0.0174 0.1327 0.0000 0.0245 149<br />
54 87.3 0.0 -0.0 -0.0 0.0 -0.0411 0.0897 0.0000 0.0859 220<br />
4 88.0 0.0 -0.0 -0.0 0.0 -0.1844 0.0064 0.0000 0.0259 181<br />
20 88.3 0.0 -0.0 -0.0 0.0 -0.4567 -0.0489 0.0000 0.0117 132<br />
36 91.3 0.0 0.0 0.0 0.0 0.0139 -0.0180 0.0000 0.0487 173<br />
52 91.5 0.0 -0.0 -0.0 0.0 -0.0171 0.0912 0.0000 0.1275 214<br />
8 95.7 0.0 0.0 0.0 0.0 -0.2010 0.0856 0.0000 0.0154 135<br />
24 98.1 0.0 -0.0 0.0 0.0 -0.0407 0.0099 0.0000 0.0173 136<br />
56 98.5 0.0 0.0 0.0 0.0 -0.3501 0.1192 0.0000 0.0006 164<br />
11 98.8 0.0 -0.0 0.0 0.0 -0.6019 0.2924 0.0000 0.0674 105<br />
27 100.0 0.0 0.0 -0.0 0.0 -0.1156 -0.0207 0.0000 0.0031 77<br />
40 100.2 0.0 -0.0 0.0 0.0 -0.0046 -0.1166 0.0000 0.0777 115<br />
59 102.0 0.0 -0.0 -0.0 0.0 -0.3510 0.1389 0.0000 0.0482 164<br />
43 102.4 0.0 0.0 -0.0 0.0 0.2029 0.0691 0.0000 0.0392 98<br />
10 111.6 0.0 0.0 0.0 0.0 -0.5535 0.3760 0.0000 0.0674 47<br />
26 113.4 0.0 -0.0 0.0 0.0 -0.1173 -0.0040 0.0000 0.0031 43<br />
58 114.7 0.0 0.0 -0.0 0.0 -0.3274 0.1879 0.0000 0.0482 82<br />
42 115.7 0.0 0.0 -0.0 0.0 0.2108 0.0392 0.0000 0.0392 50<br />
Weighted mean 0.0 -0.0 0.0 0.0 -0.0571 -0.0152 0.0000 0.0415<br />
Weighted stddev 0.0 0.0 0.0 0.0 0.1674 0.1345 0.0000 0.0377<br />
-----------------------------------------------------------------------------------------------------------<br />
Detector shifts summary<br />
---------------------------------------------------------------------------------------------------------------------------------------------------------------<br />
Hierarchy Delta XY Delta XY R Offsets R Offsets T Offsets T Offsets Z Offsets Z Offsets dR Norm dR Norm dT Norm dT Norm Local dNorm Local dNorm Rot Z Rot Z<br />
Level Sigma Sigma Sigma Sigma Sigma Sigma Sigma Sigma<br />
(microns) (microns) (microns) (microns) (microns) (microns) (microns) (microns) (deg) (deg) (deg) (deg) (deg) (deg) (deg) (deg)<br />
---------------------------------------------------------------------------------------------------------------------------------------------------------------<br />
0 4.6 0.0 3.5 0.0 -2.9 0.0 -74.0 0.0 -0.0003 0.0000 -0.0213 0.0000 0.0213 0.0000 0.0003 0.0000<br />
1 219.8 57.9 -11.7 204.3 0.3 161.0 1369.3 122.4 -0.0291 0.0500 -0.0527 0.0731 0.0880 0.0524 0.0432 0.0494<br />
2 17.6 16.1 -0.1 16.5 -4.7 16.8 12.6 51.4 -0.0687 0.1702 -0.0164 0.1286 0.1883 0.1140 0.0415 0.0381<br />
3 0.0 0.0 -0.0 0.0 0.0 0.0 0.0 0.0 -0.0571 0.1674 -0.0152 0.1345 0.0000 0.0000 0.0415 0.0377<br />
---------------------------------------------------------------------------------------------------------------------------------------------------------------<br />
<br />
<br />
For each hierarchy level, the average shifts in are computed among objects at that level, weighted by the number of reflections recorded on each object. For example, for a four quadrant detector, the average Z shift will be the average of the four quadrant Z values, each weighted by the number of reflections on that quadrant.<br />
<br />
-------------------<br />
Column descriptions<br />
-------------------<br />
<br />
Individual hierarchy level tables only:<br />
PanelG id: ID of the panel group.<br />
BC dist: distance of the panel group from the beam center.<br />
N Refls: number of reflections on this panel group<br />
<br />
All tables:<br />
Delta XY: magnitude of the shift in the local XY frame.<br />
R, T offsets: shifts relative to the parent object's location in the radial and transverse directions (relative to the detector center).<br />
Z offsets: relative shifts in the local frame in the local Z direction.<br />
R, T Norm: angle between normal vectors in lab space, projected onto the radial or transverse plane.<br />
Local dNorm: local relative angle between normal vectors.<br />
Rot Z: rotation around detector normal in lab space<br />
</pre><br />
</div><br />
<br />
== Index using v2 metrology ==<br />
<br />
In order to improve our metrology we will re-index using the new tile positions. We assume we don't have write access to the geometry file for this detector, namely /reg/d/psdm/cxi/cxi02416/calib/CsPad::CalibV1/CxiDs2.0:Cspad.0/geometry/0-end.data. To that end, we need a copy of the calibration directory for the experiment so we can modify it:<br />
<br />
cd <a subfolder in your home directory><br />
mkdir -p cxi02416/calib<br />
cd cxi02416/calib<br />
cp -r /reg/d/psdm/cxi/cxi02416/calib/* .<br />
<br />
Now, we can link in v2, backing up v1 first:<br />
<br />
cd CsPad::CalibV1/CxiDs2.0:Cspad.0/geometry<br />
mv 0-end.data 0-end.data.v1<br />
ln -fns <path to 0-end.data.t000_1k_1> 0-end.data<br />
<br />
We optionally use softlinks here to avoid duplicating data. We can now reprocess the data as trial 1:<br />
<br />
for i in `seq 2 16`; do '''cxi.mpi_submit''' input.experiment=cxi02416 \<br />
output.output_dir=/reg/d/psdm/cxi/cxi02416/ftc/brewster/dials \<br />
mp.nproc=36 mp.queue=psanaq output.split_logs=True \<br />
input.dispatcher=cctbx.xfel.xtc_process \<br />
input.target=[[cxi02416-lyso-t001.phil]] input.trial=1 input.run_num=$i \<br />
input.cfg=cxi02416-calibdir.cfg; done<br />
<br />
To instruct psana to use the modified calibration directory, we add a psana config file named cxi02416-calibdir.cfg with these lines:<br />
<br />
[psana]<br />
calib-dir = <a subfolder in your home directory>/cxi02416/calib<br />
<br />
If you have write permissions to your geometry folder, you don't need to use this config file or make a copy of your calibration directory. We recommend saving the original metrology like we did in the above example, by renaming it to 0-end.data.v1, so you can return to it as needed.<br />
<br />
The [[cxi02416-lyso-t001.phil | input phil file]] is the same as before, with a slight change to the unit cell dimensions. The program '''cspad.detector_statistics''' (described below) reports an average unit cell after joint refinement, which we use to update our indexing cell target. <br />
<br />
Indexing with the new metrology yielded 5305 indexed images.<br />
<br />
== Refine metrology (v3) ==<br />
<br />
We now refine the metrology generated from the indexed images from trial 3 and call it metrology version 3 (v3). Now that we have a better idea of the unit cell, we add a phil file with a restraint for all the crystals to that unit cell. We also increase the number images used to 2000.<br />
<br />
bsub -q psanaq -o t001_2k.out '''cspad.cbf_metrology''' tag=t001_2k \<br />
[[cxi02416-refine.phil]] reflections=indexed ../r0*/001/out \<br />
n_subset=2000 split_dataset=True<br />
<br />
Now that we have a good estimate of the unit cell, we use [[cxi02416-refine.phil | a refinement phil file]], to add a restraint to the joint refinement such the refined unit cells for each crystal have a similar set of dimensions. This assumes the crystals are isomorphous. <br />
<br />
After refinement, use '''cxi.display_metrology''', '''dev.dials.plot_detector_shifts''', and/or '''cspad.detector_shifts''' to evaluate how much the tiles moved:<br />
<br />
'''cspad.detector_shifts''' \<br />
t000_1k_1_refined_experiments_level2.json t000_1k_1_refined_reflections_level2.pickle \<br />
t001_2k_1_refined_experiments_level2.json t001_2k_1_refined_reflections_level2.pickle<br />
<br />
The Detector XY column shows the detector moved 4.3 microns in the XY plane, the quadrants on averaged moved 10.2 +/- 5.8 microns and the 2x1 sensors moved 11.7 +/- 11.5 microns. The tile positions moved substantially less between v2 and v3 then they did between v1 and v2.<br />
<br />
== Evaluate metrology ==<br />
<br />
Iterative hierarchical joint refinement should proceed until convergence, meaning until rounds of indexing and refinement do not improve the model. '''cxi.display_metrology''', '''dev.dials.plot_detector_shfits''', and '''cspad.detector_shifts''' measure the magnitude of changes during refinement. An additional tool is available, '''cspad.detector_statistics''', that can be used to evaluate the precision of refinement, and the state of the cspad after refinement.<br />
<br />
'''cspad.detector_statistics''' tag=t001_2k<br />
<br />
Warning, this dumps a lot of output. The program examines the current directory for files from cspad.cbf_metrology. For each hierarchy level 0-3, the program creates three tables of statistics:<br />
<br />
1) Detector congruence. The two half datasets from each level of refinement are compared to each other, and agreement between the two independent refinement runs is reported.<br />
<br />
2) Detector statistics. Using the two half datasets as independent measurements, statistics about the cspad are reported, such as normal vector tilts and XYZ offsets. Weighted means and standard deviations are reported as well, and can be used to estimate the overall precision of refinement.<br />
<br />
3) RMSDs by detector number. For each of the half datasets, overall, radial and transverse RMSDs are shown.<br />
<br />
For this walkthrough, we will extract a few statistics only. Look for the third set of tables, delineated with <br />
********************************************************************************<br />
Showing statistics for detector at level 2 (sensors, I.E. 2x1s)<br />
********************************************************************************<br />
<br />
In the second table, under Detector statistics, a few of the columns are reproduced here (click expand to show the table):<br />
<div class="toccolours mw-collapsible mw-collapsed"><br />
{| class="wikitable"<br />
|-<br />
|Panel G ID<br />
|Dist (mm)<br />
|F Offset Sigma (microns)<br />
|S Offset Sigma (microns)<br />
|N refls<br />
|-<br />
|1<br />
|21.8<br />
|1.457<br />
|3.795<br />
|6916<br />
|-<br />
|9<br />
|23.4<br />
|3.055<br />
|1.747<br />
|6498<br />
|-<br />
|25<br />
|24.5<br />
|0.614<br />
|1.891<br />
|6428<br />
|-<br />
|17<br />
|26<br />
|1.007<br />
|6.704<br />
|5772<br />
|-<br />
|0<br />
|40.7<br />
|2.003<br />
|1.707<br />
|5712<br />
|-<br />
|8<br />
|41.1<br />
|2.507<br />
|7.437<br />
|5882<br />
|-<br />
|24<br />
|43.7<br />
|0.139<br />
|4.121<br />
|5934<br />
|-<br />
|16<br />
|44.3<br />
|3.088<br />
|5.562<br />
|5470<br />
|-<br />
|7<br />
|55.3<br />
|1.048<br />
|5.097<br />
|5221<br />
|-<br />
|31<br />
|57.5<br />
|4.456<br />
|2.402<br />
|4837<br />
|-<br />
|15<br />
|57.8<br />
|1.447<br />
|0.87<br />
|4768<br />
|-<br />
|23<br />
|59.7<br />
|1.39<br />
|1.814<br />
|5165<br />
|-<br />
|11<br />
|71.9<br />
|0.38<br />
|10.784<br />
|3203<br />
|-<br />
|3<br />
|73<br />
|9.648<br />
|1.552<br />
|3238<br />
|-<br />
|19<br />
|75.2<br />
|9.258<br />
|3.055<br />
|2610<br />
|-<br />
|6<br />
|76.3<br />
|1.85<br />
|9.05<br />
|2921<br />
|-<br />
|27<br />
|76.3<br />
|3.867<br />
|12.665<br />
|3329<br />
|-<br />
|2<br />
|77.6<br />
|11.894<br />
|1.966<br />
|2781<br />
|-<br />
|10<br />
|78<br />
|1.809<br />
|5.188<br />
|2537<br />
|-<br />
|30<br />
|78.4<br />
|1.374<br />
|4.348<br />
|2265<br />
|-<br />
|14<br />
|78.7<br />
|1.845<br />
|1.796<br />
|2187<br />
|-<br />
|22<br />
|80.4<br />
|0.119<br />
|2.188<br />
|2574<br />
|-<br />
|18<br />
|81<br />
|19.347<br />
|9.268<br />
|2256<br />
|-<br />
|26<br />
|81.1<br />
|3.51<br />
|15.827<br />
|2767<br />
|-<br />
|4<br />
|87.5<br />
|2.449<br />
|8.722<br />
|1844<br />
|-<br />
|12<br />
|89.6<br />
|23.074<br />
|14.689<br />
|1636<br />
|-<br />
|28<br />
|90.5<br />
|12.99<br />
|19.289<br />
|1615<br />
|-<br />
|20<br />
|91.8<br />
|2.404<br />
|0.28<br />
|1548<br />
|-<br />
|5<br />
|104.8<br />
|32.112<br />
|16.115<br />
|797<br />
|-<br />
|13<br />
|106<br />
|10.282<br />
|19.482<br />
|559<br />
|-<br />
|29<br />
|107.8<br />
|8.739<br />
|12.337<br />
|762<br />
|-<br />
|21<br />
|108.5<br />
|23.591<br />
|18.247<br />
|682<br />
|-<br />
|All<br />
|<br />
|3.786<br />
|5.281<br />
|<br />
|-<br />
|Mean<br />
|<br />
|<br />
|<br />
|3459.8<br />
|}<br />
</div><br />
<br />
Here are the meaning of the columns:<br />
* Panel G ID: here we are looking at the sensors, of which there are 32.<br />
* Dist (mm): distance from the sensor center to the beam center. The table is sorted by this number.<br />
* F Offset sigma (microns): weighted standard deviation of the two measurements of the sensor's fast coordinate.<br />
* S Offset sigma (microns): weighted standard deviation of the two measurements of the sensor's slow coordinate.<br />
* N refls: sum of the number of reflections recorded on the sensor between the two half dataset. Used as a weighting term.<br />
<br />
The weighted mean of the fast and slow offset sigmas is 3.8 and 5.3 microns, respectively. This measure of the precision of this refinement indicates it quite precise compared to the pixel size of the detector (110 microns). However, there are still several panels with high fast or slow offset sigmas. That, plus the fact that the number of indexed images increased while using v3 metrology implies that refinement has not converged.<br />
<br />
== Further indexing and refinement (v4) ==<br />
<br />
Reindexing the data using v3 and re-refining the data to create v4 metrology proceeds as described in the above steps. After generating v4, we evaluate it as described above and see this table:<br />
<div class="toccolours mw-collapsible mw-collapsed"><br />
{| class="wikitable"<br />
|-<br />
!PanelG ID<br />
!Dist (mm)<br />
!F Offset Sigma (microns)<br />
!S Offset Sigma (microns)<br />
!N refls<br />
|-<br />
|1<br />
|21.8<br />
|1.1<br />
|1.7<br />
|6731<br />
|-<br />
|9<br />
|23.4<br />
|0.2<br />
|0.1<br />
|6681<br />
|-<br />
|25<br />
|24.5<br />
|0.9<br />
|1.2<br />
|6321<br />
|-<br />
|17<br />
|26<br />
|1.1<br />
|2.5<br />
|6144<br />
|-<br />
|0<br />
|40.7<br />
|0.1<br />
|1.9<br />
|5533<br />
|-<br />
|8<br />
|41<br />
|1.7<br />
|4.7<br />
|5501<br />
|-<br />
|24<br />
|43.7<br />
|1.0<br />
|1.5<br />
|5576<br />
|-<br />
|16<br />
|44.3<br />
|4.7<br />
|1.3<br />
|5707<br />
|-<br />
|7<br />
|55.3<br />
|0.5<br />
|0.5<br />
|4764<br />
|-<br />
|31<br />
|57.5<br />
|2.0<br />
|1.6<br />
|4722<br />
|-<br />
|15<br />
|57.8<br />
|2.2<br />
|1.8<br />
|4804<br />
|-<br />
|23<br />
|59.7<br />
|3.7<br />
|4.1<br />
|4681<br />
|-<br />
|11<br />
|71.9<br />
|1.7<br />
|8.7<br />
|2887<br />
|-<br />
|3<br />
|73<br />
|16.1<br />
|0.1<br />
|3234<br />
|-<br />
|19<br />
|75.1<br />
|7.3<br />
|0.1<br />
|2984<br />
|-<br />
|6<br />
|76.3<br />
|2.4<br />
|1.1<br />
|2664<br />
|-<br />
|27<br />
|76.3<br />
|3.7<br />
|16.8<br />
|2903<br />
|-<br />
|2<br />
|77.6<br />
|3.1<br />
|4.1<br />
|2866<br />
|-<br />
|10<br />
|78<br />
|2.5<br />
|5.2<br />
|2388<br />
|-<br />
|30<br />
|78.4<br />
|1.1<br />
|0.6<br />
|2413<br />
|-<br />
|14<br />
|78.7<br />
|14.2<br />
|1.7<br />
|2460<br />
|-<br />
|22<br />
|80.4<br />
|2.3<br />
|2.0<br />
|2255<br />
|-<br />
|18<br />
|81<br />
|10.2<br />
|4.6<br />
|2348<br />
|-<br />
|26<br />
|81.1<br />
|4.8<br />
|21.8<br />
|2545<br />
|-<br />
|4<br />
|87.5<br />
|8.0<br />
|4.9<br />
|1832<br />
|-<br />
|12<br />
|89.6<br />
|11.8<br />
|2.3<br />
|1559<br />
|-<br />
|28<br />
|90.4<br />
|9.7<br />
|14.8<br />
|1497<br />
|-<br />
|20<br />
|91.8<br />
|19.3<br />
|17.7<br />
|1434<br />
|-<br />
|5<br />
|104.8<br />
|47.9<br />
|35.9<br />
|875<br />
|-<br />
|13<br />
|106.1<br />
|6.2<br />
|9.9<br />
|550<br />
|-<br />
|29<br />
|107.8<br />
|15.4<br />
|24.2<br />
|748<br />
|-<br />
|21<br />
|108.5<br />
|17.7<br />
|7.2<br />
|698<br />
|-<br />
|All<br />
|<br />
|4.0<br />
|3.9<br />
|<br />
|-<br />
|Mean<br />
|<br />
|<br />
|<br />
|3384.5<br />
|}<br />
</div><br />
<br />
The fast and slow offset sigmas are lower generally, but not substantially. 5475 images were indexed, an increase of 1% over v3 metrology. The change in tile position is minimal, as shown by '''cspad.detector_shifts''' (delta XY movements (microns): detector: 1.5, quadrants: 4.0+/2.3, sensors: 5.5+/-5.9). The metrology has likely converged.<br />
<br />
== Deploy metrology ==<br />
<br />
v4 metrology is now ready for use. Deploy it or use it directly for indexing as described in [[Cxi0216 calibration#Index using v2 metrology | indexing using v2 metrology]].</div>Aaron