cctbx_xfel - Recent changes [en]

Main Page

Aaron — Wed, 15 Jul 2026 21:14:25 GMT

← Older revision		Revision as of 21:14, 15 July 2026
Line 1:		Line 1:
	News! See our [[https://doi.org/10.~~1107~~/~~S2059798318009191~~ new ~~paper~~]]! This paper shows current best practices using ''cctbx.xfel'' with ''DIALS'' for serial crystallographic data~~. Also see the [https://github.com/phyy-nx/dials_refinement_brewster2018/wiki companion set of tutorials].~~		News! See our [[https://doi.org/10.1101/2025.05.04.652045 new preprint]]! This paper shows current best practices using ''cctbx.xfel'' with ''DIALS'' for serial crystallographic data.

	~~News! See our [[http://cci.lbl.gov/publications/download/CCN_2019_p22_Brewster.pdf newsletter article]] that describes the XFEL GUI, an interactive program for handling large XFEL datasets~~.

	= Open-source tools for serial crystallographic data processing =		= Open-source tools for serial crystallographic data processing =

	''cctbx.xfel'' is a suite of software tools designed to process diffraction data from serial femtosecond crystallography (SFX) measurements at an X-ray free-electron laser (XFEL) or a synchrotron. Built on the Computational Crystallographic Toolbox ([~~http~~://cctbx.~~sourceforge~~.~~net~~ ''cctbx'']), the same toolbox on which [http://www.phenix-online.org ''PHENIX''], [http://dials.github.io DIALS~~], [http://adder.lbl.gov/labelit ''LABELIT''], and post-refinement and merging program, [http://viper.lbl.gov/cctbx.xfel/index.php/Cctbx.prime ''PRIME''~~] are built, it enables the user to solve difficult problems relating to processing serial crystallographic data. The programs and modules provided by ''cctbx.xfel'' can reduce a large set of still diffraction images recorded at Stanford’s Linac Coherent Light Source ([http://lcls.slac.stanford.edu LCLS]), [http://sacla.xfel.jp SACLA], the [https://www.xfel.eu/ European XFEL], or a synchrotron, to a single MTZ file containing merged reflection intensities suitable for structure solution.		''cctbx.xfel'' is a suite of software tools designed to process diffraction data from serial femtosecond crystallography (SFX) measurements at an X-ray free-electron laser (XFEL) or a synchrotron. Built on the Computational Crystallographic Toolbox ([https://cctbx.github.io/ ''cctbx'']), the same toolbox on which [http://www.phenix-online.org ''PHENIX''], and [http://dials.github.io DIALS] are built, it enables the user to solve difficult problems relating to processing serial crystallographic data. The programs and modules provided by ''cctbx.xfel'' can reduce a large set of still diffraction images recorded at Stanford’s Linac Coherent Light Source ([http://lcls.slac.stanford.edu LCLS]), [http://sacla.xfel.jp SACLA], the [https://www.xfel.eu/ European XFEL], or a synchrotron, to a single MTZ file containing merged reflection intensities suitable for structure solution.

	== ''cctbx.xfel'' resources ==		== ''cctbx.xfel'' resources ==

	The ~~tutorials on this wiki provide detailed instructions~~ for ~~indexing~~ and ~~integrating still diffraction images extracted from~~ the ~~raw data streams recorded at the LCLS, including pre-processing steps such as dark pedestal generation~~ and ~~refinement~~ of the ~~detector geometry~~ of the ~~Cornell–SLAC pixel array detectors~~ (~~CSPAD~~) ~~in use at the CXI~~ and ~~XPP end stations~~. ~~The tutorials also cover tools~~ to ~~efficiently leverage~~ the ~~LCLS computing cluster to process the thousands~~ to ~~millions of diffraction images that can be recorded in~~ a ~~short time~~. ~~Finally~~, ~~processing approaches~~ for ~~data collection at non-LCLS sources are also provided~~.		The current best resource is our pre-print, [https://doi.org/10.1101/2025.05.04.652045 ''cctbx.xfel: a suite for processing serial crystallographic data.'']. This paper describes the algorithms and ideas behind the implementation and execution of the two main programs of ''cctbx.xfel'':

			* ''dials.stills_process'': this program runs through all the first steps of processing data using dials: import (reading image headers), spot finding, indexing, geometry refinement, and integration. Serial images are independent from each other, so this program is optimized for massively parallel systems, but can be adapted to a variety of single and multi-processing modes.
			* ''cctbx.xfel.merge'': this program runs through the remaining steps of processing: scaling and merging to produce a full dataset (.mtz file). For documentation, see our [https://github.com/cctbx/cctbx_project/tree/master/xfel/merging Github project]

			Additional resources:

			* Our [http://cci.lbl.gov/publications/download/CCN_2019_p22_Brewster.pdf 2019 newsletter article] describes how to use the ''cctbx.xfel'' GUI, an interactive program for handling large XFEL datasets.

	* [[Overview]] to the system architecture at LCLS and real-time progress monitoring of data processing		* [[Overview]] to the system architecture at LCLS and real-time progress monitoring of data processing
Line 17:		Line 22:
	** Build a psana/cctbx.xfel build [https://exafel.github.io/docs/psana-cctbx-install from scratch]. Use this option if you want to process LCLS data outside of LCLS or want your own build.		** Build a psana/cctbx.xfel build [https://exafel.github.io/docs/psana-cctbx-install from scratch]. Use this option if you want to process LCLS data outside of LCLS or want your own build.
	* [[Tutorials]] on pre-processing, data reduction, and merging.		* [[Tutorials]] on pre-processing, data reduction, and merging.
	* ''[[cctbx.prime]]'': tutorials on post-refinement using PRIME.
	* ''[[IOTA]]'': tutorial on spotfinding optimization using ''IOTA''.

	Other related information:		Other related information:

	* [[Serial XFEL Crystallography References]]		* [[Serial XFEL Crystallography References]]
	* [[Data Processing at SACLA]]
	* [[2017 Tutorials]]. Berkeley Lab Serial Crystallography Workshop, 16-17 Feb 2017.
	* [[Experiment-day guidelines]]: a short, very hands-on set of notes for online monitoring and data-processing using cctbx.xfel.
	* [[File formats]]: for developers, information on file formats ''cctbx.xfel'' uses
	* [[Cppxfel]]: XFEL data processing with the CPPXFEL package from Diamond/Wellcome Trust.
	* [[Ha14 documentation]]: tutorials and support for processing using the [https://doi.org/10.1038/nmeth.2887 Hattne 2014] methods.

	This project is under active development. For any assistance, please contact the authors.		This project is under active development. For any assistance, please contact the authors.

Serial XFEL Crystallography References

Aaron — Wed, 15 Jul 2026 20:51:32 GMT

← Older revision		Revision as of 20:51, 15 July 2026
Line 1:		Line 1:
	Following is a list of references useful for learning about serial XFEL crystallography in general, and ''cctbx.xfel'' in particular.		Following is a list of references useful for learning about serial XFEL crystallography in general, and ''cctbx.xfel'' in particular.

	* Uervirojnangkoorn M, ''et. al.'' (2015): [http://elifesciences.org/content/4/e05421 Enabling X-ray free electron laser crystallography for challenging biological systems from a limited number of crystals.] ''eLife'' '''4~~(March)~~''', e05421.		* Brewster AS ''et al.'' (2025): [https://doi.org/10.1101/2025.05.04.652045 cctbx.xfel: a suite for processing serial crystallographic data.] ''Preprint''.

			* Mittan-Moreau DW ''et al.'' (2025): [https://doi.org/10.1107/S2059798325002852 Robust error calibration for serial crystallography.] ''Acta Crystallogr D Struct Biol'' '''81''', 265-275.

			* Blaschke JP ''et al.'' (2024): [https://doi.org/10.3389/fhpcp.2024.1414569 ExaFEL: extreme-scale real-time data processing for X-ray free electron laser science.] ''Frontiers in High Performance Computing'' '''2'''.

			* Blaschke JP ''et al.'' (2024): [https://doi.org/10.1002/cpe.8019 Real‐time XFEL data analysis at SLAC and NERSC: A trial run of nascent exascale experimental data analysis.] ''Concurrency and Computation: Practice and Experience'' '''36'''(12):e8019.

			* Schriber EA, Paley DW, ''et al.'' (2022): [https://dx.doi.org/10.1038/s41586-021-04218-3 Chemical crystallography by serial femtosecond X-ray diffraction.] ''Nature'' '''601''', 360-365.

			* Butryn A ''et al.'' (2021): [https://dx.doi.org/10.1038/s41467-021-24757-7 An on-demand, drop-on-drop method for studying enzyme catalysis by serial crystallography.] ''Nat Commun'' '''12''', 4461.

			* Brewster AS ''et al.'' (2019): [http://www.phenix-online.org/newsletter Processing serial crystallographic data from XFELs or synchrotrons using the cctbx.xfel GUI.] ''Computational Crystallography Newsletter'' '''10''', 22-39.

			* Brewster AS, ''et al.'' (2019): [https://doi.org/10.1107/S2059798319012877 SAD phasing of XFEL data depends critically on the error model.] ''Acta Crystallogr D Struct Biol'' '''75''', 959-968.

			* Brewster AS, ''et al.'' (2018): [https://doi.org/10.1107/S2059798318009191 Improving signal strength in serial crystallography with DIALS geometry refinement]. ''Acta Crystallogr D Struct Biol'' '''74''', 877-894.

			* Winter G, ''et al.'' (2018): [https://doi.org/10.1107/S2059798317017235 DIALS: implementation and evaluation of a new integration package] ''Acta Crystallogr D Struct Biol'' '''7''4, 85-97.

			* Uervirojnangkoorn M, ''et al.'' (2015): [http://elifesciences.org/content/4/e05421 Enabling X-ray free electron laser crystallography for challenging biological systems from a limited number of crystals.] ''eLife'' '''4''', e05421.

	* Sauter NK, Hattne J, Brewster AS, Echols N, Zwart PH, Adams PD (2014): [http://dx.doi.org/10.1107/S1399004714024134 Improved crystal orientation and physical properties from single-shot XFEL stills.] ''Acta Crystallographica Section D: Biological. Crystallography D'' '''70''', 3299-309. [PMID:25478847] [PMC reprint: [http://ncbi.nlm.nih.gov/pmc/articles/PMC4257623/ PMC4257623] [http://cci.lbl.gov/publications/download/Mosaicity_wa5077.pdf (Reprint)]		* Sauter NK, Hattne J, Brewster AS, Echols N, Zwart PH, Adams PD (2014): [http://dx.doi.org/10.1107/S1399004714024134 Improved crystal orientation and physical properties from single-shot XFEL stills.] ''Acta Crystallographica Section D: Biological. Crystallography D'' '''70''', 3299-309. [PMID:25478847] [PMC reprint: [http://ncbi.nlm.nih.gov/pmc/articles/PMC4257623/ PMC4257623] [http://cci.lbl.gov/publications/download/Mosaicity_wa5077.pdf (Reprint)]

	* Sauter NK, ''et. al.'' (2013): [http://dx.doi.org/10.1107/S0907444913000863 New Python-based methods for data processing.] ''Acta Crystallographica Section D: Biological Crystallography D'' '''69''', 1274-1282. [http://cci.lbl.gov/publications/download/ba5193_reprint.pdf (Reprint)]		* Sauter NK, ''et al.'' (2013): [http://dx.doi.org/10.1107/S0907444913000863 New Python-based methods for data processing.] ''Acta Crystallographica Section D: Biological Crystallography D'' '''69''', 1274-1282. [http://cci.lbl.gov/publications/download/ba5193_reprint.pdf (Reprint)]

	* Kern J, ''et. al.'' (2013): [http://www.sciencemag.org/content/early/2013/02/13/science.1234273 Simultaneous Femtosecond X-ray Spectroscopy and Diffraction of Photosystem II at Room Temperature.] ''Science'' '''340''', 491-495.		* Kern J, ''et al.'' (2013): [http://www.sciencemag.org/content/early/2013/02/13/science.1234273 Simultaneous Femtosecond X-ray Spectroscopy and Diffraction of Photosystem II at Room Temperature.] ''Science'' '''340''', 491-495.

	* Kern J, ''et. al.'' (2012): [http://www.pnas.org/content/early/2012/05/31/1204598109.abstract Room temperature femtosecond X-ray diffraction of photosytem II microcrystals]. ''Proc. Natl. Acad. Sci.'' USA '''109''', 9721-9726. [http://cci.lbl.gov/publications/download/PNAS-2012-Kern-9721-6.pdf (Reprint)]		* Kern J, ''et. al.'' (2012): [http://www.pnas.org/content/early/2012/05/31/1204598109.abstract Room temperature femtosecond X-ray diffraction of photosytem II microcrystals]. ''Proc. Natl. Acad. Sci.'' USA '''109''', 9721-9726. [http://cci.lbl.gov/publications/download/PNAS-2012-Kern-9721-6.pdf (Reprint)]

	* Sierra RG ''et. al.'' (2012): [http://dx.doi.org/10.1107/S0907444912038152 Nanoflow electrospinning serial femtosecond crystallography]. ''Acta Cryst. D'' '''68''', 1584-1587. [http://cci.lbl.gov/publications/download/lv5021.pdf (Reprint)]		* Sierra RG ''et al.'' (2012): [http://dx.doi.org/10.1107/S0907444912038152 Nanoflow electrospinning serial femtosecond crystallography]. ''Acta Cryst. D'' '''68''', 1584-1587. [http://cci.lbl.gov/publications/download/lv5021.pdf (Reprint)]

	* Alonso-Mori R, ''et. al.'' (2012): [http://dx.doi.org/10.1073/pnas.1211384109Energy-dispersive X-ray emission spectroscopy using an X-ray free electron laser in a shot-by-shot mode]. ''Proc. Natl. Acad. Sci.'' USA '''109''', 19103-19107. [http://cci.lbl.gov/publications/download/PNAS-2012-Alonso-Mori-19103-7.pdf (Reprint)]		* Alonso-Mori R, ''et al.'' (2012): [http://dx.doi.org/10.1073/pnas.1211384109Energy-dispersive X-ray emission spectroscopy using an X-ray free electron laser in a shot-by-shot mode]. ''Proc. Natl. Acad. Sci.'' USA '''109''', 19103-19107. [http://cci.lbl.gov/publications/download/PNAS-2012-Alonso-Mori-19103-7.pdf (Reprint)]

	* H. N. ~~Chapmanet~~ ''et. al.'' (2011): [http://www.ncbi.nlm.nih.gov/pubmed/21293373 Femtosecond x-ray protein nanocrystallography]. ''Nature'' '''470''', 73.		* H. N. Chapman ''et al.'' (2011): [http://www.ncbi.nlm.nih.gov/pubmed/21293373 Femtosecond x-ray protein nanocrystallography]. ''Nature'' '''470''', 73.

Template:Archived

Aaron — Wed, 15 Jul 2026 20:24:28 GMT

Created page with "This page has been archived."

New page

This page has been archived.