Overview - Revision history

Aaron at 22:41, 18 August 2026

2026-08-18T22:41:51Z

← Older revision		Revision as of 22:41, 18 August 2026
Line 51:		Line 51:
	''cctbx.xfel'' collects processing parameters using its GUI, then when submitting jobs it writes these configuration settings to a phil file. These settings will include things such as thresholds for determining hits (number of spots on an image, spot brightness cutoff, etc.), unit cell targets for indexing, resolution cutoffs, and so forth.		''cctbx.xfel'' collects processing parameters using its GUI, then when submitting jobs it writes these configuration settings to a phil file. These settings will include things such as thresholds for determining hits (number of spots on an image, spot brightness cutoff, etc.), unit cell targets for indexing, resolution cutoffs, and so forth.

	Technical information regarding ''phil'': ~~<ref>~~https://cci.lbl.gov/docs/cctbx/doc_low_phil/~~</ref>~~		Technical information regarding ''phil'': [https://cci.lbl.gov/docs/cctbx/doc_low_phil/]

	Specific ''phil'' files used in this tutorial: ''[[phil]]''		Specific ''phil'' files used in this tutorial: ''[[phil]]''
Line 61:		Line 61:
	Introduction: [http://cctbx.sourceforge.net/current/]		Introduction: [http://cctbx.sourceforge.net/current/]
	Homepage: [http://cctbx.sourceforge.net/]		Homepage: [http://cctbx.sourceforge.net/]

	~~== References ==~~
	~~<references />~~

Aaron: /* References */

2026-08-18T22:40:23Z

References

← Older revision		Revision as of 22:40, 18 August 2026
Line 63:		Line 63:

	== References ==		== References ==
	<references/>		<references />

Aaron: /* Phil */

2026-08-18T22:40:06Z

Phil

← Older revision		Revision as of 22:40, 18 August 2026
Line 51:		Line 51:
	''cctbx.xfel'' collects processing parameters using its GUI, then when submitting jobs it writes these configuration settings to a phil file. These settings will include things such as thresholds for determining hits (number of spots on an image, spot brightness cutoff, etc.), unit cell targets for indexing, resolution cutoffs, and so forth.		''cctbx.xfel'' collects processing parameters using its GUI, then when submitting jobs it writes these configuration settings to a phil file. These settings will include things such as thresholds for determining hits (number of spots on an image, spot brightness cutoff, etc.), unit cell targets for indexing, resolution cutoffs, and so forth.

	Technical information regarding ''phil'': ~~[http~~://~~cctbx~~.~~sourceforge~~.~~net~~/~~libtbx_phil.html]~~		Technical information regarding ''phil'': <ref>https://cci.lbl.gov/docs/cctbx/doc_low_phil/</ref>

	Specific ''phil'' files used in this tutorial: ''[[phil]]''		Specific ''phil'' files used in this tutorial: ''[[phil]]''

Aaron at 22:38, 18 August 2026

2026-08-18T22:38:04Z

← Older revision		Revision as of 22:38, 18 August 2026
Line 5:		Line 5:
	The LCLS at full capacity operates at 120 Hz. The incident photon packets are delivered in ≈40 femtosecond wide pulses, each containing ≈10<sup>12</sup> photons. This high repetition rate and compact beam delivery time necessitated the construction of a new detector [http://www-public.slac.stanford.edu/sciDoc/docMeta.aspx?slacPubNumber=SLAC-PUB-15284 Hart, P The Cornell-SLAC Pixel Array Detector at LCLS. <i>SLAC Scientific Documents</i> (2012).], where the work of reading out and streaming recorded data at these high speeds is accomplished through the use of 64 sensors, arranged in a quadrangular pattern around a central hole (in the place of a beam stop). Each of the 4 quadrants, containing 16 of the sensors, was adjustable on rails radially away from the central hole to adjust the size of this hole.		The LCLS at full capacity operates at 120 Hz. The incident photon packets are delivered in ≈40 femtosecond wide pulses, each containing ≈10<sup>12</sup> photons. This high repetition rate and compact beam delivery time necessitated the construction of a new detector [http://www-public.slac.stanford.edu/sciDoc/docMeta.aspx?slacPubNumber=SLAC-PUB-15284 Hart, P The Cornell-SLAC Pixel Array Detector at LCLS. <i>SLAC Scientific Documents</i> (2012).], where the work of reading out and streaming recorded data at these high speeds is accomplished through the use of 64 sensors, arranged in a quadrangular pattern around a central hole (in the place of a beam stop). Each of the 4 quadrants, containing 16 of the sensors, was adjustable on rails radially away from the central hole to adjust the size of this hole.

	The CSPAD was replaced with two detectors: the Rayonix ~~[[MX340-XFEL]~~[https://www.rayonix.com/product/mx340-xfel/] (recently retired), and the ~~[Jungfrau 16M]~~[https://pmc.ncbi.nlm.nih.gov/articles/PMC7044001/] (in current use).		The CSPAD was replaced with two detectors: the Rayonix [https://www.rayonix.com/product/mx340-xfel/ MX340-XFEL] (recently retired), and the [https://pmc.ncbi.nlm.nih.gov/articles/PMC7044001/ Jungfrau 16M] (in current use).

	Indexing, predicting spot locations using a crystal orientation matrix, and integrating reflection intensities requires precise knowledge of the location of the sensors in these detectors in three-dimensional space. For this reason, a portion of this wiki describes the calibration and refinement of the tile metrology.		Indexing, predicting spot locations using a crystal orientation matrix, and integrating reflection intensities requires precise knowledge of the location of the sensors in these detectors in three-dimensional space. For this reason, a portion of this wiki describes the calibration and refinement of the tile metrology.
Line 27:		Line 27:
	Note that the ''cctbx.xfel'' GUI provides convenient ways to specify the parameters need to the user needs not create these locator files directly.		Note that the ''cctbx.xfel'' GUI provides convenient ways to specify the parameters need to the user needs not create these locator files directly.

	''psana'' itself is designed with computational parallelization in mind. As each image is independent, processing of each image can be done by separate computer cores. ~~[[MPI]~~[https://en.wikipedia.org/wiki/Message_Passing_Interface] is required, which is configured automatically in LCLS/psana/cctbx builds.		''psana'' itself is designed with computational parallelization in mind. As each image is independent, processing of each image can be done by separate computer cores. [https://en.wikipedia.org/wiki/Message_Passing_Interface MPI] is required, which is configured automatically in LCLS/psana/cctbx builds.

	More information about psana: [https://pswww.slac.stanford.edu/swdoc/releases/ana-current/psana-ref/html/psana/]		More information about psana: [https://pswww.slac.stanford.edu/swdoc/releases/ana-current/psana-ref/html/psana/]
Line 33:		Line 33:
	== S3DF ==		== S3DF ==

	SLAC provides access to the ~~[[SLAC Shared Science Data Facility (S3DF)]~~[https://isdci.slac.stanford.edu/initiative-areas/s3df]]. There are several pools of computers available:		SLAC provides access to the [https://isdci.slac.stanford.edu/initiative-areas/s3df SLAC Shared Science Data Facility (S3DF)]. There are several pools of computers available:

	* <code>s3dflogin</code>: login nodes, not used for processing or computation. Reach these using <code>ssh -YAC <username>@s3dflogin.slac.stanford.edu</code>		* <code>s3dflogin</code>: login nodes, not used for processing or computation. Reach these using <code>ssh -YAC <username>@s3dflogin.slac.stanford.edu</code>
Line 40:		Line 40:
	* Queues: generally users should use the <code>milano</code> queue for routine processing.		* Queues: generally users should use the <code>milano</code> queue for routine processing.

	Queing is done using ~~[[slurm]~~[https://slurm.schedmd.com/overview.html]]. Of note are these commands:		Queing is done using [https://slurm.schedmd.com/overview.html slurm]. Of note are these commands:
	* <code>sbatch</code>: used to submit jobs to the queuing system		* <code>sbatch</code>: used to submit jobs to the queuing system
	* <code>squeue -u <username></code>: used to list the jobs being run by the current user		* <code>squeue -u <username></code>: used to list the jobs being run by the current user
Line 61:		Line 61:
	Introduction: [http://cctbx.sourceforge.net/current/]		Introduction: [http://cctbx.sourceforge.net/current/]
	Homepage: [http://cctbx.sourceforge.net/]		Homepage: [http://cctbx.sourceforge.net/]

			== References ==
			<references/>

Aaron: /* The X-ray detector */

2026-08-18T22:33:26Z

The X-ray detector

← Older revision		Revision as of 22:33, 18 August 2026
Line 3:		Line 3:
	== The X-ray detector ==		== The X-ray detector ==
	[[File:750.png\|200px\|thumb\|right\|Example CSPAD image]]		[[File:750.png\|200px\|thumb\|right\|Example CSPAD image]]
	The LCLS at full capacity operates at 120 Hz. The incident photon packets are delivered in ≈40 femtosecond wide pulses, each containing ≈10<sup>12</sup> photons. This high repetition rate and compact beam delivery time necessitated the construction of a new detector~~<ref>~~[http://www-public.slac.stanford.edu/sciDoc/docMeta.aspx?slacPubNumber=SLAC-PUB-15284 Hart, P The Cornell-SLAC Pixel Array Detector at LCLS. <i>SLAC Scientific Documents</i> (2012).]~~</ref>~~, where the work of reading out and streaming recorded data at these high speeds is accomplished through the use of 64 sensors, arranged in a quadrangular pattern around a central hole (in the place of a beam stop). Each of the 4 quadrants, containing 16 of the sensors, was adjustable on rails radially away from the central hole to adjust the size of this hole.		The LCLS at full capacity operates at 120 Hz. The incident photon packets are delivered in ≈40 femtosecond wide pulses, each containing ≈10<sup>12</sup> photons. This high repetition rate and compact beam delivery time necessitated the construction of a new detector [http://www-public.slac.stanford.edu/sciDoc/docMeta.aspx?slacPubNumber=SLAC-PUB-15284 Hart, P The Cornell-SLAC Pixel Array Detector at LCLS. <i>SLAC Scientific Documents</i> (2012).], where the work of reading out and streaming recorded data at these high speeds is accomplished through the use of 64 sensors, arranged in a quadrangular pattern around a central hole (in the place of a beam stop). Each of the 4 quadrants, containing 16 of the sensors, was adjustable on rails radially away from the central hole to adjust the size of this hole.

	The CSPAD was replaced with two detectors: the Rayonix [[MX340-XFEL][https://www.rayonix.com/product/mx340-xfel/] (recently retired), and the [Jungfrau 16M][https://pmc.ncbi.nlm.nih.gov/articles/PMC7044001/] (in current use).		The CSPAD was replaced with two detectors: the Rayonix [[MX340-XFEL][https://www.rayonix.com/product/mx340-xfel/] (recently retired), and the [Jungfrau 16M][https://pmc.ncbi.nlm.nih.gov/articles/PMC7044001/] (in current use).

	Indexing, predicting spot locations using a crystal orientation matrix, and integrating reflection intensities requires precise knowledge of the location of the sensors in these detectors in three-dimensional space. For this reason, a portion of this wiki describes the calibration and refinement of the tile metrology.		Indexing, predicting spot locations using a crystal orientation matrix, and integrating reflection intensities requires precise knowledge of the location of the sensors in these detectors in three-dimensional space. For this reason, a portion of this wiki describes the calibration and refinement of the tile metrology.


	== ''psana'' ==		== ''psana'' ==

Aaron at 22:31, 18 August 2026

2026-08-18T22:31:58Z

← Older revision		Revision as of 22:31, 18 August 2026
Line 62:		Line 62:
	Introduction: [http://cctbx.sourceforge.net/current/]		Introduction: [http://cctbx.sourceforge.net/current/]
	Homepage: [http://cctbx.sourceforge.net/]		Homepage: [http://cctbx.sourceforge.net/]

	~~== References ==~~
	~~<references/>~~

Aaron: /* psana */

2026-08-18T22:30:30Z

psana

← Older revision		Revision as of 22:30, 18 August 2026
Line 24:		Line 24:
	detector=rayonix		detector=rayonix

	And the parameter <code>load_models=False makes reading and display the data faster. Further details about the detector or experiment can be provided in the locator.		And the parameter <code>load_models=False</code> makes reading and display the data faster. Further details about the detector or experiment can be provided in the locator.

	Note that the ''cctbx.xfel'' GUI provides convenient ways to specify the parameters need to the user needs not create these locator files directly.		Note that the ''cctbx.xfel'' GUI provides convenient ways to specify the parameters need to the user needs not create these locator files directly.

Aaron at 22:29, 18 August 2026

2026-08-18T22:29:53Z

← Older revision		Revision as of 22:29, 18 August 2026
Line 1:		Line 1:
	LCLS ~~provides several frameworks for~~ data ~~analysis~~, ~~and also provides methods for translating collected data to HDF5~~, ~~a general and portable container~~ format ~~for storage of large amounts of numerical~~ data. ''~~cctbx.xfel~~'' ~~extends~~ the LCLS ~~analysis packages with a set of analysis modules. In particular~~, the ''cctbx.xfel'' ~~analysis modules are run through ''pyana'', the Python-implementation of LCLS's analysis framework~~. Since analysis proceeds directly from the raw data, no intermediate conversion is necessary, and it can be done while an experiment is running.		Data collection and processing using LCLS involves recording data from an X-ray detector (Jungfrau, Rayonix, CSPAD, etc.) in XTC format, reading the data using ''DIALS'' libraries that wrap the LCLS-provided library ''psana'', processing on either the SLAC Shared Science Data Facility (S3DF) or NERSC, and coordinating the processing using ''cctbx.xfel'' itself. Since analysis proceeds directly from the raw data, no intermediate conversion is necessary, and it can be done while an experiment is running.

	~~''cctbx~~.~~xfel''~~ at ~~LCLS is built on five systems:~~ the ~~CSPAD~~ detector, ~~pyana~~, ~~LCLS’s queuing system~~, ~~phil~~, ~~and fundamentally~~, ~~cctbx~~.		== The X-ray detector ==
			[[File:750.png\|200px\|thumb\|right\|Example CSPAD image]]
			The LCLS at full capacity operates at 120 Hz. The incident photon packets are delivered in ≈40 femtosecond wide pulses, each containing ≈10<sup>12</sup> photons. This high repetition rate and compact beam delivery time necessitated the construction of a new detector<ref>[http://www-public.slac.stanford.edu/sciDoc/docMeta.aspx?slacPubNumber=SLAC-PUB-15284 Hart, P The Cornell-SLAC Pixel Array Detector at LCLS. <i>SLAC Scientific Documents</i> (2012).]</ref>, where the work of reading out and streaming recorded data at these high speeds is accomplished through the use of 64 sensors, arranged in a quadrangular pattern around a central hole (in the place of a beam stop). Each of the 4 quadrants, containing 16 of the sensors, was adjustable on rails radially away from the central hole to adjust the size of this hole.

	== The CSPAD ~~detector ==~~		The CSPAD was replaced with two detectors: the Rayonix [[MX340-XFEL][https://www.rayonix.com/product/mx340-xfel/] (recently retired), and the [Jungfrau 16M][https://pmc.ncbi.nlm.nih.gov/articles/PMC7044001/] (in current use).
	[[~~File~~:~~750~~.~~png\|200px\|thumb\|right\|Example CSPAD image~~]]
	~~The LCLS at full capacity operates at 120 Hz. The incident photon packets are delivered in ≈40 femtosecond wide pulses~~, ~~each containing ≈10<sup>12</sup> photons. This high repetition rate~~ and ~~compact beam delivery time necessitated~~ the ~~construction of a new detector<ref>~~[~~http~~://~~www-public~~.~~slac~~.~~stanford~~.~~edu~~/~~sciDoc~~/~~docMeta.aspx?slacPubNumber=SLAC-PUB-15284 Hart, P The Cornell-SLAC Pixel Array Detector at LCLS. <i>SLAC Scientific Documents<~~/~~i> (2012).~~]~~</ref>, where the work of reading out and streaming recorded data at these high speeds is accomplished through the use of 64 sensors, arranged in a quadrangular pattern around a central hole~~ (in ~~the place of a beam stop~~). ~~Each of the 4 quadrants, containing 16 of the sensors, is adjustable on rails radially away from the central hole to adjust the size of this hole.~~ Indexing, predicting spot locations using a crystal orientation matrix, and integrating reflection intensities requires precise knowledge of the location of these ~~sensors~~ in three-dimensional space. For this reason, a portion of this ~~tutorial~~ describes the calibration and refinement of the tile metrology.		Indexing, predicting spot locations using a crystal orientation matrix, and integrating reflection intensities requires precise knowledge of the location of the sensors in these detectors in three-dimensional space. For this reason, a portion of this wiki describes the calibration and refinement of the tile metrology.


	== ''psana'' ==		== ''psana'' ==

	The LCLS data acquisition systems stream the terabytes of diffraction data collected from the ~~CSPAD~~ detector to container files in XTC format. XTC is a linear, sequential-access file format, where individual images can be recorded rapidly by the file system as they are collected. The programmatic interface to interact with these files at LCLS is ''psana'', a C++/Python-based interface.		The LCLS data acquisition systems stream the terabytes of diffraction data collected from the X-ray detector to container files in XTC format. XTC is a linear, sequential-access file format, where individual images can be recorded rapidly by the file system as they are collected. The programmatic interface to interact with these files at LCLS is ''psana'', a C++/Python-based interface.

			''cctbx.xfel'' wraps access to XTC using small, custom text files called locators, bypassing the need for users to work with ''psana'' directly. For example, the ''DIALS'' image viewer can be used to view images using a command like this:

			dials.image_viewer run100.loc load_models=False

			Where <code>run100.loc</code> looks like this

	''psana'' is driven by configuration files to process frames individually, and is designed with computational parallelization in mind. As each image is independent, processing of each image can be done by separate computer cores. ''cctbx.xfel'' uses ''psana'' and ''psana'''s configuration files to read and process image files stored in XTC format. The user specifies how each image is to be processed in the configuration file, and the passes the configuration file and the path to the XTC streams of interest to ''cctbx.xfel'', which calls ''psana'' and submits the job to the queuing system.		experiment=mfx0000000
			run=100
			detector=rayonix

	~~For example, if~~ the ~~user wanted to filter an XTC stream for hits, index~~ the ~~hits and then integrate images which successfully indexed,~~ the ~~user would supply a configuration file which specified~~ ''cctbx.xfel'' ~~modules that did these tasks, provide options~~ to these ~~modules, and submit the job~~. ~~Specific details are in the tutorials.~~		And the parameter <code>load_models=False makes reading and display the data faster. Further details about the detector or experiment can be provided in the locator.

			Note that the ''cctbx.xfel'' GUI provides convenient ways to specify the parameters need to the user needs not create these locator files directly.

	~~During~~ processing~~, hits are extracted from the XTC stream and written to separate files for~~ each ~~individual~~ image. ~~At the moment these separate files are in a in a Python-programming language friendly format called pickle format~~. ~~However~~, ~~by the end of 2014, ''~~cctbx~~.xfel'' will be exclusively using CBF and HDF5 formats to output results~~.		''psana'' itself is designed with computational parallelization in mind. As each image is independent, processing of each image can be done by separate computer cores. [[MPI][https://en.wikipedia.org/wiki/Message_Passing_Interface] is required, which is configured automatically in LCLS/psana/cctbx builds.

	More information about psana: [https://~~confluence~~.slac.stanford.edu/~~display~~/~~PSDM~~/psana+-~~+Python+Script+Analysis+Manual~~]		More information about psana: [https://pswww.slac.stanford.edu/swdoc/releases/ana-current/psana-ref/html/psana/]

	== ~~LCLS queuing system~~ ==		== S3DF ==

	SLAC ~~maintains several computing clusters available~~ to ~~its users for processing data. While detailed knowledge of their workings isn't required for ''cctbx.xfel'' operation, an overview of these systems is provided here:~~ [https://~~confluence~~.slac.stanford.edu/~~display~~/~~PCDS/Computing~~]. ~~Specific commands for submitting ''cctbx.xfel'' jobs to the cluster~~ are ~~given in the tutorials.~~		SLAC provides access to the [[SLAC Shared Science Data Facility (S3DF)][https://isdci.slac.stanford.edu/initiative-areas/s3df]]. There are several pools of computers available:

	~~General instructions~~ for ~~submitting batch jobs can be found here: [https://confluence~~.slac.stanford.edu/~~display/PCDS/Submitting+Batch+Jobs]. Of note are these commands:~~		* <code>s3dflogin</code>: login nodes, not used for processing or computation. Reach these using <code>ssh -YAC <username>@s3dflogin.slac.stanford.edu</code>
	* ~~bsub~~: ~~used to submit jobs~~ to the ~~queuing system~~		* <code>psana</code>: not to be confused with the ''psana'' library, these are used for small computation/compilation and for running graphical programs.
	* ~~bjobs~~: ~~used to list~~ the ~~jobs being run by the current user~~		* <code>s3dfnx</code>: NoMachine gateway, use the full address <code>s3dfnx.slac.stanford.edu</code> with NoMachine for faster graphical displays.
	* ~~bkill~~: ~~used to stop a job that is running~~		* Queues: generally users should use the <code>milano</code> queue for routine processing.

	~~All of~~ these commands ~~have extensive man pages available at LCLS.~~		Queing is done using [[slurm][https://slurm.schedmd.com/overview.html]]. Of note are these commands:
			* <code>sbatch</code>: used to submit jobs to the queuing system
			* <code>squeue -u <username></code>: used to list the jobs being run by the current user
			* <code>scancel</code>: used to stop a job that is running

	== ''Phil'' ==		== ''Phil'' ==

	~~While ''psana'' is configured using its own configuration files,~~ ''cctbx.xfel'' ~~itself is~~ driven using Python-based hierarchical interchange language (''phil'') files, the same format that drives ~~Labelit~~ and ''PHENIX'' (though ''PHENIX'' calls them .eff files). The format is intuitive and allows easy specification of per-processing run parameters.		''cctbx.xfel'' provides a GUI but the programs it wraps are driven using Python-based hierarchical interchange language (''phil'') files, the same format that drives ''DIALS'' and ''PHENIX'' (though ''PHENIX'' calls them .eff files). The format is intuitive and allows easy specification of per-processing run parameters.

	~~A user's ''psana'' configuration file will have an entry called xfel_target. This entry will provide a phil filename that contains~~ ''cctbx.xfel'' configuration settings. These settings will include ~~thing~~ such as thresholds for determining hits (number of spots on an image, spot brightness cutoff, etc.), unit cell targets for indexing, resolution cutoffs, and so forth.		''cctbx.xfel'' collects processing parameters using its GUI, then when submitting jobs it writes these configuration settings to a phil file. These settings will include things such as thresholds for determining hits (number of spots on an image, spot brightness cutoff, etc.), unit cell targets for indexing, resolution cutoffs, and so forth.

	Technical information regarding ''phil'': [http://cctbx.sourceforge.net/libtbx_phil.html]		Technical information regarding ''phil'': [http://cctbx.sourceforge.net/libtbx_phil.html]

Aaron: /* Phil */

2014-12-12T00:25:23Z

Phil

← Older revision		Revision as of 00:25, 12 December 2014
Line 33:		Line 33:
	== ''Phil'' ==		== ''Phil'' ==

	While ''~~pyana~~'' is configured using its own configuration files, ''cctbx.xfel'' itself is driven using Python-based hierarchical interchange language (''phil'') files, the same format that drives Labelit and ''PHENIX'' (though ''PHENIX'' calls them .eff files). The format is intuitive and allows easy specification of per-processing run parameters.		While ''psana'' is configured using its own configuration files, ''cctbx.xfel'' itself is driven using Python-based hierarchical interchange language (''phil'') files, the same format that drives Labelit and ''PHENIX'' (though ''PHENIX'' calls them .eff files). The format is intuitive and allows easy specification of per-processing run parameters.

	A user's ''~~pyana~~'' configuration file will have an entry called xfel_target. This entry will provide a phil filename that contains ''cctbx.xfel'' configuration settings. These settings will include thing such as thresholds for determining hits (number of spots on an image, spot brightness cutoff, etc.), unit cell targets for indexing, resolution cutoffs, and so forth.		A user's ''psana'' configuration file will have an entry called xfel_target. This entry will provide a phil filename that contains ''cctbx.xfel'' configuration settings. These settings will include thing such as thresholds for determining hits (number of spots on an image, spot brightness cutoff, etc.), unit cell targets for indexing, resolution cutoffs, and so forth.

	Technical information regarding ''phil'': [http://cctbx.sourceforge.net/libtbx_phil.html]		Technical information regarding ''phil'': [http://cctbx.sourceforge.net/libtbx_phil.html]

Aaron: /* pyana */ changed to psana

2014-12-12T00:24:41Z

pyana: changed to psana

← Older revision		Revision as of 00:24, 12 December 2014
Line 8:		Line 8:


	== ''~~pyana~~'' ==		== ''psana'' ==

	The LCLS data acquisition systems stream the terabytes of diffraction data collected from the CSPAD detector to container files in XTC format. XTC is a linear, sequential-access file format, where individual images can be recorded rapidly by the file system as they are collected. The programmatic ~~interfaces~~ to interact with these files at LCLS ~~are ''psana'' and ''pyana''.~~ ''psana'' is a C++ ~~interface. ''cctbx.xfel'' uses the~~ Python-based ~~pyana~~.		The LCLS data acquisition systems stream the terabytes of diffraction data collected from the CSPAD detector to container files in XTC format. XTC is a linear, sequential-access file format, where individual images can be recorded rapidly by the file system as they are collected. The programmatic interface to interact with these files at LCLS is ''psana'', a C++/Python-based interface.

	''~~pyana~~'' is driven by configuration files to process frames individually, and is designed with computational parallelization in mind. As each image is independent, processing of each image can be done by separate computer cores. ''cctbx.xfel'' uses ''~~pyana~~'' and ''~~pyana~~'''s configuration files to read and process image files stored in XTC format. The user specifies how each image is to be processed in the configuration file, and the passes the configuration file and the path to the XTC streams of interest to ''cctbx.xfel'', which calls ''~~pyana~~'' and submits the job to the queuing system.		''psana'' is driven by configuration files to process frames individually, and is designed with computational parallelization in mind. As each image is independent, processing of each image can be done by separate computer cores. ''cctbx.xfel'' uses ''psana'' and ''psana'''s configuration files to read and process image files stored in XTC format. The user specifies how each image is to be processed in the configuration file, and the passes the configuration file and the path to the XTC streams of interest to ''cctbx.xfel'', which calls ''psana'' and submits the job to the queuing system.

	For example, if the user wanted to filter an XTC stream for hits, index the hits and then integrate images which successfully indexed, the user would supply a configuration file which specified ''cctbx.xfel'' modules that did these tasks, provide options to these modules, and submit the job. Specific details are in the tutorials.		For example, if the user wanted to filter an XTC stream for hits, index the hits and then integrate images which successfully indexed, the user would supply a configuration file which specified ''cctbx.xfel'' modules that did these tasks, provide options to these modules, and submit the job. Specific details are in the tutorials.
Line 18:		Line 18:
	During processing, hits are extracted from the XTC stream and written to separate files for each individual image. At the moment these separate files are in a in a Python-programming language friendly format called pickle format. However, by the end of 2014, ''cctbx.xfel'' will be exclusively using CBF and HDF5 formats to output results.		During processing, hits are extracted from the XTC stream and written to separate files for each individual image. At the moment these separate files are in a in a Python-programming language friendly format called pickle format. However, by the end of 2014, ''cctbx.xfel'' will be exclusively using CBF and HDF5 formats to output results.

	More information about ~~pyana~~: [~~http~~://~~www-public~~.slac.stanford.edu/~~sciDoc~~/~~docMeta.aspx?slacPubNumber=SLAC-PUB~~-~~15284~~]		More information about psana: [https://confluence.slac.stanford.edu/display/PSDM/psana+-+Python+Script+Analysis+Manual]

	== LCLS queuing system ==		== LCLS queuing system ==