File formats - Revision history

Aaron at 23:10, 6 November 2018

2018-11-06T23:10:13Z

← Older revision		Revision as of 23:10, 6 November 2018
Line 1:		Line 1:
	~~This page is under construction~~

	== Introduction ==		== Introduction ==

Aaron: /* A note on coordinate systems */

2017-07-21T19:22:48Z

A note on coordinate systems

← Older revision		Revision as of 19:22, 21 July 2017
Line 61:		Line 61:
	=== A note on coordinate systems ===		=== A note on coordinate systems ===

	Internally, our software uses the ImageCIF coordinate system with the exception that no goniometer is typically available at XFELs, therefore Z is along the beam (source ~~to sample~~), Y points against gravity and X completes a right-handed coordinate system. In the image pickles, the origin of the detector pixel readout is assumed to be in the upper left of the image, therefore the beam center is specified as positive offsets from that point. In example above, assuming the sample is at the laboratory origin, the vector pointing to the beam center on the detector in the ImageCIF coordinate system would be (0, 0, -119.002). A vector from the laboratory origin to the pixel readout origin (upper-left of the detector) would be -97.075, 97.075, -119.002). dxtbx.print_header is a useful tool to get these vectors from any file format understood by ''cctbx.xfel''. For more information see [http://cci.lbl.gov/publications/download/jo5001_reprint.pdf Parkhurst et al, 2014].		Internally, our software uses the ImageCIF coordinate system with the exception that no goniometer is typically available at XFELs, therefore Z is along the beam (sample to source), Y points against gravity and X completes a right-handed coordinate system. In the image pickles, the origin of the detector pixel readout is assumed to be in the upper left of the image, therefore the beam center is specified as positive offsets from that point. In example above, assuming the sample is at the laboratory origin, the vector pointing to the beam center on the detector in the ImageCIF coordinate system would be (0, 0, -119.002). A vector from the laboratory origin to the pixel readout origin (upper-left of the detector) would be -97.075, 97.075, -119.002). dxtbx.print_header is a useful tool to get these vectors from any file format understood by ''cctbx.xfel''. For more information see [http://cci.lbl.gov/publications/download/jo5001_reprint.pdf Parkhurst et al, 2014].

	=== Creating image pickles from xtc streams ===		=== Creating image pickles from xtc streams ===

Aaron: /* Image pickles */

2016-11-03T00:17:39Z

Image pickles

Aaron: /* Creating image pickles from xtc streams */

2016-11-01T22:40:08Z

Creating image pickles from xtc streams

← Older revision		Revision as of 22:40, 1 November 2016
Line 61:		Line 61:
	=== Creating image pickles from xtc streams ===		=== Creating image pickles from xtc streams ===

	The program cctbx.xfel.xtc_dump can be used to convert xtc streams to image pickle format. Here we provide an example for the Rayonix detector that is often found at XPP and MFX.		The program cctbx.xfel.xtc_dump can be used to convert xtc streams to image pickle format. Here we provide an example for the Rayonix detector that is often found at XPP and MFX. The psana module mod_image_dict is used to read the data and generate the images (example: [[mod_image_dict\|mod_image_dict.cfg]]). After setting up a release directory as described in [[Setup]]:
	~~<ol>~~
	~~<li>Set up a release directory as described in [[Setup]]~~		cd ~/myrelease # or wherever your release is
	~~<li>~~The psana module mod_image_dict is used to read the data and generate the images~~. An~~ example: [[mod_image_dict\|mod_image_dict.cfg]]		sit_setup
	~~<li>~~cd /~~path/to/test/~~release		cctbx.xfel.xtc_dump experiment=mfxm9316 run_num=14 address="MfxEndstation-0\|Rayonix-0" file_format=pickle cfg=mod_image_dict.cfg
	~~<li>~~sit_setup
	~~<li>~~cctbx.xfel.xtc_dump experiment=mfxm9316 run_num=14 address="MfxEndstation-0\|Rayonix-0" file_format=pickle cfg=mod_image_dict.cfg
	~~</ol>~~
	This will fill your working folder with image pickles. You can change your output directory with output.output_dir and you can limit the number of images saved with dispatch.max_events.		This will fill your working folder with image pickles. You can change your output directory with output.output_dir and you can limit the number of images saved with dispatch.max_events.

Aaron: /* CSPAD CBFs */

2016-11-01T22:34:32Z

CSPAD CBFs

← Older revision		Revision as of 22:34, 1 November 2016
Line 79:		Line 79:
	== CSPAD CBFs ==		== CSPAD CBFs ==

	CSPAD CBFs are ~~not~~ used ~~directly by~~ ''cctbx.xfel'' ~~at this time. This aspect of the project is under development~~. The complete specification for CSPAD CBFs is laid out in an article in the Computational Crystallographic Newsletter: "XFEL Detectors and ImageCIF", Computational Crystallography Newsletter 5, 19-24. ([http://cci.lbl.gov/publications/download/CCN_2014_p19.pdf Reprint]). CSPAD ~~CBFs~~		CSPAD CBFs are used while processing CSPAD images using the DIALS backend of ''cctbx.xfel''. The complete specification for CSPAD CBFs is laid out in an article in the Computational Crystallographic Newsletter: "XFEL Detectors and ImageCIF", Computational Crystallography Newsletter 5, 19-24. ([http://cci.lbl.gov/publications/download/CCN_2014_p19.pdf Reprint]). Documentation for using DIALS to index and integrate CSPAD data can be found in an article in the Computational Crystallographic Newsletter: "Processing XFEL data with cctbx.xfel and DIALS", Computational Crystallography Newsletter 7, 32-53 ([http://www.phenix-online.org/newsletter/CCN_2016_07.pdf#page=18 Reprint]).

	=== Creating CSPAD CBFs from XTC streams ===		=== Creating CSPAD CBFs from XTC streams ===

Aaron: /* Creating image pickles from xtc streams */

2016-11-01T22:27:00Z

Creating image pickles from xtc streams

← Older revision		Revision as of 22:27, 1 November 2016
Line 64:		Line 64:
	<ol>		<ol>
	<li>Set up a release directory as described in [[Setup]]		<li>Set up a release directory as described in [[Setup]]
	<li>The psana module mod_image_dict is used to read the data and generate the images. An example: [[mod_image_dict.cfg]]		<li>The psana module mod_image_dict is used to read the data and generate the images. An example: [[mod_image_dict\|mod_image_dict.cfg]]
	<li>cd /path/to/test/release		<li>cd /path/to/test/release
	<li>sit_setup		<li>sit_setup

Aaron: /* Integration pickles */

2016-11-01T22:25:26Z

Integration pickles

← Older revision		Revision as of 22:25, 1 November 2016
Line 75:		Line 75:
	== Integration pickles ==		== Integration pickles ==

	Integration pickles are the output of cctbx.xfel indexing and integration. They are like image pickles in that they are serialized python dictionaries and can be inspected with '''cxi.print_pickle'''. They are the primary input to '''cxi.merge''' and '''prime.~~postrefine~~'''.		Integration pickles are the output of cctbx.xfel indexing and integration. They are like image pickles in that they are serialized python dictionaries and can be inspected with '''cxi.print_pickle'''. They are the primary input to '''cxi.merge''' and '''prime.run'''.

	== CSPAD CBFs ==		== CSPAD CBFs ==

Aaron: /* Image pickles */

2016-11-01T22:25:05Z

Image pickles

@@ Line 58: / Line 58: @@
 * 64 active areas: the active areas are coordinates that specify where the 64 tiles are in the image.  They are in the form of quartets of numbers [x1,y1,x2,y2], one quartet for each of the 64 tiles.  The first active area is shown.  Note, for non-CSPAD image pickles, there will only be one active area and it will be the size of the image entire.
 * 64 translated active areas: if corrections were available for this image, the first corrected active area is shown here.
 == Integration pickles ==

Aaron: /* Integration pickles */

2016-03-30T22:06:23Z

Integration pickles

← Older revision		Revision as of 22:06, 30 March 2016
Line 61:		Line 61:
	== Integration pickles ==		== Integration pickles ==

	~~Coming soon~~		Integration pickles are the output of cctbx.xfel indexing and integration. They are like image pickles in that they are serialized python dictionaries and can be inspected with '''cxi.print_pickle'''. They are the primary input to '''cxi.merge''' and '''prime.postrefine'''.

	== CSPAD CBFs ==		== CSPAD CBFs ==

Aaron: /* Image pickles */

2016-03-30T22:03:43Z

Image pickles

@@ Line 12: / Line 12: @@
 == Image pickles ==
-Coming soon
+''cctbx.xfel'' image pickles are a binary file format containing pixel data and image metadata.  Under the hood these are serialized python dictionaries, but for general users it's sufficient to know that the blob of data consists of name/value pairs.  The contents of an image pickle can be inspected with this command:
 == Integration pickles ==

← Older revision		Revision as of 00:17, 3 November 2016
Line 45:		Line 45:
	Each of the lines here is a name/value pair of data or metadata for the image. LABELIT, DIALS, cctbx.image_viewer, are all programs capable of processing these files directly. Here's a description of each of the name/value pairs:		Each of the lines here is a name/value pair of data or metadata for the image. LABELIT, DIALS, cctbx.image_viewer, are all programs capable of processing these files directly. Here's a description of each of the name/value pairs:
	* Detector format version: CXI 10.1. Technically this value is not stored in the image pickle. The detector format version is used internally by cctbx.xfel for data collected up to LCLS run 11, and is a way of looking up tile corrections stored in the software. After run 11, this information should be stored in the phil file used to processes the data. The detector format version is looked up based on the detector address and event timestamp. All known detector format versions can be displayed with the command '''cxi.detector_format_versions'''.		* Detector format version: CXI 10.1. Technically this value is not stored in the image pickle. The detector format version is used internally by cctbx.xfel for data collected up to LCLS run 11, and is a way of looking up tile corrections stored in the software. After run 11, this information should be stored in the phil file used to processes the data. The detector format version is looked up based on the detector address and event timestamp. All known detector format versions can be displayed with the command '''cxi.detector_format_versions'''.
	* DISTANCE: detector distance		* DISTANCE: detector distance (mm)
	* DETECTOR_ADDRESS: string to identify the detector in the XTC stream. Recorded here as a unique identifier for the data when coupled with the event timestamp.		* DETECTOR_ADDRESS: string to identify the detector in the XTC stream. Recorded here as a unique identifier for the data when coupled with the event timestamp.
	* SIZE1 and SIZE2: image dimensions		* SIZE1 and SIZE2: image dimensions (pixels)
	* TIMESTAMP: unique time stamp for this event in the form year-month-day'''T'''hour:minute'''Z'''second.microsecond		* TIMESTAMP: unique time stamp for this event in the form year-month-day'''T'''hour:minute'''Z'''second.microsecond
	* CCD_IMAGE_SATURATION and SATURATED_VALUE: overload value		* CCD_IMAGE_SATURATION and SATURATED_VALUE: overload value
	* PIXEL_SIZE: pixel size in mm		* PIXEL_SIZE: pixel size in mm
	* BEAM_CENTER_X, BEAM_CENTER_Y: beam center		* BEAM_CENTER_X, BEAM_CENTER_Y: beam center, specified as positive mm offsets from the origin of the detector (upper-left corner)
	* MIN_TRUSTED_VALUE: underload		* MIN_TRUSTED_VALUE: underload
	* WAVELENGTH: wavelength as recorded in the XTC stream		* WAVELENGTH: wavelength as recorded in the XTC stream (angstroms)
	* DATA: pixel data. Some vital statistics are shown.		* DATA: pixel data. Some vital statistics are shown.
	* xtal_target and SEQUENCE_NUMBER: unused		* xtal_target and SEQUENCE_NUMBER: unused
	* 64 active areas: the active areas are coordinates that specify where the 64 tiles are in the image. They are in the form of quartets of numbers [x1,y1,x2,y2], one quartet for each of the 64 tiles. The first active area is shown. Note, for non-CSPAD image pickles, there will only be one active area and it will be the size of the image entire.		* 64 active areas: the active areas are coordinates that specify where the 64 tiles are in the image. They are in the form of quartets of numbers [x1,y1,x2,y2], one quartet for each of the 64 tiles. The first active area is shown. Note, for non-CSPAD image pickles, there will only be one active area and it will be the size of the image entire.
	* 64 translated active areas: if corrections were available for this image, the first corrected active area is shown here.		* 64 translated active areas: if corrections were available for this image, the first corrected active area is shown here.

			=== A note on coordinate systems ===

			Internally, our software uses the ImageCIF coordinate system with the exception that no goniometer is typically available at XFELs, therefore Z is along the beam (source to sample), Y points against gravity and X completes a right-handed coordinate system. In the image pickles, the origin of the detector pixel readout is assumed to be in the upper left of the image, therefore the beam center is specified as positive offsets from that point. In example above, assuming the sample is at the laboratory origin, the vector pointing to the beam center on the detector in the ImageCIF coordinate system would be (0, 0, -119.002). A vector from the laboratory origin to the pixel readout origin (upper-left of the detector) would be -97.075, 97.075, -119.002). dxtbx.print_header is a useful tool to get these vectors from any file format understood by ''cctbx.xfel''. For more information see [http://cci.lbl.gov/publications/download/jo5001_reprint.pdf Parkhurst et al, 2014].

	=== Creating image pickles from xtc streams ===		=== Creating image pickles from xtc streams ===