Difference between revisions of "File formats"

From cctbx_xfel
Jump to: navigation, search
(Creating CSPAD CBFs from XTC streams)
(Creating CSPAD CBFs from XTC streams)
Line 32: Line 32:
 
     format.cbf.detz_offset=100 input.override_energy=7000
 
     format.cbf.detz_offset=100 input.override_energy=7000
  
Here, one image is created by from experiment xpptut15, run 54.  You can display the image with cctbx.image_viewer xpptut15/out/*.cbf.  Note the pink gaps between the tiles. This results from the segmented nature of the CSPAD, preserved in the CBF file.
+
Here, one image is created from experiment xpptut15, run 54.  You can display the image with cctbx.image_viewer xpptut15/out/*.cbf.  Note the pink gaps between the tiles. This results from the segmented nature of the CSPAD, preserved in the CBF file.
  
xpptut15 is data from XPP's CSPAD collected without xray's on.  Hence format.cbf.detz_offset=100 input.override_energy=7000 are set to fake values in this command.
+
xpptut15 is data from XPP's CSPAD and was collected without xray's on.  Hence format.cbf.detz_offset=100 input.override_energy=7000 are set to fake values in this command.
  
 
=== Converting from SLAC's metrology to CBF ===
 
=== Converting from SLAC's metrology to CBF ===

Revision as of 11:11, 7 December 2015

This page is under construction

Introduction

cctbx.xfel's current native file formats, image pickles and integration pickles, are mainly intermediate file formats useful for debugging and software development. These binary pickle files are serialized python dictionaries optimized for machine readability. The rationale here is that the raw data from an experiment at LCLS is stored in the xtc streams. We process these in memory without needing to write images to disk at all, but provide image pickles if requested for use with the cctbx image viewer for diagnostic purposes. Integration pickles contain integrated intensities and crystal cell and orientation parameters and are read by the merging programs cxi.merge and prime.postrefine.

In an effort to provide more human-readable data files and conform to international standards, we have worked with representatives of ImageCIF to create 64 tile segmented data in CBF format, the same format used in a wide variety of detectors. These CSPAD CBFs are designed to be used with other crystallographic software packages.

Format specifications for these data files are provided below.

Image pickles

Coming soon

Integration pickles

Coming soon

CSPAD CBFs

CSPAD CBFs are not used directly by cctbx.xfel at this time. This aspect of the project is under development. The complete specification for CSPAD CBFs is laid out in an article in the Computational Crystallographic Newsletter: "XFEL Detectors and ImageCIF", Computational Crystallography Newsletter 5, 19-24. (Reprint). CSPAD CBFs

Creating CSPAD CBFs from XTC streams

cctbx.xfel.xtc_dump is useful for converting XTC streams to CBF files as needed. Use -c to get a listing of all options, and -c -a 2 to get a full listing with help strings. Example command:

 cd ~/myrelease # or wherever your release is
 sit_setup
 cctbx.xfel.xtc_dump dispatch.max_events=1 input.experiment=xpptut15 input.address=cspad \
   input.run_num=54 format.file_format=cbf output.output_dir=xpptut15/out \
   format.cbf.detz_offset=100 input.override_energy=7000

Here, one image is created from experiment xpptut15, run 54. You can display the image with cctbx.image_viewer xpptut15/out/*.cbf. Note the pink gaps between the tiles. This results from the segmented nature of the CSPAD, preserved in the CBF file.

xpptut15 is data from XPP's CSPAD and was collected without xray's on. Hence format.cbf.detz_offset=100 input.override_energy=7000 are set to fake values in this command.

Converting from SLAC's metrology to CBF

The tile positions of the hierarchical CSPAD detector are specified by SLAC in a geometry file in the calib folder of each experiment. Typically this file is named something like 0-end.data. If desired, this file can be converted to just the human readable header portion of a CSPAD cbf using the cctbx.xfel command cxi.slaccalib2cbfheader. Example:

 cxi.slaccalib2cbfheader metrology_file=0-end.data out=tmp.cbf

This cbf header can be converted back to SLAC format using cxi.cbfheader2slaccalib:

 cxi.cbfheader2slaccalib cbf_header=tmp.cbf out_metrology_file=tmp.data

Displaying metrology files

cxi.display_metrology <filename> can be used to show a plot of tile positions. It accepts SLAC metrology files, CSPAD CBFs and image pickles.