IOTA: Difference between revisions
No edit summary |
|||
Line 19: | Line 19: | ||
iota.run /path/to/image/files/ | iota.run /path/to/image/files/ | ||
The path may contain a tree of folders in any configuration. IOTA will then carry out a conversion step if the source folder contains raw diffraction images. The converted image pickles will be saved in the current folder under the subfolder "converted_pickles". Inside that folder, converted pickles will be saved separately for each ''IOTA'' run, under subfolders named "001", "002", "003", etc. Alternatively, once raw images have been successfully converted to image pickles, ''IOTA'' can be pointed to the image pickles instead, e.g.: | The path may contain a tree of folders in any configuration. ''IOTA'' will then carry out a conversion step if the source folder contains raw diffraction images. The converted image pickles will be saved in the current folder under the subfolder "converted_pickles". Inside that folder, converted pickles will be saved separately for each ''IOTA'' run, under subfolders named "001", "002", "003", etc. Alternatively, once raw images have been successfully converted to image pickles, ''IOTA'' can be pointed to the image pickles instead, e.g.: | ||
iota.run ./converted_pickles/001/ | iota.run ./converted_pickles/001/ | ||
Line 51: | Line 51: | ||
''IOTA'' will automatically create two script files: iota.param (which contains settings for running ''IOTA'') and target.phil (a ''cctbx.xfel'' target file), which can be modified by a user to fine-tune various settings. The output will be collected in the folder named "integration", which will contain subfolders for each integration run, titled "001", "002", "003", etc. Each run generates a folder named "final" with the final integrated pickles as well as individual ''cctbx.xfel'' logs for each image. Furthermore, lists of files that have been successfully integrated (integrated.lst), failed integration (not_integrated.lst), etc. can be found there. Finally, a pre-populated script for ''PRIME'' (prime.phil) can be found there as well. (Currently, the user must manually edit prime.phil to specify the number of residues - "n_residues" - in order to run ''PRIME'' successfully.) | ''IOTA'' will automatically create two script files: iota.param (which contains settings for running ''IOTA'') and target.phil (a ''cctbx.xfel'' target file), which can be modified by a user to fine-tune various settings. The output will be collected in the folder named "integration", which will contain subfolders for each integration run, titled "001", "002", "003", etc. Each run generates a folder named "final" with the final integrated pickles as well as individual ''cctbx.xfel'' logs for each image. Furthermore, lists of files that have been successfully integrated (integrated.lst), failed integration (not_integrated.lst), etc. can be found there. Finally, a pre-populated script for ''PRIME'' (prime.phil) can be found there as well. (Currently, the user must manually edit prime.phil to specify the number of residues - "n_residues" - in order to run ''PRIME'' successfully.) | ||
== Running ''IOTA'': Script Mode == | == Running ''IOTA'': Script Mode == |
Revision as of 00:18, 14 January 2016
IOTA: integration optimization, triage and analysis
IOTA is a user-friendly front end for the cctbx.xfel suite of serial diffraction data processing programs. It is comprised of three main modules:
- Raw image import, conversion, pre-processing and triage
- Image indexing and integration using cctbx.xfel modules (with optimization of spot-finding parameters)
- Analysis of the integrated dataset
These modules are united by a command-line interface and are run in sequence. The user can point IOTA towards a single diffraction image, a folder of images, a list of image paths or a modified script file. If no script file is supplied, IOTA will automatically generate reasonable defaults (currently optimized for a MARCCD detector, but also usable with Pilatus detectors).
Running IOTA: Auto Mode
The simplest way to run IOTA is in Auto Mode. To do so, simply issue:
iota.run /path/to/image/files/
The path may contain a tree of folders in any configuration. IOTA will then carry out a conversion step if the source folder contains raw diffraction images. The converted image pickles will be saved in the current folder under the subfolder "converted_pickles". Inside that folder, converted pickles will be saved separately for each IOTA run, under subfolders named "001", "002", "003", etc. Alternatively, once raw images have been successfully converted to image pickles, IOTA can be pointed to the image pickles instead, e.g.:
iota.run ./converted_pickles/001/
Alternatively, if a list of images exists, IOTA can accept that list of images as input (IOTA creates the input list automatically and saves it under ./integration/###/input_images.lst):
iota.run ./integration/001/input_images.lst
Once running, IOTA will display a program logo, some information about the configuration of the run and a progress bar for each major step, e.g.:
-bash-4.1$ iota.run converted_pickles/003/ IIIIII OOOOOOO TTTTTTTTTT A II O O TT A A II O O TT A A >------INTEGRATION----OPTIMIZATION--------TRIAGE-------ANALYSIS---------> II O O TT A A II O O TT A A IIIIII OOOOOOO TT A A v2.23 with CCTBX.XFEL Wednesday, Nov 11, 2015. 06:36 PM IOTA will run in AUTO mode using /net/cci-filer2/raid1/home/art/iota/test_v2.23/converted_pickles/003: Reading files from data folder -- DONE.....................................0.11s IMPORTING IMAGES: 9% [ - ] [ ===> ]
IOTA will automatically create two script files: iota.param (which contains settings for running IOTA) and target.phil (a cctbx.xfel target file), which can be modified by a user to fine-tune various settings. The output will be collected in the folder named "integration", which will contain subfolders for each integration run, titled "001", "002", "003", etc. Each run generates a folder named "final" with the final integrated pickles as well as individual cctbx.xfel logs for each image. Furthermore, lists of files that have been successfully integrated (integrated.lst), failed integration (not_integrated.lst), etc. can be found there. Finally, a pre-populated script for PRIME (prime.phil) can be found there as well. (Currently, the user must manually edit prime.phil to specify the number of residues - "n_residues" - in order to run PRIME successfully.)
Running IOTA: Script Mode
IOTA can be run using a script file, e.g.:
iota.run script.param
The script contains settings in PHIL format, e.g.:
description = "IOTA run #5, with some modified settings" input = "/path/to/raw/images/" target = "target.phil" cctbx { grid_search { type = None *brute_force smart area_median = 5 area_range = 2 height_median = 4 height_range = 2 sig_height_search = False } selection { select_only { flag_on = False grid_search_path = None } min_sigma = 5 select_by = *epv mosaicity prefilter { flag_on = False target_pointgroup = None target_unit_cell = None target_uc_tolerance = 0.05 min_reflections = 0 min_resolution = None } } } . . .
The script can be auto-generated (with an accompanying target.phil file with some default cctbx.xfel settings) via a "dry run" by issuing
iota.run -d
The same "-d" command-line option will print to terminal the full IOTA script file with help statements (which will also be included at the end of this page).
Additionally, IOTA settings can be modified by command-line statements, e.g.:
iota.run script.param cctbx.grid_search.type=smart cctbx.grid_search.area_median=7
Running IOTA: Single-Image Mode
IOTA can accept a single image as input:
iota.run images/img_00001.pickle
Alternatively, IOTA can be run in bare-bones "single-image mode"
iota.single_image images/img_00001.pickle
These options are best for testing purposes.
Running IOTA: Command-line Options
In addition to a command script, IOTA runs can be modified by command-line options:
-h, --help show help message and exit --version Prints version info -l, --list Output a file (input_images.lst) with input image paths and exit -c, --convert Convert raw images to pickles and exit -d, --default Generate default iota.param and target.phil files and exit -p PREFIX, --prefix PREFIX Specify custom prefix for converted pickles (e.g. -p user) -s, --select Selection only, no grid search -r RANDOM Run IOTA with a random subset of images, e.g. "-r 5" -n NPROC Specify a number of cores for a multiprocessor run" --mpi [MPI] Specify stage of process - for MPI only --analyze [ANALYZE] Use for analysis only; specify run number or folder with results
These options can be shown by issuing:
iota.run -h
Perhaps the most useful of these are -r and -n options, as they allow the user to adjust an IOTA run in Auto-mode on the fly. Alternatively, both of these settings can be changed within the script file.
All of the options in the script can be introduced as command-line statements by using a "compressed" PHIL format. Thus:
cctbx { grid_search { type = None *brute_force smart } }
translates into
iota.run script.param cctbx.grid_search.type=brute_force
IOTA Output
Due to IOTA's flexibility, there are several types of output that co-exist simultaneously and can be somewhat disconnected from one another. It helps to think of them as three separate stages of the process: pre-processing, grid-search / integration, and post-processing / analysis.
In pre-processing, raw images are read in and converted to cctbx.xfel "pickle" format (i.e. pickled arrays of pixel intensity values). These are saved under the "converted_pickles" folder in the format <prefix>_<run_no>_<#####>.pickle; each cycle of pre-processing is assigned a run number (e.g. "001", "002", "003", etc.). Pre-processing is only triggered if a) the read-in image is not already in pickle format or b) the image has to be modified in some way (e.g. override beamXY coordinates, change detector distance, etc.). Thus, if converted and modified pickles are submitted to IOTA, the "converted_pickles" folder will not be created. The purpose of this is to allow the user to experiment with image modification, then subsequently select the converted pickles that best fit the user's needs.
The output of the other two steps (grid-search / integration and post-processing / analysis) can be found under the "integration" folder. The grid-search results are saved to the "integration/###/image_objects" folder in the format <filename>.int. These are pickled dictionaries which contain all the information about the individual images (without the pixel values or integrated intensities), such as raw image filename, converted pickle filename, the details of the grid search, etc. These can be used for some of the advanced options, such as experimentation with the selection process without repeating the grid search.
The integrated pickles are collected under "integration/###/final" folder, in the format int_<filename>.pickle. Only successfully integrated images are saved this way. For each of the input images, however, a log of cctbx.xfel output is saved in the same folder, in the format <filename>.log. This log documents each integration attempt from the grid-search, with the final integration attempt at the end, and can be used for troubleshooting.
If the user chooses to output any charts (e.g. grid-search heatmap, beam center plot, image visualization, etc.), these will be found under "integration/###/visualization" folder.
Finally, the "integration/###" folder itself contains text files with lists of images, e.g. input images, all integrated images, all images that failed integration, major clusters from the unit cell-clustering module, etc. The main logfile (iota.log) is also found here, as is the default input file for PRIME (prime.phil).