Difference between revisions of "IOTA"

Revision as of 23:51, 13 January 2016

IOTA: integration optimization, triage and analysis

IOTA is a user-friendly front end for the cctbx.xfel suite of serial diffraction data processing programs. It is comprised of three main modules:

1. Raw image import, conversion, pre-processing and triage
2. Image indexing and integration using cctbx.xfel modules (with optimization of spot-finding parameters)
3. Analysis of the integrated dataset

These modules are united by a command-line interface and are run in sequence. The user can point IOTA towards a single diffraction image, a folder of images, a list of image paths or a modified script file. If no script file is supplied, IOTA will automatically generate reasonable defaults (currently optimized for a MARCCD detector, but also usable with Pilatus detectors).

Running IOTA: Auto Mode

The simplest way to run IOTA is in Auto Mode. To do so, simply issue:

 iota.run /path/to/image/files/

The path may contain a tree of folders in any configuration. IOTA will then carry out a conversion step if the source folder contains raw diffraction images. The converted image pickles will be saved in the current folder under the subfolder "converted_pickles". Inside that folder, converted pickles will be saved separately for each IOTA run, under subfolders named "001", "002", "003", etc. Alternatively, once raw images have been successfully converted to image pickles, IOTA can be pointed to the image pickles instead, e.g.:

 iota.run ./converted_pickles/001/

Alternatively, if a list of images exists, IOTA can accept that list of images as input (IOTA creates the input list automatically and saves it under ./integration/###/input_images.lst):

 iota.run ./integration/001/input_images.lst

Once running, IOTA will display a program logo, some information about the configuration of the run and a progress bar for each major step, e.g.:

 -bash-4.1$ iota.run converted_pickles/003/
 
 
      IIIIII            OOOOOOO        TTTTTTTTTT          A              
        II             O       O           TT             A A             
        II             O       O           TT            A   A            
 >------INTEGRATION----OPTIMIZATION--------TRIAGE-------ANALYSIS--------->
        II             O       O           TT          A       A          
        II             O       O           TT         A         A         
      IIIIII            OOOOOOO            TT        A           A   v2.23  
                                                           with CCTBX.XFEL
 
 
 Wednesday, Nov 11, 2015. 06:36 PM
 
 
 IOTA will run in AUTO mode using /net/cci-filer2/raid1/home/art/iota/test_v2.23/converted_pickles/003:
 
 Reading files from data folder -- DONE.....................................0.11s
 IMPORTING IMAGES:   9% [ -    ] [ ===>                                         ]

IOTA will automatically create two script files: iota.param (which contains settings for running IOTA) and target.phil (a cctbx.xfel target file), which can be modified by a user to fine-tune various settings. The output will be collected in the folder named "integration", which will contain subfolders for each integration run, titled "001", "002", "003", etc. Each run generates a folder named "final" with the final integrated pickles as well as individual cctbx.xfel logs for each image. Furthermore, lists of files that have been successfully integrated (integrated.lst), failed integration (not_integrated.lst), etc. can be found there. Finally, a pre-populated script for PRIME (prime.phil) can be found there as well. (Currently, the user must manually edit prime.phil to specify the number of residues - "n_residues" - in order to run PRIME successfully.)

Running IOTA: Script Mode

IOTA can be run using a script file, e.g.:

 iota.run script.param

The script contains settings in PHIL format, e.g.:

 description = "IOTA run #5, with some modified settings"
 input = "/path/to/raw/images/"
 target = "target.phil"
 cctbx {
   grid_search {
     type = None *brute_force smart
     area_median = 5
     area_range = 2
     height_median = 4
     height_range = 2
     sig_height_search = False
   }
   selection {
     select_only {
       flag_on = False
       grid_search_path = None
     }
     min_sigma = 5
     select_by = *epv mosaicity
     prefilter {
       flag_on = False
       target_pointgroup = None
       target_unit_cell = None
       target_uc_tolerance = 0.05
       min_reflections = 0
       min_resolution = None
     }
   }
 }
 .
 .
 .

The script can be auto-generated (with an accompanying target.phil file with some default cctbx.xfel settings) via a "dry run" by issuing

 iota.run -d

The same "-d" command-line option will print to terminal the full IOTA script file with help statements (which will also be included at the end of this page).

Additionally, IOTA settings can be modified by command-line statements, e.g.:

 iota.run script.param cctbx.grid_search.type=smart cctbx.grid_search.area_median=7

Running IOTA: Single-Image Mode

IOTA can accept a single image as input:

 iota.run images/img_00001.pickle 

Alternatively, IOTA can be run in bare-bones "single-image mode"

 iota.single_image images/img_00001.pickle 

These options are best for testing purposes.

Running IOTA: Command-line Options

In addition to a command script, IOTA runs can be modified by command-line options:

 -h, --help                                show help message and exit
 --version                                 Prints version info
 -l, --list                                   Output a file (input_images.lst) with input image paths and exit
 -c, --convert                          Convert raw images to pickles and exit
 -d, --default                           Generate default iota.param and target.phil files and exit
 -p PREFIX, --prefix PREFIX   Specify custom prefix for converted pickles (e.g. -p user)
 -s, --select                             Selection only, no grid search
 -r RANDOM                            Run IOTA with a random subset of images, e.g. "-r 5"
 -n NPROC                              Specify a number of cores for a multiprocessor run"
 --mpi [MPI]                            Specify stage of process - for MPI only
 --analyze [ANALYZE]            Use for analysis only; specify run number or folder with results