IOTA - Revision history

Artem: /* IOTA: integration optimization, triage and analysis */

2019-08-03T00:25:31Z

IOTA: integration optimization, triage and analysis

← Older revision		Revision as of 00:25, 3 August 2019
Line 13:		Line 13:
	[1] [http://www.ncbi.nlm.nih.gov/pubmed/27275148 ''IOTA'': integration optimization, triage and analysis tool for the processing of XFEL diffraction images.] Lyubimov AY, Uervirojnangkoorn M, Zeldin OB, Brewster AS, Murray TD, Sauter NK, Berger JM, Weis WI, Brunger AT. J Appl Crystallogr. 2016 May 11;49(Pt 3):1057-1064		[1] [http://www.ncbi.nlm.nih.gov/pubmed/27275148 ''IOTA'': integration optimization, triage and analysis tool for the processing of XFEL diffraction images.] Lyubimov AY, Uervirojnangkoorn M, Zeldin OB, Brewster AS, Murray TD, Sauter NK, Berger JM, Weis WI, Brunger AT. J Appl Crystallogr. 2016 May 11;49(Pt 3):1057-1064

	[2] [~~http~~://www.ncbi.nlm.nih.gov/pubmed/~~27050135 Diffraction-geometry refinement in the~~ DIALS ~~framework~~.] Waterman DG, ~~Winter~~ G, Gildea RJ, ~~Parkhurst JM~~, ~~Brewster AS~~, Sauter NK, Evans G. ''Acta ~~Crystallogr D Struct Biol~~.~~'' 2016 Apr;72(Pt 4):558~~-75.		[2] [https://www.ncbi.nlm.nih.gov/pubmed/29533234 DIALS: implementation and evaluation of a new integration package.] Winter, G., Waterman, D. G., Parkhurst, J. M., Brewster, A. S., Gildea, R. J., Gerstel, M., Fuentes-Montero, L., Vollmar, M., Michels-Clark, T., Young, I. D., Sauter, N. K. & Evans, G. (2018). Acta Cryst. D74, 85-97.

	[3] [https://www.ncbi.nlm.nih.gov/pubmed/30198898 Improving signal strength in serial crystallography with DIALS geometry refinement.] Brewster, A. S., Waterman, D. G., Parkhurst, J. M., Gildea, R. J., Young, I. D., O'Riordan, L. J., Yano, J., Winter, G., Evans, G. & Sauter, N. K. (2018). Acta Cryst. D74, 877-894.		[3] [https://www.ncbi.nlm.nih.gov/pubmed/30198898 Improving signal strength in serial crystallography with DIALS geometry refinement.] Brewster, A. S., Waterman, D. G., Parkhurst, J. M., Gildea, R. J., Young, I. D., O'Riordan, L. J., Yano, J., Winter, G., Evans, G. & Sauter, N. K. (2018). Acta Cryst. D74, 877-894.

Artem: /* IOTA: integration optimization, triage and analysis */

2019-08-03T00:23:04Z

IOTA: integration optimization, triage and analysis

← Older revision		Revision as of 00:23, 3 August 2019
Line 1:		Line 1:
	= ''IOTA'': '''i'''ntegration '''o'''ptimization, '''t'''riage and '''a'''nalysis =		= ''IOTA'': '''i'''ntegration '''o'''ptimization, '''t'''riage and '''a'''nalysis =

	''IOTA'' is a user-friendly front end for ~~the ''cctbx~~.~~xfel''and ''DIALS'' suites of~~ serial diffraction data processing ~~programs~~. It is comprised of three main modules:		''IOTA'' is a user-friendly front end for <code>dials.stills_process</code>: a serial diffraction data processing program. ''IOTA'' is comprised of three main modules:

	# Raw image import~~, conversion~~, pre-processing and triage		# Raw image import, pre-processing and triage
	# Image indexing and integration using ~~''cctbx~~.~~xfel'' (with optimization of spot-finding parameters) or ''DIALS'' (this is currently in the process of being adapted for diffraction stills)~~		# Image indexing, lattice model refinement, and integration using <code>dials.stills_process</code>
	# Analysis of the integrated dataset		# Analysis of the integrated dataset

	''IOTA'' can be run as a GUI or from the command-line; scripts can be used for both, interchangeably. The GUI has the advantage of displaying useful statistics; it can also be run in "monitor mode" during live data collection, during which the program will wait for new images to be written into the specified input folder. The command-line mode is useful if the program is run remotely on servers that do not, for some reason, support graphics.		''IOTA'' can be run as a GUI or from the command-line; scripts can be used for both, interchangeably. The GUI has the advantage of displaying useful statistics; it can also be run in "monitor mode" during live data collection, during which the program will wait for new images to be written into the specified input folder. The command-line mode is useful if the program is run remotely on servers that do not, for some reason, support graphics.

	Please note that ''IOTA'' is a front-end for ~~(currently) two pieces of data~~ processing software~~: ''cctbx.xfel'' and ''DIALS''~~. Therefore, the preferred construction for citation should be something like "~~diffraction~~ data were processed with ''IOTA'' [1] using data reduction algorithms implemented in ''~~cctbx.xfel~~'' [2~~] (or ''DIALS'' [~~3])".		Please note that ''IOTA'' is a front-end for other processing software. Therefore, the preferred construction for citation should be something like "data were processed with ''IOTA'' [1] using serial diffraction data reduction algorithms implemented in ''DIALS'' [2, 3]".

	[1] [http://www.ncbi.nlm.nih.gov/pubmed/27275148 ''IOTA'': integration optimization, triage and analysis tool for the processing of XFEL diffraction images.] Lyubimov AY, Uervirojnangkoorn M, Zeldin OB, Brewster AS, Murray TD, Sauter NK, Berger JM, Weis WI, Brunger AT. J Appl Crystallogr. 2016 May 11;49(Pt 3):1057-1064		[1] [http://www.ncbi.nlm.nih.gov/pubmed/27275148 ''IOTA'': integration optimization, triage and analysis tool for the processing of XFEL diffraction images.] Lyubimov AY, Uervirojnangkoorn M, Zeldin OB, Brewster AS, Murray TD, Sauter NK, Berger JM, Weis WI, Brunger AT. J Appl Crystallogr. 2016 May 11;49(Pt 3):1057-1064

	[2] [http://www.ncbi.nlm.nih.gov/pubmed/~~24633409 Accurate macromolecular structures using minimal measurements from X~~-~~ray free-electron lasers~~.] ~~Accurate macromolecular structures using minimal measurements from X-ray free-electron lasers. Hattne J~~, ~~Echols N, Tran R, Kern J~~, Gildea RJ, Brewster AS, ~~Alonso-Mori R, Glöckner C, Hellmich J, Laksmono H, Sierra RG, Lassalle-Kaiser B~~, ~~Lampe A, Han~~ G, Gul S, DiFiore D, Milathianaki D, Fry AR, Miahnahri A, White WE, Schafer DW, Seibert MM, Koglin JE, Sokaras D, Weng TC, Sellberg J, Latimer MJ, Glatzel P, Zwart PH, Grosse-Kunstleve RW, Bogan MJ, Messerschmidt M, Williams GJ, Boutet S, Messinger J, Zouni A, Yano J, Bergmann U, Yachandra VK, Adams PD, Sauter NK. ''~~Nat Methods~~.'' ~~2014 May~~;11(5):~~545~~-8.		[2] [http://www.ncbi.nlm.nih.gov/pubmed/27050135 Diffraction-geometry refinement in the DIALS framework.] Waterman DG, Winter G, Gildea RJ, Parkhurst JM, Brewster AS, Sauter NK, Evans G. ''Acta Crystallogr D Struct Biol.'' 2016 Apr;72(Pt 4):558-75.

	[3] [~~http~~://www.ncbi.nlm.nih.gov/pubmed/~~27050135 Diffraction-~~geometry refinement ~~in the DIALS framework~~.] Waterman DG, ~~Winter~~ G, Gildea RJ, ~~Parkhurst JM~~, ~~Brewster AS~~, ~~Sauter NK~~, Evans G. ~~''Acta Crystallogr D Struct Biol~~.~~'' 2016 Apr;72~~(~~Pt 4~~)~~:558~~-75.		[3] [https://www.ncbi.nlm.nih.gov/pubmed/30198898 Improving signal strength in serial crystallography with DIALS geometry refinement.] Brewster, A. S., Waterman, D. G., Parkhurst, J. M., Gildea, R. J., Young, I. D., O'Riordan, L. J., Yano, J., Winter, G., Evans, G. & Sauter, N. K. (2018). Acta Cryst. D74, 877-894.

	= ''IOTA'' GUI =		= ''IOTA'' GUI =

Artem: /* Output */

2019-08-03T00:15:30Z

Output

← Older revision		Revision as of 00:15, 3 August 2019
Line 195:		Line 195:
	Due to ''IOTA's'' flexibility, there are several types of output that co-exist simultaneously and can be somewhat disconnected from one another. It helps to think of them as three separate stages of the process: pre-processing, integration, and post-processing / analysis.		Due to ''IOTA's'' flexibility, there are several types of output that co-exist simultaneously and can be somewhat disconnected from one another. It helps to think of them as three separate stages of the process: pre-processing, integration, and post-processing / analysis.

	In pre-processing, raw images are read in and converted to Python pickles. These are saved under the "converted_pickles" folder in the format <prefix>_<run_no>_<#####>.pickle; each cycle of pre-processing is assigned a run number (e.g. "001", "002", "003", etc.). Pre-processing is only triggered if a) the read-in image is not already pickled or b) the image has to be modified in some way (e.g. override beamXY coordinates, change detector distance, etc.). Thus, if converted and modified pickles are submitted to IOTA, the "converted_pickles" folder will not be created. The purpose of this is to allow the user to experiment with image modification, then subsequently select the converted pickles that best fit the user's needs. (<b>NOTE:</b> conversion to Python pickle format is only necessary for the deprecated "HA14" processing backend. The new <code>dials.stills_process</code> program can read any image format directly and doesn't require format conversion. Likewise, spotfinding parameter grid search is only applicable to the "HA14" backend and is not used with <code>dials.stills_process</code>.)		In pre-processing, raw images are read in and converted to Python pickles. These are saved under the <code>converted_pickles</code> folder in the format <prefix>_<run_no>_<#####>.pickle; each cycle of pre-processing is assigned a run number (e.g. "001", "002", "003", etc.). Pre-processing is only triggered if a) the read-in image is not already pickled or b) the image has to be modified in some way (e.g. override beamXY coordinates, change detector distance, etc.). Thus, if converted and modified pickles are submitted to IOTA, the "converted_pickles" folder will not be created. The purpose of this is to allow the user to experiment with image modification, then subsequently select the converted pickles that best fit the user's needs. (<b>NOTE:</b> conversion to Python pickle format is only necessary for the deprecated "HA14" processing backend. The new <code>dials.stills_process</code> program can read any image format directly and doesn't require format conversion. Likewise, spotfinding parameter grid search is only applicable to the "HA14" backend and is not used with <code>dials.stills_process</code>.)

	The output of the other two steps (integration and post-processing / analysis) can be found under the <code>integration</code> folder. It mostly contains a lot of files that are necessary for IOTA to keep track of the processed data, including a JSON dictionary file <code>proc.info</code> that contains all the information about the current processing run. The integrated pickles are collected under <code>integration/###/final</code> folder, in the format int_<filename>.pickle. Only successfully integrated images are saved this way. For each of the input images, however, a log of the backend output is saved in the <code>logs</code> folder, in the format <filename>.log. This file is not for the faint of heart, as it contains the raw output of <code>dials.stills_process</code> and can be used for troubleshooting by advanced users.		The output of the other two steps (integration and post-processing / analysis) can be found under the <code>integration</code> folder. It mostly contains a lot of files that are necessary for IOTA to keep track of the processed data, including a JSON dictionary file <code>proc.info</code> that contains all the information about the current processing run. The integrated pickles are collected under <code>integration/###/final</code> folder, in the format int_<filename>.pickle. Only successfully integrated images are saved this way. For each of the input images, however, a log of the backend output is saved in the <code>logs</code> folder, in the format <filename>.log. This file is not for the faint of heart, as it contains the raw output of <code>dials.stills_process</code> and can be used for troubleshooting by advanced users.

Artem: /* Output */

2019-08-03T00:15:10Z

Output

← Older revision		Revision as of 00:15, 3 August 2019
Line 193:		Line 193:
	= Output =		= Output =

	Due to ''IOTA's'' flexibility, there are several types of output that co-exist simultaneously and can be somewhat disconnected from one another. It helps to think of them as three separate stages of the process: pre-processing, ~~grid-search /~~ integration, and post-processing / analysis.		Due to ''IOTA's'' flexibility, there are several types of output that co-exist simultaneously and can be somewhat disconnected from one another. It helps to think of them as three separate stages of the process: pre-processing, integration, and post-processing / analysis.

	In pre-processing, raw images are read in and converted to Python pickles. These are saved under the "converted_pickles" folder in the format <prefix>_<run_no>_<#####>.pickle; each cycle of pre-processing is assigned a run number (e.g. "001", "002", "003", etc.). Pre-processing is only triggered if a) the read-in image is not already pickled or b) the image has to be modified in some way (e.g. override beamXY coordinates, change detector distance, etc.). Thus, if converted and modified pickles are submitted to IOTA, the "converted_pickles" folder will not be created. The purpose of this is to allow the user to experiment with image modification, then subsequently select the converted pickles that best fit the user's needs.		In pre-processing, raw images are read in and converted to Python pickles. These are saved under the "converted_pickles" folder in the format <prefix>_<run_no>_<#####>.pickle; each cycle of pre-processing is assigned a run number (e.g. "001", "002", "003", etc.). Pre-processing is only triggered if a) the read-in image is not already pickled or b) the image has to be modified in some way (e.g. override beamXY coordinates, change detector distance, etc.). Thus, if converted and modified pickles are submitted to IOTA, the "converted_pickles" folder will not be created. The purpose of this is to allow the user to experiment with image modification, then subsequently select the converted pickles that best fit the user's needs. (<b>NOTE:</b> conversion to Python pickle format is only necessary for the deprecated "HA14" processing backend. The new <code>dials.stills_process</code> program can read any image format directly and doesn't require format conversion. Likewise, spotfinding parameter grid search is only applicable to the "HA14" backend and is not used with <code>dials.stills_process</code>.)

	The output of the other two steps (~~grid-search /~~ integration and post-processing / analysis) can be found under the "integration" folder. ~~The grid-search results~~ are ~~saved~~ to the ~~"integration/###/image_objects" folder in the format~~ <~~filename~~>.~~int. These are pickled dictionaries which contain~~ all the information about the individual images (without the pixel values or integrated intensities), such as raw image filename, converted pickle filename, the details of the grid search, etc. These can be used for some of the advanced operations, such as experimentation with the selection process without repeating the grid search.		The output of the other two steps (integration and post-processing / analysis) can be found under the <code>integration</code> folder. It mostly contains a lot of files that are necessary for IOTA to keep track of the processed data, including a JSON dictionary file <code>proc.info</code> that contains all the information about the current processing run. The integrated pickles are collected under <code>integration/###/final</code> folder, in the format int_<filename>.pickle. Only successfully integrated images are saved this way. For each of the input images, however, a log of the backend output is saved in the <code>logs</code> folder, in the format <filename>.log. This file is not for the faint of heart, as it contains the raw output of <code>dials.stills_process</code> and can be used for troubleshooting by advanced users.

	The integrated pickles are collected under "integration/###/final" folder, in the format int_<filename>.pickle. Only successfully integrated images are saved this way. For each of the input images, however, a log of ~~''cctbx.xfel'' or ''DIALS''~~ output is saved in the ~~same~~ folder, in the format <filename>.log. This ~~log documents each integration attempt from~~ the ~~grid-search with~~ the ~~final integration attempt at the end (for ''cctbx~~.~~xfel'') or the linear indexing~~/~~integration output (for ''DIALS'')~~ and can be used for troubleshooting.

	~~If the user chooses to output any charts (e.g. grid-search heatmap, beam center plot, image visualization, etc.), these will be found under "integration/###/visualization" folder.~~

	Finally, the "integration/###" folder itself contains text files with lists of images, e.g. input images, all integrated images, all images that failed integration, major clusters from the unit cell-clustering module, etc. The main logfile (iota.log) is also found here, as is the default input file for ''PRIME'' (prime.phil).

			If the user chooses to output any charts (e.g. beam center plot, image visualization, etc.), these will be found under <code>integration/###/visualization</code> folder.

			Finally, the <code>integration/###</code> folder itself contains text files with lists of images, e.g. input images, all integrated images, all images that failed integration, major clusters from the unit cell-clustering module, etc. The main logfile (<code>iota.log</code>) is also found here, as are the script files for IOTA (<code>iota_r#.param</code>) and the backend (<code>target.phil</code>), and the default input file for ''PRIME'' (prime.phil).

	= ''IOTA'' Tutorial =		= ''IOTA'' Tutorial =

Artem: /* Command-line Options */

2019-08-03T00:04:06Z

Command-line Options

← Older revision		Revision as of 00:04, 3 August 2019
Line 183:		Line 183:
	All of the options in the script can be introduced as command-line statements by using a "compressed" PHIL format. Thus:		All of the options in the script can be introduced as command-line statements by using a "compressed" PHIL format. Thus:

	~~cctbx~~ {		cctbx_xfel {
	~~grid_search {~~		target_space_group = P422
	~~type~~ = ~~None *brute_force smart~~
	}
	}		}

	translates into		translates into

	iota.run script.param ~~cctbx~~.~~grid_search.type~~=~~brute_force~~		iota.run script.param cctbx_xfel.target_space_group=P422

	= Output =		= Output =

Artem: /* Command-line Options */

2019-08-03T00:02:58Z

Command-line Options

← Older revision		Revision as of 00:02, 3 August 2019
Line 180:		Line 180:

	iota.run -h		iota.run -h

	Perhaps the most useful of these are -r and -n options, as they allow the user to adjust an IOTA run in Auto-mode on the fly. Alternatively, both of these settings can be changed within the script file.

	All of the options in the script can be introduced as command-line statements by using a "compressed" PHIL format. Thus:		All of the options in the script can be introduced as command-line statements by using a "compressed" PHIL format. Thus:

Artem: /* Command-line Options */

2019-08-03T00:02:15Z

Command-line Options

← Older revision		Revision as of 00:02, 3 August 2019
Line 165:		Line 165:
	In addition to a command script, IOTA runs can be modified by command-line options:		In addition to a command script, IOTA runs can be modified by command-line options:

	-h, --help show help message and exit		-h, --help show this help message and exit
	--version Prints version info		--version Prints version info of IOTA
	-l, --~~list Output a file (input_images.lst) with input image paths~~ and ~~exit~~		-d, --default Generate default settings files and stop
	-c, -~~-convert~~ ~~Convert raw images to pickles and exit~~		--ha14 Run IOTA with old HA14 backend
	-d, -~~-default Generate default iota.param and target.phil files and exit~~		--random RANDOM Size of randomized subset, e.g. "--random 10"
	~~-p PREFIX~~, ~~--prefix PREFIX Specify custom prefix for converted pickles (~~e.g. -~~p user)~~		--range [RANGE] Range of images, e.g."--range 1-5,25,200-250"
	~~-s,~~ --~~select Selection only, no grid search~~		-n NPROC, --nproc NPROC
	~~-r RANDOM Run IOTA with a random subset~~ of images, e.g. "-r 5"		Specify a number of cores for a multiprocessor run"
	-n NPROC Specify a number of cores for a multiprocessor run"		--analyze [ANALYZE] Use for analysis only; specify run number or folder
	~~--mpi [MPI] Specify stage of process - for MPI only~~		--tmp TMP Path to temp folder
	--analyze [ANALYZE] Use for analysis only; specify run number or folder ~~with results~~		--silent Run IOTA in silent mode

	These options can be shown by issuing:		These options can be shown by issuing:
Line 194:		Line 194:

	iota.run script.param cctbx.grid_search.type=brute_force		iota.run script.param cctbx.grid_search.type=brute_force


	= Output =		= Output =

Artem: /* Script Mode */

2019-08-02T23:58:12Z

Script Mode

← Older revision		Revision as of 23:58, 2 August 2019
Line 146:		Line 146:
	Additionally, IOTA settings can be modified by command-line statements, e.g.:		Additionally, IOTA settings can be modified by command-line statements, e.g.:

	iota.run script.param ~~cctbx~~.~~grid_search~~.~~type~~=~~smart cctbx~~.~~grid_search~~.~~area_median~~=7		iota.run script.param data_selection.image_range.flag_on=True data_selection.image_range.range=1-100

	== Single-Image Mode ==		== Single-Image Mode ==

Artem: /* Script Mode */

2019-08-02T23:56:33Z

Script Mode

← Older revision		Revision as of 23:56, 2 August 2019
Line 116:		Line 116:
	The script contains settings in PHIL format, e.g.:		The script contains settings in PHIL format, e.g.:

	description = "IOTA ~~run #5~~, ~~with some modified settings~~"		description = "IOTA parameters auto-generated on Friday, Aug 02, 2019. 04:40 PM"
	input = "/~~path~~/to/~~raw~~/~~images~~/"		input = "/Users/art/Science/iota_tutorial/hewl_Br_data"
	output = "/~~path/to~~/~~iota~~/~~output~~/"		output = "/Users/art/Science/iota_tutorial"
	~~image_conversion~~ {		data_selection {
	~~rename_pickle = None keep_file_structure *auto_filename custom_filename~~		image_triage {
	~~rename_pickle_prefix = None~~		flag_on = True
	~~convert_only = False~~		minimum_Bragg_peaks = 10
	~~square_mode = None no_modification *pad crop~~		strong_sigma = 5
	~~mask = None~~
	~~invert_boolean_mask = False~~
	~~beamstop~~ = 0
	~~distance = 0~~
	~~beam_center {~~
	x = 0
	y = 0
	}		}
	}		image_range {
	~~image_triage {~~		flag_on = False
	~~type~~ = None ~~no_triage *simple grid_search~~		range = None
	~~min_Bragg_peaks = 10~~		}
	~~grid_search~~ {		random_sample {
	~~area_min = 6~~		flag_on = False
	~~area_max = 24~~		number = 0
	~~height_min = 2~~
	~~height_max~~ = 20
	~~step_size~~ = 4
	}		}
	}		}

Artem: /* Target Files */

2019-08-02T23:55:23Z

Target Files

← Older revision		Revision as of 23:55, 2 August 2019
Line 98:		Line 98:
	== Target Files ==		== Target Files ==

	''IOTA'' itself is a front-end to the data processing ~~programs ''cctbx~~.~~xfel'' and ''DIALS''~~. ~~These programs require their~~ own set of parameters, distinct from ''IOTA'' parameters, which are located in so-called "target" files: text files containing parameters encoded in Python-based hierarchical interchange language or PHIL. When run in AUTO mode, ''IOTA'' generates an appropriate target file ~~for ''cctbx.xfel'' or ''DIALS''~~ using defaults deemed reasonable for most serial crystallography projects. ~~These~~ default target ~~files~~ can also serve as a starting point for the user to modify those settings as they see fit. The user has the option to provide their own target file (perhaps generated during a previous data processing attempt). The user can edit the ''IOTA'' settings to specify the target file		''IOTA'' itself is a front-end to the data processing program <code>dials.stills_process</code>. This program requires its own set of parameters, distinct from ''IOTA'' parameters, which are located in so-called "target" files: text files containing parameters encoded in Python-based hierarchical interchange language or PHIL. When run in AUTO mode, ''IOTA'' generates an appropriate target file using defaults deemed reasonable for most serial crystallography projects. This default target file can also serve as a starting point for the user to modify those settings as they see fit. The user has the option to provide their own target file (perhaps generated during a previous data processing attempt). The user can edit the ''IOTA'' settings to specify the target file

	cctbx {		cctbx {
	target = "~~dials~~.phil"		target = "user_params.phil"
	}		}