Phil - Revision history

Nicksauter at 16:54, 18 September 2015

2015-09-18T16:54:06Z

← Older revision		Revision as of 16:54, 18 September 2015
Line 79:		Line 79:
	* mosaicity_limit: maximum moisicity before a frame is rejected		* mosaicity_limit: maximum moisicity before a frame is rejected
	* distl_minimum_number_spots_for_indexing: indexing will not proceed unless there are at least this many good spots found on the image		* distl_minimum_number_spots_for_indexing: indexing will not proceed unless there are at least this many good spots found on the image
	* Subcategory <code>distl</code>: parameters specific to spot finding. '''NOTE: More extensive documentation on spotfinder and other parameters is at the [http://~~adder~~.lbl.gov/labelit/html/spotfinder.html spot finder web site]'''		* Subcategory <code>distl</code>: parameters specific to spot finding. '''NOTE: More extensive documentation on spotfinder and other parameters is at the [http://cci.lbl.gov/labelit/html/spotfinder.html spot finder web site]'''
	** res.outer: resolution limit for ''spotfinder'' [Note this is #3 of 3 resolution flags that all must be set together]		** res.outer: resolution limit for ''spotfinder'' [Note this is #3 of 3 resolution flags that all must be set together]
	** minimum_signal_height: in units of background noise sigma, how much signal is needed for a spot		** minimum_signal_height: in units of background noise sigma, how much signal is needed for a spot

Aaron at 18:27, 16 March 2015

2015-03-16T18:27:19Z

← Older revision		Revision as of 18:27, 16 March 2015
Line 135:		Line 135:
	*** rotations: angles in degrees for each tile (64)		*** rotations: angles in degrees for each tile (64)
	*** translation: x/y translations for each tile (128)		*** translation: x/y translations for each tile (128)

			== Comparing two phil files ==

			Often phil files are edited repeatedly by users. cxi.mpi_submit saves copies of phil files as they are used as a record of which trials were processed using which parameters. Now, say you have two different phil files that you want to see the differences between them. Due to comments or rearrangements, diffing the two files isn't making your life easier. Do this:

			$ cxi.parameters > default.phil
			$ libtbx.phil --diff default.phil file1.phil > a.txt
			$ libtbx.phil --diff default.phil file2.phil > b.txt
			$ diff a.txt b.txt

			This will show the true differences between the phil files. Note also that cxi.parameters is a useful tool for seeing all the default parameters available in ''cctbx.xfel''.

Nicksauter: /* Hitfinding/indexing/integration phil file */

2014-08-22T20:11:14Z

Hitfinding/indexing/integration phil file

New page

As described in the [[overview]], ''phil'' files contain parameters used during hitfinding, indexing and integration. This tutorial uses two ''phil'' files during the indexing and integration step: Ls04-lysozyme.phil and metrology-7.1.phil. The former specifies parameters specific to the processing run, while the latter specifies whole-pixel and sub-pixel metrology corrections applied to the 64 CSPAD sensor tiles.

== Hitfinding/indexing/integration ''phil'' file ==

For the tutorial, Ls04-lysozyme.phil, stored in the /reg/d/ffb/cxi/temp/cctbx/tutorials/indexing directory but copied to your ~/myrelease directory during use, contains configuration settings we worked out that best process this data. The file will look like this:

<pre>
# -*- mode: Conf -*-

include file metrology-7.1.phil

# From looking at 35 images integrated with detz_offset = 581 and
# without target_cell set.
target_cell = 38 79 79 90 90 90
target_cell_centring_type = P
known_setting = 9

distl_highres_limit = 2.0
force_method2_resolution_limit = 2.0

mosaicity_limit = 1

# Set to True to pick up second lattice, if present.
#outlier_detection_switch = True

# TEST
distl_minimum_number_spots_for_indexing = 20

distl {
res.outer = 2.0
minimum_signal_height = 5
#minimum_spot_height = 10
minimum_spot_height = 5
minimum_spot_area = 1
spot_area_maximum_factor = 20
compactness_filter = False
#method2_cutoff_percentage = 5
method2_cutoff_percentage = 2.5

# Avoids intensity filter.
#peak_intensity_maximum_factor = 10000
peak_intensity_maximum_factor = 100
}

indexing {
# Set to True to generate correction vectors.
verbose_cv = True
}

integration {
background_factor = 2

#detector_gain = 7.5
detector_gain = 1.0

#model = use_case_3_simulated_annealing_7
model = user_supplied

signal_penetration = 0.5
#spot_shape_verbose = False [This time consuming option prints out a detailed shape analysis of every spot]
spotfinder_subset = spots_non-ice
}
</pre>

Several lines of parameters are given, then a few parameter blocks are specified, enclosed in {} brackets. The parameters in detail:
* include file metrology-7.1.phil: this line specifies another ''phil'' file to include. In this case, the included file specifies metrology corrections (see below).
* target_cell: the known unit cell for this sample. In the form a, b, c, alpha, beta, gamma. Must be in the primitive setting, or specify a target_cell_centring_type. Or, convert the dimensions first to the primitive setting with ''iotbx.lattice_symmetry''.
* target_cell_centring_type: the centering type of the Bravais lattice: P (default), C, I, R, or F.
* known_setting: this is the setting number of the true Bravais symmetry as output by the autoindexing procedure in ''LABELIT''. The indexing process fundamentally determines the triclinic cell, but also a list of possible Bravais lattices. Normally we associate the following setting numbers with the crystal systems:
**Triclinic=1
**Monoclinic=2
**Orthorhombic=5
**Rhombohedral=5
**Tetragonal=9
**Hexagonal=12
**Cubic=22
**However, the assumptions break down if there is pseudosymmetry, i.e., if a higher-symmetry Bravais lattice is close to the true Bravais symmetry. In these cases, a careful inspection of the ''LABELIT'' output is necessary to choose the setting, and it may be necessary to implement new code in ''cctbx.xfel'' for this (contact the authors).
* distl_highres_limit and force_method2_resolution_limit: only process to this resolution limit. [Note these are #1 and #2 of 3 resolution flags that all must be set together]
* mosaicity_limit: maximum moisicity before a frame is rejected
* distl_minimum_number_spots_for_indexing: indexing will not proceed unless there are at least this many good spots found on the image
* Subcategory <code>distl</code>: parameters specific to spot finding. '''NOTE: More extensive documentation on spotfinder and other parameters is at the [http://adder.lbl.gov/labelit/html/spotfinder.html spot finder web site]'''
** res.outer: resolution limit for ''spotfinder'' [Note this is #3 of 3 resolution flags that all must be set together]
** minimum_signal_height: in units of background noise sigma, how much signal is needed for a spot
** minimum_spot_height: minimum height for a pixel to be considered a maximum (after it's determined to be signal)
** minimum_spot_area: minimum area in pixels for each spot. NOTE: there is a longstanding bug in Spotfinder; increment this number by +1 to get the actually-used minimum area.
** spot_area_maximum_factor: in multiples of minimum spot area, how large spots are allowed to be
** compactness_filter: (Use False). This is an experimental algorithm that insists that spots need to be compact; e.g., a line 4 pixels long is not an acceptable spot. However, it turns out that many XFEL spots are not compact, so this algorithm degrades XFEL performance.
** method2_cutoff_percentage: Controls how method2 (see the Spotfinder paper) chooses the resolution cutoff. Synchrotron data: 25 is OK. XFEL stills: requires low values, try 5. Lower numbers result in the determination of more expansive values for the resolution cutoff.
** peak_intensity_maximum_factor: a peak intensity filter
* Subcategory <code>indexing</code>:
** verbose_cv: if true, correction vectors are generated. This is verbose output required for the metrology cutoff. Not for routine users.
* Subcategory <code>integration</code>:
** background_factor: ratio of # pixels used for background subtraction : # pixels in spot
** detector_gain: ADU units per photon
** model: ''labelit'' has several integration models, and allows users to provide their own. The model listed here is custom for these xfel applications
** signal_penetration: thickness of the CSPAD sensors.
** spotfinder_subset: which spots found by ''spotfinder'' to use. Choose from: < add >

== Metrology phil ==

'''Update 12/11/13: quad translations and tile translations should no longer be specified in the phil file. These live in the code for now. We are re-working the file storage type and in the future we won't need these at all.'''

''cctbx.xfel'' applies 4 levels of metrology:
* Optical metrology: this is supplied by LCLS. The pyana config file specifies a path to a directory where initial tile placements are specified.
* Quadrant translations: adjustments to each of the 4 quadrants as a whole. ''(12/11/13: specified in the codebase)''
* Unit-pixel or whole-pixel metrology: a series of whole-pixel translations applied to each tile. ''(12/11/13: specified in the codebase)''
* Sub-pixel metrology: fractional corrections including translations and rotations of each tile. Used during integration.

Sub-pixel metrology corrections are contained in the metrology ''phil'' file. For this tutorial that is metrology-7.1.phil:

<pre>
distl {
detector_tiling = None
peripheral_margin = 1
)
integration {
# This is L785_119.
subpixel_joint_model {
rotations = \
0.1056719175 0.1080664532 -0.2151669703 -0.2099097279 \
...
translations = \
0.2700895641 -0.6117579705 -0.5111648376 -0.8676594815 \
...
}
}
</pre>

These parameters are in detail:
* distl subsection: quadrant translations and single-pixel tile translations
** detector_tiling: < add >
** peripheral_margin: how many pixels to leave as a border around the final image
* integration subsection: sub-pixel adjustments applied during reflection integration:
** subpixel_joint_model: root tag for the sub-pixel corrections
*** rotations: angles in degrees for each tile (64)
*** translation: x/y translations for each tile (128)

Aaron: /* Metrology phil */

2013-12-12T01:11:07Z

Metrology phil

New page

As described in the [[overview]], ''phil'' files contain parameters used during hitfinding, indexing and integration. This tutorial uses two ''phil'' files during the indexing and integration step: Ls04-lysozyme.phil and metrology-7.1.phil. The former specifies parameters specific to the processing run, while the latter specifies whole-pixel and sub-pixel metrology corrections applied to the 64 CSPAD sensor tiles.

== Hitfinding/indexing/integration ''phil'' file ==

For the tutorial, Ls04-lysozyme.phil, stored in the /reg/d/ffb/cxi/temp/cctbx/tutorials/indexing directory but copied to your ~/myrelease directory during use, contains configuration settings we worked out that best process this data. The file will look like this:

<pre>
# -*- mode: Conf -*-

include file metrology-7.1.phil

# From looking at 35 images integrated with detz_offset = 581 and
# without target_cell set.
target_cell = 38 79 79 90 90 90
target_cell_centring_type = P
known_setting = 9

distl_highres_limit = 2.0
force_method2_resolution_limit = 2.0

mosaicity_limit = 1

# Set to True to pick up second lattice, if present.
#outlier_detection_switch = True

# TEST
distl_minimum_number_spots_for_indexing = 20

distl {
res.outer = 2.0
minimum_signal_height = 5
#minimum_spot_height = 10
minimum_spot_height = 5
minimum_spot_area = 1
spot_area_maximum_factor = 20
compactness_filter = False
#method2_cutoff_percentage = 5
method2_cutoff_percentage = 2.5

# Avoids intensity filter.
#peak_intensity_maximum_factor = 10000
peak_intensity_maximum_factor = 100
}

indexing {
# Set to True to generate correction vectors.
verbose_cv = True
}

integration {
background_factor = 2

#detector_gain = 7.5
detector_gain = 1.0

#model = use_case_3_simulated_annealing_7
model = user_supplied

signal_penetration = 0.5
#spot_shape_verbose = True
spotfinder_subset = spots_non-ice
}
</pre>

Several lines of parameters are given, then a few parameter blocks are specified, enclosed in {} brackets. The parameters in detail:
* include file metrology-7.1.phil: this line specifies another ''phil'' file to include. In this case, the included file specifies metrology corrections (see below).
* target_cell: the known unit cell for this sample. In the form a, b, c, alpha, beta, gamma. Must be in the primitive setting, or specify a target_cell_centring_type. Or, convert the dimensions first to the primitive setting with ''iotbx.lattice_symmetry''.
* target_cell_centring_type: the centering type of the Bravais lattice: P (default), C, I, R, or F.
* known_setting: this is the setting number of the true Bravais symmetry as output by the autoindexing procedure in ''LABELIT''. The indexing process fundamentally determines the triclinic cell, but also a list of possible Bravais lattices. Normally we associate the following setting numbers with the crystal systems:
**Triclinic=1
**Monoclinic=2
**Orthorhombic=5
**Rhombohedral=5
**Tetragonal=9
**Hexagonal=12
**Cubic=22
**However, the assumptions break down if there is pseudosymmetry, i.e., if a higher-symmetry Bravais lattice is close to the true Bravais symmetry. In these cases, a careful inspection of the ''LABELIT'' output is necessary to choose the setting, and it may be necessary to implement new code in ''cctbx.xfel'' for this (contact the authors).
* distl_highres_limit and force_method2_resolution_limit: only process to this resolution limit. [Note these are #1 and #2 of 3 resolution flags that all must be set together]
* mosaicity_limit: maximum moisicity before a frame is rejected
* distl_minimum_number_spots_for_indexing: indexing will not proceed unless there are at least this many good spots found on the image
* Subcategory distl: parameters specific to spot finding
** res.outer: resolution limit for ''spotfinder'' [Noet this is #3 of 3 resolution flags that all must be set together]
** minimum_signal_height: in units of background noise sigma, how much signal is needed for a spot
** minimum_spot_height: minimum height for a pixel to be considered a maximum (after it's determined to be signal)
** minimum_spot_area: minimum area in pixels for each spot. NOTE: there is a longstanding bug in Spotfinder; increment this number by +1 to get the actually-used minimum area.
** spot_area_maximum_factor: in multiples of minimum spot area, how large spots are allowed to be
** compactness_filter: (Use False). This is an experimental algorithm that insists that spots need to be compact; e.g., a line 4 pixels long is not an acceptable spot. However, it turns out that many XFEL spots are not compact, so this algorithm degrades XFEL performance.
** method2_cutoff_percentage: Controls how method2 (see the Spotfinder paper) chooses the resolution cutoff. Synchrotron data: 25 is OK. XFEL stills: requires low values, try 5. Lower numbers result in the determination of more expansive values for the resolution cutoff.
** peak_intensity_maximum_factor: a peak intensity filter
* Subcategory indexing:
** verbose_cv: if true, correction vectors are generated. This is verbose output required for the metrology cutoff. Not for routine users.
* Subcategory: integration
** background_factor:
** detector_gain: ADU units per photon
** model: ''labelit'' has several integration models, and allows users to provide their own. The model listed here is custom for these xfel applications
** signal_penetration: thickness of the CSPAD sensors.
** spotfinder_subset: which spots found by ''spotfinder'' to use. Choose from: < add >

== Metrology phil ==

'''Update 12/11/13: quad translations and tile translations should no longer be specified in the phil file. These live in the code for now. We are re-working the file storage type and in the future we won't need these at all.'''

''cctbx.xfel'' applies 4 levels of metrology:
* Optical metrology: this is supplied by LCLS. The pyana config file specifies a path to a directory where initial tile placements are specified.
* Quadrant translations: adjustments to each of the 4 quadrants as a whole. ''(12/11/13: specified in the codebase)''
* Unit-pixel or whole-pixel metrology: a series of whole-pixel translations applied to each tile. ''(12/11/13: specified in the codebase)''
* Sub-pixel metrology: fractional corrections including translations and rotations of each tile. Used during integration.

Sub-pixel metrology corrections are contained in the metrology ''phil'' file. For this tutorial that is metrology-7.1.phil:

<pre>
distl {
detector_tiling = None
peripheral_margin = 1
)
integration {
# This is L785_119.
subpixel_joint_model {
rotations = \
0.1056719175 0.1080664532 -0.2151669703 -0.2099097279 \
...
translations = \
0.2700895641 -0.6117579705 -0.5111648376 -0.8676594815 \
...
}
}
</pre>

These parameters are in detail:
* distl subsection: quadrant translations and single-pixel tile translations
** detector_tiling: < add >
** peripheral_margin: how many pixels to leave as a border around the final image
* integration subsection: sub-pixel adjustments applied during reflection integration:
** subpixel_joint_model: root tag for the sub-pixel corrections
*** rotations: angles in degrees for each tile (64)
*** translation: x/y translations for each tile (128)

Aaron: /* Hitfinding/indexing/integration phil file */

2013-10-03T22:48:38Z

Hitfinding/indexing/integration phil file

@@ Line 15: / Line 15: @@
 known_setting = 9
 distl_highres_limit = 2.0
 force_method2_resolution_limit = 2.0
@@ Line 65: / Line 63: @@
 Several lines of parameters are given, then a few parameter blocks are specified, enclosed in {} brackets.  The parameters in detail:
 * target_cell: known unit cell for this sample.  In the form a, b, c, alpha, beta, gamma
 * known_setting: < Nick will add a description and how to derive this soon >
@@ Line 86: / Line 85: @@
 ** model: ''labelit'' has several integration models, and allows users to provide their own.  The model listed here is custom for these xfel applications
 ** signal_penetration: thickness of the CSPAD sensors.
-** spotfinder_subset: which spots found by ''spotfinder'' to use.  Choose from:
+** spotfinder_subset: which spots found by ''spotfinder'' to use.  Choose from: < add >

Aaron: Created page with "As described in the overview, ''phil'' files contain parameters used during hitfinding, indexing and integration. This tutorial uses two ''phil'' files during the indexin..."

2013-10-03T22:22:26Z

Created page with "As described in the overview, ''phil'' files contain parameters used during hitfinding, indexing and integration. This tutorial uses two ''phil'' files during the indexin..."

New page