Indexing and integration - Revision history

Aaron at 23:03, 18 August 2026

2026-08-18T23:03:54Z

← Older revision		Revision as of 23:03, 18 August 2026
Line 1:		Line 1:
			{{Archived}}

	In principle, one could attempt to index and integrate every single image recorded during a "diffract-and-destroy" experiment. Typically only a small fraction of the recorded images exhibit an indexable lattice, and in practice a ''hit-finder'' is needed to triage the stream of images, and preserve computational resources.		In principle, one could attempt to index and integrate every single image recorded during a "diffract-and-destroy" experiment. Typically only a small fraction of the recorded images exhibit an indexable lattice, and in practice a ''hit-finder'' is needed to triage the stream of images, and preserve computational resources.

Aaron: /* Processing the tutorial data */

2015-05-08T17:35:33Z

Processing the tutorial data

← Older revision		Revision as of 17:35, 8 May 2015
Line 116:		Line 116:
	-c Ls04-lysozyme.cfg -i /reg/d/ana11/cxi/data/Mar2013calib/xtc \		-c Ls04-lysozyme.cfg -i /reg/d/ana11/cxi/data/Mar2013calib/xtc \
	-o /reg/g/cctbx/tutorials/scratch/<username>/lysozyme \		-o /reg/g/cctbx/tutorials/scratch/<username>/lysozyme \
	-p 6 -q ~~psanacsq~~ -r 3		-p 6 -q psanaq -r 3
	</pre>		</pre>

Aaron at 03:00, 12 December 2014

2014-12-12T03:00:32Z

← Older revision		Revision as of 03:00, 12 December 2014
Line 3:		Line 3:
	Indexing and integration are tightly coupled; it is impossible to integrate an image without first obtaining an indexing solution. The result of successfully indexing an image is essentially a model of crystal–beam interaction. The model not only describes how the crystal is oriented with respect to the direction of the X-ray pulse, but also its degree of imperfection (''i.e.'' mosaicity), and qualities of the illuminating pulse (''e.g.'' bandwidth). The integration algorithm uses this model to predict where Bragg spots appear on the image, and integrates the intensities around those predications. The integration algorithm thus exploits the regularity of the underlying crystal lattice to recover signal where it was too weak to be detected.		Indexing and integration are tightly coupled; it is impossible to integrate an image without first obtaining an indexing solution. The result of successfully indexing an image is essentially a model of crystal–beam interaction. The model not only describes how the crystal is oriented with respect to the direction of the X-ray pulse, but also its degree of imperfection (''i.e.'' mosaicity), and qualities of the illuminating pulse (''e.g.'' bandwidth). The integration algorithm uses this model to predict where Bragg spots appear on the image, and integrates the intensities around those predications. The integration algorithm thus exploits the regularity of the underlying crystal lattice to recover signal where it was too weak to be detected.

	== <code>~~pyana~~</code> section ==		== <code>psana</code> section ==

	For historical reasons, hit-finding and indexing/integration are both implemented in the same analysis module, called <code>mod_hitfind</code>. To chain the two functions together, ''~~pyana~~'' must be configured to pass each event through two different instances of the same module. The two instances are differentiated by their name following the colon. In the example below, the first instance is called <code>hitfind</code>, and the second is <code>index</code>.		For historical reasons, hit-finding and indexing/integration are both implemented in the same analysis module, called <code>mod_hitfind</code>. To chain the two functions together, ''psana'' must be configured to pass each event through two different instances of the same module. The two instances are differentiated by their name following the colon. In the example below, the first instance is called <code>hitfind</code>, and the second is <code>index</code>.
	<pre>		<pre>
	[~~pyana~~]		[psana]
	modules = my_ana_pkg.mod_hitfind:hitfind \		modules = my_ana_pkg.mod_hitfind:hitfind \
	my_ana_pkg.mod_hitfind:index		my_ana_pkg.mod_hitfind:index
	</pre>		</pre>
	Any instance of an analysis module can signal ''~~pyana~~'' to skip further processing of the event. In the case above, the <code>hitfind</code> instance of <code>mod_hitfind</code> would signal ''~~pyana~~'' to skip the event if it determines that the image associated with the event is not a hit. In that case, the second instance, <code>index</code>, will never see the event. Otherwise, the event is fed to the second instance, which will attempt indexing and integration on the associated image.		Any instance of an analysis module can signal ''psana'' to skip further processing of the event. In the case above, the <code>hitfind</code> instance of <code>mod_hitfind</code> would signal ''psana'' to skip the event if it determines that the image associated with the event is not a hit. In that case, the second instance, <code>index</code>, will never see the event. Otherwise, the event is fed to the second instance, which will attempt indexing and integration on the associated image.

	The <code>~~pyana~~</code>-section of the processing configuration used for this part of the tutorial has one more feature.		The <code>psana</code>-section of the processing configuration used for this part of the tutorial has one more feature.
	<pre>		<pre>
	[~~pyana~~]		[psana]
	modules = my_ana_pkg.mod_hitfind:hitfind \		modules = my_ana_pkg.mod_hitfind:hitfind \
	my_ana_pkg.mod_dump:hit \		my_ana_pkg.mod_dump:hit \

Hattne: Replaced ./lsf.sh with cxi.lsf.

2014-02-11T00:31:31Z

Replaced ./lsf.sh with cxi.lsf.

← Older revision		Revision as of 00:31, 11 February 2014
Line 113:		Line 113:
	$ sit_setup		$ sit_setup
	$ cp /reg/g/cctbx/tutorials/indexing/Ls04-lysozyme.cfg .		$ cp /reg/g/cctbx/tutorials/indexing/Ls04-lysozyme.cfg .
	$ ./lsf~~.sh~~ \		$ cxi.lsf \
	-c Ls04-lysozyme.cfg -i /reg/d/ana11/cxi/data/Mar2013calib/xtc \		-c Ls04-lysozyme.cfg -i /reg/d/ana11/cxi/data/Mar2013calib/xtc \
	-o /reg/g/cctbx/tutorials/scratch/<username>/lysozyme \		-o /reg/g/cctbx/tutorials/scratch/<username>/lysozyme \

Aaron: /* Processing the tutorial data */

2014-02-04T22:48:28Z

Processing the tutorial data

← Older revision		Revision as of 22:48, 4 February 2014
Line 112:		Line 112:
	$ cd ~/myrelease		$ cd ~/myrelease
	$ sit_setup		$ sit_setup
	$ cp /reg/~~d/ffb/cxi/temp~~/cctbx/tutorials/indexing/Ls04-lysozyme.cfg .		$ cp /reg/g/cctbx/tutorials/indexing/Ls04-lysozyme.cfg .
	$ ~~cp /reg/d/ffb/cxi/temp/cctbx/tutorials/indexing/Ls04-lysozyme.phil .~~		$ ./lsf.sh \
	~~$ cp /reg/d/ffb/cxi/temp/cctbx/tutorials/indexing/metrology-7.1.phil .~~		-c Ls04-lysozyme.cfg -i /reg/d/ana11/cxi/data/Mar2013calib/xtc \
	~~$ cp /reg/d/ffb/cxi/temp/cctbx/tutorials/indexing/hitfind-7~~.~~1.phil .~~		-o /reg/g/cctbx/tutorials/scratch/<username>/lysozyme \
	~~$ /reg/d/ffb/cxi/temp/cctbx/cctbx_xfel/sources/cctbx_project/xfel/cxi~~/lsf.sh \		-p 6 -q psanacsq -r 3
	-c Ls04-lysozyme.cfg -i /reg/d/~~ffb~~/cxi/~~temp~~/xtc \
	-o /reg/~~d/ffb/cxi/temp~~/cctbx/tutorials/scratch/<username>/lysozyme \
	-p 6 -q ~~psfehq~~ -r 3
	</pre>		</pre>

Aaron: /* Indexing and integration */

2014-02-04T22:45:34Z

Indexing and integration

← Older revision		Revision as of 22:45, 4 February 2014
Line 85:		Line 85:
	<pre>		<pre>
	address = CxiDs1-0\|Cspad-0		address = CxiDs1-0\|Cspad-0
	calib_dir = /reg/~~d/ffb/cxi/temp~~/cctbx~~/cctbx_xfel~~/sources/cctbx_project/xfel/metrology/CSPad/run4/CxiDs1.0_Cspad.0		calib_dir = /reg/g/cctbx/sources/cctbx_project/xfel/metrology/CSPad/run4/CxiDs1.0_Cspad.0
	dark_path = /reg/~~d/ffb/cxi/temp~~/cctbx/tutorials/darks/Ds1-r0002-avg.pickle		dark_path = /reg/g/cctbx/tutorials/darks/Ds1-r0002-avg.pickle
	dark_stddev = /reg/~~d/ffb/cxi/temp~~/cctbx/tutorials/darks/Ds1-r0002-stddev.pickle		dark_stddev = /reg/g/cctbx/tutorials/darks/Ds1-r0002-stddev.pickle
	detz_offset = 581		detz_offset = 581
	dispatch = index		dispatch = index
	integration_dirname = ~~/reg/d/ffb/cxi/temp/cctbx/tutorials/scratch/<username>/lysozyme~~		integration_dirname = integration
	integration_basename = int-		integration_basename = int-
	xtal_target = Ls04-lysozyme.phil		xtal_target = Ls04-lysozyme.phil

Aaron: /* Image output */

2014-02-04T22:44:46Z

Image output

← Older revision		Revision as of 22:44, 4 February 2014
Line 68:		Line 68:
	[my_ana_pkg.mod_dump:hit]		[my_ana_pkg.mod_dump:hit]
	address = CxiDs1-0\|Cspad-0		address = CxiDs1-0\|Cspad-0
	calib_dir = /reg/~~d/ffb/cxi/temp~~/cctbx~~/cctbx_xfel~~/sources/cctbx_project/xfel/metrology/CSPad/run4/CxiDs1.0_Cspad.0		calib_dir = /reg/g/cctbx/sources/cctbx_project/xfel/metrology/CSPad/run4/CxiDs1.0_Cspad.0
	out_dirname = ~~/reg/d/ffb/cxi/temp/cctbx/tutorials/scratch/<username>/lysozyme~~		out_dirname = out
	out_basename = shot-		out_basename = shot-
	</pre>		</pre>

Aaron: /* Hit-finding */

2014-02-04T22:43:28Z

Hit-finding

← Older revision		Revision as of 22:43, 4 February 2014
Line 31:		Line 31:
	[my_ana_pkg.mod_hitfind:hitfind]		[my_ana_pkg.mod_hitfind:hitfind]
	address = CxiDs1-0\|Cspad-0		address = CxiDs1-0\|Cspad-0
	calib_dir = /reg/~~d/ffb/cxi/temp~~/cctbx~~/cctbx_xfel~~/sources/cctbx_project/xfel/metrology/CSPad/run4/CxiDs1.0_Cspad.0		calib_dir = /reg/g/cctbx/sources/cctbx_project/xfel/metrology/CSPad/run4/CxiDs1.0_Cspad.0
	dark_path = /reg/~~d/ffb/cxi/temp~~/cctbx/tutorials/darks/Ds1-r0002-avg.pickle		dark_path = /reg/g/cctbx/tutorials/darks/Ds1-r0002-avg.pickle
	dark_stddev = /reg/~~d/ffb/cxi/temp~~/cctbx/tutorials/darks/Ds1-r0002-stddev.pickle		dark_stddev = /reg/g/cctbx/tutorials/darks/Ds1-r0002-stddev.pickle
	detz_offset = 581		detz_offset = 581
	dispatch = nop		dispatch = nop
Line 39:		Line 39:
	distl_min_peaks = 16		distl_min_peaks = 16
	threshold = 450		threshold = 450
	xtal_target = /reg/~~d/ffb/cxi/temp~~/cctbx/~~cctbx_xfel/sources/labelit_regression~~/~~xfel~~/hitfind-7.1.phil		xtal_target = /reg/g/cctbx/tutorials/indexing/hitfind-7.1.phil
	</pre>		</pre>

Hattne: Fixed path to lsf.sh.

2013-10-04T22:41:32Z

Fixed path to lsf.sh.

← Older revision		Revision as of 22:41, 4 October 2013
Line 116:		Line 116:
	$ cp /reg/d/ffb/cxi/temp/cctbx/tutorials/indexing/metrology-7.1.phil .		$ cp /reg/d/ffb/cxi/temp/cctbx/tutorials/indexing/metrology-7.1.phil .
	$ cp /reg/d/ffb/cxi/temp/cctbx/tutorials/indexing/hitfind-7.1.phil .		$ cp /reg/d/ffb/cxi/temp/cctbx/tutorials/indexing/hitfind-7.1.phil .
	$ ./lsf.sh -c Ls04-lysozyme.cfg -i /reg/d/ffb/cxi/temp/xtc \		$ /reg/d/ffb/cxi/temp/cctbx/cctbx_xfel/sources/cctbx_project/xfel/cxi/lsf.sh \
			-c Ls04-lysozyme.cfg -i /reg/d/ffb/cxi/temp/xtc \
	-o /reg/d/ffb/cxi/temp/cctbx/tutorials/scratch/<username>/lysozyme \		-o /reg/d/ffb/cxi/temp/cctbx/tutorials/scratch/<username>/lysozyme \
	-p 6 -q psfehq -r 3		-p 6 -q psfehq -r 3

Hattne: Cautionary note on number of jobs.

2013-10-04T11:07:54Z

Cautionary note on number of jobs.

← Older revision		Revision as of 11:07, 4 October 2013
Line 116:		Line 116:
	$ cp /reg/d/ffb/cxi/temp/cctbx/tutorials/indexing/metrology-7.1.phil .		$ cp /reg/d/ffb/cxi/temp/cctbx/tutorials/indexing/metrology-7.1.phil .
	$ cp /reg/d/ffb/cxi/temp/cctbx/tutorials/indexing/hitfind-7.1.phil .		$ cp /reg/d/ffb/cxi/temp/cctbx/tutorials/indexing/hitfind-7.1.phil .
	$ ./lsf.sh -c Ls04-lysozyme.cfg -o /reg/d/ffb/cxi/temp/cctbx/tutorials/scratch/<username>/lysozyme \		$ ./lsf.sh -c Ls04-lysozyme.cfg -i /reg/d/ffb/cxi/temp/xtc \
	-p 6 -q psfehq -r ~~4 -i /reg/d/ffb/cxi/temp/xtc~~		-o /reg/d/ffb/cxi/temp/cctbx/tutorials/scratch/<username>/lysozyme \
			-p 6 -q psfehq -r 3
	</pre>		</pre>

			To minimize the load on the cluster during the tutorial, please do not submit more than one run.

			<!--
	How to monitor progress, check on number of hits, number of integrated images. How to inspect sample images. etc		How to monitor progress, check on number of hits, number of integrated images. How to inspect sample images. etc
			-->