Preparatory steps - Revision history

Aaron: /* Running and testing the command */

2014-12-19T03:43:44Z

Running and testing the command

← Older revision		Revision as of 03:43, 19 December 2014
Line 40:		Line 40:
	To average a dark run, the user can use cxi.mpi_average. Run it with no parameters to get a full help listing with examples. To test it on 10 events before using the queuing system, run it as follows:		To average a dark run, the user can use cxi.mpi_average. Run it with no parameters to get a full help listing with examples. To test it on 10 events before using the queuing system, run it as follows:

	$ cxi.mpi_average -c '''''expname'''''/dark.cfg -x '''''expname''''' -r '''''runnumber''''' -a '''''~~addres~~''''' -d '''''detzoffset''''' -n 10 -p		$ cxi.mpi_average -c '''''expname'''''/dark.cfg -x '''''expname''''' -r '''''runnumber''''' -a '''''address''''' -d '''''detzoffset''''' -n 10 -p

	The options are described as follows:		The options are described as follows:

Aaron: /* Creating a mask image */

2014-12-12T01:58:57Z

Creating a mask image

← Older revision		Revision as of 01:58, 12 December 2014
Line 84:		Line 84:
	* Maximum projection from a lighted run (''i.e.'' an experimental run): pixels < 300 are considered non-bonded or in a shadow. [N.B.–this assumes a lot–300 may be appropriate for XPP data collected at atmosphere; not CXI data <i>in vacuo</i>–NKS.] The presence of diffracting data is not needed, but also will not interfere.		* Maximum projection from a lighted run (''i.e.'' an experimental run): pixels < 300 are considered non-bonded or in a shadow. [N.B.–this assumes a lot–300 may be appropriate for XPP data collected at atmosphere; not CXI data <i>in vacuo</i>–NKS.] The presence of diffracting data is not needed, but also will not interfere.

	To create the mask, execute ~~these commands~~:		To create the mask, execute this command:
	~~$ cd /reg/g/cctbx/tutorials/scratch/<b><i><username></i></b>/~~
	~~$ mkdir masks~~
	~~$ cd masks~~
	$ cxi.make_mask -m 10 <b><i><name of dark average></i></b>.pickle <b><i><name of dark standard deviation></i></b>.pickle \		$ cxi.make_mask -m 10 <b><i><name of dark average></i></b>.pickle <b><i><name of dark standard deviation></i></b>.pickle \
	<b><i><name of light maximum projection></i></b>.pickle		<b><i><name of light maximum projection></i></b>.pickle

Aaron: /* Create a dark average */

2014-12-12T01:55:48Z

Create a dark average

← Older revision		Revision as of 01:55, 12 December 2014
Line 40:		Line 40:
	To average a dark run, the user can use cxi.mpi_average. Run it with no parameters to get a full help listing with examples. To test it on 10 events before using the queuing system, run it as follows:		To average a dark run, the user can use cxi.mpi_average. Run it with no parameters to get a full help listing with examples. To test it on 10 events before using the queuing system, run it as follows:

	cxi.mpi_average -c '''''expname'''''/dark.cfg -x '''''expname''''' -r '''''runnumber''''' -a '''''addres''''' -d '''''detzoffset''''' -n 10 -p		$ cxi.mpi_average -c '''''expname'''''/dark.cfg -x '''''expname''''' -r '''''runnumber''''' -a '''''addres''''' -d '''''detzoffset''''' -n 10 -p

	The options are described as follows:		The options are described as follows:
Line 55:		Line 55:
	Now, to submit the averaging job to the processing queue using many processor and all the events, use bsub:		Now, to submit the averaging job to the processing queue using many processor and all the events, use bsub:

	bsub -a mympi -n 100 -q psanacsq -o dark_average.log cxi.mpi_average -c '''''expname'''''/dark.cfg -x '''''expname''''' -r '''''runnumber''''' -a '''''address''''' -d '''''detzoffset''''' -p		$ bsub -a mympi -n 100 -q psanacsq -o dark_average.log cxi.mpi_average -c '''''expname'''''/dark.cfg -x '''''expname''''' -r '''''runnumber''''' -a '''''address''''' -d '''''detzoffset''''' -p

	The job will be scheduled in the queue called <code>psanacsq</code>, which is one of several available queues, using 100 cores. Note that the job must be submitted from the <code>~/myrelease</code> directory.		The job will be scheduled in the queue called <code>psanacsq</code>, which is one of several available queues, using 100 cores. Note that the job must be submitted from the <code>~/myrelease</code> directory.

Aaron: /* Averaging an experimental light run */

2014-12-12T01:52:49Z

Averaging an experimental light run

@@ Line 67: / Line 67: @@
 == Averaging an experimental light run ==
-Once the dark average is completed, experimental, lighted, runs can be averaged to see the extent of diffraction and potential pathologies in the run.  Copy the <code>average.cfg</code> file from the tutorials directory to your pyana directory:
+Experimental, lighted, runs can be averaged to see the extent of diffraction and potential pathologies in the run.  Here, you will need this config file [[average.cfg]].  Copy the file from to your myrelease/'''''expname''''' directory.
-The configuration file will look like this:
+The configuration file uses two new modules, [https://confluence.slac.stanford.edu/display/PSDM/psana+-+Module+Catalog#psana-ModuleCatalog-Modulecspad_mod.CsPadCalib cspad_mod.CsPadCalib] and pyimgalgos.cspad_gainmap (currently undocumented).  The former uses darks prepared by [[calibman]] for dark subtraction and performs common mode correction.  The latter applies any high/low gain pixel mask applied during data collection.  Note how the data from one module is passed to the next module using inkey and outkey.  cxi.mpi_average looks for the final data using the outkey ''image0''.
 == Information contained in the light average ==

Aaron: /* Create a dark average */

2014-12-12T01:40:41Z

Create a dark average

← Older revision		Revision as of 01:40, 12 December 2014
Line 26:		Line 26:
	The CSPAD detector at LCLS is sensitive to temperature distortions and, unlike images collected at a synchrotron source, its output is not automatically dark-subtracted. Because of this it is necessary to periodically record a dark run with the shutters closed during an LCLS experiment. The dark run should contain on the order of at least 1000 images. Before an experimental run can be processed, this dark run must be averaged so that subsequent experimental data can have the dark subtracted from each image. Furthermore, events during collection can necessitate a new dark, such as ice in the beam over-saturating creating dead pixels or a change in temperature.		The CSPAD detector at LCLS is sensitive to temperature distortions and, unlike images collected at a synchrotron source, its output is not automatically dark-subtracted. Because of this it is necessary to periodically record a dark run with the shutters closed during an LCLS experiment. The dark run should contain on the order of at least 1000 images. Before an experimental run can be processed, this dark run must be averaged so that subsequent experimental data can have the dark subtracted from each image. Furthermore, events during collection can necessitate a new dark, such as ice in the beam over-saturating creating dead pixels or a change in temperature.

	To create the average, you will need this config file [dark.cfg]. Save it to your myrelease/<b><i><expname></i></b> folder.		To create the average, you will need this config file: [[dark.cfg]]. Save it to your myrelease/<b><i><expname></i></b> folder.

	The configuration file is broken into two sections, the [[#''psana'' section]] and the [[#Modules section \| modules section]].		The configuration file is broken into two sections, the [[#psana section \| ''psana'' section]] and the [[#Modules section \| modules section]].

			=== ''psana'' section ===
			The ''psana'' section specifies which modules to run. In this case we are running the built in module <code>CSPadPixCoords.CSPadNDArrProducer</code>. This module is fully documented in [https://confluence.slac.stanford.edu/display/PSDM/psana+-+Module+Catalog#psana-ModuleCatalog-ModuleCSPadPixCoords::CSPadNDArrProducer SLAC's module catalog]. If we were running multiple modules, we could chain them together here, and each module would be executed in sequence using the parameters specified in the corresponding module section.

			=== Module section ===
			The module section specifies parameters specific to the CSPadPixCoords.CSPadNDArrProducer. The user should only ever need to change the source, which dictates which detector is going to be averaged. This string can be obtained from the beamline operator.

			=== Running and testing the command ===

	~~===~~ ''~~psana~~'' ~~section ===~~		To average a dark run, the user can use cxi.mpi_average. Run it with no parameters to get a full help listing with examples. To test it on 10 events before using the queuing system, run it as follows:
	~~The~~ ''~~pyana~~'' ~~section specifies which modules to run~~. ~~In this case we are running~~ ''~~cctbx.xfel~~'''s <code>mod_average</code> module. If we were running multiple modules, we could chain them together here, and each module would be executed in sequence using the parameters specified in the corresponding module section. We also request a ~~number of processes to parallelize the job with. Even though averaging must be done on a single host, on that host the processing can be split up among multiple cores. Note, other aspects of~~ ''~~cctbx.xfel~~''~~, such~~ as ~~indexing and integration, do not have this restriction.~~
			cxi.mpi_average -c '''''expname'''''/dark.cfg -x '''''expname''''' -r '''''runnumber''''' -a '''''addres''''' -d '''''detzoffset''''' -n 10 -p

			The options are described as follows:

			* -c: path to the dark config file
			* -x: your experiment name
			* -r: the dark run number
			* -a: the address string for the detector. Will be something like CxiDs1.0:Cspad.0
			* -d: The detz offset. This is the offset (in mm) from the sample interaction region to the back the CSPAD detector rail at CXI. Typically it's on the order of between 560 and 580 mm. The beamline operator will provide you with a value. At XPP you must specify the detector distance directly as their CSPAD is on a robot arm.
			* -p: write the output as image pickle files. Currently the standard file format for ''cctbx.xfel''.

	~~=== Module section ===~~		When complete, three new .pickle files will be present in your directory. The dark average will be named '''''expname'''''_avg-r'''''runnum'''''.pickle. A similar stddev and max file will be found as well. Use the -o option to put these averages somewhere else, such as a location available for all users to access for your experiment, e.g. /reg/d/psdm/'''''instrumentname'''''/'''''expname'''''/res/darks
	~~The module section specifies parameters specific to averaging:~~
	~~;calib_dir~~
	~~: this directory is provided by LCLS and includes the layout information of the 64 tiles of the CSPAD detector~~
	~~;avg_dirname~~
	~~: directory to place the average image~~
	~~;stddev_dirname~~
	~~: directory to place the standard deviation image~~
	~~;max_dirname~~
	~~: directory to place the maximum projection image~~
	~~;detz_offset~~
	~~: experiment-specific number used to calculate the detector distance~~. ~~This is the distance~~ in ~~millimeters between the sample interaction region and the far end of the detector stage~~. The ~~XTC stream contains the distance from the detector stage to the detector~~. ~~These two numbers are subtracted to calculate the distance between sample interaction region~~ and ~~the detector~~. ~~We determine this number experimentally using a grid search, testing a range of potential detz_offsets and choosing~~ the ~~one that shows the greatest success at indexing images~~
	~~;address~~
	~~: the name of the detector in the XTC stream~~
	~~;avg_basename~~
	~~: this name is prepended~~ to a ~~date stamp~~ to ~~name the average image~~
	~~;stddev_basename~~
	~~: this name is prepended to a date stamp to name the standard deviation image~~
	~~;max_basename~~
	~~: this name is prepended to a date stamp to name the maximum projection image image~~

			Now, to submit the averaging job to the processing queue using many processor and all the events, use bsub:

	~~=== Running and testing the command ===~~		bsub -a mympi -n 100 -q psanacsq -o dark_average.log cxi.mpi_average -c '''''expname'''''/dark.cfg -x '''''expname''''' -r '''''runnumber''''' -a '''''address''''' -d '''''detzoffset''''' -p

	Before starting the averaging job the command, replace <code><b><i><username></i></b></code> in the configuration file with your user name. To average run 2 using the modified configuration file, execute		The job will be scheduled in the queue called <code>psanacsq</code>, which is one of several available queues, using 100 cores. Note that the job must be submitted from the <code>~/myrelease</code> directory.
	~~$ cxi.lsf -c ~/myrelease/dark.cfg -i /reg/d/ana11/cxi/data/Mar2013calib/xtc \~~
	~~-o /reg/g/cctbx/tutorials/scratch/<b><i><username></i></b> -p 6 -q psanacsq -r 2 -s~~
	This will request 6 processors, overriding the <code>num-cpu</code> parameter set in the configuration file. The directory specified in the <code>-i</code> option will be searched for XTC streams, and output will be arranged in the directory defined by the value of the <code>-o</code> option. The <code>-s</code> option tells ''cxi.lsf'' to process all the streams on the same host—this is a requirement for averaging. The job will be scheduled in the queue called <code>psanacsq</code>, which is one of several available queues. Note that the job must be submitted from the <code>~/myrelease</code> directory.

	Type <code>bjobs</code> to see if the job is running. When it is, follow the output thusly:		Type <code>bjobs</code> to see if the job is running. When it is, follow the output thusly:
	$ tail -f ~~/reg/g/cctbx/tutorials/scratch/<b><i><username></i></b>/r0002/000/stdout/sNN.out~~		$ tail -f dark_average.log
	Note the presence of the <code>000</code> directory. ''cxi.lsf'' places the output of each job in a separate <i>trial</i> directory, and prints this directory just before exiting. By default, trial numbers start at zero and increase by one with each subsequent analysis of the same run. To use any other, previously unused, trial number, use ''cxi.lsf'''s <code>-t</code> option.

	When the average has been completed, the output can be inspected using the ''cctbx'' image viewer:		When the average has been completed, the output can be inspected using the ''cctbx'' image viewer:
	$ cctbx.image_viewer ~~/reg/g/cctbx/tutorials/scratch/~~<~~b><i><username></i~~>~~</b>/r0002/000/out/Ds1-r0002-*.pickle~~		$ cctbx.image_viewer <path to pickle files>


	== Averaging an experimental light run ==		== Averaging an experimental light run ==

Aaron at 01:05, 12 December 2014

2014-12-12T01:05:48Z

← Older revision		Revision as of 01:05, 12 December 2014
Line 1:		Line 1:
	This tutorial covers averaging dark and light runs, and creating mask images.		This tutorial covers averaging dark and light runs, and creating mask images.

	~~UNDER CONSTRUCTION~~		== Setup your scratch area ==

			Create a scratch folder and populate it with some directories that will be used during this tutorial. Usually you will work within the ftc directory of your experiment, for example:

	~~== Setup your scratch area ==~~		/reg/d/psdm/cxi/cxi49812/ftc

	~~Create~~ a ~~scratch folder and populate it with some directories that will be used during this tutorial~~, changing <code><b><i><username></i></b></code> to your SLAC account name:		In that directory, add a directory for your username, changing <code><b><I>expname</i></b></code> to your experiment name and <code><b><i><username></i></b></code> to your SLAC account name:

	$ cd /reg/g/~~cctbx~~/~~tutorials~~/~~scratch~~/		$ cd /reg/d/psdm/cxi/<b><i><expname></i></b>/ftc
	$ mkdir <b><i><username></i></b>		$ mkdir <b><i><username></i></b>
	$ cd <b><i><username></i></b>		$ cd <b><i><username></i></b>
Line 15:		Line 16:
	$ mkdir results		$ mkdir results

	~~Or you~~ may use a directory in your home folder.		You may use a directory in your home folder for saving results but this is not advised as it will fill quickly.

			Finally, create a directory in your myrelease folder to hold config files and phil files needed during processing:
			$ cd ~/myrelease
			$ mkdir <b><i><expname></i></b>

	== Create a dark average ==		== Create a dark average ==
Line 22:		Line 26:
	The CSPAD detector at LCLS is sensitive to temperature distortions and, unlike images collected at a synchrotron source, its output is not automatically dark-subtracted. Because of this it is necessary to periodically record a dark run with the shutters closed during an LCLS experiment. The dark run should contain on the order of at least 1000 images. Before an experimental run can be processed, this dark run must be averaged so that subsequent experimental data can have the dark subtracted from each image. Furthermore, events during collection can necessitate a new dark, such as ice in the beam over-saturating creating dead pixels or a change in temperature.		The CSPAD detector at LCLS is sensitive to temperature distortions and, unlike images collected at a synchrotron source, its output is not automatically dark-subtracted. Because of this it is necessary to periodically record a dark run with the shutters closed during an LCLS experiment. The dark run should contain on the order of at least 1000 images. Before an experimental run can be processed, this dark run must be averaged so that subsequent experimental data can have the dark subtracted from each image. Furthermore, events during collection can necessitate a new dark, such as ice in the beam over-saturating creating dead pixels or a change in temperature.

	To create the average, ~~start by copying the <code>dark.cfg</code>~~ file ~~to your pyana directory:~~		To create the average, you will need this config file [dark.cfg]. Save it to your myrelease/<b><i><expname></i></b> folder.
	~~$ cd ~/myrelease~~
	~~$ sit_setup~~
	~~$ cp /reg/g/cctbx/tutorials/preprocessing/~~dark.cfg .

	~~Open <code>dark.cfg</code>. It will look something like this:~~
	~~[pyana~~]
	~~modules = my_ana_pkg~~.~~mod_average~~
	~~num-cpu = 8~~

	~~[my_ana_pkg.mod_average]~~
	~~calib_dir = /reg/g/cctbx/sources/cctbx_project/xfel/metrology/CSPad/run4/CxiDs1.0_Cspad.0~~
	~~avg_dirname = /reg/g/cctbx/tutorials/scratch~~/<b><i><~~username~~></i></b>~~/darks~~
	~~stddev_dirname = /reg/g/cctbx/tutorials/scratch/<b><i><username></i></b>/darks~~
	~~max_dirname = /reg/g/cctbx/tutorials/scratch/<b><i><username></i></b>/darks~~
	~~detz_offset = 581~~
	~~address = CxiDs1-0\|Cspad-0~~
	~~avg_basename = Ds1-r0002-avg~~
	~~stddev_basename = Ds1-r0002-stddev~~
	~~max_basename = Ds1-r0002-max~~

	The configuration file is broken into two sections, the [[#''~~pyana'' section \| ''pyana~~'' section]] and the [[#Modules section \| modules section]].		The configuration file is broken into two sections, the [[#''psana'' section]] and the [[#Modules section \| modules section]].


	=== ''~~pyana~~'' section ===		=== ''psana'' section ===
	The ''pyana'' section specifies which modules to run. In this case we are running ''cctbx.xfel'''s <code>mod_average</code> module. If we were running multiple modules, we could chain them together here, and each module would be executed in sequence using the parameters specified in the corresponding module section. We also request a number of processes to parallelize the job with. Even though averaging must be done on a single host, on that host the processing can be split up among multiple cores. Note, other aspects of ''cctbx.xfel'', such as indexing and integration, do not have this restriction.		The ''pyana'' section specifies which modules to run. In this case we are running ''cctbx.xfel'''s <code>mod_average</code> module. If we were running multiple modules, we could chain them together here, and each module would be executed in sequence using the parameters specified in the corresponding module section. We also request a number of processes to parallelize the job with. Even though averaging must be done on a single host, on that host the processing can be split up among multiple cores. Note, other aspects of ''cctbx.xfel'', such as indexing and integration, do not have this restriction.

Aaron at 00:46, 12 December 2014

2014-12-12T00:46:36Z

Show changes

Aaron: Blanked the page

2014-12-12T00:46:02Z

Blanked the page

← Older revision		Revision as of 00:46, 12 December 2014
Line 1:		Line 1:
	~~#REDIRECT [[Preparatory steps pyana version]]~~

Aaron: Aaron moved page Preparatory steps to Preparatory steps pyana version

2014-12-12T00:40:55Z

Aaron moved page Preparatory steps to Preparatory steps pyana version

New page

#REDIRECT [[Preparatory steps pyana version]]