Difference between revisions of "Ha14 installation"

From cctbx_xfel
Jump to: navigation, search
(Configure the source tree)
(Compile the sources)
(One intermediate revision by one other user not shown)
Line 1: Line 1:
IMPORTANT: installing ''cctbx.xfel'' is not necessary for most users, as ''cctbx.xfel'' is installed at SLAC already and is generally accessible.  General users should instead go to [[setup]] to configure their unix account at SLAC to run ''cctbx.xfel.''
 
 
This installation can be performed by any regular user, no root privileges are required.
 
 
 
== Prerequisite: install and set up the PSDM suite ==
 
== Prerequisite: install and set up the PSDM suite ==
If you are installing somewhere other than SLAC, follow [https://confluence.slac.stanford.edu/display/PCDS/PSDM+software+distribution these instructions] to install SLAC's PSDM suite.
 
  
All users will need to follow these [https://confluence.slac.stanford.edu/display/PCDS/Analysis+Workbook.+Quick+Tour further instructions] to configure ''pyana''.
+
It is assumed that the [[Set up PSDM software | PSDM software distribution has been set up]].  Developers may additionally want to [[Set up ssh-agent | set up an ssh-agent]].  Note again that several sites already have ''cctbx.xfel'' installed, and that this step can be skipped for regular users not involved in the development of the softwareOnce ''cctbx.xfel'' has been installed it must be [[Setup | set up]] before it can be used.
   
+
It is assumed that ana_env.sh or ana_env.csh has been sourced, a release directory has been configured in ~/myrelease or ~analisys-rel, and that an empty analysis package has been set up in ~/myrelease/my_ana_pkg or ~/analisys-rel/my_ana_pkg
+
  
In addition, these software packages need to be available: reportlab, h5py, etc. FIXME
+
== Download and extract a source tree ==
  
== Download and extract ''cctbx'' and ''LABELIT'' source bundles ==
+
This step has to be performed on a host with Internet access. Not all hosts at SLAC have that, but the members of <i>e.g.</i> psexport pool do. In what follows, the absolute path <code><b><i>/path/to/cctbx/sources</i></b></code> should be replaced with something more appropriate (a site-wide install will benefit from a location on a shared disk, whereas <code>~/projects/phenix-src</code> is a common choice among developers). The directory must accessible everywhere ''cctbx.xfel'' is to be run.
This step has to be performed on a host with Internet access. Not all hosts at SLAC have that, but the members of e.g. psexport pool do. In what follows, the absolute path /path/to/cctbx/directory should be replaced with something more appropriate. The directory should accessible everywhere ''cctbx.xfel'' is to be run.
+
$ mkdir /path/to/cctbx/directory
+
$ cd /path/to/cctbx/directory
+
$ wget http://cci.lbl.gov/cctbx_build/results/2013_08_13_0005/cctbx_bundle.tar.gz
+
$ wget http://cci.lbl.gov/~hattne/labelit/labelit_bundle_20130814.tar.gz
+
Note: new bundles are automatically prepared nightly. It is, however, not advisable to automatically update from the latest bundle. Please check with the program authors before upgrading a critical installation.
+
  
Extract the sources:
+
=== Regular users ===
 +
$ mkdir -p <b><i>/path/to/cctbx/sources</i></b>
 +
$ cd <b><i>/path/to/cctbx/sources</i></b>
 +
$ wget http://cci.lbl.gov/cctbx_build/results/current/cctbx_bundle.tar.gz
 +
$ wget http://adder.lbl.gov/cctbx.xfel/downloads/labelit_bundle_current.tar.gz
 
  $ mkdir cctbx_project
 
  $ mkdir cctbx_project
 
  $ tar -xpvzf cctbx_bundle.tar.gz -C cctbx_project --strip-components=1
 
  $ tar -xpvzf cctbx_bundle.tar.gz -C cctbx_project --strip-components=1
  $ tar -xpvzf labelit_bundle_20130814.tar.gz
+
  $ tar -xpvzf labelit_bundle_current.tar.gz
  
== Configure the source tree ==
+
<!--
This step and the next require access to a C++ compiler and a python interpreter, version 2.7.  At SLAC, the members of the psexport pool do not have any compilers installed, but the interactive nodes, reachable via psanacs, do. Furthermore, the Python interpreter used to configure the source tree must be the same interpreter the PSDM suite uses. At SLAC this interpreter is located somewhere under /reg/g/psdm. To check which file is executed when python is typed on the command line, one can use which python. If this is not the python used by the PSDM suite, the full path to the proper interpreter, e.g./reg/g/psdm/sw/external/python/2.7.2/x86_64-rhel5-gcc41-opt/bin/python, should be given instead.
+
{{ mbox
 +
| type = speedy
 +
| text = New bundles are automatically prepared nightly. It is, however, not advisable to automatically update from the latest bundle. Please check with the program authors before upgrading a critical installation.
 +
}}
 +
-->
  
First, add the cxi_user directory and its __init__.pyThis configures the integration routines used by ''cctbx.xfel'':
+
<!-- Note to editors: nightly bundles are stored in /net/cci/auto_build/cctbx/results.  The nightly LABELIT bundles are generated by a cron job on adder and stored at /net/adder/raid1/hattne/work/downloadsThe script to generate the LABELIT bundles is at /net/adder/raid1/hattne/work/make_labelit_source_bundle.sh. -->
  
 +
=== Developers ===
 +
A user account on cci.lbl.gov is required, the name of which is to be substituted for <code><b><i>user</i></b></code> below.
 +
$ mkdir -p <b><i>/path/to/cctbx/sources</i></b>
 +
$ cd <b><i>/path/to/cctbx/sources</i></b>
 +
$ svn export svn+ssh://<b><i>user</i></b>@cci.lbl.gov/phenix_regression/trunk/phenix_svn_getting_started.csh
 +
$ ./phenix_svn_getting_started.csh <b><i>user</i></b>
 +
$ rm phenix_svn_getting_started.csh
 +
 +
 +
== Create the <code>cxi_user</code> directory ==
 +
 +
Add the <code>cxi_user</code> directory and its <code>__init__.py</code> module.  This defines the default integration algorithm used by ''cctbx.xfel''.
 
  $ mkdir cxi_user
 
  $ mkdir cxi_user
  $ printf "from xfel.mono_simulation.mono_treatment import post_outlier_rejection\nfrom xfel.mono_simulation.mono_treatment import pre_get_predictions\n\n" > cxi_user/__init__.py
+
  $ cat > cxi_user/__init__.py << EOF
 +
from xfel.mono_simulation.mono_treatment import post_outlier_rejection
 +
from xfel.mono_simulation.mono_treatment import pre_get_predictions
 +
EOF
  
Next, create your build folder and compile the sources:
+
== Configure the source tree ==
 +
To avoid problems with run-time dynamic linking of Python extensions, the Python interpreter required for the next step must be the one provided by the PSDM software distribution.  That interpreter can be located using find, <i>e.g.</i>
 +
$ find $SIT_ROOT/sw/external/python -perm /0111 -type f -wholename "*/$SIT_ARCH/*/python" 2> /dev/null
 +
At SLAC this interpreter is located somewhere under <code>/reg/g/psdm/sw/external/python</code>.  Then create and initialize the build directory, <code><b><i>/path/to/cctbx/build</i></b></code> below, using the Python interpreter located using the above find command.  The build directory is often located next to the source directory (<code>~/projects/phenix-build</code> is a common developer choice).
  
  $ mkdir cctbx_build
+
=== Regular users ===
  $ cd cctbx_build
+
  $ mkdir -p <b><i>/path/to/cctbx/build</i></b>
  $ python ../cctbx_project/libtbx/configure.py xfel
+
  $ cd <b><i>/path/to/cctbx/build</i></b>
 +
  $ <b><i>python</i></b> <b><i>/path/to/cctbx/sources</i></b>/cctbx_project/libtbx/configure.py xfel
 +
 
 +
=== Developers ===
 +
$ mkdir -p <b><i>/path/to/cctbx/build</i></b>
 +
$ cd <b><i>/path/to/cctbx/build</i></b>
 +
$ <b><i>python</i></b> <b><i>/path/to/cctbx/sources</i></b>/cctbx_project/libtbx/configure.py cxi_xdr_xes labelit_regression phenix xfel
 +
<!-- XXX at NERSC, it’s installed at /project/projectdirs/lcls XXX Note that we may require tntbx for singular value decomposition.  -->
  
Now, initialise the running shell using the newly created configuration files. bash(1) should do
 
$ . setpaths.sh
 
while csh(1) users should instead run
 
$ source setpaths.csh
 
  
 
== Compile the sources ==
 
== Compile the sources ==
 +
 +
Initialise the running shell using the newly created configuration files.  bash-users should
 +
$ . setpaths.sh
 +
while csh-users will instead need to run
 +
% source setpaths.csh
 +
 +
The next step requires a C++ compiler.  At SLAC, the members of the psexport pool do not have any compilers installed, but the interactive nodes, reachable <i>via</i> psanacs, do. 
 +
$ cd <b><i>/path/to/cctbx/build</i></b>
 
  $ make
 
  $ make
 
  $ make
 
  $ make
On SLAC's interactive nodes, this takes just over 6 minutes.
+
Please be sure to run make twice, as shown. On SLAC's interactive nodes, this takes just over 6 minutes.  To finalize the installation, see [[Setup]].
 +
 
 +
<!-- XXX Long-standing gremlin: it appears that for initial installs, it is necessary to run make twice!  XXX Maybe should include a bit on running the regression tests here? -->
 +
 
 +
== External links ==
 +
 
 +
* [http://cctbx.sourceforge.net/current_cvs/installation.html#manually-building-from-sources-under-unix Manually building from sources under Unix]
 +
 
 +
 
 +
<!-- POSSBILE ADDITIONS:
 +
 
 +
COMMON PITFALLS:
 +
 
 +
forget to source the shell configuration files
 +
forget to run sit_setup
 +
running cxi.pyana from outside the myrelease directory
 +
 
 +
DEBUGGING:
 +
 
 +
And a few more things to check: output of "ls -lR arch/*", output of "which python2.7", and the "scons.out" file that results from "scons TRACE=7 > scons.out 2>&1".
  
== Edit shell configuration files ==
+
What Linux distribution are you running this on (could you past the output of "uname -a" and "cat /etc/issue")?  Could you also tell me which scons you are using (output of "which scons"). Last, could you attach a dump of the environment (env > env.out)?
To make the changes to the environment persistent, bash(1) users are encouraged to add these lines to ~/.bashrc
+
test -r "/path/to/cctbx/directory/cctbx-build/setpaths.sh" && \
+
  . "/path/to/cctbx/directory/cctbx-build/setpaths.sh"
+
  
csh(1) users should instead add these lines to ~/.cshrc
+
Andy's information request on debugging: full output from these commands:
test -r "/path/to/cctbx/directory/cctbx-build/setpaths.csh" && \
+
  source "/path/to/cctbx/directory/cctbx-build/setpaths.csh"
+
Note: to avoid surprises when running ''cctbx.xfel'' from a non-interactive shell, it is recommended that bash(1) users edit ~/.bash_profile, and put this line near the top:
+
test -r "${HOME}/.bashrc" && . "${HOME}/.bashrc"
+
A job submitted to the cluster will typically run cctbx.xfel from a non-interactive shell.
+
  
== Link ''cctbx.xfel'' modules to the PSDM suite. ==
+
% rm -rf arch build
$ cd ~/myrelease/my_ana_pkg
+
% printenv
$ ln -s /path/to/cctbx/directory/cctbx_project/xfel/cxi/cspad_ana src
+
% which scons
$ cd ..
+
% which python
$ scons
+
% which python2.7
 +
% scons TRACE=9
 +
% ls -lR arch
 +
% ls -lR my_ana_pkg
  
== Test the installation ==
+
-->
At SLAC, test that it is working by e.g.
+
$ cd ~/myrelease
+
$ cxi.pyana -c /reg/d/ffb/cxi/temp/cctbx/tutorials/setup/test.cfg /reg/d/ffb/cxi/temp/Feb2013calib/e290-r0069-s00-c00.xtc
+
This should start a viewer, displaying dark-subtracted averages from an XTC stream.
+

Revision as of 21:26, 11 April 2014

Prerequisite: install and set up the PSDM suite

It is assumed that the PSDM software distribution has been set up. Developers may additionally want to set up an ssh-agent. Note again that several sites already have cctbx.xfel installed, and that this step can be skipped for regular users not involved in the development of the software. Once cctbx.xfel has been installed it must be set up before it can be used.

Download and extract a source tree

This step has to be performed on a host with Internet access. Not all hosts at SLAC have that, but the members of e.g. psexport pool do. In what follows, the absolute path /path/to/cctbx/sources should be replaced with something more appropriate (a site-wide install will benefit from a location on a shared disk, whereas ~/projects/phenix-src is a common choice among developers). The directory must accessible everywhere cctbx.xfel is to be run.

Regular users

$ mkdir -p /path/to/cctbx/sources
$ cd /path/to/cctbx/sources
$ wget http://cci.lbl.gov/cctbx_build/results/current/cctbx_bundle.tar.gz
$ wget http://adder.lbl.gov/cctbx.xfel/downloads/labelit_bundle_current.tar.gz
$ mkdir cctbx_project
$ tar -xpvzf cctbx_bundle.tar.gz -C cctbx_project --strip-components=1
$ tar -xpvzf labelit_bundle_current.tar.gz


Developers

A user account on cci.lbl.gov is required, the name of which is to be substituted for user below.

$ mkdir -p /path/to/cctbx/sources
$ cd /path/to/cctbx/sources
$ svn export svn+ssh://user@cci.lbl.gov/phenix_regression/trunk/phenix_svn_getting_started.csh
$ ./phenix_svn_getting_started.csh user
$ rm phenix_svn_getting_started.csh


Create the cxi_user directory

Add the cxi_user directory and its __init__.py module. This defines the default integration algorithm used by cctbx.xfel.

$ mkdir cxi_user
$ cat > cxi_user/__init__.py << EOF
from xfel.mono_simulation.mono_treatment import post_outlier_rejection
from xfel.mono_simulation.mono_treatment import pre_get_predictions
EOF

Configure the source tree

To avoid problems with run-time dynamic linking of Python extensions, the Python interpreter required for the next step must be the one provided by the PSDM software distribution. That interpreter can be located using find, e.g.

$ find $SIT_ROOT/sw/external/python -perm /0111 -type f -wholename "*/$SIT_ARCH/*/python" 2> /dev/null

At SLAC this interpreter is located somewhere under /reg/g/psdm/sw/external/python. Then create and initialize the build directory, /path/to/cctbx/build below, using the Python interpreter located using the above find command. The build directory is often located next to the source directory (~/projects/phenix-build is a common developer choice).

Regular users

$ mkdir -p /path/to/cctbx/build
$ cd /path/to/cctbx/build
$ python /path/to/cctbx/sources/cctbx_project/libtbx/configure.py xfel

Developers

$ mkdir -p /path/to/cctbx/build
$ cd /path/to/cctbx/build
$ python /path/to/cctbx/sources/cctbx_project/libtbx/configure.py cxi_xdr_xes labelit_regression phenix xfel


Compile the sources

Initialise the running shell using the newly created configuration files. bash-users should

$ . setpaths.sh

while csh-users will instead need to run

% source setpaths.csh

The next step requires a C++ compiler. At SLAC, the members of the psexport pool do not have any compilers installed, but the interactive nodes, reachable via psanacs, do.

$ cd /path/to/cctbx/build
$ make
$ make

Please be sure to run make twice, as shown. On SLAC's interactive nodes, this takes just over 6 minutes. To finalize the installation, see Setup.


External links