Ha14 installation: Difference between revisions

From cctbx_xfel
Jump to navigation Jump to search
mNo edit summary
No edit summary
Line 1: Line 1:
Note, these instructions are out of date.  Draft instructions are being prepared here: [[Installation Draft]].  That page will replace this one shortly.
== Prerequisites ==


== Prerequisite: install and set up the PSDM suite ==
While these tutorials assume you wish to process XTC streams at SLAC, some users have stills collected from other sources and do not need the full PSDM suite.  If this is the case, see the installation directions below for installing on a non PSDM system.  Otherwise, it is assumed that the [[Set up PSDM software | PSDM software distribution has been set up]].  Note again that several sites already have ''cctbx.xfel'' installed, and so regular users not involved in the development of the software will not need the instructions here.


It is assumed that the [[Set up PSDM software | PSDM software distribution has been set up]].  Developers may additionally want to [[Set up ssh-agent | set up an ssh-agent]].  Note again that several sites already have ''cctbx.xfel'' installed, and that this step can be skipped for regular users not involved in the development of the software.  Once ''cctbx.xfel'' has been installed it must be [[Setup | set up]] before it can be used.
Once ''cctbx.xfel'' has been installed it must be [[Setup | set up]] before it can be used.  Developers may additionally want to [[Set up ssh-agent | set up an ssh-agent]].   


== Download and extract a source tree ==
Finally, you must have a user account on cci.lbl.gov in order to proceed, or else you will not be able to download the sources.


This step has to be performed on a host with Internet access. Not all hosts at SLAC have that, but the members of <i>e.g.</i> psexport pool do. In what follows, the absolute path <code><b><i>/path/to/cctbx/sources</i></b></code> should be replaced with something more appropriate (a site-wide install will benefit from a location on a shared disk, whereas <code>~/projects/phenix-src</code> is a common choice among developers). The directory must accessible everywhere ''cctbx.xfel'' is to be run.
== Standard installation using PSDM ==


=== Regular users ===
This step has to be performed on a host with Internet access. Not all hosts at SLAC have that, but the members of <i>e.g.</i> pslogin pool do. Make and change to a working directory to contain the new source code and build (this directory should be accessible from any computing nodes ''cctbx.xfel'' will be run. Then download these bootstrap modules:
$ mkdir -p <b><i>/path/to/cctbx/sources</i></b>
$ cd <b><i>/path/to/cctbx/sources</i></b>
$ wget http://cci.lbl.gov/cctbx_build/results/current/cctbx_bundle.tar.gz
$ wget http://adder.lbl.gov/cctbx.xfel/downloads/labelit_bundle_current.tar.gz
$ mkdir cctbx_project
$ tar -xpvzf cctbx_bundle.tar.gz -C cctbx_project --strip-components=1
$ tar -xpvzf labelit_bundle_current.tar.gz


<!--
  svn export svn://svn.code.sf.net/p/cctbx/code/trunk/libtbx/auto_build/bootstrap.py
{{ mbox
| type = speedy
| text = New bundles are automatically prepared nightly. It is, however, not advisable to automatically update from the latest bundle. Please check with the program authors before upgrading a critical installation.
}}
-->


<!-- Note to editors: nightly bundles are stored in /net/cci/auto_build/cctbx/results.  The nightly LABELIT bundles are generated by a cron job on adder and stored at /net/adder/raid1/hattne/work/downloads.  The script to generate the LABELIT bundles is at /net/adder/raid1/hattne/work/make_labelit_source_bundle.sh. -->
Then:


=== Developers ===
  python bootstrap.py hot update --builder=xfel --cciuser=<cciusername> --sfuser=<sourceforgeusername>
A user account on cci.lbl.gov is required, the name of which is to be substituted for <code><b><i>user</i></b></code> below.
$ mkdir -p <b><i>/path/to/cctbx/sources</i></b>
$ cd <b><i>/path/to/cctbx/sources</i></b>
$ svn export svn+ssh://<b><i>user</i></b>@cci.lbl.gov/phenix_regression/trunk/phenix_svn_getting_started.csh
$ ./phenix_svn_getting_started.csh <b><i>user</i></b>
$ rm phenix_svn_getting_started.csh


Developers may also want to check out the bleeding edge [http://www.bernstein-plus-sons.com/software/CBF CBFlib] sources, as well as sources and auxiliary data for the [http://dials.sourceforge.net DIALS] project.
This command instructs bootstrap.py to download static packages required for ''cctbx.xfel'' (hot), and then to checkout the rest of the sources from cci.lbl.gov and sourceforge.net (update), using the user accounts specified by the cciuser and sfuser parameters, respectively.
  $ svn checkout "http://svn.code.sf.net/p/cbflib/code-0/trunk/CBFlib_bleeding_edge" cbflib
  $ svn checkout "http://svn.code.sf.net/p/dials/code/trunk" dials
  $ svn checkout "svn+ssh://user@cci.lbl.gov/dials_regression/trunk" dials_regression


== Create the <code>cxi_user</code> directory ==
After downloading the sources, you need to be sure you have the appropriate compilers before executing the next command.  At SLAC, that means you need to ssh to one of the psana nodes, as the pslogin nodes do not have the requisite compilers.  When ready, configure and compile thusly:


Add the <code>cxi_user</code> directory and its <code>__init__.py</code> module.  This defines the default integration algorithm used by ''cctbx.xfel''.
  python bootstrap.py build --builder=xfel --with-python=`which python`
$ mkdir cxi_user
$ cat > cxi_user/__init__.py << EOF
from xfel.mono_simulation.mono_treatment import post_outlier_rejection
from xfel.mono_simulation.mono_treatment import pre_get_predictions
EOF


== Configure the source tree ==
On SLAC's interactive nodes, this takes just over 6 minutes. To avoid problems with run-time dynamic linking of Python extensions, the Python interpreter required for the above command must be the one provided by the PSDM software distribution.  That interpreter can be located using find, <i>e.g.</i>
To avoid problems with run-time dynamic linking of Python extensions, the Python interpreter required for the next step must be the one provided by the PSDM software distribution.  That interpreter can be located using find, <i>e.g.</i>
  $ find $SIT_ROOT/sw/external/python -perm /0111 -type f -wholename "*/$SIT_ARCH/*/python" 2> /dev/null
  $ find $SIT_ROOT/sw/external/python -perm /0111 -type f -wholename "*/$SIT_ARCH/*/python" 2> /dev/null
At SLAC this interpreter is located somewhere under <code>/reg/g/psdm/sw/external/python</code>.  Then create and initialize the build directory, <code><b><i>/path/to/cctbx/build</i></b></code> below, using the Python interpreter located using the above find command.  The build directory is often located next to the source directory (<code>~/projects/phenix-build</code> is a common developer choice).
At SLAC this interpreter is located somewhere under <code>/reg/g/psdm/sw/external/python</code>.  Here we use `which python` to get this path automatically.   


=== Regular users ===
Initialise the running shell using the newly created configuration files.  bash-users should
  $ mkdir -p <b><i>/path/to/cctbx/build</i></b>
  $ . build/setpaths.sh
  $ cd <b><i>/path/to/cctbx/build</i></b>
while csh-users will instead need to run
$ <b><i>python</i></b> <b><i>/path/to/cctbx/sources</i></b>/cctbx_project/libtbx/configure.py xfel
  % source build/setpaths.csh
 
To finalize the installation, see [[Setup]].
 
Note, you may also follow this procedure if you are on a machine where you have your own python that you want to use, instead of the one provided by PSDM.
 
== Installation on a non-PSDM system ==
 
If you find yourself on a machine without PSDM and won't be dealing with XTC directly, for example if you have your own data collected as stills from a non-XFEL source, you can use the below commands in a new directory (again assuming you have a cci user account):


=== Developers ===
  svn export svn://svn.code.sf.net/p/cctbx/code/trunk/libtbx/auto_build/bootstrap.py
$ mkdir -p <b><i>/path/to/cctbx/build</i></b>
  python bootstrap.py hot update base build --builder=xfel --cciuser=<cciusername> --sfuser=<sourceforgeusername>
$ cd <b><i>/path/to/cctbx/build</i></b>
$ <b><i>python</i></b> <b><i>/path/to/cctbx/sources</i></b>/cctbx_project/libtbx/configure.py cxi_xdr_xes labelit_regression phenix xfel
<!-- XXX at NERSC, it’s installed at /project/projectdirs/lcls XXX Note that we may require tntbx for singular value decomposition.  -->


Here, in addition to hot, update and build, there's a command called 'base', which downloads a python and needed packages and compiles it.  After this process completes, source the installation (for details see above), and then add scipy to it:


== Compile the sources ==
  . build/setpaths.sh
  libtbx.python -m easy_install scipy


Initialise the running shell using the newly created configuration files.  bash-users should
Please contact the authors if with any issues.  Some gotchas:
$ . setpaths.sh
while csh-users will instead need to run
  % source setpaths.csh


The next step requires a C++ compiler.  At SLAC, the members of the psexport pool do not have any compilers installed, but the interactive nodes, reachable <i>via</i> psanacs, do.   
1) On a Mac, you will need to make sure you have the latest version of xcode installed.  Consider the command xcode-select --install.
$ cd <b><i>/path/to/cctbx/build</i></b>
$ make
$ make
Please be sure to run make twice, as shown. On SLAC's interactive nodes, this takes just over 6 minutes.  To finalize the installation, see [[Setup]].


<!-- XXX Long-standing gremlin: it appears that for initial installs, it is necessary to run make twice!  XXX Maybe should include a bit on running the regression tests here? -->
2) On linux you will likely need to install some dependencies, including gFortran.


== External links ==
== External links ==

Revision as of 19:07, 5 March 2015

Prerequisites

While these tutorials assume you wish to process XTC streams at SLAC, some users have stills collected from other sources and do not need the full PSDM suite. If this is the case, see the installation directions below for installing on a non PSDM system. Otherwise, it is assumed that the PSDM software distribution has been set up. Note again that several sites already have cctbx.xfel installed, and so regular users not involved in the development of the software will not need the instructions here.

Once cctbx.xfel has been installed it must be set up before it can be used. Developers may additionally want to set up an ssh-agent.

Finally, you must have a user account on cci.lbl.gov in order to proceed, or else you will not be able to download the sources.

Standard installation using PSDM

This step has to be performed on a host with Internet access. Not all hosts at SLAC have that, but the members of e.g. pslogin pool do. Make and change to a working directory to contain the new source code and build (this directory should be accessible from any computing nodes cctbx.xfel will be run. Then download these bootstrap modules: 

 svn export svn://svn.code.sf.net/p/cctbx/code/trunk/libtbx/auto_build/bootstrap.py

Then: 

 python bootstrap.py hot update --builder=xfel --cciuser=<cciusername> --sfuser=<sourceforgeusername>

This command instructs bootstrap.py to download static packages required for cctbx.xfel (hot), and then to checkout the rest of the sources from cci.lbl.gov and sourceforge.net (update), using the user accounts specified by the cciuser and sfuser parameters, respectively.

After downloading the sources, you need to be sure you have the appropriate compilers before executing the next command. At SLAC, that means you need to ssh to one of the psana nodes, as the pslogin nodes do not have the requisite compilers. When ready, configure and compile thusly: 

 python bootstrap.py build --builder=xfel --with-python=`which python`

On SLAC's interactive nodes, this takes just over 6 minutes. To avoid problems with run-time dynamic linking of Python extensions, the Python interpreter required for the above command must be the one provided by the PSDM software distribution. That interpreter can be located using find, e.g. 

$ find $SIT_ROOT/sw/external/python -perm /0111 -type f -wholename "*/$SIT_ARCH/*/python" 2> /dev/null

At SLAC this interpreter is located somewhere under /reg/g/psdm/sw/external/python. Here we use `which python` to get this path automatically.

Initialise the running shell using the newly created configuration files. bash-users should 

$ . build/setpaths.sh

while csh-users will instead need to run 

% source build/setpaths.csh

To finalize the installation, see Setup.

Note, you may also follow this procedure if you are on a machine where you have your own python that you want to use, instead of the one provided by PSDM.

Installation on a non-PSDM system

If you find yourself on a machine without PSDM and won't be dealing with XTC directly, for example if you have your own data collected as stills from a non-XFEL source, you can use the below commands in a new directory (again assuming you have a cci user account): 

 svn export svn://svn.code.sf.net/p/cctbx/code/trunk/libtbx/auto_build/bootstrap.py
 python bootstrap.py hot update base build --builder=xfel --cciuser=<cciusername> --sfuser=<sourceforgeusername>

Here, in addition to hot, update and build, there's a command called 'base', which downloads a python and needed packages and compiles it. After this process completes, source the installation (for details see above), and then add scipy to it: 

 . build/setpaths.sh
 libtbx.python -m easy_install scipy

Please contact the authors if with any issues. Some gotchas:

1) On a Mac, you will need to make sure you have the latest version of xcode installed. Consider the command xcode-select --install.

2) On linux you will likely need to install some dependencies, including gFortran.

External links


Retrieved from "http://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Ha14_installation&oldid=917"