Difference between revisions of "Ha14 installation"

From cctbx_xfel
Jump to: navigation, search
m (Aaron moved page Installation to Ha14 installation: deprecated installation instructions)
 
(One intermediate revision by one other user not shown)
Line 1: Line 1:
−
IMPORTANT: installing ''cctbx.xfel'' is not necessary for most users, as ''cctbx.xfel'' is installed at SLAC already and is generally accessible.  General users should instead go to [[setup]] to configure their unix account at SLAC to run ''cctbx.xfel.''
+
== Prerequisites ==
  
−
This installation can be performed by any regular user, no root privileges are required.
+
While these tutorials assume you wish to process XTC streams at SLAC, some users have stills collected from other sources and do not need the full PSDM suite.  If this is the case, see the installation directions below for installing on a non PSDM system.  Otherwise, it is assumed that the [[Set up PSDM software | PSDM software distribution has been set up]].  Note again that several sites already have ''cctbx.xfel'' installed, and so regular users not involved in the development of the software will not need the instructions here.
  
−
== Prerequisite: install and set up the PSDM suite ==
+
Once ''cctbx.xfel'' has been installed it must be [[Setup | set up]] before it can be used.  Developers may additionally want to [[Set up ssh-agent | set up an ssh-agent]].
−
If you are installing somewhere other than SLAC, follow [https://confluence.slac.stanford.edu/display/PCDS/PSDM+software+distribution these instructions] to install SLAC's PSDM suite.
+
  
−
All users will need to follow these [https://confluence.slac.stanford.edu/display/PCDS/Analysis+Workbook.+Quick+Tour further instructions] to configure ''pyana''.
+
Finally, you must have a user account on cci.lbl.gov in order to proceed, or else you will not be able to download the sources. A sourceforge account is only needed if you wish to commit changes back to the repository. If you don't have or don't want to use a sourceforge account, you can leave it off in the below commands where it is specified.
 +
 
 +
== Standard installation using PSDM  ==
 +
===Quick start===
 +
This step has to be performed on a host with Internet access. Not all hosts at SLAC have that, but the members of <i>e.g.</i> pslogin pool do. Make and change to a working directory to contain the new source code and build (this directory should be accessible from any computing nodes ''cctbx.xfel'' will be run. Then download these bootstrap modules:
 +
 
 +
  wget https://raw.githubusercontent.com/cctbx/cctbx_project/master/libtbx/auto_build/bootstrap.py
 +
 
 +
Then:
 +
 
 +
  python bootstrap.py hot update --builder=xfel --cciuser=<cciusername> --sfuser=<githubusername>
 +
 
 +
This command instructs bootstrap.py to download static packages required for ''cctbx.xfel'' (hot), and then to checkout the rest of the sources from cci.lbl.gov and github (update), using the user accounts specified by the cciuser and sfuser parameters, respectively.
 +
 
 +
===Alternate instructions for non-LBNL developers===
 +
Our non-LBNL colleagues will now realize that the "python bootstrap" step does not work without cci username credentials, needed to obtain source code for the program LABELIT.  Developers are instructed to use the following alternate procedure:
 +
 
 +
  python bootstrap.py hot --builder=xfel --sfuser=<githubusername> # download the static packages
 +
  python bootstrap.py update --builder=dials --sfuser=<githubusername> # download the github code for dials & cctbx
 +
  mkdir modules/cxi_xdr_xes # create a stub directory for experiment-specific python code
 
   
 
   
−
It is assumed that ana_env.sh or ana_env.csh has been sourced, a release directory has been configured in ~/myrelease or ~analisys-rel, and that an empty analysis package has been set up in ~/myrelease/my_ana_pkg or ~/analisys-rel/my_ana_pkg
+
Then download phenix from http://phenix-online.org and untar the package.  Locate the modules directory in the phenix directory tree, and copy-paste the labelit subdirectory into the modules directory of your current working area.
 +
 
 +
===Finish up the build===
 +
After downloading the sources, you need to be sure you have the appropriate compilers before executing the next command.  At SLAC, that means you need to ssh to one of the psana nodes, as the pslogin nodes do not have the requisite compilers.  When ready, configure and compile thusly:
 +
 
 +
  python bootstrap.py build --builder=xfel --with-python=`which python` --nproc=<# cores available for compile>
 +
 
 +
On SLAC's interactive nodes, this takes just over 6 minutes. To avoid problems with run-time dynamic linking of Python extensions, the Python interpreter required for the above command must be the one provided by the PSDM software distribution.  That interpreter can be located using find, <i>e.g.</i>
 +
$ find $SIT_ROOT/sw/external/python -perm /0111 -type f -wholename "*/$SIT_ARCH/*/python" 2> /dev/null
 +
At SLAC this interpreter is located somewhere under <code>/reg/g/psdm/sw/external/python</code>.  Here we use `which python` to get this path automatically. 
 +
 
 +
Initialise the running shell using the newly created configuration files.  bash-users should
 +
$ . build/setpaths.sh
 +
while csh-users will instead need to run
 +
% source build/setpaths.csh
 +
 
 +
To finalize the installation, see [[Setup]].
 +
 
 +
Note, you may also follow this procedure if you are on a machine where you have your own python that you want to use, instead of the one provided by PSDM.
 +
 
 +
== Installation on a non-PSDM system ==
 +
 
 +
If you find yourself on a machine without PSDM and won't be dealing with XTC directly, for example if you have your own data collected as stills from a non-XFEL source, you can use the below commands in a new directory (again assuming you have a cci user account):
 +
 
 +
  # On Linux:
 +
  wget https://raw.githubusercontent.com/cctbx/cctbx_project/master/libtbx/auto_build/bootstrap.py
 +
  # On MacOSX:
 +
  curl https://raw.githubusercontent.com/cctbx/cctbx_project/master/libtbx/auto_build/bootstrap.py > bootstrap.py
 +
 
 +
  python bootstrap.py hot update base build --builder=xfel --cciuser=<cciusername> --sfuser=<sourceforgeusername>
 +
 
 +
Here, in addition to hot, update and build, there's a command called 'base', which downloads a python and needed packages and compiles it.  After this process completes, source the installation (for details see above), and then add scipy to it:
 +
 
 +
  . build/setpaths.sh
 +
  libtbx.python -m easy_install scipy
 +
 
 +
Please contact the authors if with any issues.  Some gotchas:
 +
 
 +
1) On a Mac, you will need to make sure you have the latest version of xcode installed.  Consider the command xcode-select --install. [NOTE: Xcode v. 8.2.1. for Mac OS 10.12 (Sierra) does not include a gFortran compiler. You will need to install gFortran separately if you need scipy.]
 +
 
 +
2) On linux you will likely need to install some dependencies, including gFortran. [Note: gFortran is needed only for the scipy installation, not for the core build based on cctbx.  Therefore, only install gFortran if you need scipy.]
 +
 
 +
== External links ==
 +
 
 +
* [http://cctbx.sourceforge.net/current_cvs/installation.html#manually-building-from-sources-under-unix Manually building from sources under Unix]
 +
 
  
−
In addition, these software packages need to be available: reportlab, h5py, etc. FIXME
+
<!-- POSSBILE ADDITIONS:
  
−
== Download and extract ''cctbx'' and ''LABELIT'' source bundles ==
+
COMMON PITFALLS:
−
This step has to be performed on a host with Internet access. Not all hosts at SLAC have that, but the members of e.g. psexport pool do. In what follows, the absolute path /path/to/cctbx/directory should be replaced with something more appropriate. The directory should accessible everywhere ''cctbx.xfel'' is to be run.
+
−
$ mkdir /path/to/cctbx/directory
+
−
$ cd /path/to/cctbx/directory
+
−
$ wget http://cci.lbl.gov/cctbx_build/results/2013_08_13_0005/cctbx_bundle.tar.gz
+
−
$ wget http://cci.lbl.gov/~hattne/labelit/labelit_bundle_20130814.tar.gz
+
−
Note: new bundles are automatically prepared nightly. It is, however, not advisable to automatically update from the latest bundle. Please check with the program authors before upgrading a critical installation.
+
  
−
Extract the sources:
+
forget to source the shell configuration files
−
$ mkdir cctbx_project
+
forget to run sit_setup
−
$ tar -xpvzf cctbx_bundle.tar.gz -C cctbx_project --strip-components=1
+
running cxi.pyana from outside the myrelease directory
−
$ tar -xpvzf labelit_bundle_20130814.tar.gz
+
  
−
== Configure the source tree ==
+
DEBUGGING:
−
This step and the next require access to a C++ compiler and a python interpreter, version 2.7.  At SLAC, the members of the psexport pool do not have any compilers installed, but the interactive nodes, reachable via psanacs, do. Furthermore, the Python interpreter used to configure the source tree must be the same interpreter the PSDM suite uses. At SLAC this interpreter is located somewhere under /reg/g/psdm. To check which file is executed when python is typed on the command line, one can use which python. If this is not the python used by the PSDM suite, the full path to the proper interpreter, e.g./reg/g/psdm/sw/external/python/2.7.2/x86_64-rhel5-gcc41-opt/bin/python, should be given instead.
+
−
$ mkdir cctbx_build
+
−
$ cd cctbx_build
+
−
$ python ../cctbx_project/libtbx/configure.py xfel
+
−
Next, initialise the running shell using the newly created configuration files. bash(1) should do
+
−
$ . setpaths.sh
+
−
while csh(1) users should instead run
+
−
$ source setpaths.csh
+
  
−
== Compile the sources ==
+
And a few more things to check: output of "ls -lR arch/*", output of "which python2.7", and the "scons.out" file that results from "scons TRACE=7 > scons.out 2>&1".
−
$ make
+
−
$ make
+
−
On SLAC's interactive nodes, this takes just over 6 minutes.
+
  
−
== Edit shell configuration files ==
+
What Linux distribution are you running this on (could you past the output of "uname -a" and "cat /etc/issue")?  Could you also tell me which scons you are using (output of "which scons"). Last, could you attach a dump of the environment (env > env.out)?
−
To make the changes to the environment persistent, bash(1) users are encouraged to add these lines to ~/.bashrc
+
−
test -r "/path/to/cctbx/directory/cctbx-build/setpaths.sh" && \
+
−
  . "/path/to/cctbx/directory/cctbx-build/setpaths.sh"
+
  
−
csh(1) users should instead add these lines to ~/.cshrc
+
Andy's information request on debugging: full output from these commands:
−
test -r "/path/to/cctbx/directory/cctbx-build/setpaths.csh" && \
+
−
  source "/path/to/cctbx/directory/cctbx-build/setpaths.csh"
+
−
Note: to avoid surprises when running ''cctbx.xfel'' from a non-interactive shell, it is recommended that bash(1) users edit ~/.bash_profile, and put this line near the top:
+
−
test -r "${HOME}/.bashrc" && . "${HOME}/.bashrc"
+
−
A job submitted to the cluster will typically run cctbx.xfel from a non-interactive shell.
+
  
−
== Link ''cctbx.xfel'' modules to the PSDM suite. ==
+
% rm -rf arch build
−
$ cd ~/myrelease/my_ana_pkg
+
% printenv
−
$ ln -s /path/to/cctbx/directory/cctbx_project/xfel/cxi/cspad_ana src
+
% which scons
−
$ cd ..
+
% which python
−
$ scons
+
% which python2.7
 +
% scons TRACE=9
 +
% ls -lR arch
 +
% ls -lR my_ana_pkg
  
−
== Test the installation ==
+
-->
−
At SLAC, test that it is working by e.g.
+
−
$ cd ~/myrelease
+
−
$ cxi.pyana -c /reg/d/ffb/cxi/temp/cctbx/tutorials/setup/test.cfg /reg/d/ffb/cxi/temp/Feb2013calib/e290-r0069-s00-c00.xtc
+
−
This should start a viewer, displaying dark-subtracted averages from an XTC stream.
+

Latest revision as of 23:13, 6 November 2018

Prerequisites

While these tutorials assume you wish to process XTC streams at SLAC, some users have stills collected from other sources and do not need the full PSDM suite. If this is the case, see the installation directions below for installing on a non PSDM system. Otherwise, it is assumed that the PSDM software distribution has been set up. Note again that several sites already have cctbx.xfel installed, and so regular users not involved in the development of the software will not need the instructions here.

Once cctbx.xfel has been installed it must be set up before it can be used. Developers may additionally want to set up an ssh-agent.

Finally, you must have a user account on cci.lbl.gov in order to proceed, or else you will not be able to download the sources. A sourceforge account is only needed if you wish to commit changes back to the repository. If you don't have or don't want to use a sourceforge account, you can leave it off in the below commands where it is specified.

Standard installation using PSDM

Quick start

This step has to be performed on a host with Internet access. Not all hosts at SLAC have that, but the members of e.g. pslogin pool do. Make and change to a working directory to contain the new source code and build (this directory should be accessible from any computing nodes cctbx.xfel will be run. Then download these bootstrap modules: 

 wget https://raw.githubusercontent.com/cctbx/cctbx_project/master/libtbx/auto_build/bootstrap.py

Then: 

 python bootstrap.py hot update --builder=xfel --cciuser=<cciusername> --sfuser=<githubusername>

This command instructs bootstrap.py to download static packages required for cctbx.xfel (hot), and then to checkout the rest of the sources from cci.lbl.gov and github (update), using the user accounts specified by the cciuser and sfuser parameters, respectively.

Alternate instructions for non-LBNL developers

Our non-LBNL colleagues will now realize that the "python bootstrap" step does not work without cci username credentials, needed to obtain source code for the program LABELIT. Developers are instructed to use the following alternate procedure: 

 python bootstrap.py hot --builder=xfel --sfuser=<githubusername> # download the static packages
 python bootstrap.py update --builder=dials --sfuser=<githubusername> # download the github code for dials & cctbx
 mkdir modules/cxi_xdr_xes # create a stub directory for experiment-specific python code

Then download phenix from http://phenix-online.org and untar the package. Locate the modules directory in the phenix directory tree, and copy-paste the labelit subdirectory into the modules directory of your current working area.

Finish up the build

After downloading the sources, you need to be sure you have the appropriate compilers before executing the next command. At SLAC, that means you need to ssh to one of the psana nodes, as the pslogin nodes do not have the requisite compilers. When ready, configure and compile thusly: 

 python bootstrap.py build --builder=xfel --with-python=`which python` --nproc=<# cores available for compile>

On SLAC's interactive nodes, this takes just over 6 minutes. To avoid problems with run-time dynamic linking of Python extensions, the Python interpreter required for the above command must be the one provided by the PSDM software distribution. That interpreter can be located using find, e.g. 

$ find $SIT_ROOT/sw/external/python -perm /0111 -type f -wholename "*/$SIT_ARCH/*/python" 2> /dev/null

At SLAC this interpreter is located somewhere under /reg/g/psdm/sw/external/python. Here we use `which python` to get this path automatically.

Initialise the running shell using the newly created configuration files. bash-users should 

$ . build/setpaths.sh

while csh-users will instead need to run 

% source build/setpaths.csh

To finalize the installation, see Setup.

Note, you may also follow this procedure if you are on a machine where you have your own python that you want to use, instead of the one provided by PSDM.

Installation on a non-PSDM system

If you find yourself on a machine without PSDM and won't be dealing with XTC directly, for example if you have your own data collected as stills from a non-XFEL source, you can use the below commands in a new directory (again assuming you have a cci user account): 

 # On Linux:
 wget https://raw.githubusercontent.com/cctbx/cctbx_project/master/libtbx/auto_build/bootstrap.py
 # On MacOSX:
 curl https://raw.githubusercontent.com/cctbx/cctbx_project/master/libtbx/auto_build/bootstrap.py > bootstrap.py

 python bootstrap.py hot update base build --builder=xfel --cciuser=<cciusername> --sfuser=<sourceforgeusername>

Here, in addition to hot, update and build, there's a command called 'base', which downloads a python and needed packages and compiles it. After this process completes, source the installation (for details see above), and then add scipy to it: 

 . build/setpaths.sh
 libtbx.python -m easy_install scipy

Please contact the authors if with any issues. Some gotchas:

1) On a Mac, you will need to make sure you have the latest version of xcode installed. Consider the command xcode-select --install. [NOTE: Xcode v. 8.2.1. for Mac OS 10.12 (Sierra) does not include a gFortran compiler. You will need to install gFortran separately if you need scipy.]

2) On linux you will likely need to install some dependencies, including gFortran. [Note: gFortran is needed only for the scipy installation, not for the core build based on cctbx. Therefore, only install gFortran if you need scipy.]

External links


Retrieved from "http://viper.lbl.gov:8080/cctbx.xfel/index.php?title=Ha14_installation&oldid=1617"