cctbx_xfel - User contributions [en]

2017 prime tutorial

2017-02-14T20:38:21Z

Mona:

Post-refine and Merge Sample Data Set with PRIME (2017 Tutorial)

In this tutorial, we will work on the integration results from the first part of Tutorial 2 (Myoglobin Data). Before proceeding to running the program, we'll consider making the input file for PRIME based on the situation of this data set.

== Generating Input File ==

PRIME input files contain information necessary for successful post-refinement and merging steps. You can access and review the list of input parameters by running prime.run or prime.run -h to view the description of these parameters. For this tutorial we'll start building it from scratch.

* Location of integration results
In this case, we know the location where the integration results (pickle files) are. We can then set,

data = /net/viper/raid1/mu238/XfelProject/dials17/extracted

Note that you can supply data parameter as multiple arguments. The value of the parameter can be a file containing list of integration results or a folder.

* Unit cell information
You can obtain the mean (or median) unit-cell dimensions from either IOTA or DIALS. In case of IOTA, prime .phil file is auto generated and this information is readily available in there. For n_residues, enter number of residues in asymmetric unit of your molecule.

target_unit_cell = 91.7 91.7 46 90 90 120
target_space_group = P6
n_residues = 128

* Detector information
pixel_size_mm = 0.172

* Post-refinement and Scaling information
This is where you specify the optimal resolution cutoffs for post-refinement and merging. Note that when running for the first time on you newly collected data, you can choose the "expected" values (resolution which you see the spots at the corner or on the edge). You can then adjust these parameters when analyzing merging statistics based on the I/sigI values in the high resolution shells and rerun the program again. Note that sigma cutoffs are set to 1.5 in scaling and post-refinement steps while it's set to -3.0 so we can include negative values in the merged reflection set.

scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
}
postref {
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
allparams {
flag_on = True
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
merge {
d_min = 2.5
d_max = 20
sigma_min = -3.0
partiality_min = 0.1
uc_tolerance = 5
}

* Indexing ambiguity
For other sets that are not in polar space or have indexing ambiguity (when one or more of the unit-cell dimensions are very similar but not the same!), you can very well use the .phil file parameters thus far to proceed and run post-refinement. However, this data set is in P6 (polar space group) and therefore, the indexing ambiguity needs to be resolved prior to other refinement and merging steps.

Other point worth noting is for any polar space groups, PRIME will automatically solve the ambiguity based on the default parameters. However, this data set has about 5,000 integration results so we want to make sure that we modify the number of images used for random and best selections.

indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
d_min = 3.0
d_max = 10.0
sigma_min = 1.5
n_sample_frames = 1000
n_selected_frames = 100
}

We left other parameters to their default value and modified n_sample_frames to 1000 and n_selected_frames to 100.

* No. of Bin
n_bins = 10

Now we have a complete .phil file ready to run.

data = /net/viper/raid1/mu238/XfelProject/dials17/extracted
target_unit_cell = 91.7 91.7 46 90 90 120
target_space_group = P6
n_residues = 128
pixel_size_mm = 0.172
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
}
postref {
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
}
allparams {
flag_on = True
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
merge {
d_min = 2.5
d_max = 20
sigma_min = -3.0
partiality_min = 0.1
uc_tolerance = 5
}
indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
d_min = 3.0
d_max = 10.0
sigma_min = 1.5
n_sample_frames = 1000
n_selected_frames = 100
}
n_bins = 10

Copy and paste this set of parameter in an editor then save the file as "prime.phil".

== Running the Program ==
You can run the program by giving it an input file:

prime.run prime.phil

For this tutorial, PRIME will score the randomly selected 1,000 images then select the best 100 for running Brehm & Diederichs algorithm in Bootstrap mode. If you run the program with flag_plot=True, you'll see a plot showing two separated clusters, each representing images with matching assigned basis.

[https://commons.wikimedia.org/wiki/File:Dials17_myo_indexing_ambiguity.png Results of Running Indexing Ambiguity with Boostrap]

[https://commons.wikimedia.org/wiki/File:Dials17_indexing_ambiguity_clustering.gif Results of Image Clustering]

PRIME will select on of these two clusters and merge it to get a reference set for the Bootstrap step. Here, the remaining images will get assigned with a basis that makes it correlate best with the reference set.

Once all images are assigned with appropriate basis, PRIME will proceed to scaling and post-refinement steps. After three post-refinement cycles (default value), the process is done and here is the output of the program.

Isotropic B-factor: 5.30
No. of reflections
all: 7786
outside resolution: 51
outliers: 0
total left: 7735
Summary for Prime_Run_1/postref_cycle_3_merge.mtz
Bin Resolution Range Completeness N_obs |Rmerge Rsplit CC1/2 N_ind |CCiso N_ind|CCanoma N_ind| I/sigI I sigI I**2
--------------------------------------------------------------------------------------------------------------------------------------------------
01 19.88 - 5.35 100.0 807 / 807 189.78 85.75 8.87 98.43 807 0.00 0 0.00 0 4.46 684.2 136.2 3.57
02 5.35 - 4.26 100.0 782 / 782 140.90 73.61 8.89 97.67 782 0.00 0 0.00 0 5.23 794.3 140.3 2.05
03 4.26 - 3.73 100.0 788 / 788 129.32 69.32 8.70 98.11 788 0.00 0 0.00 0 5.41 878.5 150.3 1.95
04 3.73 - 3.39 100.0 765 / 765 117.75 70.72 9.67 97.55 765 0.00 0 0.00 0 4.12 712.3 162.4 1.88
05 3.39 - 3.15 100.0 770 / 770 113.22 71.61 11.22 88.54 770 0.00 0 0.00 0 2.73 500.1 173.7 2.19
06 3.15 - 2.96 100.0 767 / 767 106.12 73.19 11.07 97.21 767 0.00 0 0.00 0 2.09 404.6 183.8 2.02
07 2.96 - 2.81 100.0 766 / 766 103.73 75.79 12.53 96.62 766 0.00 0 0.00 0 1.72 345.3 193.9 1.89
08 2.81 - 2.69 100.0 745 / 745 101.51 76.11 12.84 96.21 745 0.00 0 0.00 0 1.49 317.3 204.1 1.98
09 2.69 - 2.59 100.0 786 / 786 97.88 77.59 14.19 95.40 786 0.00 0 0.00 0 1.38 299.0 209.3 1.86
10 2.59 - 2.50 100.0 759 / 759 92.53 78.61 14.72 96.51 759 0.00 0 0.00 0 1.36 312.4 218.8 2.07
--------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL 100.0 7735 / 7735 119.73 75.51 10.45 97.21 7735 0.00 0 0.00 0 3.02 527.3 176.9 2.56
--------------------------------------------------------------------------------------------------------------------------------------------------

Summary of CC1/2 on three crystal axes
Bin Resolution Range CC1/2 I N_refl
a* b* c* | a* b* c* | a* b* c*
---------------------------------------------------------------------------------------------------------
01 19.88 - 5.35 97.01 98.64 98.23 528.6 559.6 1216.7 42 51 47
02 5.35 - 4.26 97.64 98.43 99.08 817.6 527.5 964.9 43 44 40
03 4.26 - 3.73 96.31 98.02 97.68 605.7 682.9 856.0 39 39 41
04 3.73 - 3.39 98.49 98.55 97.73 961.9 532.6 729.1 42 37 45
05 3.39 - 3.15 96.88 98.38 92.69 449.5 492.1 721.6 39 39 40
06 3.15 - 2.96 98.48 93.58 98.61 389.9 303.4 391.7 39 37 39
07 2.96 - 2.81 96.98 98.02 95.35 361.3 331.7 383.2 42 37 43
08 2.81 - 2.69 95.29 94.02 94.69 290.8 194.7 292.7 41 35 36
09 2.69 - 2.59 96.55 91.88 98.57 265.7 341.4 290.2 41 35 44
10 2.59 - 2.50 94.44 97.81 96.67 249.0 400.1 236.2 42 36 40
----------------------------------------------------------------------------------------------------------
total 97.57 97.74 94.58 494.1 446.0 619.4 410 390 415
----------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 4733
No. bad cc frames: 113
No. bad G frames) : 109
No. bad unit cell frames: 20
No. bad gamma_e frames: 22
No. bad SE: 2
No. observations: 935265
Mean target value (BEFORE: Mean Median (Std.))
post-refinement: 301.22 259.10 ( 171.56)
(x,y) restraints: 1679.63 1573.15 ( 657.49)
Mean target value (AFTER: Mean Median (Std.))
post-refinement: 300.02 257.53 ( 170.98)
(x,y) restraints: 1679.90 1572.77 ( 660.19)
SE: 1915.60 776.84 ( 33765.97)
G: 1.000e+00 8.971e-01 ( 8.15e-01)
B: 11.83 14.45 ( 11.95)
Rot.x: -0.08 0.00 ( 12.10)
Rot.y: 0.14 0.00 ( 9.62)
gamma_y: 0.00000 0.00000 ( 0.00000)
gamma_z: 0.00000 0.00000 ( 0.00000)
gamma_0: 0.03793 0.00019 ( 0.60820)
gamma_e: -0.12824 0.00145 ( 0.60227)
voigt_nu: 0.50000 0.50000 ( 0.00000)
unit cell
a: 91.45 91.45 ( 0.11)
b: 91.45 91.45 ( 0.11)
c: 45.96 45.96 ( 0.12)
alpha: 90.00 90.00 ( 0.00)
beta: 90.00 90.00 ( 0.00)
gamma: 120.00 120.00 ( 0.00)
Parmeters from integration (not-refined)
Wavelength: 0.96861 0.96861 ( 0.00000)
Detector distance: 303.81868 303.81868 ( 0.00000)
* (standard deviation)

Total calculation time: 542.00 seconds
Finished: Tuesday 14. February 2017 10:53:18

== Obtaining the Output ==

Your output will be in Prime_Run_n (where n is the number of run).

-bash-4.1$ ls Prime_Run_1/ -l
total 9076
-rw-r--r-- 1 mu238 camb 879638 Feb 14 10:53 crystal.o
drwxr-xr-x 2 mu238 camb 104 Feb 14 10:46 index_ambiguity
drwxr-xr-x 2 mu238 camb 6 Feb 14 10:44 isoform_cluster
-rw-r--r-- 1 mu238 camb 32704 Feb 14 10:53 log.txt
-rw-r--r-- 1 mu238 camb 324556 Feb 14 10:47 mean_scaled_merge.hkl
-rw-r--r-- 1 mu238 camb 157260 Feb 14 10:47 mean_scaled_merge.mtz
-rw-r--r-- 1 mu238 camb 15753 Feb 14 10:53 pickle.stat
-rw-r--r-- 1 mu238 camb 324381 Feb 14 10:49 postref_cycle_1_merge.hkl
-rw-r--r-- 1 mu238 camb 157260 Feb 14 10:49 postref_cycle_1_merge.mtz
-rw-r--r-- 1 mu238 camb 324515 Feb 14 10:51 postref_cycle_2_merge.hkl
-rw-r--r-- 1 mu238 camb 157260 Feb 14 10:51 postref_cycle_2_merge.mtz
-rw-r--r-- 1 mu238 camb 324716 Feb 14 10:53 postref_cycle_3_merge.hkl
-rw-r--r-- 1 mu238 camb 157340 Feb 14 10:53 postref_cycle_3_merge.mtz
-rw-r--r-- 1 mu238 camb 6412200 Feb 14 10:53 rejections.txt

File log.txt contains all the merging stats. The final merged reflection set is postref_cycle_3_merge.mtz (or .hkl).

2017 prime tutorial

2017-02-14T20:37:23Z

Mona: /* Generating Input File */

Post-refine and Merge Sample Data Set with PRIME (2017 Tutorial)

In this tutorial, we will work on the integration results from the first of Tutorial 2 (Myoglobin Data). Before proceeding to running the program, we'll consider making the input file for PRIME based on the situation of this data set.

== Generating Input File ==

PRIME input files contain information necessary for successful post-refinement and merging steps. You can access and review the list of input parameters by running prime.run or prime.run -h to view the description of these parameters. For this tutorial we'll start building it from scratch.

* Location of integration results
In this case, we know the location where the integration results (pickle files) are. We can then set,

data = /net/viper/raid1/mu238/XfelProject/dials17/extracted

Note that you can supply data parameter as multiple arguments. The value of the parameter can be a file containing list of integration results or a folder.

* Unit cell information
You can obtain the mean (or median) unit-cell dimensions from either IOTA or DIALS. In case of IOTA, prime .phil file is auto generated and this information is readily available in there. For n_residues, enter number of residues in asymmetric unit of your molecule.

target_unit_cell = 91.7 91.7 46 90 90 120
target_space_group = P6
n_residues = 128

* Detector information
pixel_size_mm = 0.172

* Post-refinement and Scaling information
This is where you specify the optimal resolution cutoffs for post-refinement and merging. Note that when running for the first time on you newly collected data, you can choose the "expected" values (resolution which you see the spots at the corner or on the edge). You can then adjust these parameters when analyzing merging statistics based on the I/sigI values in the high resolution shells and rerun the program again. Note that sigma cutoffs are set to 1.5 in scaling and post-refinement steps while it's set to -3.0 so we can include negative values in the merged reflection set.

scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
}
postref {
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
allparams {
flag_on = True
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
merge {
d_min = 2.5
d_max = 20
sigma_min = -3.0
partiality_min = 0.1
uc_tolerance = 5
}

* Indexing ambiguity
For other sets that are not in polar space or have indexing ambiguity (when one or more of the unit-cell dimensions are very similar but not the same!), you can very well use the .phil file parameters thus far to proceed and run post-refinement. However, this data set is in P6 (polar space group) and therefore, the indexing ambiguity needs to be resolved prior to other refinement and merging steps.

Other point worth noting is for any polar space groups, PRIME will automatically solve the ambiguity based on the default parameters. However, this data set has about 5,000 integration results so we want to make sure that we modify the number of images used for random and best selections.

indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
d_min = 3.0
d_max = 10.0
sigma_min = 1.5
n_sample_frames = 1000
n_selected_frames = 100
}

We left other parameters to their default value and modified n_sample_frames to 1000 and n_selected_frames to 100.

* No. of Bin
n_bins = 10

Now we have a complete .phil file ready to run.

data = /net/viper/raid1/mu238/XfelProject/dials17/extracted
target_unit_cell = 91.7 91.7 46 90 90 120
target_space_group = P6
n_residues = 128
pixel_size_mm = 0.172
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
}
postref {
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
}
allparams {
flag_on = True
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
merge {
d_min = 2.5
d_max = 20
sigma_min = -3.0
partiality_min = 0.1
uc_tolerance = 5
}
indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
d_min = 3.0
d_max = 10.0
sigma_min = 1.5
n_sample_frames = 1000
n_selected_frames = 100
}
n_bins = 10

Copy and paste this set of parameter in an editor then save the file as "prime.phil".

== Running the Program ==
You can run the program by giving it an input file:

prime.run prime.phil

For this tutorial, PRIME will score the randomly selected 1,000 images then select the best 100 for running Brehm & Diederichs algorithm in Bootstrap mode. If you run the program with flag_plot=True, you'll see a plot showing two separated clusters, each representing images with matching assigned basis.

[https://commons.wikimedia.org/wiki/File:Dials17_myo_indexing_ambiguity.png Results of Running Indexing Ambiguity with Boostrap]

[https://commons.wikimedia.org/wiki/File:Dials17_indexing_ambiguity_clustering.gif Results of Image Clustering]

PRIME will select on of these two clusters and merge it to get a reference set for the Bootstrap step. Here, the remaining images will get assigned with a basis that makes it correlate best with the reference set.

Once all images are assigned with appropriate basis, PRIME will proceed to scaling and post-refinement steps. After three post-refinement cycles (default value), the process is done and here is the output of the program.

Isotropic B-factor: 5.30
No. of reflections
all: 7786
outside resolution: 51
outliers: 0
total left: 7735
Summary for Prime_Run_1/postref_cycle_3_merge.mtz
Bin Resolution Range Completeness N_obs |Rmerge Rsplit CC1/2 N_ind |CCiso N_ind|CCanoma N_ind| I/sigI I sigI I**2
--------------------------------------------------------------------------------------------------------------------------------------------------
01 19.88 - 5.35 100.0 807 / 807 189.78 85.75 8.87 98.43 807 0.00 0 0.00 0 4.46 684.2 136.2 3.57
02 5.35 - 4.26 100.0 782 / 782 140.90 73.61 8.89 97.67 782 0.00 0 0.00 0 5.23 794.3 140.3 2.05
03 4.26 - 3.73 100.0 788 / 788 129.32 69.32 8.70 98.11 788 0.00 0 0.00 0 5.41 878.5 150.3 1.95
04 3.73 - 3.39 100.0 765 / 765 117.75 70.72 9.67 97.55 765 0.00 0 0.00 0 4.12 712.3 162.4 1.88
05 3.39 - 3.15 100.0 770 / 770 113.22 71.61 11.22 88.54 770 0.00 0 0.00 0 2.73 500.1 173.7 2.19
06 3.15 - 2.96 100.0 767 / 767 106.12 73.19 11.07 97.21 767 0.00 0 0.00 0 2.09 404.6 183.8 2.02
07 2.96 - 2.81 100.0 766 / 766 103.73 75.79 12.53 96.62 766 0.00 0 0.00 0 1.72 345.3 193.9 1.89
08 2.81 - 2.69 100.0 745 / 745 101.51 76.11 12.84 96.21 745 0.00 0 0.00 0 1.49 317.3 204.1 1.98
09 2.69 - 2.59 100.0 786 / 786 97.88 77.59 14.19 95.40 786 0.00 0 0.00 0 1.38 299.0 209.3 1.86
10 2.59 - 2.50 100.0 759 / 759 92.53 78.61 14.72 96.51 759 0.00 0 0.00 0 1.36 312.4 218.8 2.07
--------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL 100.0 7735 / 7735 119.73 75.51 10.45 97.21 7735 0.00 0 0.00 0 3.02 527.3 176.9 2.56
--------------------------------------------------------------------------------------------------------------------------------------------------

Summary of CC1/2 on three crystal axes
Bin Resolution Range CC1/2 I N_refl
a* b* c* | a* b* c* | a* b* c*
---------------------------------------------------------------------------------------------------------
01 19.88 - 5.35 97.01 98.64 98.23 528.6 559.6 1216.7 42 51 47
02 5.35 - 4.26 97.64 98.43 99.08 817.6 527.5 964.9 43 44 40
03 4.26 - 3.73 96.31 98.02 97.68 605.7 682.9 856.0 39 39 41
04 3.73 - 3.39 98.49 98.55 97.73 961.9 532.6 729.1 42 37 45
05 3.39 - 3.15 96.88 98.38 92.69 449.5 492.1 721.6 39 39 40
06 3.15 - 2.96 98.48 93.58 98.61 389.9 303.4 391.7 39 37 39
07 2.96 - 2.81 96.98 98.02 95.35 361.3 331.7 383.2 42 37 43
08 2.81 - 2.69 95.29 94.02 94.69 290.8 194.7 292.7 41 35 36
09 2.69 - 2.59 96.55 91.88 98.57 265.7 341.4 290.2 41 35 44
10 2.59 - 2.50 94.44 97.81 96.67 249.0 400.1 236.2 42 36 40
----------------------------------------------------------------------------------------------------------
total 97.57 97.74 94.58 494.1 446.0 619.4 410 390 415
----------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 4733
No. bad cc frames: 113
No. bad G frames) : 109
No. bad unit cell frames: 20
No. bad gamma_e frames: 22
No. bad SE: 2
No. observations: 935265
Mean target value (BEFORE: Mean Median (Std.))
post-refinement: 301.22 259.10 ( 171.56)
(x,y) restraints: 1679.63 1573.15 ( 657.49)
Mean target value (AFTER: Mean Median (Std.))
post-refinement: 300.02 257.53 ( 170.98)
(x,y) restraints: 1679.90 1572.77 ( 660.19)
SE: 1915.60 776.84 ( 33765.97)
G: 1.000e+00 8.971e-01 ( 8.15e-01)
B: 11.83 14.45 ( 11.95)
Rot.x: -0.08 0.00 ( 12.10)
Rot.y: 0.14 0.00 ( 9.62)
gamma_y: 0.00000 0.00000 ( 0.00000)
gamma_z: 0.00000 0.00000 ( 0.00000)
gamma_0: 0.03793 0.00019 ( 0.60820)
gamma_e: -0.12824 0.00145 ( 0.60227)
voigt_nu: 0.50000 0.50000 ( 0.00000)
unit cell
a: 91.45 91.45 ( 0.11)
b: 91.45 91.45 ( 0.11)
c: 45.96 45.96 ( 0.12)
alpha: 90.00 90.00 ( 0.00)
beta: 90.00 90.00 ( 0.00)
gamma: 120.00 120.00 ( 0.00)
Parmeters from integration (not-refined)
Wavelength: 0.96861 0.96861 ( 0.00000)
Detector distance: 303.81868 303.81868 ( 0.00000)
* (standard deviation)

Total calculation time: 542.00 seconds
Finished: Tuesday 14. February 2017 10:53:18

== Obtaining the Output ==

Your output will be in Prime_Run_n (where n is the number of run).

-bash-4.1$ ls Prime_Run_1/ -l
total 9076
-rw-r--r-- 1 mu238 camb 879638 Feb 14 10:53 crystal.o
drwxr-xr-x 2 mu238 camb 104 Feb 14 10:46 index_ambiguity
drwxr-xr-x 2 mu238 camb 6 Feb 14 10:44 isoform_cluster
-rw-r--r-- 1 mu238 camb 32704 Feb 14 10:53 log.txt
-rw-r--r-- 1 mu238 camb 324556 Feb 14 10:47 mean_scaled_merge.hkl
-rw-r--r-- 1 mu238 camb 157260 Feb 14 10:47 mean_scaled_merge.mtz
-rw-r--r-- 1 mu238 camb 15753 Feb 14 10:53 pickle.stat
-rw-r--r-- 1 mu238 camb 324381 Feb 14 10:49 postref_cycle_1_merge.hkl
-rw-r--r-- 1 mu238 camb 157260 Feb 14 10:49 postref_cycle_1_merge.mtz
-rw-r--r-- 1 mu238 camb 324515 Feb 14 10:51 postref_cycle_2_merge.hkl
-rw-r--r-- 1 mu238 camb 157260 Feb 14 10:51 postref_cycle_2_merge.mtz
-rw-r--r-- 1 mu238 camb 324716 Feb 14 10:53 postref_cycle_3_merge.hkl
-rw-r--r-- 1 mu238 camb 157340 Feb 14 10:53 postref_cycle_3_merge.mtz
-rw-r--r-- 1 mu238 camb 6412200 Feb 14 10:53 rejections.txt

File log.txt contains all the merging stats. The final merged reflection set is postref_cycle_3_merge.mtz (or .hkl).

2017 prime tutorial

2017-02-14T20:02:38Z

Mona: /* Running the Program */

Post-refine and Merge Sample Data Set with PRIME (2017 Tutorial)

In this tutorial, we will work on the integration results from the first of Tutorial 2 (Myoglobin Data). Before proceeding to running the program, we'll consider making the input file for PRIME based on the situation of this data set.

== Generating Input File ==

PRIME input files contain information necessary for successful post-refinement and merging steps. You can access and review the list of input parameters by running prime.run or prime.run -h to view the description of these parameters. For this tutorial we'll start building it from scratch.

* Location of integration results
In this case, we know the location where the integration results (pickle files) are. We can then set,

data = /net/viper/raid1/mu238/XfelProject/dials17/extracted

Note that you supply data as a multiple arguments. The value of the parameter can be a file containing list of integration results, a folder, or a wildcard argument.

* Unit cell information
You can obtain the mean (or median) unit-cell dimensions from either IOTA or DIALS. In case of IOTA, prime .phil file is auto generated and this information is readily available in there. For n_residues, enter number of residues in asymmetric unit of your molecule.

target_unit_cell = 91.7 91.7 46 90 90 120
target_space_group = P6
n_residues = 128

* Detector information
pixel_size_mm = 0.172

* Post-refinement and Scaling information
This is where you specify the optimal resolution cutoffs for post-refinement and merging. Note that when running for the first time on you newly collected data, you can choose the "expected" values (resolution which you see the spots at the corner or on the edge). You can then adjust these parameters when analyzing merging statistics based on the I/sigI values in the high resolution shells and rerun the program again. Note that sigma cutoffs are set to 1.5 in scaling and post-refinement steps while it's set to -3.0 so we can include negative values in the merged reflection set.

scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
}
postref {
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
allparams {
flag_on = True
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
merge {
d_min = 2.5
d_max = 20
sigma_min = -3.0
partiality_min = 0.1
uc_tolerance = 5
}

* Indexing ambiguity
For other sets that are not in polar space or have indexing ambiguity (when one or more of the unit-cell dimensions are very similar but not the same!), you can very well use the .phil file parameters thus far to proceed and run post-refinement. However, this data set is in P6 (polar space group) and therefore, the indexing ambiguity needs to be resolved prior to other refinement and merging steps.

Other point worth noting is for any polar space groups, PRIME will automatically solve the ambiguity based on the default parameters. However, this data set has about 5,000 integration results so we want to make sure that we modify the number of images used for random and best selections.

indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
d_min = 3.0
d_max = 10.0
sigma_min = 1.5
n_sample_frames = 1000
n_selected_frames = 100
}

We left other parameters to their default value and modified n_sample_frames to 1000 and n_selected_frames to 100.

* No. of Bin
n_bins = 10

Now we have a complete .phil file ready to run.

data = /net/viper/raid1/mu238/XfelProject/dials17/extracted
target_unit_cell = 91.7 91.7 46 90 90 120
target_space_group = P6
n_residues = 128
pixel_size_mm = 0.172
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
}
postref {
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
}
allparams {
flag_on = True
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
merge {
d_min = 2.5
d_max = 20
sigma_min = -3.0
partiality_min = 0.1
uc_tolerance = 5
}
indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
d_min = 3.0
d_max = 10.0
sigma_min = 1.5
n_sample_frames = 1000
n_selected_frames = 100
}
n_bins = 10

Copy and paste this set of parameter in an editor then save the file as "prime.phil".

== Running the Program ==
You can run the program by giving it an input file:

prime.run prime.phil

For this tutorial, PRIME will score the randomly selected 1,000 images then select the best 100 for running Brehm & Diederichs algorithm in Bootstrap mode. If you run the program with flag_plot=True, you'll see a plot showing two separated clusters, each representing images with matching assigned basis.

[https://commons.wikimedia.org/wiki/File:Dials17_myo_indexing_ambiguity.png Results of Running Indexing Ambiguity with Boostrap]

[https://commons.wikimedia.org/wiki/File:Dials17_indexing_ambiguity_clustering.gif Results of Image Clustering]

PRIME will select on of these two clusters and merge it to get a reference set for the Bootstrap step. Here, the remaining images will get assigned with a basis that makes it correlate best with the reference set.

Once all images are assigned with appropriate basis, PRIME will proceed to scaling and post-refinement steps. After three post-refinement cycles (default value), the process is done and here is the output of the program.

Isotropic B-factor: 5.30
No. of reflections
all: 7786
outside resolution: 51
outliers: 0
total left: 7735
Summary for Prime_Run_1/postref_cycle_3_merge.mtz
Bin Resolution Range Completeness N_obs |Rmerge Rsplit CC1/2 N_ind |CCiso N_ind|CCanoma N_ind| I/sigI I sigI I**2
--------------------------------------------------------------------------------------------------------------------------------------------------
01 19.88 - 5.35 100.0 807 / 807 189.78 85.75 8.87 98.43 807 0.00 0 0.00 0 4.46 684.2 136.2 3.57
02 5.35 - 4.26 100.0 782 / 782 140.90 73.61 8.89 97.67 782 0.00 0 0.00 0 5.23 794.3 140.3 2.05
03 4.26 - 3.73 100.0 788 / 788 129.32 69.32 8.70 98.11 788 0.00 0 0.00 0 5.41 878.5 150.3 1.95
04 3.73 - 3.39 100.0 765 / 765 117.75 70.72 9.67 97.55 765 0.00 0 0.00 0 4.12 712.3 162.4 1.88
05 3.39 - 3.15 100.0 770 / 770 113.22 71.61 11.22 88.54 770 0.00 0 0.00 0 2.73 500.1 173.7 2.19
06 3.15 - 2.96 100.0 767 / 767 106.12 73.19 11.07 97.21 767 0.00 0 0.00 0 2.09 404.6 183.8 2.02
07 2.96 - 2.81 100.0 766 / 766 103.73 75.79 12.53 96.62 766 0.00 0 0.00 0 1.72 345.3 193.9 1.89
08 2.81 - 2.69 100.0 745 / 745 101.51 76.11 12.84 96.21 745 0.00 0 0.00 0 1.49 317.3 204.1 1.98
09 2.69 - 2.59 100.0 786 / 786 97.88 77.59 14.19 95.40 786 0.00 0 0.00 0 1.38 299.0 209.3 1.86
10 2.59 - 2.50 100.0 759 / 759 92.53 78.61 14.72 96.51 759 0.00 0 0.00 0 1.36 312.4 218.8 2.07
--------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL 100.0 7735 / 7735 119.73 75.51 10.45 97.21 7735 0.00 0 0.00 0 3.02 527.3 176.9 2.56
--------------------------------------------------------------------------------------------------------------------------------------------------

Summary of CC1/2 on three crystal axes
Bin Resolution Range CC1/2 I N_refl
a* b* c* | a* b* c* | a* b* c*
---------------------------------------------------------------------------------------------------------
01 19.88 - 5.35 97.01 98.64 98.23 528.6 559.6 1216.7 42 51 47
02 5.35 - 4.26 97.64 98.43 99.08 817.6 527.5 964.9 43 44 40
03 4.26 - 3.73 96.31 98.02 97.68 605.7 682.9 856.0 39 39 41
04 3.73 - 3.39 98.49 98.55 97.73 961.9 532.6 729.1 42 37 45
05 3.39 - 3.15 96.88 98.38 92.69 449.5 492.1 721.6 39 39 40
06 3.15 - 2.96 98.48 93.58 98.61 389.9 303.4 391.7 39 37 39
07 2.96 - 2.81 96.98 98.02 95.35 361.3 331.7 383.2 42 37 43
08 2.81 - 2.69 95.29 94.02 94.69 290.8 194.7 292.7 41 35 36
09 2.69 - 2.59 96.55 91.88 98.57 265.7 341.4 290.2 41 35 44
10 2.59 - 2.50 94.44 97.81 96.67 249.0 400.1 236.2 42 36 40
----------------------------------------------------------------------------------------------------------
total 97.57 97.74 94.58 494.1 446.0 619.4 410 390 415
----------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 4733
No. bad cc frames: 113
No. bad G frames) : 109
No. bad unit cell frames: 20
No. bad gamma_e frames: 22
No. bad SE: 2
No. observations: 935265
Mean target value (BEFORE: Mean Median (Std.))
post-refinement: 301.22 259.10 ( 171.56)
(x,y) restraints: 1679.63 1573.15 ( 657.49)
Mean target value (AFTER: Mean Median (Std.))
post-refinement: 300.02 257.53 ( 170.98)
(x,y) restraints: 1679.90 1572.77 ( 660.19)
SE: 1915.60 776.84 ( 33765.97)
G: 1.000e+00 8.971e-01 ( 8.15e-01)
B: 11.83 14.45 ( 11.95)
Rot.x: -0.08 0.00 ( 12.10)
Rot.y: 0.14 0.00 ( 9.62)
gamma_y: 0.00000 0.00000 ( 0.00000)
gamma_z: 0.00000 0.00000 ( 0.00000)
gamma_0: 0.03793 0.00019 ( 0.60820)
gamma_e: -0.12824 0.00145 ( 0.60227)
voigt_nu: 0.50000 0.50000 ( 0.00000)
unit cell
a: 91.45 91.45 ( 0.11)
b: 91.45 91.45 ( 0.11)
c: 45.96 45.96 ( 0.12)
alpha: 90.00 90.00 ( 0.00)
beta: 90.00 90.00 ( 0.00)
gamma: 120.00 120.00 ( 0.00)
Parmeters from integration (not-refined)
Wavelength: 0.96861 0.96861 ( 0.00000)
Detector distance: 303.81868 303.81868 ( 0.00000)
* (standard deviation)

Total calculation time: 542.00 seconds
Finished: Tuesday 14. February 2017 10:53:18

== Obtaining the Output ==

Your output will be in Prime_Run_n (where n is the number of run).

-bash-4.1$ ls Prime_Run_1/ -l
total 9076
-rw-r--r-- 1 mu238 camb 879638 Feb 14 10:53 crystal.o
drwxr-xr-x 2 mu238 camb 104 Feb 14 10:46 index_ambiguity
drwxr-xr-x 2 mu238 camb 6 Feb 14 10:44 isoform_cluster
-rw-r--r-- 1 mu238 camb 32704 Feb 14 10:53 log.txt
-rw-r--r-- 1 mu238 camb 324556 Feb 14 10:47 mean_scaled_merge.hkl
-rw-r--r-- 1 mu238 camb 157260 Feb 14 10:47 mean_scaled_merge.mtz
-rw-r--r-- 1 mu238 camb 15753 Feb 14 10:53 pickle.stat
-rw-r--r-- 1 mu238 camb 324381 Feb 14 10:49 postref_cycle_1_merge.hkl
-rw-r--r-- 1 mu238 camb 157260 Feb 14 10:49 postref_cycle_1_merge.mtz
-rw-r--r-- 1 mu238 camb 324515 Feb 14 10:51 postref_cycle_2_merge.hkl
-rw-r--r-- 1 mu238 camb 157260 Feb 14 10:51 postref_cycle_2_merge.mtz
-rw-r--r-- 1 mu238 camb 324716 Feb 14 10:53 postref_cycle_3_merge.hkl
-rw-r--r-- 1 mu238 camb 157340 Feb 14 10:53 postref_cycle_3_merge.mtz
-rw-r--r-- 1 mu238 camb 6412200 Feb 14 10:53 rejections.txt

File log.txt contains all the merging stats. The final merged reflection set is postref_cycle_3_merge.mtz (or .hkl).

2017 prime tutorial

2017-02-14T20:02:10Z

Mona: /* Running the Program */

Post-refine and Merge Sample Data Set with PRIME (2017 Tutorial)

In this tutorial, we will work on the integration results from the first of Tutorial 2 (Myoglobin Data). Before proceeding to running the program, we'll consider making the input file for PRIME based on the situation of this data set.

== Generating Input File ==

PRIME input files contain information necessary for successful post-refinement and merging steps. You can access and review the list of input parameters by running prime.run or prime.run -h to view the description of these parameters. For this tutorial we'll start building it from scratch.

* Location of integration results
In this case, we know the location where the integration results (pickle files) are. We can then set,

data = /net/viper/raid1/mu238/XfelProject/dials17/extracted

Note that you supply data as a multiple arguments. The value of the parameter can be a file containing list of integration results, a folder, or a wildcard argument.

* Unit cell information
You can obtain the mean (or median) unit-cell dimensions from either IOTA or DIALS. In case of IOTA, prime .phil file is auto generated and this information is readily available in there. For n_residues, enter number of residues in asymmetric unit of your molecule.

target_unit_cell = 91.7 91.7 46 90 90 120
target_space_group = P6
n_residues = 128

* Detector information
pixel_size_mm = 0.172

* Post-refinement and Scaling information
This is where you specify the optimal resolution cutoffs for post-refinement and merging. Note that when running for the first time on you newly collected data, you can choose the "expected" values (resolution which you see the spots at the corner or on the edge). You can then adjust these parameters when analyzing merging statistics based on the I/sigI values in the high resolution shells and rerun the program again. Note that sigma cutoffs are set to 1.5 in scaling and post-refinement steps while it's set to -3.0 so we can include negative values in the merged reflection set.

scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
}
postref {
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
allparams {
flag_on = True
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
merge {
d_min = 2.5
d_max = 20
sigma_min = -3.0
partiality_min = 0.1
uc_tolerance = 5
}

* Indexing ambiguity
For other sets that are not in polar space or have indexing ambiguity (when one or more of the unit-cell dimensions are very similar but not the same!), you can very well use the .phil file parameters thus far to proceed and run post-refinement. However, this data set is in P6 (polar space group) and therefore, the indexing ambiguity needs to be resolved prior to other refinement and merging steps.

Other point worth noting is for any polar space groups, PRIME will automatically solve the ambiguity based on the default parameters. However, this data set has about 5,000 integration results so we want to make sure that we modify the number of images used for random and best selections.

indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
d_min = 3.0
d_max = 10.0
sigma_min = 1.5
n_sample_frames = 1000
n_selected_frames = 100
}

We left other parameters to their default value and modified n_sample_frames to 1000 and n_selected_frames to 100.

* No. of Bin
n_bins = 10

Now we have a complete .phil file ready to run.

data = /net/viper/raid1/mu238/XfelProject/dials17/extracted
target_unit_cell = 91.7 91.7 46 90 90 120
target_space_group = P6
n_residues = 128
pixel_size_mm = 0.172
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
}
postref {
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
}
allparams {
flag_on = True
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
merge {
d_min = 2.5
d_max = 20
sigma_min = -3.0
partiality_min = 0.1
uc_tolerance = 5
}
indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
d_min = 3.0
d_max = 10.0
sigma_min = 1.5
n_sample_frames = 1000
n_selected_frames = 100
}
n_bins = 10

Copy and paste this set of parameter in an editor then save the file as "prime.phil".

== Running the Program ==
You can run the program by giving it an input file:

prime.run prime.phil

For this tutorial, PRIME will score the randomly selected 1,000 images then select the best 100 for running Brehm & Diederichs algorithm in Bootstrap mode. If you run the program with flag_plot=True, you'll see a plot showing two separated clusters, each representing images with matching assigned basis.

[https://commons.wikimedia.org/wiki/File:Dials17_myo_indexing_ambiguity.png Results of Running Indexing Ambiguity with Boostrap]
[https://commons.wikimedia.org/wiki/File:Dials17_indexing_ambiguity_clustering.gif Results of Image Clustering]

PRIME will select on of these two clusters and merge it to get a reference set for the Bootstrap step. Here, the remaining images will get assigned with a basis that makes it correlate best with the reference set.

Once all images are assigned with appropriate basis, PRIME will proceed to scaling and post-refinement steps. After three post-refinement cycles (default value), the process is done and here is the output of the program.

Isotropic B-factor: 5.30
No. of reflections
all: 7786
outside resolution: 51
outliers: 0
total left: 7735
Summary for Prime_Run_1/postref_cycle_3_merge.mtz
Bin Resolution Range Completeness N_obs |Rmerge Rsplit CC1/2 N_ind |CCiso N_ind|CCanoma N_ind| I/sigI I sigI I**2
--------------------------------------------------------------------------------------------------------------------------------------------------
01 19.88 - 5.35 100.0 807 / 807 189.78 85.75 8.87 98.43 807 0.00 0 0.00 0 4.46 684.2 136.2 3.57
02 5.35 - 4.26 100.0 782 / 782 140.90 73.61 8.89 97.67 782 0.00 0 0.00 0 5.23 794.3 140.3 2.05
03 4.26 - 3.73 100.0 788 / 788 129.32 69.32 8.70 98.11 788 0.00 0 0.00 0 5.41 878.5 150.3 1.95
04 3.73 - 3.39 100.0 765 / 765 117.75 70.72 9.67 97.55 765 0.00 0 0.00 0 4.12 712.3 162.4 1.88
05 3.39 - 3.15 100.0 770 / 770 113.22 71.61 11.22 88.54 770 0.00 0 0.00 0 2.73 500.1 173.7 2.19
06 3.15 - 2.96 100.0 767 / 767 106.12 73.19 11.07 97.21 767 0.00 0 0.00 0 2.09 404.6 183.8 2.02
07 2.96 - 2.81 100.0 766 / 766 103.73 75.79 12.53 96.62 766 0.00 0 0.00 0 1.72 345.3 193.9 1.89
08 2.81 - 2.69 100.0 745 / 745 101.51 76.11 12.84 96.21 745 0.00 0 0.00 0 1.49 317.3 204.1 1.98
09 2.69 - 2.59 100.0 786 / 786 97.88 77.59 14.19 95.40 786 0.00 0 0.00 0 1.38 299.0 209.3 1.86
10 2.59 - 2.50 100.0 759 / 759 92.53 78.61 14.72 96.51 759 0.00 0 0.00 0 1.36 312.4 218.8 2.07
--------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL 100.0 7735 / 7735 119.73 75.51 10.45 97.21 7735 0.00 0 0.00 0 3.02 527.3 176.9 2.56
--------------------------------------------------------------------------------------------------------------------------------------------------

Summary of CC1/2 on three crystal axes
Bin Resolution Range CC1/2 I N_refl
a* b* c* | a* b* c* | a* b* c*
---------------------------------------------------------------------------------------------------------
01 19.88 - 5.35 97.01 98.64 98.23 528.6 559.6 1216.7 42 51 47
02 5.35 - 4.26 97.64 98.43 99.08 817.6 527.5 964.9 43 44 40
03 4.26 - 3.73 96.31 98.02 97.68 605.7 682.9 856.0 39 39 41
04 3.73 - 3.39 98.49 98.55 97.73 961.9 532.6 729.1 42 37 45
05 3.39 - 3.15 96.88 98.38 92.69 449.5 492.1 721.6 39 39 40
06 3.15 - 2.96 98.48 93.58 98.61 389.9 303.4 391.7 39 37 39
07 2.96 - 2.81 96.98 98.02 95.35 361.3 331.7 383.2 42 37 43
08 2.81 - 2.69 95.29 94.02 94.69 290.8 194.7 292.7 41 35 36
09 2.69 - 2.59 96.55 91.88 98.57 265.7 341.4 290.2 41 35 44
10 2.59 - 2.50 94.44 97.81 96.67 249.0 400.1 236.2 42 36 40
----------------------------------------------------------------------------------------------------------
total 97.57 97.74 94.58 494.1 446.0 619.4 410 390 415
----------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 4733
No. bad cc frames: 113
No. bad G frames) : 109
No. bad unit cell frames: 20
No. bad gamma_e frames: 22
No. bad SE: 2
No. observations: 935265
Mean target value (BEFORE: Mean Median (Std.))
post-refinement: 301.22 259.10 ( 171.56)
(x,y) restraints: 1679.63 1573.15 ( 657.49)
Mean target value (AFTER: Mean Median (Std.))
post-refinement: 300.02 257.53 ( 170.98)
(x,y) restraints: 1679.90 1572.77 ( 660.19)
SE: 1915.60 776.84 ( 33765.97)
G: 1.000e+00 8.971e-01 ( 8.15e-01)
B: 11.83 14.45 ( 11.95)
Rot.x: -0.08 0.00 ( 12.10)
Rot.y: 0.14 0.00 ( 9.62)
gamma_y: 0.00000 0.00000 ( 0.00000)
gamma_z: 0.00000 0.00000 ( 0.00000)
gamma_0: 0.03793 0.00019 ( 0.60820)
gamma_e: -0.12824 0.00145 ( 0.60227)
voigt_nu: 0.50000 0.50000 ( 0.00000)
unit cell
a: 91.45 91.45 ( 0.11)
b: 91.45 91.45 ( 0.11)
c: 45.96 45.96 ( 0.12)
alpha: 90.00 90.00 ( 0.00)
beta: 90.00 90.00 ( 0.00)
gamma: 120.00 120.00 ( 0.00)
Parmeters from integration (not-refined)
Wavelength: 0.96861 0.96861 ( 0.00000)
Detector distance: 303.81868 303.81868 ( 0.00000)
* (standard deviation)

Total calculation time: 542.00 seconds
Finished: Tuesday 14. February 2017 10:53:18

== Obtaining the Output ==

Your output will be in Prime_Run_n (where n is the number of run).

-bash-4.1$ ls Prime_Run_1/ -l
total 9076
-rw-r--r-- 1 mu238 camb 879638 Feb 14 10:53 crystal.o
drwxr-xr-x 2 mu238 camb 104 Feb 14 10:46 index_ambiguity
drwxr-xr-x 2 mu238 camb 6 Feb 14 10:44 isoform_cluster
-rw-r--r-- 1 mu238 camb 32704 Feb 14 10:53 log.txt
-rw-r--r-- 1 mu238 camb 324556 Feb 14 10:47 mean_scaled_merge.hkl
-rw-r--r-- 1 mu238 camb 157260 Feb 14 10:47 mean_scaled_merge.mtz
-rw-r--r-- 1 mu238 camb 15753 Feb 14 10:53 pickle.stat
-rw-r--r-- 1 mu238 camb 324381 Feb 14 10:49 postref_cycle_1_merge.hkl
-rw-r--r-- 1 mu238 camb 157260 Feb 14 10:49 postref_cycle_1_merge.mtz
-rw-r--r-- 1 mu238 camb 324515 Feb 14 10:51 postref_cycle_2_merge.hkl
-rw-r--r-- 1 mu238 camb 157260 Feb 14 10:51 postref_cycle_2_merge.mtz
-rw-r--r-- 1 mu238 camb 324716 Feb 14 10:53 postref_cycle_3_merge.hkl
-rw-r--r-- 1 mu238 camb 157340 Feb 14 10:53 postref_cycle_3_merge.mtz
-rw-r--r-- 1 mu238 camb 6412200 Feb 14 10:53 rejections.txt

File log.txt contains all the merging stats. The final merged reflection set is postref_cycle_3_merge.mtz (or .hkl).

2017 prime tutorial

2017-02-14T19:56:35Z

Mona: /* Running the Program */

Post-refine and Merge Sample Data Set with PRIME (2017 Tutorial)

In this tutorial, we will work on the integration results from the first of Tutorial 2 (Myoglobin Data). Before proceeding to running the program, we'll consider making the input file for PRIME based on the situation of this data set.

== Generating Input File ==

PRIME input files contain information necessary for successful post-refinement and merging steps. You can access and review the list of input parameters by running prime.run or prime.run -h to view the description of these parameters. For this tutorial we'll start building it from scratch.

* Location of integration results
In this case, we know the location where the integration results (pickle files) are. We can then set,

data = /net/viper/raid1/mu238/XfelProject/dials17/extracted

Note that you supply data as a multiple arguments. The value of the parameter can be a file containing list of integration results, a folder, or a wildcard argument.

* Unit cell information
You can obtain the mean (or median) unit-cell dimensions from either IOTA or DIALS. In case of IOTA, prime .phil file is auto generated and this information is readily available in there. For n_residues, enter number of residues in asymmetric unit of your molecule.

target_unit_cell = 91.7 91.7 46 90 90 120
target_space_group = P6
n_residues = 128

* Detector information
pixel_size_mm = 0.172

* Post-refinement and Scaling information
This is where you specify the optimal resolution cutoffs for post-refinement and merging. Note that when running for the first time on you newly collected data, you can choose the "expected" values (resolution which you see the spots at the corner or on the edge). You can then adjust these parameters when analyzing merging statistics based on the I/sigI values in the high resolution shells and rerun the program again. Note that sigma cutoffs are set to 1.5 in scaling and post-refinement steps while it's set to -3.0 so we can include negative values in the merged reflection set.

scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
}
postref {
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
allparams {
flag_on = True
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
merge {
d_min = 2.5
d_max = 20
sigma_min = -3.0
partiality_min = 0.1
uc_tolerance = 5
}

* Indexing ambiguity
For other sets that are not in polar space or have indexing ambiguity (when one or more of the unit-cell dimensions are very similar but not the same!), you can very well use the .phil file parameters thus far to proceed and run post-refinement. However, this data set is in P6 (polar space group) and therefore, the indexing ambiguity needs to be resolved prior to other refinement and merging steps.

Other point worth noting is for any polar space groups, PRIME will automatically solve the ambiguity based on the default parameters. However, this data set has about 5,000 integration results so we want to make sure that we modify the number of images used for random and best selections.

indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
d_min = 3.0
d_max = 10.0
sigma_min = 1.5
n_sample_frames = 1000
n_selected_frames = 100
}

We left other parameters to their default value and modified n_sample_frames to 1000 and n_selected_frames to 100.

* No. of Bin
n_bins = 10

Now we have a complete .phil file ready to run.

data = /net/viper/raid1/mu238/XfelProject/dials17/extracted
target_unit_cell = 91.7 91.7 46 90 90 120
target_space_group = P6
n_residues = 128
pixel_size_mm = 0.172
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
}
postref {
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
}
allparams {
flag_on = True
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
merge {
d_min = 2.5
d_max = 20
sigma_min = -3.0
partiality_min = 0.1
uc_tolerance = 5
}
indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
d_min = 3.0
d_max = 10.0
sigma_min = 1.5
n_sample_frames = 1000
n_selected_frames = 100
}
n_bins = 10

Copy and paste this set of parameter in an editor then save the file as "prime.phil".

== Running the Program ==
You can run the program by giving it an input file:

prime.run prime.phil

For this tutorial, PRIME will score the randomly selected 1,000 images then select the best 100 for running Brehm & Diederichs algorithm in Bootstrap mode. If you run the program with flag_plot=True, you'll see a plot showing two separated clusters, each representing images with matching assigned basis.

[https://commons.wikimedia.org/wiki/File:Dials17_myo_indexing_ambiguity.png Results of Running Indexing Ambiguity with Boostrap]

PRIME will select on of these two clusters and merge it to get a reference set for the Bootstrap step. Here, the remaining images will get assigned with a basis that makes it correlate best with the reference set.

Once all images are assigned with appropriate basis, PRIME will proceed to scaling and post-refinement steps. After three post-refinement cycles (default value), the process is done and here is the output of the program.

Isotropic B-factor: 5.30
No. of reflections
all: 7786
outside resolution: 51
outliers: 0
total left: 7735
Summary for Prime_Run_1/postref_cycle_3_merge.mtz
Bin Resolution Range Completeness N_obs |Rmerge Rsplit CC1/2 N_ind |CCiso N_ind|CCanoma N_ind| I/sigI I sigI I**2
--------------------------------------------------------------------------------------------------------------------------------------------------
01 19.88 - 5.35 100.0 807 / 807 189.78 85.75 8.87 98.43 807 0.00 0 0.00 0 4.46 684.2 136.2 3.57
02 5.35 - 4.26 100.0 782 / 782 140.90 73.61 8.89 97.67 782 0.00 0 0.00 0 5.23 794.3 140.3 2.05
03 4.26 - 3.73 100.0 788 / 788 129.32 69.32 8.70 98.11 788 0.00 0 0.00 0 5.41 878.5 150.3 1.95
04 3.73 - 3.39 100.0 765 / 765 117.75 70.72 9.67 97.55 765 0.00 0 0.00 0 4.12 712.3 162.4 1.88
05 3.39 - 3.15 100.0 770 / 770 113.22 71.61 11.22 88.54 770 0.00 0 0.00 0 2.73 500.1 173.7 2.19
06 3.15 - 2.96 100.0 767 / 767 106.12 73.19 11.07 97.21 767 0.00 0 0.00 0 2.09 404.6 183.8 2.02
07 2.96 - 2.81 100.0 766 / 766 103.73 75.79 12.53 96.62 766 0.00 0 0.00 0 1.72 345.3 193.9 1.89
08 2.81 - 2.69 100.0 745 / 745 101.51 76.11 12.84 96.21 745 0.00 0 0.00 0 1.49 317.3 204.1 1.98
09 2.69 - 2.59 100.0 786 / 786 97.88 77.59 14.19 95.40 786 0.00 0 0.00 0 1.38 299.0 209.3 1.86
10 2.59 - 2.50 100.0 759 / 759 92.53 78.61 14.72 96.51 759 0.00 0 0.00 0 1.36 312.4 218.8 2.07
--------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL 100.0 7735 / 7735 119.73 75.51 10.45 97.21 7735 0.00 0 0.00 0 3.02 527.3 176.9 2.56
--------------------------------------------------------------------------------------------------------------------------------------------------

Summary of CC1/2 on three crystal axes
Bin Resolution Range CC1/2 I N_refl
a* b* c* | a* b* c* | a* b* c*
---------------------------------------------------------------------------------------------------------
01 19.88 - 5.35 97.01 98.64 98.23 528.6 559.6 1216.7 42 51 47
02 5.35 - 4.26 97.64 98.43 99.08 817.6 527.5 964.9 43 44 40
03 4.26 - 3.73 96.31 98.02 97.68 605.7 682.9 856.0 39 39 41
04 3.73 - 3.39 98.49 98.55 97.73 961.9 532.6 729.1 42 37 45
05 3.39 - 3.15 96.88 98.38 92.69 449.5 492.1 721.6 39 39 40
06 3.15 - 2.96 98.48 93.58 98.61 389.9 303.4 391.7 39 37 39
07 2.96 - 2.81 96.98 98.02 95.35 361.3 331.7 383.2 42 37 43
08 2.81 - 2.69 95.29 94.02 94.69 290.8 194.7 292.7 41 35 36
09 2.69 - 2.59 96.55 91.88 98.57 265.7 341.4 290.2 41 35 44
10 2.59 - 2.50 94.44 97.81 96.67 249.0 400.1 236.2 42 36 40
----------------------------------------------------------------------------------------------------------
total 97.57 97.74 94.58 494.1 446.0 619.4 410 390 415
----------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 4733
No. bad cc frames: 113
No. bad G frames) : 109
No. bad unit cell frames: 20
No. bad gamma_e frames: 22
No. bad SE: 2
No. observations: 935265
Mean target value (BEFORE: Mean Median (Std.))
post-refinement: 301.22 259.10 ( 171.56)
(x,y) restraints: 1679.63 1573.15 ( 657.49)
Mean target value (AFTER: Mean Median (Std.))
post-refinement: 300.02 257.53 ( 170.98)
(x,y) restraints: 1679.90 1572.77 ( 660.19)
SE: 1915.60 776.84 ( 33765.97)
G: 1.000e+00 8.971e-01 ( 8.15e-01)
B: 11.83 14.45 ( 11.95)
Rot.x: -0.08 0.00 ( 12.10)
Rot.y: 0.14 0.00 ( 9.62)
gamma_y: 0.00000 0.00000 ( 0.00000)
gamma_z: 0.00000 0.00000 ( 0.00000)
gamma_0: 0.03793 0.00019 ( 0.60820)
gamma_e: -0.12824 0.00145 ( 0.60227)
voigt_nu: 0.50000 0.50000 ( 0.00000)
unit cell
a: 91.45 91.45 ( 0.11)
b: 91.45 91.45 ( 0.11)
c: 45.96 45.96 ( 0.12)
alpha: 90.00 90.00 ( 0.00)
beta: 90.00 90.00 ( 0.00)
gamma: 120.00 120.00 ( 0.00)
Parmeters from integration (not-refined)
Wavelength: 0.96861 0.96861 ( 0.00000)
Detector distance: 303.81868 303.81868 ( 0.00000)
* (standard deviation)

Total calculation time: 542.00 seconds
Finished: Tuesday 14. February 2017 10:53:18

== Obtaining the Output ==

Your output will be in Prime_Run_n (where n is the number of run).

-bash-4.1$ ls Prime_Run_1/ -l
total 9076
-rw-r--r-- 1 mu238 camb 879638 Feb 14 10:53 crystal.o
drwxr-xr-x 2 mu238 camb 104 Feb 14 10:46 index_ambiguity
drwxr-xr-x 2 mu238 camb 6 Feb 14 10:44 isoform_cluster
-rw-r--r-- 1 mu238 camb 32704 Feb 14 10:53 log.txt
-rw-r--r-- 1 mu238 camb 324556 Feb 14 10:47 mean_scaled_merge.hkl
-rw-r--r-- 1 mu238 camb 157260 Feb 14 10:47 mean_scaled_merge.mtz
-rw-r--r-- 1 mu238 camb 15753 Feb 14 10:53 pickle.stat
-rw-r--r-- 1 mu238 camb 324381 Feb 14 10:49 postref_cycle_1_merge.hkl
-rw-r--r-- 1 mu238 camb 157260 Feb 14 10:49 postref_cycle_1_merge.mtz
-rw-r--r-- 1 mu238 camb 324515 Feb 14 10:51 postref_cycle_2_merge.hkl
-rw-r--r-- 1 mu238 camb 157260 Feb 14 10:51 postref_cycle_2_merge.mtz
-rw-r--r-- 1 mu238 camb 324716 Feb 14 10:53 postref_cycle_3_merge.hkl
-rw-r--r-- 1 mu238 camb 157340 Feb 14 10:53 postref_cycle_3_merge.mtz
-rw-r--r-- 1 mu238 camb 6412200 Feb 14 10:53 rejections.txt

File log.txt contains all the merging stats. The final merged reflection set is postref_cycle_3_merge.mtz (or .hkl).

2017 prime tutorial

2017-02-14T19:49:22Z

Mona: /* Running the Program */

Post-refine and Merge Sample Data Set with PRIME (2017 Tutorial)

In this tutorial, we will work on the integration results from the first of Tutorial 2 (Myoglobin Data). Before proceeding to running the program, we'll consider making the input file for PRIME based on the situation of this data set.

== Generating Input File ==

PRIME input files contain information necessary for successful post-refinement and merging steps. You can access and review the list of input parameters by running prime.run or prime.run -h to view the description of these parameters. For this tutorial we'll start building it from scratch.

* Location of integration results
In this case, we know the location where the integration results (pickle files) are. We can then set,

data = /net/viper/raid1/mu238/XfelProject/dials17/extracted

Note that you supply data as a multiple arguments. The value of the parameter can be a file containing list of integration results, a folder, or a wildcard argument.

* Unit cell information
You can obtain the mean (or median) unit-cell dimensions from either IOTA or DIALS. In case of IOTA, prime .phil file is auto generated and this information is readily available in there. For n_residues, enter number of residues in asymmetric unit of your molecule.

target_unit_cell = 91.7 91.7 46 90 90 120
target_space_group = P6
n_residues = 128

* Detector information
pixel_size_mm = 0.172

* Post-refinement and Scaling information
This is where you specify the optimal resolution cutoffs for post-refinement and merging. Note that when running for the first time on you newly collected data, you can choose the "expected" values (resolution which you see the spots at the corner or on the edge). You can then adjust these parameters when analyzing merging statistics based on the I/sigI values in the high resolution shells and rerun the program again. Note that sigma cutoffs are set to 1.5 in scaling and post-refinement steps while it's set to -3.0 so we can include negative values in the merged reflection set.

scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
}
postref {
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
allparams {
flag_on = True
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
merge {
d_min = 2.5
d_max = 20
sigma_min = -3.0
partiality_min = 0.1
uc_tolerance = 5
}

* Indexing ambiguity
For other sets that are not in polar space or have indexing ambiguity (when one or more of the unit-cell dimensions are very similar but not the same!), you can very well use the .phil file parameters thus far to proceed and run post-refinement. However, this data set is in P6 (polar space group) and therefore, the indexing ambiguity needs to be resolved prior to other refinement and merging steps.

Other point worth noting is for any polar space groups, PRIME will automatically solve the ambiguity based on the default parameters. However, this data set has about 5,000 integration results so we want to make sure that we modify the number of images used for random and best selections.

indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
d_min = 3.0
d_max = 10.0
sigma_min = 1.5
n_sample_frames = 1000
n_selected_frames = 100
}

We left other parameters to their default value and modified n_sample_frames to 1000 and n_selected_frames to 100.

* No. of Bin
n_bins = 10

Now we have a complete .phil file ready to run.

data = /net/viper/raid1/mu238/XfelProject/dials17/extracted
target_unit_cell = 91.7 91.7 46 90 90 120
target_space_group = P6
n_residues = 128
pixel_size_mm = 0.172
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
}
postref {
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
}
allparams {
flag_on = True
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
merge {
d_min = 2.5
d_max = 20
sigma_min = -3.0
partiality_min = 0.1
uc_tolerance = 5
}
indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
d_min = 3.0
d_max = 10.0
sigma_min = 1.5
n_sample_frames = 1000
n_selected_frames = 100
}
n_bins = 10

Copy and paste this set of parameter in an editor then save the file as "prime.phil".

== Running the Program ==
You can run the program by giving it an input file:

prime.run prime.phil

For this tutorial, PRIME will score the randomly selected 1,000 images then select the best 100 for running Brehm & Diederichs algorithm in Bootstrap mode. If you run the program with flag_plot=True, you'll see a plot showing two separated clusters, each representing images with matching assigned basis.

[[File:Dials17 myo indexing ambiguity.png|thumb|left|Solving indexing ambiguity program in XFEL Crystallography.]]
PRIME will select on of these two clusters and merge it to get a reference set for the Bootstrap step. Here, the remaining images will get assigned with a basis that makes it correlate best with the reference set.

Once all images are assigned with appropriate basis, PRIME will proceed to scaling and post-refinement steps. After three post-refinement cycles (default value), the process is done and here is the output of the program.

Isotropic B-factor: 5.30
No. of reflections
all: 7786
outside resolution: 51
outliers: 0
total left: 7735
Summary for Prime_Run_1/postref_cycle_3_merge.mtz
Bin Resolution Range Completeness N_obs |Rmerge Rsplit CC1/2 N_ind |CCiso N_ind|CCanoma N_ind| I/sigI I sigI I**2
--------------------------------------------------------------------------------------------------------------------------------------------------
01 19.88 - 5.35 100.0 807 / 807 189.78 85.75 8.87 98.43 807 0.00 0 0.00 0 4.46 684.2 136.2 3.57
02 5.35 - 4.26 100.0 782 / 782 140.90 73.61 8.89 97.67 782 0.00 0 0.00 0 5.23 794.3 140.3 2.05
03 4.26 - 3.73 100.0 788 / 788 129.32 69.32 8.70 98.11 788 0.00 0 0.00 0 5.41 878.5 150.3 1.95
04 3.73 - 3.39 100.0 765 / 765 117.75 70.72 9.67 97.55 765 0.00 0 0.00 0 4.12 712.3 162.4 1.88
05 3.39 - 3.15 100.0 770 / 770 113.22 71.61 11.22 88.54 770 0.00 0 0.00 0 2.73 500.1 173.7 2.19
06 3.15 - 2.96 100.0 767 / 767 106.12 73.19 11.07 97.21 767 0.00 0 0.00 0 2.09 404.6 183.8 2.02
07 2.96 - 2.81 100.0 766 / 766 103.73 75.79 12.53 96.62 766 0.00 0 0.00 0 1.72 345.3 193.9 1.89
08 2.81 - 2.69 100.0 745 / 745 101.51 76.11 12.84 96.21 745 0.00 0 0.00 0 1.49 317.3 204.1 1.98
09 2.69 - 2.59 100.0 786 / 786 97.88 77.59 14.19 95.40 786 0.00 0 0.00 0 1.38 299.0 209.3 1.86
10 2.59 - 2.50 100.0 759 / 759 92.53 78.61 14.72 96.51 759 0.00 0 0.00 0 1.36 312.4 218.8 2.07
--------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL 100.0 7735 / 7735 119.73 75.51 10.45 97.21 7735 0.00 0 0.00 0 3.02 527.3 176.9 2.56
--------------------------------------------------------------------------------------------------------------------------------------------------

Summary of CC1/2 on three crystal axes
Bin Resolution Range CC1/2 I N_refl
a* b* c* | a* b* c* | a* b* c*
---------------------------------------------------------------------------------------------------------
01 19.88 - 5.35 97.01 98.64 98.23 528.6 559.6 1216.7 42 51 47
02 5.35 - 4.26 97.64 98.43 99.08 817.6 527.5 964.9 43 44 40
03 4.26 - 3.73 96.31 98.02 97.68 605.7 682.9 856.0 39 39 41
04 3.73 - 3.39 98.49 98.55 97.73 961.9 532.6 729.1 42 37 45
05 3.39 - 3.15 96.88 98.38 92.69 449.5 492.1 721.6 39 39 40
06 3.15 - 2.96 98.48 93.58 98.61 389.9 303.4 391.7 39 37 39
07 2.96 - 2.81 96.98 98.02 95.35 361.3 331.7 383.2 42 37 43
08 2.81 - 2.69 95.29 94.02 94.69 290.8 194.7 292.7 41 35 36
09 2.69 - 2.59 96.55 91.88 98.57 265.7 341.4 290.2 41 35 44
10 2.59 - 2.50 94.44 97.81 96.67 249.0 400.1 236.2 42 36 40
----------------------------------------------------------------------------------------------------------
total 97.57 97.74 94.58 494.1 446.0 619.4 410 390 415
----------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 4733
No. bad cc frames: 113
No. bad G frames) : 109
No. bad unit cell frames: 20
No. bad gamma_e frames: 22
No. bad SE: 2
No. observations: 935265
Mean target value (BEFORE: Mean Median (Std.))
post-refinement: 301.22 259.10 ( 171.56)
(x,y) restraints: 1679.63 1573.15 ( 657.49)
Mean target value (AFTER: Mean Median (Std.))
post-refinement: 300.02 257.53 ( 170.98)
(x,y) restraints: 1679.90 1572.77 ( 660.19)
SE: 1915.60 776.84 ( 33765.97)
G: 1.000e+00 8.971e-01 ( 8.15e-01)
B: 11.83 14.45 ( 11.95)
Rot.x: -0.08 0.00 ( 12.10)
Rot.y: 0.14 0.00 ( 9.62)
gamma_y: 0.00000 0.00000 ( 0.00000)
gamma_z: 0.00000 0.00000 ( 0.00000)
gamma_0: 0.03793 0.00019 ( 0.60820)
gamma_e: -0.12824 0.00145 ( 0.60227)
voigt_nu: 0.50000 0.50000 ( 0.00000)
unit cell
a: 91.45 91.45 ( 0.11)
b: 91.45 91.45 ( 0.11)
c: 45.96 45.96 ( 0.12)
alpha: 90.00 90.00 ( 0.00)
beta: 90.00 90.00 ( 0.00)
gamma: 120.00 120.00 ( 0.00)
Parmeters from integration (not-refined)
Wavelength: 0.96861 0.96861 ( 0.00000)
Detector distance: 303.81868 303.81868 ( 0.00000)
* (standard deviation)

Total calculation time: 542.00 seconds
Finished: Tuesday 14. February 2017 10:53:18

== Obtaining the Output ==

Your output will be in Prime_Run_n (where n is the number of run).

-bash-4.1$ ls Prime_Run_1/ -l
total 9076
-rw-r--r-- 1 mu238 camb 879638 Feb 14 10:53 crystal.o
drwxr-xr-x 2 mu238 camb 104 Feb 14 10:46 index_ambiguity
drwxr-xr-x 2 mu238 camb 6 Feb 14 10:44 isoform_cluster
-rw-r--r-- 1 mu238 camb 32704 Feb 14 10:53 log.txt
-rw-r--r-- 1 mu238 camb 324556 Feb 14 10:47 mean_scaled_merge.hkl
-rw-r--r-- 1 mu238 camb 157260 Feb 14 10:47 mean_scaled_merge.mtz
-rw-r--r-- 1 mu238 camb 15753 Feb 14 10:53 pickle.stat
-rw-r--r-- 1 mu238 camb 324381 Feb 14 10:49 postref_cycle_1_merge.hkl
-rw-r--r-- 1 mu238 camb 157260 Feb 14 10:49 postref_cycle_1_merge.mtz
-rw-r--r-- 1 mu238 camb 324515 Feb 14 10:51 postref_cycle_2_merge.hkl
-rw-r--r-- 1 mu238 camb 157260 Feb 14 10:51 postref_cycle_2_merge.mtz
-rw-r--r-- 1 mu238 camb 324716 Feb 14 10:53 postref_cycle_3_merge.hkl
-rw-r--r-- 1 mu238 camb 157340 Feb 14 10:53 postref_cycle_3_merge.mtz
-rw-r--r-- 1 mu238 camb 6412200 Feb 14 10:53 rejections.txt

File log.txt contains all the merging stats. The final merged reflection set is postref_cycle_3_merge.mtz (or .hkl).

2017 prime tutorial

2017-02-14T19:47:35Z

Mona: /* Running the Program */

Post-refine and Merge Sample Data Set with PRIME (2017 Tutorial)

In this tutorial, we will work on the integration results from the first of Tutorial 2 (Myoglobin Data). Before proceeding to running the program, we'll consider making the input file for PRIME based on the situation of this data set.

== Generating Input File ==

PRIME input files contain information necessary for successful post-refinement and merging steps. You can access and review the list of input parameters by running prime.run or prime.run -h to view the description of these parameters. For this tutorial we'll start building it from scratch.

* Location of integration results
In this case, we know the location where the integration results (pickle files) are. We can then set,

data = /net/viper/raid1/mu238/XfelProject/dials17/extracted

Note that you supply data as a multiple arguments. The value of the parameter can be a file containing list of integration results, a folder, or a wildcard argument.

* Unit cell information
You can obtain the mean (or median) unit-cell dimensions from either IOTA or DIALS. In case of IOTA, prime .phil file is auto generated and this information is readily available in there. For n_residues, enter number of residues in asymmetric unit of your molecule.

target_unit_cell = 91.7 91.7 46 90 90 120
target_space_group = P6
n_residues = 128

* Detector information
pixel_size_mm = 0.172

* Post-refinement and Scaling information
This is where you specify the optimal resolution cutoffs for post-refinement and merging. Note that when running for the first time on you newly collected data, you can choose the "expected" values (resolution which you see the spots at the corner or on the edge). You can then adjust these parameters when analyzing merging statistics based on the I/sigI values in the high resolution shells and rerun the program again. Note that sigma cutoffs are set to 1.5 in scaling and post-refinement steps while it's set to -3.0 so we can include negative values in the merged reflection set.

scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
}
postref {
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
allparams {
flag_on = True
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
merge {
d_min = 2.5
d_max = 20
sigma_min = -3.0
partiality_min = 0.1
uc_tolerance = 5
}

* Indexing ambiguity
For other sets that are not in polar space or have indexing ambiguity (when one or more of the unit-cell dimensions are very similar but not the same!), you can very well use the .phil file parameters thus far to proceed and run post-refinement. However, this data set is in P6 (polar space group) and therefore, the indexing ambiguity needs to be resolved prior to other refinement and merging steps.

Other point worth noting is for any polar space groups, PRIME will automatically solve the ambiguity based on the default parameters. However, this data set has about 5,000 integration results so we want to make sure that we modify the number of images used for random and best selections.

indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
d_min = 3.0
d_max = 10.0
sigma_min = 1.5
n_sample_frames = 1000
n_selected_frames = 100
}

We left other parameters to their default value and modified n_sample_frames to 1000 and n_selected_frames to 100.

* No. of Bin
n_bins = 10

Now we have a complete .phil file ready to run.

data = /net/viper/raid1/mu238/XfelProject/dials17/extracted
target_unit_cell = 91.7 91.7 46 90 90 120
target_space_group = P6
n_residues = 128
pixel_size_mm = 0.172
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
}
postref {
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
}
allparams {
flag_on = True
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
merge {
d_min = 2.5
d_max = 20
sigma_min = -3.0
partiality_min = 0.1
uc_tolerance = 5
}
indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
d_min = 3.0
d_max = 10.0
sigma_min = 1.5
n_sample_frames = 1000
n_selected_frames = 100
}
n_bins = 10

Copy and paste this set of parameter in an editor then save the file as "prime.phil".

== Running the Program ==
You can run the program by giving it an input file:

prime.run prime.phil

For this tutorial, PRIME will score the randomly selected 1,000 images then select the best 100 for running Brehm & Diederichs algorithm in Bootstrap mode. If you run the program with flag_plot=True, you'll see a plot showing two separated clusters, each representing images with matching assigned basis.

[[File:Dials17 myo indexing ambiguity.png|thumb|left|Solving indexing ambiguity program in XFEL Crystallography.]]
PRIME will select on of these two clusters and merge it to get a reference set for the Bootstrap step. Here, the remaining images will get assigned with a basis that makes it correlate best with the reference set.

Once all images are assigned with appropriate basis, PRIME will proceed to scaling and post-refinement steps. After three post-refinement cycles (default value), the process is done and here is the output of the program.

Isotropic B-factor: 5.30
No. of reflections
all: 7786
outside resolution: 51
outliers: 0
total left: 7735
Summary for Prime_Run_1/postref_cycle_3_merge.mtz
Bin Resolution Range Completeness N_obs |Rmerge Rsplit CC1/2 N_ind |CCiso N_ind|CCanoma N_ind| I/sigI I sigI I**2
--------------------------------------------------------------------------------------------------------------------------------------------------
01 19.88 - 5.35 100.0 807 / 807 189.78 85.75 8.87 98.43 807 0.00 0 0.00 0 4.46 684.2 136.2 3.57
02 5.35 - 4.26 100.0 782 / 782 140.90 73.61 8.89 97.67 782 0.00 0 0.00 0 5.23 794.3 140.3 2.05
03 4.26 - 3.73 100.0 788 / 788 129.32 69.32 8.70 98.11 788 0.00 0 0.00 0 5.41 878.5 150.3 1.95
04 3.73 - 3.39 100.0 765 / 765 117.75 70.72 9.67 97.55 765 0.00 0 0.00 0 4.12 712.3 162.4 1.88
05 3.39 - 3.15 100.0 770 / 770 113.22 71.61 11.22 88.54 770 0.00 0 0.00 0 2.73 500.1 173.7 2.19
06 3.15 - 2.96 100.0 767 / 767 106.12 73.19 11.07 97.21 767 0.00 0 0.00 0 2.09 404.6 183.8 2.02
07 2.96 - 2.81 100.0 766 / 766 103.73 75.79 12.53 96.62 766 0.00 0 0.00 0 1.72 345.3 193.9 1.89
08 2.81 - 2.69 100.0 745 / 745 101.51 76.11 12.84 96.21 745 0.00 0 0.00 0 1.49 317.3 204.1 1.98
09 2.69 - 2.59 100.0 786 / 786 97.88 77.59 14.19 95.40 786 0.00 0 0.00 0 1.38 299.0 209.3 1.86
10 2.59 - 2.50 100.0 759 / 759 92.53 78.61 14.72 96.51 759 0.00 0 0.00 0 1.36 312.4 218.8 2.07
--------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL 100.0 7735 / 7735 119.73 75.51 10.45 97.21 7735 0.00 0 0.00 0 3.02 527.3 176.9 2.56
--------------------------------------------------------------------------------------------------------------------------------------------------

Summary of CC1/2 on three crystal axes
Bin Resolution Range CC1/2 I N_refl
a* b* c* | a* b* c* | a* b* c*
---------------------------------------------------------------------------------------------------------
01 19.88 - 5.35 97.01 98.64 98.23 528.6 559.6 1216.7 42 51 47
02 5.35 - 4.26 97.64 98.43 99.08 817.6 527.5 964.9 43 44 40
03 4.26 - 3.73 96.31 98.02 97.68 605.7 682.9 856.0 39 39 41
04 3.73 - 3.39 98.49 98.55 97.73 961.9 532.6 729.1 42 37 45
05 3.39 - 3.15 96.88 98.38 92.69 449.5 492.1 721.6 39 39 40
06 3.15 - 2.96 98.48 93.58 98.61 389.9 303.4 391.7 39 37 39
07 2.96 - 2.81 96.98 98.02 95.35 361.3 331.7 383.2 42 37 43
08 2.81 - 2.69 95.29 94.02 94.69 290.8 194.7 292.7 41 35 36
09 2.69 - 2.59 96.55 91.88 98.57 265.7 341.4 290.2 41 35 44
10 2.59 - 2.50 94.44 97.81 96.67 249.0 400.1 236.2 42 36 40
----------------------------------------------------------------------------------------------------------
total 97.57 97.74 94.58 494.1 446.0 619.4 410 390 415
----------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 4733
No. bad cc frames: 113
No. bad G frames) : 109
No. bad unit cell frames: 20
No. bad gamma_e frames: 22
No. bad SE: 2
No. observations: 935265
Mean target value (BEFORE: Mean Median (Std.))
post-refinement: 301.22 259.10 ( 171.56)
(x,y) restraints: 1679.63 1573.15 ( 657.49)
Mean target value (AFTER: Mean Median (Std.))
post-refinement: 300.02 257.53 ( 170.98)
(x,y) restraints: 1679.90 1572.77 ( 660.19)
SE: 1915.60 776.84 ( 33765.97)
G: 1.000e+00 8.971e-01 ( 8.15e-01)
B: 11.83 14.45 ( 11.95)
Rot.x: -0.08 0.00 ( 12.10)
Rot.y: 0.14 0.00 ( 9.62)
gamma_y: 0.00000 0.00000 ( 0.00000)
gamma_z: 0.00000 0.00000 ( 0.00000)
gamma_0: 0.03793 0.00019 ( 0.60820)
gamma_e: -0.12824 0.00145 ( 0.60227)
voigt_nu: 0.50000 0.50000 ( 0.00000)
unit cell
a: 91.45 91.45 ( 0.11)
b: 91.45 91.45 ( 0.11)
c: 45.96 45.96 ( 0.12)
alpha: 90.00 90.00 ( 0.00)
beta: 90.00 90.00 ( 0.00)
gamma: 120.00 120.00 ( 0.00)
Parmeters from integration (not-refined)
Wavelength: 0.96861 0.96861 ( 0.00000)
Detector distance: 303.81868 303.81868 ( 0.00000)
* (standard deviation)

Total calculation time: 542.00 seconds
Finished: Tuesday 14. February 2017 10:53:18

== Obtaining the Output ==

Your output will be in Prime_Run_n (where n is the number of run).

-bash-4.1$ ls Prime_Run_1/ -l
total 9076
-rw-r--r-- 1 mu238 camb 879638 Feb 14 10:53 crystal.o
drwxr-xr-x 2 mu238 camb 104 Feb 14 10:46 index_ambiguity
drwxr-xr-x 2 mu238 camb 6 Feb 14 10:44 isoform_cluster
-rw-r--r-- 1 mu238 camb 32704 Feb 14 10:53 log.txt
-rw-r--r-- 1 mu238 camb 324556 Feb 14 10:47 mean_scaled_merge.hkl
-rw-r--r-- 1 mu238 camb 157260 Feb 14 10:47 mean_scaled_merge.mtz
-rw-r--r-- 1 mu238 camb 15753 Feb 14 10:53 pickle.stat
-rw-r--r-- 1 mu238 camb 324381 Feb 14 10:49 postref_cycle_1_merge.hkl
-rw-r--r-- 1 mu238 camb 157260 Feb 14 10:49 postref_cycle_1_merge.mtz
-rw-r--r-- 1 mu238 camb 324515 Feb 14 10:51 postref_cycle_2_merge.hkl
-rw-r--r-- 1 mu238 camb 157260 Feb 14 10:51 postref_cycle_2_merge.mtz
-rw-r--r-- 1 mu238 camb 324716 Feb 14 10:53 postref_cycle_3_merge.hkl
-rw-r--r-- 1 mu238 camb 157340 Feb 14 10:53 postref_cycle_3_merge.mtz
-rw-r--r-- 1 mu238 camb 6412200 Feb 14 10:53 rejections.txt

File log.txt contains all the merging stats. The final merged reflection set is postref_cycle_3_merge.mtz (or .hkl).

Cctbx.prime

2017-02-14T19:25:55Z

Mona:

== Prime: '''p'''ost-'''r'''ef'''i'''nement and '''me'''rging ==
With the latest update, prime can be used to process data on multiple nodes (on a queuing system). At the moment, only LSF (bsub) is supported. See documentation below for more information on how to use the queuing system.

This major update replaces prime.postrefine with

<pre>
prime.run
</pre>

Step-by-step guidelines to post-refine and merge XFEL diffraction images. For more detail and citation, see
"Enabling X-ray Free Electron Laser Crystallography for Challenging Biological Systems from a Limited Number of Crystals"
[http://elifesciences.org/content/4/e05421 "DOI: http://dx.doi.org/10.7554/eLife.05421"]

== Prime is gui-ed ==
Thanks to Dr. Lyubimov, PRIME is also available as a Graphic User Interface program. Try it by running

<pre>
prime
</pre>

Click to see [https://commons.wikimedia.org/wiki/File:PRIME_main.png "PRIME main gui"] and [https://commons.wikimedia.org/wiki/File:PRIME_advanced_options.png "Advanced options"]

== Getting started ==
'''Generating input phil file''' 
Like most programs developed under ''cctbx'' framework, ''prime'' reads in input .phil file, which stores all the parameters needed to run post-refinement and merging steps. To generate the template .phil file, do the dry run by calling

<pre>
$ prime.run
</pre>

An example of the template .phil file:

<pre>
data = None
run_no = None
title = None
scale {
d_min = 0.1
d_max = 99
sigma_min = 1.5
}
...
</pre>

You can save the content of the output to any file name - in this tutorial, let's save it to thermolysin.phil.

'''First look at your phil file''' 
To run prime, set the required parameters to match with your experiments (you can leave other parameters with their default values - or just delete them from you .phil file). The most interesting parameters are shown below:

<pre>
data = /path/to/your/integarion/result/pickle_files
run_no = 001
title = First trial for thermolysin
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
}
postref {
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
allparams {
flag_on = False
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
}
merge {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
target_unit_cell = 93.99,93.99,130.87,90,90,120
target_space_group = P 61 2 2
pixel_size_mm = 0.102
</pre>

You should pay attention to d_min and d_max for the refinement and merging parameters. If you use IOTA to integrate the images, IOTA will output .phil file for prime that has the optimal resolution range. If not, a few trial-and-error runs may be required to get the best resolution range for your dataset. Use merging statistics output by prime and check the values of CC1/2 and I/sigI to find out your optimal resolution range.

Cell parameters (target_unit_cell and target_space_group) are required to run prime. Target cell parameter is used to remove some outlier images by controlling uc_tolerance parameter (the default value of tolerate range is 3% different). Space group parameter is used in removing outliers and merging with the given symmetry.

Don't forget also to change your pixel size in millimeters. Check what your detector is and note down its pixel size.

'''Running post-refinement''' 
Once you have the input .phil file, you can run ''prime'' by calling
<pre>
prime.run thermolysin.phil
</pre>
''Prime'' will post-refine and merge for reflection sets using three (default value) macrocycles. At the end of the run, you can obtain merging statistics in the last cycle - all other cycle statistics are also available in log.txt.

An example of merging statistics:
<pre>
Summary for 001/postref_cycle_1_merge.mtz
Bin Resolution Range Completeness <N_obs> |Rsplit CC1/2 N_ind |CCanom N_ind| <I/sigI> 
-------------------------------------------------------------------------------------------------------------
02 5.70 - 4.52 100.0 1055 / 1055 65.89 16.02 89.15 1055 0.00 0 20.17 2101.97
03 4.52 - 3.95 100.0 1032 / 1032 61.53 14.48 92.03 1032 0.00 0 20.39 2529.90
04 3.95 - 3.59 100.0 1016 / 1016 54.15 15.61 90.13 1016 0.00 0 16.69 1971.43
05 3.59 - 3.33 100.0 1004 / 1004 42.67 17.66 89.23 1004 0.00 0 14.21 1502.14
06 3.33 - 3.14 100.0 1013 / 1013 32.77 20.40 84.26 1013 0.00 0 11.76 1077.60
07 3.14 - 2.98 100.0 995 / 995 27.36 23.00 78.72 995 0.00 0 11.58 935.37
08 2.98 - 2.85 100.0 1006 / 1006 23.57 22.63 82.26 1006 0.00 0 10.56 722.62
09 2.85 - 2.74 100.0 986 / 986 16.64 28.51 72.90 985 0.00 0 10.01 591.56
10 2.74 - 2.65 99.9 989 / 990 12.41 31.35 72.95 987 0.00 0 9.91 515.07
11 2.65 - 2.56 99.7 979 / 982 9.35 37.14 65.31 970 0.00 0 9.31 438.96
12 2.56 - 2.49 98.0 979 / 999 6.06 45.98 45.37 930 0.00 0 9.45 390.05
13 2.49 - 2.42 95.1 931 / 979 4.46 50.68 34.20 834 0.00 0 8.93 334.80
14 2.42 - 2.37 91.7 896 / 977 3.35 55.66 37.15 729 0.00 0 9.27 320.17
15 2.37 - 2.31 83.9 829 / 988 2.61 56.92 43.21 600 0.00 0 9.60 296.67
16 2.31 - 2.26 72.4 702 / 969 1.97 65.81 26.89 386 0.00 0 10.29 284.39
17 2.26 - 2.22 59.1 582 / 985 1.75 64.72 31.28 275 0.00 0 9.87 284.06
18 2.22 - 2.18 52.9 513 / 970 1.51 71.27 16.86 188 0.00 0 8.93 215.31
19 2.18 - 2.14 35.7 349 / 978 1.32 62.26 68.25 90 0.00 0 8.22 199.09
20 2.14 - 2.10 23.1 227 / 981 1.20 92.14 -9.20 42 0.00 0 8.59 224.44
-------------------------------------------------------------------------------------------------------------
TOTAL 85.9 17224 / 20046 27.11 21.11 92.07 15305 0.00 0 12.87 999.53
-------------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 1809
No. bad cc frames: 153
No. bad G frames) : 0
No. bad unit cell frames: 5
No. bad gamma_e frames: 0
No. bad SE: 0
No. observations: 466997

</pre>

== Solving indexing ambiguity (New) ==
* SOFTWARE UPDATE REQUIRED *
With the latest version (Aug 31, 2016), you can solve the indexing ambiguity problem directly in prime. The Brehm & Diederichs algorithms ([http://dx.doi.org/10.1107/S1399004713025431 "doi:10.1107/S1399004713025431"]) have been implemented with bootstrap capability to handle large dataset.
 
'''Merohedral Twinning''' 
For merohedral twinning (27 space groups e.g. P6), the indexing choices will be determined automatically in prime. Use this default setting in your .phil file,

<pre>
indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
n_sample_frames = 300
n_selected_frames = 100
}
</pre>

The n_sample_frames parameter indicates no. of images that will be used for the calculation of the scoring function. After that, only n_selected_images will be used in the B&D algorithms. This saves a lot of computing time since only the selected images will be used for the determination of the ambiguity. You can change these two parameters to fit with your experiments. The default values are 300 and 100 (give 300 - use 100).

'''Pseudo-Merohedral Twinning''' 
For pseudo-merohedral twinning, due to different possibilities for the indexing choice, prime doesn't determine these choices automatically. If you suspect that you may have pseudo twinning (b and c are similar, beta angle is almost 90 degree but not quite), you have an option to force prime to determine the ambiguity according to your choices.
<pre>
indexing_ambiguity {
mode = Forced
index_basis_in = None
assigned_basis = -h,l,k
assigned_basis = -k, l, h
n_sample_frames = 300
n_selected_frames = 100
}
</pre>

When you set indexing_ambiguity.mode to Forced, you can assign indexing choices according to your problem. In this example, two more choices (-h, l, k and -k, l, h) were assigned as the indexing choice.

'''Reusing the solution''' 
At the end of the run, your solution pickle is saved to your_run_no/index_ambiguity/solution_pickle.pickle. If you don't want to spend time solving the ambiguity again in the next run, you can reuse this solution pickle by setting these parameters:
<pre>
indexing_ambiguity {
mode = Auto
index_basis_in = your_run_no/index_ambiguity/solution_pickle.pickle
}
</pre>

This will bypass the indexing ambiguity module. Prime will use the solution file to perform normal post-refinement and merging.

'''Using an external reference set''' 
To use another isomorphous dataset (e.g. from a synchrotron experiment) as a reference set to solve the ambiguity, you can specify an mtz file as part of these parameters:
<pre>
indexing_ambiguity {
mode = Auto
index_basis_in = path/to/your/mtz/file.mtz
}
</pre>
Again, you can choose to do Auto or Forced (with a list of assigned_basis parameters) depending on your problem.

== More detail with input parameters ==
Now that you have your first trial merged data set, you can explore different parameter settings to merge or to obtain the Bijvoet pairs (I+/I-) for your anomalous data set.

'''Anomalous data:'''
<pre>
target_anomalous_flag = True
</pre>
In the last cycle, prime will output a reflection set with I+ and I-.

'''Number of micro- and macrocycles'''
<pre>
n_postref_cycle = 3
n_postref_sub_cycle = 1
</pre>

'''Number of bins for merging statistics'''
<pre>
n_bins = 20
</pre>

== Help with input parameters ==

Most input parameters are self-explained. However, you can run -h switch to view help information for each parameter.
<pre>
prime.run -h
</pre>

== Running on multiple nodes ==
For LCLS users (or other users with LSF bsub), you can use psana (or your) queuing system to parallelize the entire process. For example, if you want to run your job on 100 nodes using psanq, you can specify:
<pre>
queue {
mode = bsub
qname = psanaq
n_nodes = 100
}
timeout_seconds = 300
</pre>
Prime will divide all the images into 100 batches and submit them to different nodes. It will wait until all images in every batches are done before returning to the merging step (or the exit step in the manual mode). You can control timeout_seconds parameter to tell prime how long it should wait for all the image batches to finish. Usually, this timeout parameter is not used (all images should return before 300 seconds) but in case, you need to wait longer or shorter, you can modify this parameter.

Cctbx.prime

2017-02-14T19:24:16Z

Mona: /* Getting started */

== Prime: '''p'''ost-'''r'''ef'''i'''nement and '''me'''rging ==
With the latest update, prime can be used to process data on multiple nodes (on a queuing system). At the moment, only LSF (bsub) is supported. See documentation below for more information on how to use the queuing system.

This major update replaces prime.postrefine with

<pre>
prime.run
</pre>

Step-by-step guidelines to post-refine and merge XFEL diffraction images. For more detail and citation, see
"Enabling X-ray Free Electron Laser Crystallography for Challenging Biological Systems from a Limited Number of Crystals"
[http://elifesciences.org/content/4/e05421 "DOI: http://dx.doi.org/10.7554/eLife.05421"]

== Prime is gui-ed ==
Thanks to Dr. Lyubimov, PRIME is also available as a Graphic User Interface program. Try it by running

<pre>
prime
</pre>

Click to see [https://commons.wikimedia.org/wiki/File:PRIME_main.png "PRIME main gui"] and [https://commons.wikimedia.org/wiki/File:PRIME_advanced_options.png "Advanced options"]

== Getting started ==
'''Generating input phil file''' 
Like most programs developed under ''cctbx'' framework, ''prime'' reads in input .phil file, which stores all the parameters needed to run post-refinement and merging steps. To generate the template .phil file, do the dry run by calling

<pre>
$ prime.run
</pre>

An example of the template .phil file:

<pre>
data = None
run_no = None
title = None
scale {
d_min = 0.1
d_max = 99
sigma_min = 1.5
}
...
</pre>

You can save the content of the output to any file name - in this tutorial, let's save it to thermolysin.phil.

'''First look at your phil file''' 
To run prime, set the required parameters to match with your experiments (you can leave other parameters with their default values - or just delete them from you .phil file). The most interesting parameters are shown below:

<pre>
data = /path/to/your/integarion/result/pickle_files
run_no = 001
title = First trial for thermolysin
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
}
postref {
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
allparams {
flag_on = False
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
}
merge {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
target_unit_cell = 93.99,93.99,130.87,90,90,120
target_space_group = P 61 2 2
pixel_size_mm = 0.102
</pre>

You should pay attention to d_min and d_max for the refinement and merging parameters. If you use IOTA to integrate the images, IOTA will output .phil file for prime that has the optimal resolution range. If not, a few trial-and-error runs may be required to get the best resolution range for your dataset. Use merging statistics output by prime and check the values of CC1/2 and I/sigI to find out your optimal resolution range.

Cell parameters (target_unit_cell and target_space_group) are required to run prime. Target cell parameter is used to remove some outlier images by controlling uc_tolerance parameter (the default value of tolerate range is 3% different). Space group parameter is used in removing outliers and merging with the given symmetry.

Don't forget also to change your pixel size in millimeters. Check what your detector is and note down its pixel size.

'''Running post-refinement''' 
Once you have the input .phil file, you can run ''prime'' by calling
<pre>
prime.run thermolysin.phil
</pre>
''Prime'' will post-refine and merge for reflection sets using three (default value) macrocycles. At the end of the run, you can obtain merging statistics in the last cycle - all other cycle statistics are also available in log.txt.

An example of merging statistics:
<pre>
Summary for 001/postref_cycle_1_merge.mtz
Bin Resolution Range Completeness <N_obs> |Rsplit CC1/2 N_ind |CCanom N_ind| <I/sigI> 
-------------------------------------------------------------------------------------------------------------
02 5.70 - 4.52 100.0 1055 / 1055 65.89 16.02 89.15 1055 0.00 0 20.17 2101.97
03 4.52 - 3.95 100.0 1032 / 1032 61.53 14.48 92.03 1032 0.00 0 20.39 2529.90
04 3.95 - 3.59 100.0 1016 / 1016 54.15 15.61 90.13 1016 0.00 0 16.69 1971.43
05 3.59 - 3.33 100.0 1004 / 1004 42.67 17.66 89.23 1004 0.00 0 14.21 1502.14
06 3.33 - 3.14 100.0 1013 / 1013 32.77 20.40 84.26 1013 0.00 0 11.76 1077.60
07 3.14 - 2.98 100.0 995 / 995 27.36 23.00 78.72 995 0.00 0 11.58 935.37
08 2.98 - 2.85 100.0 1006 / 1006 23.57 22.63 82.26 1006 0.00 0 10.56 722.62
09 2.85 - 2.74 100.0 986 / 986 16.64 28.51 72.90 985 0.00 0 10.01 591.56
10 2.74 - 2.65 99.9 989 / 990 12.41 31.35 72.95 987 0.00 0 9.91 515.07
11 2.65 - 2.56 99.7 979 / 982 9.35 37.14 65.31 970 0.00 0 9.31 438.96
12 2.56 - 2.49 98.0 979 / 999 6.06 45.98 45.37 930 0.00 0 9.45 390.05
13 2.49 - 2.42 95.1 931 / 979 4.46 50.68 34.20 834 0.00 0 8.93 334.80
14 2.42 - 2.37 91.7 896 / 977 3.35 55.66 37.15 729 0.00 0 9.27 320.17
15 2.37 - 2.31 83.9 829 / 988 2.61 56.92 43.21 600 0.00 0 9.60 296.67
16 2.31 - 2.26 72.4 702 / 969 1.97 65.81 26.89 386 0.00 0 10.29 284.39
17 2.26 - 2.22 59.1 582 / 985 1.75 64.72 31.28 275 0.00 0 9.87 284.06
18 2.22 - 2.18 52.9 513 / 970 1.51 71.27 16.86 188 0.00 0 8.93 215.31
19 2.18 - 2.14 35.7 349 / 978 1.32 62.26 68.25 90 0.00 0 8.22 199.09
20 2.14 - 2.10 23.1 227 / 981 1.20 92.14 -9.20 42 0.00 0 8.59 224.44
-------------------------------------------------------------------------------------------------------------
TOTAL 85.9 17224 / 20046 27.11 21.11 92.07 15305 0.00 0 12.87 999.53
-------------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 1809
No. bad cc frames: 153
No. bad G frames) : 0
No. bad unit cell frames: 5
No. bad gamma_e frames: 0
No. bad SE: 0
No. observations: 466997

</pre>

== Solving indexing ambiguity (New) ==
* SOFTWARE UPDATE REQUIRED *
With the latest version (Aug 31, 2016), you can solve the indexing ambiguity problem directly in prime. The Brehm & Diederichs algorithms ([http://dx.doi.org/10.1107/S1399004713025431 "doi:10.1107/S1399004713025431"]) have been implemented with bootstrap capability to handle large dataset.
 
'''Merohedral Twinning''' 
For merohedral twinning (27 space groups e.g. P6), the indexing choices will be determined automatically in prime. Use this default setting in your .phil file,

<pre>
indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
n_sample_frames = 300
n_selected_frames = 100
}
</pre>

The n_sample_frames parameter indicates no. of images that will be used for the calculation of the scoring function. After that, only n_selected_images will be used in the B&D algorithms. This saves a lot of computing time since only the selected images will be used for the determination of the ambiguity. You can change these two parameters to fit with your experiments. The default values are 300 and 100 (give 300 - use 100).

'''Pseudo-Merohedral Twinning''' 
For pseudo-merohedral twinning, due to different possibilities for the indexing choice, prime doesn't determine these choices automatically. If you suspect that you may have pseudo twinning (b and c are similar, beta angle is almost 90 degree but not quite), you have an option to force prime to determine the ambiguity according to your choices.
<pre>
indexing_ambiguity {
mode = Forced
index_basis_in = None
assigned_basis = -h,l,k
assigned_basis = -k, l, h
n_sample_frames = 300
n_selected_frames = 100
}
</pre>

When you set indexing_ambiguity.mode to Forced, you can assign indexing choices according to your problem. In this example, two more choices (-h, l, k and -k, l, h) were assigned as the indexing choice.

'''Reusing the solution''' 
At the end of the run, your solution pickle is saved to your_run_no/index_ambiguity/solution_pickle.pickle. If you don't want to spend time solving the ambiguity again in the next run, you can reuse this solution pickle by setting these parameters:
<pre>
indexing_ambiguity {
mode = Auto
index_basis_in = your_run_no/index_ambiguity/solution_pickle.pickle
}
</pre>

This will bypass the indexing ambiguity module. Prime will use the solution file to perform normal post-refinement and merging.

'''Using an external reference set''' 
To use another isomorphous dataset (e.g. from a synchrotron experiment) as a reference set to solve the ambiguity, you can specify an mtz file as part of these parameters:
<pre>
indexing_ambiguity {
mode = Auto
index_basis_in = path/to/your/mtz/file.mtz
}
</pre>
Again, you can choose to do Auto or Forced (with a list of assigned_basis parameters) depending on your problem.

== More detail with input parameters ==
Now that you have your first trial merged data set, you can explore different parameter settings to merge or to obtain the Bijvoet pairs (I+/I-) for your anomalous data set.

'''Anomalous data:'''
<pre>
target_anomalous_flag = True
</pre>
In the last cycle, prime will output a reflection set with I+ and I-.

'''Number of micro- and macrocycles'''
<pre>
n_postref_cycle = 3
n_postref_sub_cycle = 1
</pre>

'''Number of bins for merging statistics'''
<pre>
n_bins = 20
</pre>

== Help with input parameters ==

Most input parameters are self-explained. However, you can run -h switch to view help information for each parameter.
<pre>
prime.run -h
</pre>

== Running in manual mode ==
With the same phil file, you can run prime manually. This gives you more freedom in terms of parameter settings at different stages (generating reference set, post-refining images, and merging) or at different cycle of post-refinement.

'''Example A''': I want to generate a reference set then post-refine all the images on the '''scale factors only''' for '''three cycles''' then refine '''all parameters''' in the '''4th cycle'''. To do this, you can follow these steps:

To generate a reference set,
<pre>
prime.genref prime.phil
</pre>

To post-refine on scale factors only, modify your .phil file so that all parameters are turned ''off''.
<pre>
...
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
}
postref {
residual_threshold = 5
residual_threshold_xy = 5
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
allparams {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 3
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To refine all parameters one more cycle, update your .phil file again (flag_on = True)
<pre>
...
allparams {
flag_on = True
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 1
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To obtain the final merged mtz, run
<pre>
prime.merge prime.phil
</pre>

== Running on multiple nodes ==
For LCLS users (or other users with LSF bsub), you can use psana (or your) queuing system to parallelize the entire process. For example, if you want to run your job on 100 nodes using psanq, you can specify:
<pre>
queue {
mode = bsub
qname = psanaq
n_nodes = 100
}
timeout_seconds = 300
</pre>
Prime will divide all the images into 100 batches and submit them to different nodes. It will wait until all images in every batches are done before returning to the merging step (or the exit step in the manual mode). You can control timeout_seconds parameter to tell prime how long it should wait for all the image batches to finish. Usually, this timeout parameter is not used (all images should return before 300 seconds) but in case, you need to wait longer or shorter, you can modify this parameter.

Cctbx.prime

2017-02-14T19:23:39Z

Mona: /* Prime: post-refinement and merging */

== Prime: '''p'''ost-'''r'''ef'''i'''nement and '''me'''rging ==
With the latest update, prime can be used to process data on multiple nodes (on a queuing system). At the moment, only LSF (bsub) is supported. See documentation below for more information on how to use the queuing system.

This major update replaces prime.postrefine with

<pre>
prime.run
</pre>

Step-by-step guidelines to post-refine and merge XFEL diffraction images. For more detail and citation, see
"Enabling X-ray Free Electron Laser Crystallography for Challenging Biological Systems from a Limited Number of Crystals"
[http://elifesciences.org/content/4/e05421 "DOI: http://dx.doi.org/10.7554/eLife.05421"]

== Prime is gui-ed ==
Thanks to Dr. Lyubimov, PRIME is also available as a Graphic User Interface program. Try it by running

<pre>
prime
</pre>

Click to see [https://commons.wikimedia.org/wiki/File:PRIME_main.png "PRIME main gui"] and [https://commons.wikimedia.org/wiki/File:PRIME_advanced_options.png "Advanced options"]

== Getting started ==
'''Generating input phil file''' 
Like most programs developed under ''cctbx'' framework, ''prime'' reads in input .phil file, which stores all the parameters needed to run post-refinement and merging steps. To generate the template .phil file, do the dry run by calling

<pre>
$ prime.run
</pre>

An example of the template .phil file:

<pre>
data = None
run_no = None
title = None
scale {
d_min = 0.1
d_max = 99
sigma_min = 1.5
}
...
</pre>

You can save the content of the output to any file name - in this tutorial, let's save it to thermolysin.phil.

'''First look at your phil file''' 
To run prime, set the required parameters to match with your experiments (you can leave other parameters with their default values - or just delete them from you .phil file). The most interesting parameters are shown below:

<pre>
data = /path/to/your/integarion/result/pickle_files
run_no = 001
title = First trial for thermolysin
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
}
postref {
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
allparams {
flag_on = False
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
}
merge {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
target_unit_cell = 93.99,93.99,130.87,90,90,120
target_space_group = P 61 2 2
pixel_size_mm = 0.102
</pre>

You should pay attention to d_min and d_max for the refinement and merging parameters. If you use IOTA to integrate the images, IOTA will output .phil file for prime that has the optimal resolution range. If not, a few trial-and-error runs may be required to get the best resolution range for your dataset. Use merging statistics output by prime and check the values of CC1/2 and I/sigI to find out your optimal resolution range.

Cell parameters (target_unit_cell and target_space_group) are required to run prime. Target cell parameter is used to remove some outlier images by controlling uc_tolerance parameter (the default value of tolerate range is 3% different). Space group parameter is used in removing outliers and merging with the given symmetry.

Don't forget also to change your pixel size in millimeters. Check what your detector is and note down its pixel size.

'''Running post-refinement in automatic mode''' 
Once you have the input .phil file, you can run ''prime'' by calling
<pre>
prime.run thermolysin.phil
</pre>
''Prime'' will post-refine and merge for reflection sets using three (default value) macrocycles. At the end of the run, you can obtain merging statistics in the last cycle - all other cycle statistics are also available in log.txt.

An example of merging statistics:
<pre>
Summary for 001/postref_cycle_1_merge.mtz
Bin Resolution Range Completeness <N_obs> |Rsplit CC1/2 N_ind |CCanom N_ind| <I/sigI> 
-------------------------------------------------------------------------------------------------------------
02 5.70 - 4.52 100.0 1055 / 1055 65.89 16.02 89.15 1055 0.00 0 20.17 2101.97
03 4.52 - 3.95 100.0 1032 / 1032 61.53 14.48 92.03 1032 0.00 0 20.39 2529.90
04 3.95 - 3.59 100.0 1016 / 1016 54.15 15.61 90.13 1016 0.00 0 16.69 1971.43
05 3.59 - 3.33 100.0 1004 / 1004 42.67 17.66 89.23 1004 0.00 0 14.21 1502.14
06 3.33 - 3.14 100.0 1013 / 1013 32.77 20.40 84.26 1013 0.00 0 11.76 1077.60
07 3.14 - 2.98 100.0 995 / 995 27.36 23.00 78.72 995 0.00 0 11.58 935.37
08 2.98 - 2.85 100.0 1006 / 1006 23.57 22.63 82.26 1006 0.00 0 10.56 722.62
09 2.85 - 2.74 100.0 986 / 986 16.64 28.51 72.90 985 0.00 0 10.01 591.56
10 2.74 - 2.65 99.9 989 / 990 12.41 31.35 72.95 987 0.00 0 9.91 515.07
11 2.65 - 2.56 99.7 979 / 982 9.35 37.14 65.31 970 0.00 0 9.31 438.96
12 2.56 - 2.49 98.0 979 / 999 6.06 45.98 45.37 930 0.00 0 9.45 390.05
13 2.49 - 2.42 95.1 931 / 979 4.46 50.68 34.20 834 0.00 0 8.93 334.80
14 2.42 - 2.37 91.7 896 / 977 3.35 55.66 37.15 729 0.00 0 9.27 320.17
15 2.37 - 2.31 83.9 829 / 988 2.61 56.92 43.21 600 0.00 0 9.60 296.67
16 2.31 - 2.26 72.4 702 / 969 1.97 65.81 26.89 386 0.00 0 10.29 284.39
17 2.26 - 2.22 59.1 582 / 985 1.75 64.72 31.28 275 0.00 0 9.87 284.06
18 2.22 - 2.18 52.9 513 / 970 1.51 71.27 16.86 188 0.00 0 8.93 215.31
19 2.18 - 2.14 35.7 349 / 978 1.32 62.26 68.25 90 0.00 0 8.22 199.09
20 2.14 - 2.10 23.1 227 / 981 1.20 92.14 -9.20 42 0.00 0 8.59 224.44
-------------------------------------------------------------------------------------------------------------
TOTAL 85.9 17224 / 20046 27.11 21.11 92.07 15305 0.00 0 12.87 999.53
-------------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 1809
No. bad cc frames: 153
No. bad G frames) : 0
No. bad unit cell frames: 5
No. bad gamma_e frames: 0
No. bad SE: 0
No. observations: 466997

</pre>

== Solving indexing ambiguity (New) ==
* SOFTWARE UPDATE REQUIRED *
With the latest version (Aug 31, 2016), you can solve the indexing ambiguity problem directly in prime. The Brehm & Diederichs algorithms ([http://dx.doi.org/10.1107/S1399004713025431 "doi:10.1107/S1399004713025431"]) have been implemented with bootstrap capability to handle large dataset.
 
'''Merohedral Twinning''' 
For merohedral twinning (27 space groups e.g. P6), the indexing choices will be determined automatically in prime. Use this default setting in your .phil file,

<pre>
indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
n_sample_frames = 300
n_selected_frames = 100
}
</pre>

The n_sample_frames parameter indicates no. of images that will be used for the calculation of the scoring function. After that, only n_selected_images will be used in the B&D algorithms. This saves a lot of computing time since only the selected images will be used for the determination of the ambiguity. You can change these two parameters to fit with your experiments. The default values are 300 and 100 (give 300 - use 100).

'''Pseudo-Merohedral Twinning''' 
For pseudo-merohedral twinning, due to different possibilities for the indexing choice, prime doesn't determine these choices automatically. If you suspect that you may have pseudo twinning (b and c are similar, beta angle is almost 90 degree but not quite), you have an option to force prime to determine the ambiguity according to your choices.
<pre>
indexing_ambiguity {
mode = Forced
index_basis_in = None
assigned_basis = -h,l,k
assigned_basis = -k, l, h
n_sample_frames = 300
n_selected_frames = 100
}
</pre>

When you set indexing_ambiguity.mode to Forced, you can assign indexing choices according to your problem. In this example, two more choices (-h, l, k and -k, l, h) were assigned as the indexing choice.

'''Reusing the solution''' 
At the end of the run, your solution pickle is saved to your_run_no/index_ambiguity/solution_pickle.pickle. If you don't want to spend time solving the ambiguity again in the next run, you can reuse this solution pickle by setting these parameters:
<pre>
indexing_ambiguity {
mode = Auto
index_basis_in = your_run_no/index_ambiguity/solution_pickle.pickle
}
</pre>

This will bypass the indexing ambiguity module. Prime will use the solution file to perform normal post-refinement and merging.

'''Using an external reference set''' 
To use another isomorphous dataset (e.g. from a synchrotron experiment) as a reference set to solve the ambiguity, you can specify an mtz file as part of these parameters:
<pre>
indexing_ambiguity {
mode = Auto
index_basis_in = path/to/your/mtz/file.mtz
}
</pre>
Again, you can choose to do Auto or Forced (with a list of assigned_basis parameters) depending on your problem.

== More detail with input parameters ==
Now that you have your first trial merged data set, you can explore different parameter settings to merge or to obtain the Bijvoet pairs (I+/I-) for your anomalous data set.

'''Anomalous data:'''
<pre>
target_anomalous_flag = True
</pre>
In the last cycle, prime will output a reflection set with I+ and I-.

'''Number of micro- and macrocycles'''
<pre>
n_postref_cycle = 3
n_postref_sub_cycle = 1
</pre>

'''Number of bins for merging statistics'''
<pre>
n_bins = 20
</pre>

== Help with input parameters ==

Most input parameters are self-explained. However, you can run -h switch to view help information for each parameter.
<pre>
prime.run -h
</pre>

== Running in manual mode ==
With the same phil file, you can run prime manually. This gives you more freedom in terms of parameter settings at different stages (generating reference set, post-refining images, and merging) or at different cycle of post-refinement.

'''Example A''': I want to generate a reference set then post-refine all the images on the '''scale factors only''' for '''three cycles''' then refine '''all parameters''' in the '''4th cycle'''. To do this, you can follow these steps:

To generate a reference set,
<pre>
prime.genref prime.phil
</pre>

To post-refine on scale factors only, modify your .phil file so that all parameters are turned ''off''.
<pre>
...
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
}
postref {
residual_threshold = 5
residual_threshold_xy = 5
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
allparams {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 3
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To refine all parameters one more cycle, update your .phil file again (flag_on = True)
<pre>
...
allparams {
flag_on = True
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 1
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To obtain the final merged mtz, run
<pre>
prime.merge prime.phil
</pre>

== Running on multiple nodes ==
For LCLS users (or other users with LSF bsub), you can use psana (or your) queuing system to parallelize the entire process. For example, if you want to run your job on 100 nodes using psanq, you can specify:
<pre>
queue {
mode = bsub
qname = psanaq
n_nodes = 100
}
timeout_seconds = 300
</pre>
Prime will divide all the images into 100 batches and submit them to different nodes. It will wait until all images in every batches are done before returning to the merging step (or the exit step in the manual mode). You can control timeout_seconds parameter to tell prime how long it should wait for all the image batches to finish. Usually, this timeout parameter is not used (all images should return before 300 seconds) but in case, you need to wait longer or shorter, you can modify this parameter.

2017 prime tutorial

2017-02-14T19:22:04Z

Mona: /* Generating Input File */

Post-refine and Merge Sample Data Set with PRIME (2017 Tutorial)

In this tutorial, we will work on the integration results from the first of Tutorial 2 (Myoglobin Data). Before proceeding to running the program, we'll consider making the input file for PRIME based on the situation of this data set.

== Generating Input File ==

PRIME input files contain information necessary for successful post-refinement and merging steps. You can access and review the list of input parameters by running prime.run or prime.run -h to view the description of these parameters. For this tutorial we'll start building it from scratch.

* Location of integration results
In this case, we know the location where the integration results (pickle files) are. We can then set,

data = /net/viper/raid1/mu238/XfelProject/dials17/extracted

Note that you supply data as a multiple arguments. The value of the parameter can be a file containing list of integration results, a folder, or a wildcard argument.

* Unit cell information
You can obtain the mean (or median) unit-cell dimensions from either IOTA or DIALS. In case of IOTA, prime .phil file is auto generated and this information is readily available in there. For n_residues, enter number of residues in asymmetric unit of your molecule.

target_unit_cell = 91.7 91.7 46 90 90 120
target_space_group = P6
n_residues = 128

* Detector information
pixel_size_mm = 0.172

* Post-refinement and Scaling information
This is where you specify the optimal resolution cutoffs for post-refinement and merging. Note that when running for the first time on you newly collected data, you can choose the "expected" values (resolution which you see the spots at the corner or on the edge). You can then adjust these parameters when analyzing merging statistics based on the I/sigI values in the high resolution shells and rerun the program again. Note that sigma cutoffs are set to 1.5 in scaling and post-refinement steps while it's set to -3.0 so we can include negative values in the merged reflection set.

scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
}
postref {
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
allparams {
flag_on = True
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
merge {
d_min = 2.5
d_max = 20
sigma_min = -3.0
partiality_min = 0.1
uc_tolerance = 5
}

* Indexing ambiguity
For other sets that are not in polar space or have indexing ambiguity (when one or more of the unit-cell dimensions are very similar but not the same!), you can very well use the .phil file parameters thus far to proceed and run post-refinement. However, this data set is in P6 (polar space group) and therefore, the indexing ambiguity needs to be resolved prior to other refinement and merging steps.

Other point worth noting is for any polar space groups, PRIME will automatically solve the ambiguity based on the default parameters. However, this data set has about 5,000 integration results so we want to make sure that we modify the number of images used for random and best selections.

indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
d_min = 3.0
d_max = 10.0
sigma_min = 1.5
n_sample_frames = 1000
n_selected_frames = 100
}

We left other parameters to their default value and modified n_sample_frames to 1000 and n_selected_frames to 100.

* No. of Bin
n_bins = 10

Now we have a complete .phil file ready to run.

data = /net/viper/raid1/mu238/XfelProject/dials17/extracted
target_unit_cell = 91.7 91.7 46 90 90 120
target_space_group = P6
n_residues = 128
pixel_size_mm = 0.172
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
}
postref {
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
}
allparams {
flag_on = True
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
merge {
d_min = 2.5
d_max = 20
sigma_min = -3.0
partiality_min = 0.1
uc_tolerance = 5
}
indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
d_min = 3.0
d_max = 10.0
sigma_min = 1.5
n_sample_frames = 1000
n_selected_frames = 100
}
n_bins = 10

Copy and paste this set of parameter in an editor then save the file as "prime.phil".

== Running the Program ==
You can run the program by giving it an input file:

prime.run prime.phil

For this tutorial, PRIME will score the randomly selected 1,000 images then select the best 100 for running Brehm & Diederichs algorithm in Bootstrap mode. If you run the program with flag_plot=True, you'll see a plot showing two separated clusters, each representing images with matching assigned basis.

PRIME will select on of these two clusters and merge it to get a reference set for the Bootstrap step. Here, the remaining images will get assigned with a basis that makes it correlate best with the reference set.

Once all images are assigned with appropriate basis, PRIME will proceed to scaling and post-refinement steps. After three post-refinement cycles (default value), the process is done and here is the output of the program.

Isotropic B-factor: 5.30
No. of reflections
all: 7786
outside resolution: 51
outliers: 0
total left: 7735
Summary for Prime_Run_1/postref_cycle_3_merge.mtz
Bin Resolution Range Completeness N_obs |Rmerge Rsplit CC1/2 N_ind |CCiso N_ind|CCanoma N_ind| I/sigI I sigI I**2
--------------------------------------------------------------------------------------------------------------------------------------------------
01 19.88 - 5.35 100.0 807 / 807 189.78 85.75 8.87 98.43 807 0.00 0 0.00 0 4.46 684.2 136.2 3.57
02 5.35 - 4.26 100.0 782 / 782 140.90 73.61 8.89 97.67 782 0.00 0 0.00 0 5.23 794.3 140.3 2.05
03 4.26 - 3.73 100.0 788 / 788 129.32 69.32 8.70 98.11 788 0.00 0 0.00 0 5.41 878.5 150.3 1.95
04 3.73 - 3.39 100.0 765 / 765 117.75 70.72 9.67 97.55 765 0.00 0 0.00 0 4.12 712.3 162.4 1.88
05 3.39 - 3.15 100.0 770 / 770 113.22 71.61 11.22 88.54 770 0.00 0 0.00 0 2.73 500.1 173.7 2.19
06 3.15 - 2.96 100.0 767 / 767 106.12 73.19 11.07 97.21 767 0.00 0 0.00 0 2.09 404.6 183.8 2.02
07 2.96 - 2.81 100.0 766 / 766 103.73 75.79 12.53 96.62 766 0.00 0 0.00 0 1.72 345.3 193.9 1.89
08 2.81 - 2.69 100.0 745 / 745 101.51 76.11 12.84 96.21 745 0.00 0 0.00 0 1.49 317.3 204.1 1.98
09 2.69 - 2.59 100.0 786 / 786 97.88 77.59 14.19 95.40 786 0.00 0 0.00 0 1.38 299.0 209.3 1.86
10 2.59 - 2.50 100.0 759 / 759 92.53 78.61 14.72 96.51 759 0.00 0 0.00 0 1.36 312.4 218.8 2.07
--------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL 100.0 7735 / 7735 119.73 75.51 10.45 97.21 7735 0.00 0 0.00 0 3.02 527.3 176.9 2.56
--------------------------------------------------------------------------------------------------------------------------------------------------

Summary of CC1/2 on three crystal axes
Bin Resolution Range CC1/2 I N_refl
a* b* c* | a* b* c* | a* b* c*
---------------------------------------------------------------------------------------------------------
01 19.88 - 5.35 97.01 98.64 98.23 528.6 559.6 1216.7 42 51 47
02 5.35 - 4.26 97.64 98.43 99.08 817.6 527.5 964.9 43 44 40
03 4.26 - 3.73 96.31 98.02 97.68 605.7 682.9 856.0 39 39 41
04 3.73 - 3.39 98.49 98.55 97.73 961.9 532.6 729.1 42 37 45
05 3.39 - 3.15 96.88 98.38 92.69 449.5 492.1 721.6 39 39 40
06 3.15 - 2.96 98.48 93.58 98.61 389.9 303.4 391.7 39 37 39
07 2.96 - 2.81 96.98 98.02 95.35 361.3 331.7 383.2 42 37 43
08 2.81 - 2.69 95.29 94.02 94.69 290.8 194.7 292.7 41 35 36
09 2.69 - 2.59 96.55 91.88 98.57 265.7 341.4 290.2 41 35 44
10 2.59 - 2.50 94.44 97.81 96.67 249.0 400.1 236.2 42 36 40
----------------------------------------------------------------------------------------------------------
total 97.57 97.74 94.58 494.1 446.0 619.4 410 390 415
----------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 4733
No. bad cc frames: 113
No. bad G frames) : 109
No. bad unit cell frames: 20
No. bad gamma_e frames: 22
No. bad SE: 2
No. observations: 935265
Mean target value (BEFORE: Mean Median (Std.))
post-refinement: 301.22 259.10 ( 171.56)
(x,y) restraints: 1679.63 1573.15 ( 657.49)
Mean target value (AFTER: Mean Median (Std.))
post-refinement: 300.02 257.53 ( 170.98)
(x,y) restraints: 1679.90 1572.77 ( 660.19)
SE: 1915.60 776.84 ( 33765.97)
G: 1.000e+00 8.971e-01 ( 8.15e-01)
B: 11.83 14.45 ( 11.95)
Rot.x: -0.08 0.00 ( 12.10)
Rot.y: 0.14 0.00 ( 9.62)
gamma_y: 0.00000 0.00000 ( 0.00000)
gamma_z: 0.00000 0.00000 ( 0.00000)
gamma_0: 0.03793 0.00019 ( 0.60820)
gamma_e: -0.12824 0.00145 ( 0.60227)
voigt_nu: 0.50000 0.50000 ( 0.00000)
unit cell
a: 91.45 91.45 ( 0.11)
b: 91.45 91.45 ( 0.11)
c: 45.96 45.96 ( 0.12)
alpha: 90.00 90.00 ( 0.00)
beta: 90.00 90.00 ( 0.00)
gamma: 120.00 120.00 ( 0.00)
Parmeters from integration (not-refined)
Wavelength: 0.96861 0.96861 ( 0.00000)
Detector distance: 303.81868 303.81868 ( 0.00000)
* (standard deviation)

Total calculation time: 542.00 seconds
Finished: Tuesday 14. February 2017 10:53:18

== Obtaining the Output ==

Your output will be in Prime_Run_n (where n is the number of run).

-bash-4.1$ ls Prime_Run_1/ -l
total 9076
-rw-r--r-- 1 mu238 camb 879638 Feb 14 10:53 crystal.o
drwxr-xr-x 2 mu238 camb 104 Feb 14 10:46 index_ambiguity
drwxr-xr-x 2 mu238 camb 6 Feb 14 10:44 isoform_cluster
-rw-r--r-- 1 mu238 camb 32704 Feb 14 10:53 log.txt
-rw-r--r-- 1 mu238 camb 324556 Feb 14 10:47 mean_scaled_merge.hkl
-rw-r--r-- 1 mu238 camb 157260 Feb 14 10:47 mean_scaled_merge.mtz
-rw-r--r-- 1 mu238 camb 15753 Feb 14 10:53 pickle.stat
-rw-r--r-- 1 mu238 camb 324381 Feb 14 10:49 postref_cycle_1_merge.hkl
-rw-r--r-- 1 mu238 camb 157260 Feb 14 10:49 postref_cycle_1_merge.mtz
-rw-r--r-- 1 mu238 camb 324515 Feb 14 10:51 postref_cycle_2_merge.hkl
-rw-r--r-- 1 mu238 camb 157260 Feb 14 10:51 postref_cycle_2_merge.mtz
-rw-r--r-- 1 mu238 camb 324716 Feb 14 10:53 postref_cycle_3_merge.hkl
-rw-r--r-- 1 mu238 camb 157340 Feb 14 10:53 postref_cycle_3_merge.mtz
-rw-r--r-- 1 mu238 camb 6412200 Feb 14 10:53 rejections.txt

File log.txt contains all the merging stats. The final merged reflection set is postref_cycle_3_merge.mtz (or .hkl).

2017 prime tutorial

2017-02-14T19:18:28Z

Mona:

Post-refine and Merge Sample Data Set with PRIME (2017 Tutorial)

In this tutorial, we will work on the integration results from the first of Tutorial 2 (Myoglobin Data). Before proceeding to running the program, we'll consider making the input file for PRIME based on the situation of this data set.

== Generating Input File ==

PRIME input files contain information necessary for successful post-refinement and merging steps. You can access and review the list of input parameters by running prime.run or prime.run -h to view the description of these parameters. For this tutorial we'll start building it from scratch.

* Location of integration results
In this case, we know the location where the integration results (pickle files) are. We can then set,

data = /net/viper/raid1/mu238/XfelProject/dials17/extracted

Note that you supply data as a multiple arguments. The value of the parameter can be a file containing list of integration results, a folder, or a wildcard argument.

* Unit cell information
You can obtain the mean (or median) unit-cell dimensions from either IOTA or DIALS. In case of IOTA, prime .phil file is auto generated and this information is readily available in there. For n_residues, enter number of residues in asymmetric unit of your molecule.

target_unit_cell = 91.7 91.7 46 90 90 120
target_space_group = P6
n_residues = 128

* Detector information
pixel_size_mm = 0.172

* Post-refinement and Scaling information
This is where you specify the optimal resolution cutoffs for post-refinement and merging. Note that when running for the first time on you newly collected data, you can choose the "expected" values (resolution which you see the spots at the corner or on the edge). You can then adjust these parameters when analyzing merging statistics based on the I/sigI values in the high resolution shells and rerun the program again. Note that sigma cutoffs are set to 1.5 in scaling and post-refinement steps while it's set to -3.0 so we can include negative values in the merged reflection set.

scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
}
postref {
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
allparams {
flag_on = True
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
merge {
d_min = 2.5
d_max = 20
sigma_min = -3.0
partiality_min = 0.1
uc_tolerance = 5
}

* Indexing ambiguity
For other sets that are not in polar space or have indexing ambiguity (when one or more of the unit-cell dimensions are very similar but not the same!), you can very well use the .phil file parameters thus far to proceed and run post-refinement. However, this data set is in P6 (polar space group) and therefore, the indexing ambiguity needs to be resolved prior to other refinement and merging steps.

Other point worth noting is for any polar space groups, PRIME will automatically solve the ambiguity based on the default parameters. However, this data set has about 5,000 integration results so we want to make sure that we modify the number of images used for random and best selections.

indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
d_min = 3.0
d_max = 10.0
sigma_min = 1.5
n_sample_frames = 1000
n_selected_frames = 100
}

We left other parameters to their default value and modified n_sample_frames to 1000 and n_selected_frames to 100.

* No. of Bin
n_bins = 10

Now we have a complete .phil file ready to run.

data = /net/viper/raid1/mu238/XfelProject/dials17/extracted
target_unit_cell = 91.7 91.7 46 90 90 120
target_space_group = P6
n_residues = 128
pixel_size_mm = 0.172
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
}
postref {
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
}
allparams {
flag_on = True
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
merge {
d_min = 2.5
d_max = 20
sigma_min = -3.0
partiality_min = 0.1
uc_tolerance = 5
}

indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
d_min = 3.0
d_max = 10.0
sigma_min = 1.5
n_sample_frames = 1000
n_selected_frames = 100
}
n_bins = 10

Copy and paste this set of parameter in an editor then save the file as "prime.phil".

== Running the Program ==
You can run the program by giving it an input file:

prime.run prime.phil

For this tutorial, PRIME will score the randomly selected 1,000 images then select the best 100 for running Brehm & Diederichs algorithm in Bootstrap mode. If you run the program with flag_plot=True, you'll see a plot showing two separated clusters, each representing images with matching assigned basis.

PRIME will select on of these two clusters and merge it to get a reference set for the Bootstrap step. Here, the remaining images will get assigned with a basis that makes it correlate best with the reference set.

Once all images are assigned with appropriate basis, PRIME will proceed to scaling and post-refinement steps. After three post-refinement cycles (default value), the process is done and here is the output of the program.

Isotropic B-factor: 5.30
No. of reflections
all: 7786
outside resolution: 51
outliers: 0
total left: 7735
Summary for Prime_Run_1/postref_cycle_3_merge.mtz
Bin Resolution Range Completeness N_obs |Rmerge Rsplit CC1/2 N_ind |CCiso N_ind|CCanoma N_ind| I/sigI I sigI I**2
--------------------------------------------------------------------------------------------------------------------------------------------------
01 19.88 - 5.35 100.0 807 / 807 189.78 85.75 8.87 98.43 807 0.00 0 0.00 0 4.46 684.2 136.2 3.57
02 5.35 - 4.26 100.0 782 / 782 140.90 73.61 8.89 97.67 782 0.00 0 0.00 0 5.23 794.3 140.3 2.05
03 4.26 - 3.73 100.0 788 / 788 129.32 69.32 8.70 98.11 788 0.00 0 0.00 0 5.41 878.5 150.3 1.95
04 3.73 - 3.39 100.0 765 / 765 117.75 70.72 9.67 97.55 765 0.00 0 0.00 0 4.12 712.3 162.4 1.88
05 3.39 - 3.15 100.0 770 / 770 113.22 71.61 11.22 88.54 770 0.00 0 0.00 0 2.73 500.1 173.7 2.19
06 3.15 - 2.96 100.0 767 / 767 106.12 73.19 11.07 97.21 767 0.00 0 0.00 0 2.09 404.6 183.8 2.02
07 2.96 - 2.81 100.0 766 / 766 103.73 75.79 12.53 96.62 766 0.00 0 0.00 0 1.72 345.3 193.9 1.89
08 2.81 - 2.69 100.0 745 / 745 101.51 76.11 12.84 96.21 745 0.00 0 0.00 0 1.49 317.3 204.1 1.98
09 2.69 - 2.59 100.0 786 / 786 97.88 77.59 14.19 95.40 786 0.00 0 0.00 0 1.38 299.0 209.3 1.86
10 2.59 - 2.50 100.0 759 / 759 92.53 78.61 14.72 96.51 759 0.00 0 0.00 0 1.36 312.4 218.8 2.07
--------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL 100.0 7735 / 7735 119.73 75.51 10.45 97.21 7735 0.00 0 0.00 0 3.02 527.3 176.9 2.56
--------------------------------------------------------------------------------------------------------------------------------------------------

Summary of CC1/2 on three crystal axes
Bin Resolution Range CC1/2 I N_refl
a* b* c* | a* b* c* | a* b* c*
---------------------------------------------------------------------------------------------------------
01 19.88 - 5.35 97.01 98.64 98.23 528.6 559.6 1216.7 42 51 47
02 5.35 - 4.26 97.64 98.43 99.08 817.6 527.5 964.9 43 44 40
03 4.26 - 3.73 96.31 98.02 97.68 605.7 682.9 856.0 39 39 41
04 3.73 - 3.39 98.49 98.55 97.73 961.9 532.6 729.1 42 37 45
05 3.39 - 3.15 96.88 98.38 92.69 449.5 492.1 721.6 39 39 40
06 3.15 - 2.96 98.48 93.58 98.61 389.9 303.4 391.7 39 37 39
07 2.96 - 2.81 96.98 98.02 95.35 361.3 331.7 383.2 42 37 43
08 2.81 - 2.69 95.29 94.02 94.69 290.8 194.7 292.7 41 35 36
09 2.69 - 2.59 96.55 91.88 98.57 265.7 341.4 290.2 41 35 44
10 2.59 - 2.50 94.44 97.81 96.67 249.0 400.1 236.2 42 36 40
----------------------------------------------------------------------------------------------------------
total 97.57 97.74 94.58 494.1 446.0 619.4 410 390 415
----------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 4733
No. bad cc frames: 113
No. bad G frames) : 109
No. bad unit cell frames: 20
No. bad gamma_e frames: 22
No. bad SE: 2
No. observations: 935265
Mean target value (BEFORE: Mean Median (Std.))
post-refinement: 301.22 259.10 ( 171.56)
(x,y) restraints: 1679.63 1573.15 ( 657.49)
Mean target value (AFTER: Mean Median (Std.))
post-refinement: 300.02 257.53 ( 170.98)
(x,y) restraints: 1679.90 1572.77 ( 660.19)
SE: 1915.60 776.84 ( 33765.97)
G: 1.000e+00 8.971e-01 ( 8.15e-01)
B: 11.83 14.45 ( 11.95)
Rot.x: -0.08 0.00 ( 12.10)
Rot.y: 0.14 0.00 ( 9.62)
gamma_y: 0.00000 0.00000 ( 0.00000)
gamma_z: 0.00000 0.00000 ( 0.00000)
gamma_0: 0.03793 0.00019 ( 0.60820)
gamma_e: -0.12824 0.00145 ( 0.60227)
voigt_nu: 0.50000 0.50000 ( 0.00000)
unit cell
a: 91.45 91.45 ( 0.11)
b: 91.45 91.45 ( 0.11)
c: 45.96 45.96 ( 0.12)
alpha: 90.00 90.00 ( 0.00)
beta: 90.00 90.00 ( 0.00)
gamma: 120.00 120.00 ( 0.00)
Parmeters from integration (not-refined)
Wavelength: 0.96861 0.96861 ( 0.00000)
Detector distance: 303.81868 303.81868 ( 0.00000)
* (standard deviation)

Total calculation time: 542.00 seconds
Finished: Tuesday 14. February 2017 10:53:18

== Obtaining the Output ==

Your output will be in Prime_Run_n (where n is the number of run).

-bash-4.1$ ls Prime_Run_1/ -l
total 9076
-rw-r--r-- 1 mu238 camb 879638 Feb 14 10:53 crystal.o
drwxr-xr-x 2 mu238 camb 104 Feb 14 10:46 index_ambiguity
drwxr-xr-x 2 mu238 camb 6 Feb 14 10:44 isoform_cluster
-rw-r--r-- 1 mu238 camb 32704 Feb 14 10:53 log.txt
-rw-r--r-- 1 mu238 camb 324556 Feb 14 10:47 mean_scaled_merge.hkl
-rw-r--r-- 1 mu238 camb 157260 Feb 14 10:47 mean_scaled_merge.mtz
-rw-r--r-- 1 mu238 camb 15753 Feb 14 10:53 pickle.stat
-rw-r--r-- 1 mu238 camb 324381 Feb 14 10:49 postref_cycle_1_merge.hkl
-rw-r--r-- 1 mu238 camb 157260 Feb 14 10:49 postref_cycle_1_merge.mtz
-rw-r--r-- 1 mu238 camb 324515 Feb 14 10:51 postref_cycle_2_merge.hkl
-rw-r--r-- 1 mu238 camb 157260 Feb 14 10:51 postref_cycle_2_merge.mtz
-rw-r--r-- 1 mu238 camb 324716 Feb 14 10:53 postref_cycle_3_merge.hkl
-rw-r--r-- 1 mu238 camb 157340 Feb 14 10:53 postref_cycle_3_merge.mtz
-rw-r--r-- 1 mu238 camb 6412200 Feb 14 10:53 rejections.txt

File log.txt contains all the merging stats. The final merged reflection set is postref_cycle_3_merge.mtz (or .hkl).

2017 prime tutorial

2017-02-14T19:12:54Z

Mona: /* Running the Program */

Post-refine and Merge Sample Data Set with PRIME (2017 Tutorial)

In this tutorial, we will work on the integration results from the first of Tutorial 2 (Myoglobin Data). Before proceeding to running the program, we'll consider making the input file for PRIME based on the situation of this data set.

== Generating Input File ==

PRIME input files contain information necessary for successful post-refinement and merging steps. You can access and review the list of input parameters by running prime.run or prime.run -h to view the description of these parameters. For this tutorial we'll start building it from scratch.

* Location of integration results
In this case, we know the location where the integration results (pickle files) are. We can then set,

data = /net/viper/raid1/mu238/XfelProject/dials17/extracted

Note that you supply data as a multiple arguments. The value of the parameter can be a file containing list of integration results, a folder, or a wildcard argument.

* Unit cell information
You can obtain the mean (or median) unit-cell dimensions from either IOTA or DIALS. In case of IOTA, prime .phil file is auto generated and this information is readily available in there. For n_residues, enter number of residues in asymmetric unit of your molecule.

target_unit_cell = 91.7 91.7 46 90 90 120
target_space_group = P6
n_residues = 128

* Detector information
pixel_size_mm = 0.172

* Post-refinement and Scaling information
This is where you specify the optimal resolution cutoffs for post-refinement and merging. Note that when running for the first time on you newly collected data, you can choose the "expected" values (resolution which you see the spots at the corner or on the edge). You can then adjust these parameters when analyzing merging statistics based on the I/sigI values in the high resolution shells and rerun the program again. Note that sigma cutoffs are set to 1.5 in scaling and post-refinement steps while it's set to -3.0 so we can include negative values in the merged reflection set.

scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
}
postref {
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
allparams {
flag_on = True
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
merge {
d_min = 2.5
d_max = 20
sigma_min = -3.0
partiality_min = 0.1
uc_tolerance = 5
}

* Indexing ambiguity
For other sets that are not in polar space or have indexing ambiguity (when one or more of the unit-cell dimensions are very similar but not the same!), you can very well use the .phil file parameters thus far to proceed and run post-refinement. However, this data set is in P6 (polar space group) and therefore, the indexing ambiguity needs to be resolved prior to other refinement and merging steps.

Other point worth noting is for any polar space groups, PRIME will automatically solve the ambiguity based on the default parameters. However, this data set has about 5,000 integration results so we want to make sure that we modify the number of images used for random and best selections.

indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
d_min = 3.0
d_max = 10.0
sigma_min = 1.5
n_sample_frames = 1000
n_selected_frames = 100
}

We left other parameters to their default value and modified n_sample_frames to 1000 and n_selected_frames to 100.

* No. of Bin
n_bins = 10

Now we have a complete .phil file ready to run.

data = /net/viper/raid1/mu238/XfelProject/dials17/extracted
target_unit_cell = 91.7 91.7 46 90 90 120
target_space_group = P6
n_residues = 128
pixel_size_mm = 0.172
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
}
postref {
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
}
allparams {
flag_on = True
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
merge {
d_min = 2.5
d_max = 20
sigma_min = -3.0
partiality_min = 0.1
uc_tolerance = 5
}

indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
d_min = 3.0
d_max = 10.0
sigma_min = 1.5
n_sample_frames = 1000
n_selected_frames = 100
}
n_bins = 10

Copy and paste this set of parameter in an editor then save the file as "prime.phil".

== Running the Program ==
You can run the program by giving it an input file:

prime.run prime.phil

For this tutorial, PRIME will score the randomly selected 1,000 images then select the best 100 for running Brehm & Diederichs algorithm in Bootstrap mode. If you run the program with flag_plot=True, you'll see a plot showing two separated clusters, each representing images with matching assigned basis.

PRIME will select on of these two clusters and merge it to get a reference set for the Bootstrap step. Here, the remaining images will get assigned with a basis that makes it correlate best with the reference set.

Once all images are assigned with appropriate basis, PRIME will proceed to scaling and post-refinement steps. After three post-refinement cycles (default value), the process is done and here is the output of the program.

Isotropic B-factor: 5.30
No. of reflections
all: 7786
outside resolution: 51
outliers: 0
total left: 7735
Summary for Prime_Run_1/postref_cycle_3_merge.mtz
Bin Resolution Range Completeness N_obs |Rmerge Rsplit CC1/2 N_ind |CCiso N_ind|CCanoma N_ind| I/sigI I sigI I**2
--------------------------------------------------------------------------------------------------------------------------------------------------
01 19.88 - 5.35 100.0 807 / 807 189.78 85.75 8.87 98.43 807 0.00 0 0.00 0 4.46 684.2 136.2 3.57
02 5.35 - 4.26 100.0 782 / 782 140.90 73.61 8.89 97.67 782 0.00 0 0.00 0 5.23 794.3 140.3 2.05
03 4.26 - 3.73 100.0 788 / 788 129.32 69.32 8.70 98.11 788 0.00 0 0.00 0 5.41 878.5 150.3 1.95
04 3.73 - 3.39 100.0 765 / 765 117.75 70.72 9.67 97.55 765 0.00 0 0.00 0 4.12 712.3 162.4 1.88
05 3.39 - 3.15 100.0 770 / 770 113.22 71.61 11.22 88.54 770 0.00 0 0.00 0 2.73 500.1 173.7 2.19
06 3.15 - 2.96 100.0 767 / 767 106.12 73.19 11.07 97.21 767 0.00 0 0.00 0 2.09 404.6 183.8 2.02
07 2.96 - 2.81 100.0 766 / 766 103.73 75.79 12.53 96.62 766 0.00 0 0.00 0 1.72 345.3 193.9 1.89
08 2.81 - 2.69 100.0 745 / 745 101.51 76.11 12.84 96.21 745 0.00 0 0.00 0 1.49 317.3 204.1 1.98
09 2.69 - 2.59 100.0 786 / 786 97.88 77.59 14.19 95.40 786 0.00 0 0.00 0 1.38 299.0 209.3 1.86
10 2.59 - 2.50 100.0 759 / 759 92.53 78.61 14.72 96.51 759 0.00 0 0.00 0 1.36 312.4 218.8 2.07
--------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL 100.0 7735 / 7735 119.73 75.51 10.45 97.21 7735 0.00 0 0.00 0 3.02 527.3 176.9 2.56
--------------------------------------------------------------------------------------------------------------------------------------------------

Summary of CC1/2 on three crystal axes
Bin Resolution Range CC1/2 I N_refl
a* b* c* | a* b* c* | a* b* c*
---------------------------------------------------------------------------------------------------------
01 19.88 - 5.35 97.01 98.64 98.23 528.6 559.6 1216.7 42 51 47
02 5.35 - 4.26 97.64 98.43 99.08 817.6 527.5 964.9 43 44 40
03 4.26 - 3.73 96.31 98.02 97.68 605.7 682.9 856.0 39 39 41
04 3.73 - 3.39 98.49 98.55 97.73 961.9 532.6 729.1 42 37 45
05 3.39 - 3.15 96.88 98.38 92.69 449.5 492.1 721.6 39 39 40
06 3.15 - 2.96 98.48 93.58 98.61 389.9 303.4 391.7 39 37 39
07 2.96 - 2.81 96.98 98.02 95.35 361.3 331.7 383.2 42 37 43
08 2.81 - 2.69 95.29 94.02 94.69 290.8 194.7 292.7 41 35 36
09 2.69 - 2.59 96.55 91.88 98.57 265.7 341.4 290.2 41 35 44
10 2.59 - 2.50 94.44 97.81 96.67 249.0 400.1 236.2 42 36 40
----------------------------------------------------------------------------------------------------------
total 97.57 97.74 94.58 494.1 446.0 619.4 410 390 415
----------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 4733
No. bad cc frames: 113
No. bad G frames) : 109
No. bad unit cell frames: 20
No. bad gamma_e frames: 22
No. bad SE: 2
No. observations: 935265
Mean target value (BEFORE: Mean Median (Std.))
post-refinement: 301.22 259.10 ( 171.56)
(x,y) restraints: 1679.63 1573.15 ( 657.49)
Mean target value (AFTER: Mean Median (Std.))
post-refinement: 300.02 257.53 ( 170.98)
(x,y) restraints: 1679.90 1572.77 ( 660.19)
SE: 1915.60 776.84 ( 33765.97)
G: 1.000e+00 8.971e-01 ( 8.15e-01)
B: 11.83 14.45 ( 11.95)
Rot.x: -0.08 0.00 ( 12.10)
Rot.y: 0.14 0.00 ( 9.62)
gamma_y: 0.00000 0.00000 ( 0.00000)
gamma_z: 0.00000 0.00000 ( 0.00000)
gamma_0: 0.03793 0.00019 ( 0.60820)
gamma_e: -0.12824 0.00145 ( 0.60227)
voigt_nu: 0.50000 0.50000 ( 0.00000)
unit cell
a: 91.45 91.45 ( 0.11)
b: 91.45 91.45 ( 0.11)
c: 45.96 45.96 ( 0.12)
alpha: 90.00 90.00 ( 0.00)
beta: 90.00 90.00 ( 0.00)
gamma: 120.00 120.00 ( 0.00)
Parmeters from integration (not-refined)
Wavelength: 0.96861 0.96861 ( 0.00000)
Detector distance: 303.81868 303.81868 ( 0.00000)
* (standard deviation)

Total calculation time: 542.00 seconds
Finished: Tuesday 14. February 2017 10:53:18

2017 prime tutorial

2017-02-14T19:12:15Z

Mona: /* Running the Program */

Post-refine and Merge Sample Data Set with PRIME (2017 Tutorial)

In this tutorial, we will work on the integration results from the first of Tutorial 2 (Myoglobin Data). Before proceeding to running the program, we'll consider making the input file for PRIME based on the situation of this data set.

== Generating Input File ==

PRIME input files contain information necessary for successful post-refinement and merging steps. You can access and review the list of input parameters by running prime.run or prime.run -h to view the description of these parameters. For this tutorial we'll start building it from scratch.

* Location of integration results
In this case, we know the location where the integration results (pickle files) are. We can then set,

data = /net/viper/raid1/mu238/XfelProject/dials17/extracted

Note that you supply data as a multiple arguments. The value of the parameter can be a file containing list of integration results, a folder, or a wildcard argument.

* Unit cell information
You can obtain the mean (or median) unit-cell dimensions from either IOTA or DIALS. In case of IOTA, prime .phil file is auto generated and this information is readily available in there. For n_residues, enter number of residues in asymmetric unit of your molecule.

target_unit_cell = 91.7 91.7 46 90 90 120
target_space_group = P6
n_residues = 128

* Detector information
pixel_size_mm = 0.172

* Post-refinement and Scaling information
This is where you specify the optimal resolution cutoffs for post-refinement and merging. Note that when running for the first time on you newly collected data, you can choose the "expected" values (resolution which you see the spots at the corner or on the edge). You can then adjust these parameters when analyzing merging statistics based on the I/sigI values in the high resolution shells and rerun the program again. Note that sigma cutoffs are set to 1.5 in scaling and post-refinement steps while it's set to -3.0 so we can include negative values in the merged reflection set.

scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
}
postref {
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
allparams {
flag_on = True
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
merge {
d_min = 2.5
d_max = 20
sigma_min = -3.0
partiality_min = 0.1
uc_tolerance = 5
}

* Indexing ambiguity
For other sets that are not in polar space or have indexing ambiguity (when one or more of the unit-cell dimensions are very similar but not the same!), you can very well use the .phil file parameters thus far to proceed and run post-refinement. However, this data set is in P6 (polar space group) and therefore, the indexing ambiguity needs to be resolved prior to other refinement and merging steps.

Other point worth noting is for any polar space groups, PRIME will automatically solve the ambiguity based on the default parameters. However, this data set has about 5,000 integration results so we want to make sure that we modify the number of images used for random and best selections.

indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
d_min = 3.0
d_max = 10.0
sigma_min = 1.5
n_sample_frames = 1000
n_selected_frames = 100
}

We left other parameters to their default value and modified n_sample_frames to 1000 and n_selected_frames to 100.

* No. of Bin
n_bins = 10

Now we have a complete .phil file ready to run.

data = /net/viper/raid1/mu238/XfelProject/dials17/extracted
target_unit_cell = 91.7 91.7 46 90 90 120
target_space_group = P6
n_residues = 128
pixel_size_mm = 0.172
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
}
postref {
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
}
allparams {
flag_on = True
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
merge {
d_min = 2.5
d_max = 20
sigma_min = -3.0
partiality_min = 0.1
uc_tolerance = 5
}

indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
d_min = 3.0
d_max = 10.0
sigma_min = 1.5
n_sample_frames = 1000
n_selected_frames = 100
}
n_bins = 10

Copy and paste this set of parameter in an editor then save the file as "prime.phil".

== Running the Program ==
You can run the program by giving it an input file:

prime.run prime.phil

For this tutorial, PRIME will score the randomly selected 1,000 images then select the best 100 for running Brehm & Diederichs algorithm in Bootstrap mode. If you run the program with flag_plot=True, you'll see a plot showing two separated clusters, each representing images with matching assigned basis.

PRIME will select on of these two clusters and merge it to get a reference set for the Bootstrap step. Here, the remaining images will get assigned with a basis that makes it correlate best with the reference set.

Once all images are assigned with appropriate basis, PRIME will proceed to scaling and post-refinement steps. After three post-refinement cycles (default value), the process is done and here is the output of the program.

Isotropic B-factor: 5.30
No. of reflections
all: 7786
outside resolution: 51
outliers: 0
total left: 7735
Summary for Prime_Run_1/postref_cycle_3_merge.mtz
Bin Resolution Range Completeness N_obs |Rmerge Rsplit CC1/2 N_ind |CCiso N_ind|CCanoma N_ind| I/sigI I sigI I**2
--------------------------------------------------------------------------------------------------------------------------------------------------
01 19.88 - 5.35 100.0 807 / 807 189.78 85.75 8.87 98.43 807 0.00 0 0.00 0 4.46 684.2 136.2 3.57
02 5.35 - 4.26 100.0 782 / 782 140.90 73.61 8.89 97.67 782 0.00 0 0.00 0 5.23 794.3 140.3 2.05
03 4.26 - 3.73 100.0 788 / 788 129.32 69.32 8.70 98.11 788 0.00 0 0.00 0 5.41 878.5 150.3 1.95
04 3.73 - 3.39 100.0 765 / 765 117.75 70.72 9.67 97.55 765 0.00 0 0.00 0 4.12 712.3 162.4 1.88
05 3.39 - 3.15 100.0 770 / 770 113.22 71.61 11.22 88.54 770 0.00 0 0.00 0 2.73 500.1 173.7 2.19
06 3.15 - 2.96 100.0 767 / 767 106.12 73.19 11.07 97.21 767 0.00 0 0.00 0 2.09 404.6 183.8 2.02
07 2.96 - 2.81 100.0 766 / 766 103.73 75.79 12.53 96.62 766 0.00 0 0.00 0 1.72 345.3 193.9 1.89
08 2.81 - 2.69 100.0 745 / 745 101.51 76.11 12.84 96.21 745 0.00 0 0.00 0 1.49 317.3 204.1 1.98
09 2.69 - 2.59 100.0 786 / 786 97.88 77.59 14.19 95.40 786 0.00 0 0.00 0 1.38 299.0 209.3 1.86
10 2.59 - 2.50 100.0 759 / 759 92.53 78.61 14.72 96.51 759 0.00 0 0.00 0 1.36 312.4 218.8 2.07
--------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL 100.0 7735 / 7735 119.73 75.51 10.45 97.21 7735 0.00 0 0.00 0 3.02 527.3 176.9 2.56
--------------------------------------------------------------------------------------------------------------------------------------------------

Summary of CC1/2 on three crystal axes
Bin Resolution Range CC1/2 I N_refl
a* b* c* | a* b* c* | a* b* c*
---------------------------------------------------------------------------------------------------------
01 19.88 - 5.35 97.01 98.64 98.23 528.6 559.6 1216.7 42 51 47
02 5.35 - 4.26 97.64 98.43 99.08 817.6 527.5 964.9 43 44 40
03 4.26 - 3.73 96.31 98.02 97.68 605.7 682.9 856.0 39 39 41
04 3.73 - 3.39 98.49 98.55 97.73 961.9 532.6 729.1 42 37 45
05 3.39 - 3.15 96.88 98.38 92.69 449.5 492.1 721.6 39 39 40
06 3.15 - 2.96 98.48 93.58 98.61 389.9 303.4 391.7 39 37 39
07 2.96 - 2.81 96.98 98.02 95.35 361.3 331.7 383.2 42 37 43
08 2.81 - 2.69 95.29 94.02 94.69 290.8 194.7 292.7 41 35 36
09 2.69 - 2.59 96.55 91.88 98.57 265.7 341.4 290.2 41 35 44
10 2.59 - 2.50 94.44 97.81 96.67 249.0 400.1 236.2 42 36 40
----------------------------------------------------------------------------------------------------------
total 97.57 97.74 94.58 494.1 446.0 619.4 410 390 415
----------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 4733
No. bad cc frames: 113
No. bad G frames) : 109
No. bad unit cell frames: 20
No. bad gamma_e frames: 22
No. bad SE: 2
No. observations: 935265
Mean target value (BEFORE: Mean Median (Std.))
post-refinement: 301.22 259.10 ( 171.56)
(x,y) restraints: 1679.63 1573.15 ( 657.49)
Mean target value (AFTER: Mean Median (Std.))
post-refinement: 300.02 257.53 ( 170.98)
(x,y) restraints: 1679.90 1572.77 ( 660.19)
SE: 1915.60 776.84 ( 33765.97)
G: 1.000e+00 8.971e-01 ( 8.15e-01)
B: 11.83 14.45 ( 11.95)
Rot.x: -0.08 0.00 ( 12.10)
Rot.y: 0.14 0.00 ( 9.62)
gamma_y: 0.00000 0.00000 ( 0.00000)
gamma_z: 0.00000 0.00000 ( 0.00000)
gamma_0: 0.03793 0.00019 ( 0.60820)
gamma_e: -0.12824 0.00145 ( 0.60227)
voigt_nu: 0.50000 0.50000 ( 0.00000)
unit cell
a: 91.45 91.45 ( 0.11)
b: 91.45 91.45 ( 0.11)
c: 45.96 45.96 ( 0.12)
alpha: 90.00 90.00 ( 0.00)
beta: 90.00 90.00 ( 0.00)
gamma: 120.00 120.00 ( 0.00)
Parmeters from integration (not-refined)
Wavelength: 0.96861 0.96861 ( 0.00000)
Detector distance: 303.81868 303.81868 ( 0.00000)
* (standard deviation)

Total calculation time: 542.00 seconds
Finished: Tuesday 14. February 2017 10:53:18

2017 prime tutorial

2017-02-14T19:10:10Z

Mona:

Post-refine and Merge Sample Data Set with PRIME (2017 Tutorial)

In this tutorial, we will work on the integration results from the first of Tutorial 2 (Myoglobin Data). Before proceeding to running the program, we'll consider making the input file for PRIME based on the situation of this data set.

== Generating Input File ==

PRIME input files contain information necessary for successful post-refinement and merging steps. You can access and review the list of input parameters by running prime.run or prime.run -h to view the description of these parameters. For this tutorial we'll start building it from scratch.

* Location of integration results
In this case, we know the location where the integration results (pickle files) are. We can then set,

data = /net/viper/raid1/mu238/XfelProject/dials17/extracted

Note that you supply data as a multiple arguments. The value of the parameter can be a file containing list of integration results, a folder, or a wildcard argument.

* Unit cell information
You can obtain the mean (or median) unit-cell dimensions from either IOTA or DIALS. In case of IOTA, prime .phil file is auto generated and this information is readily available in there. For n_residues, enter number of residues in asymmetric unit of your molecule.

target_unit_cell = 91.7 91.7 46 90 90 120
target_space_group = P6
n_residues = 128

* Detector information
pixel_size_mm = 0.172

* Post-refinement and Scaling information
This is where you specify the optimal resolution cutoffs for post-refinement and merging. Note that when running for the first time on you newly collected data, you can choose the "expected" values (resolution which you see the spots at the corner or on the edge). You can then adjust these parameters when analyzing merging statistics based on the I/sigI values in the high resolution shells and rerun the program again. Note that sigma cutoffs are set to 1.5 in scaling and post-refinement steps while it's set to -3.0 so we can include negative values in the merged reflection set.

scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
}
postref {
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
allparams {
flag_on = True
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
merge {
d_min = 2.5
d_max = 20
sigma_min = -3.0
partiality_min = 0.1
uc_tolerance = 5
}

* Indexing ambiguity
For other sets that are not in polar space or have indexing ambiguity (when one or more of the unit-cell dimensions are very similar but not the same!), you can very well use the .phil file parameters thus far to proceed and run post-refinement. However, this data set is in P6 (polar space group) and therefore, the indexing ambiguity needs to be resolved prior to other refinement and merging steps.

Other point worth noting is for any polar space groups, PRIME will automatically solve the ambiguity based on the default parameters. However, this data set has about 5,000 integration results so we want to make sure that we modify the number of images used for random and best selections.

indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
d_min = 3.0
d_max = 10.0
sigma_min = 1.5
n_sample_frames = 1000
n_selected_frames = 100
}

We left other parameters to their default value and modified n_sample_frames to 1000 and n_selected_frames to 100.

* No. of Bin
n_bins = 10

Now we have a complete .phil file ready to run.

data = /net/viper/raid1/mu238/XfelProject/dials17/extracted
target_unit_cell = 91.7 91.7 46 90 90 120
target_space_group = P6
n_residues = 128
pixel_size_mm = 0.172
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
}
postref {
scale {
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
}
allparams {
flag_on = True
d_min = 2.5
d_max = 20
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
merge {
d_min = 2.5
d_max = 20
sigma_min = -3.0
partiality_min = 0.1
uc_tolerance = 5
}

indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
d_min = 3.0
d_max = 10.0
sigma_min = 1.5
n_sample_frames = 1000
n_selected_frames = 100
}
n_bins = 10

Copy and paste this set of parameter in an editor then save the file as "prime.phil".

== Running the Program ==
You can run the program by giving it an input file:

prime.run prime.phil

For this tutorial, PRIME will score the randomly selected 1,000 images then select the best 100 for running Brehm & Diederichs algorithm in Bootstrap mode. If you run the program with flag_plot=True, you'll see a plot showing two separated clusters, each representing images with matching assigned basis.

PRIME will select on of these two clusters and merge it to get a reference set for the Bootstrap step. Here, the remaining images will get assigned with a basis that makes it correlate best with the reference set.

Once all images are assigned with appropriate basis, PRIME will proceed to scaling and post-refinement steps. After three post-refinement cycles (default value), the process is done and here is the output of the program.

Isotropic B-factor: 5.30
No. of reflections
all: 7786
outside resolution: 51
outliers: 0
total left: 7735
Summary for Prime_Run_1/postref_cycle_3_merge.mtz
Bin Resolution Range Completeness <N_obs> |Rmerge Rsplit CC1/2 N_ind |CCiso N_ind|CCanoma N_ind| <I/sigI> <sigI> <I**2>
--------------------------------------------------------------------------------------------------------------------------------------------------
01 19.88 - 5.35 100.0 807 / 807 189.78 85.75 8.87 98.43 807 0.00 0 0.00 0 4.46 684.2 136.2 3.57
02 5.35 - 4.26 100.0 782 / 782 140.90 73.61 8.89 97.67 782 0.00 0 0.00 0 5.23 794.3 140.3 2.05
03 4.26 - 3.73 100.0 788 / 788 129.32 69.32 8.70 98.11 788 0.00 0 0.00 0 5.41 878.5 150.3 1.95
04 3.73 - 3.39 100.0 765 / 765 117.75 70.72 9.67 97.55 765 0.00 0 0.00 0 4.12 712.3 162.4 1.88
05 3.39 - 3.15 100.0 770 / 770 113.22 71.61 11.22 88.54 770 0.00 0 0.00 0 2.73 500.1 173.7 2.19
06 3.15 - 2.96 100.0 767 / 767 106.12 73.19 11.07 97.21 767 0.00 0 0.00 0 2.09 404.6 183.8 2.02
07 2.96 - 2.81 100.0 766 / 766 103.73 75.79 12.53 96.62 766 0.00 0 0.00 0 1.72 345.3 193.9 1.89
08 2.81 - 2.69 100.0 745 / 745 101.51 76.11 12.84 96.21 745 0.00 0 0.00 0 1.49 317.3 204.1 1.98
09 2.69 - 2.59 100.0 786 / 786 97.88 77.59 14.19 95.40 786 0.00 0 0.00 0 1.38 299.0 209.3 1.86
10 2.59 - 2.50 100.0 759 / 759 92.53 78.61 14.72 96.51 759 0.00 0 0.00 0 1.36 312.4 218.8 2.07
--------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL 100.0 7735 / 7735 119.73 75.51 10.45 97.21 7735 0.00 0 0.00 0 3.02 527.3 176.9 2.56
--------------------------------------------------------------------------------------------------------------------------------------------------

Summary of CC1/2 on three crystal axes
Bin Resolution Range CC1/2 N_refl
a* b* c* | a* b* c* | a* b* c*
---------------------------------------------------------------------------------------------------------
01 19.88 - 5.35 97.01 98.64 98.23 528.6 559.6 1216.7 42 51 47
02 5.35 - 4.26 97.64 98.43 99.08 817.6 527.5 964.9 43 44 40
03 4.26 - 3.73 96.31 98.02 97.68 605.7 682.9 856.0 39 39 41
04 3.73 - 3.39 98.49 98.55 97.73 961.9 532.6 729.1 42 37 45
05 3.39 - 3.15 96.88 98.38 92.69 449.5 492.1 721.6 39 39 40
06 3.15 - 2.96 98.48 93.58 98.61 389.9 303.4 391.7 39 37 39
07 2.96 - 2.81 96.98 98.02 95.35 361.3 331.7 383.2 42 37 43
08 2.81 - 2.69 95.29 94.02 94.69 290.8 194.7 292.7 41 35 36
09 2.69 - 2.59 96.55 91.88 98.57 265.7 341.4 290.2 41 35 44
10 2.59 - 2.50 94.44 97.81 96.67 249.0 400.1 236.2 42 36 40
----------------------------------------------------------------------------------------------------------
total 97.57 97.74 94.58 494.1 446.0 619.4 410 390 415
----------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 4733
No. bad cc frames: 113
No. bad G frames) : 109
No. bad unit cell frames: 20
No. bad gamma_e frames: 22
No. bad SE: 2
No. observations: 935265
Mean target value (BEFORE: Mean Median (Std.))
post-refinement: 301.22 259.10 ( 171.56)
(x,y) restraints: 1679.63 1573.15 ( 657.49)
Mean target value (AFTER: Mean Median (Std.))
post-refinement: 300.02 257.53 ( 170.98)
(x,y) restraints: 1679.90 1572.77 ( 660.19)
SE: 1915.60 776.84 ( 33765.97)
G: 1.000e+00 8.971e-01 ( 8.15e-01)
B: 11.83 14.45 ( 11.95)
Rot.x: -0.08 0.00 ( 12.10)
Rot.y: 0.14 0.00 ( 9.62)
gamma_y: 0.00000 0.00000 ( 0.00000)
gamma_z: 0.00000 0.00000 ( 0.00000)
gamma_0: 0.03793 0.00019 ( 0.60820)
gamma_e: -0.12824 0.00145 ( 0.60227)
voigt_nu: 0.50000 0.50000 ( 0.00000)
unit cell
a: 91.45 91.45 ( 0.11)
b: 91.45 91.45 ( 0.11)
c: 45.96 45.96 ( 0.12)
alpha: 90.00 90.00 ( 0.00)
beta: 90.00 90.00 ( 0.00)
gamma: 120.00 120.00 ( 0.00)
Parmeters from integration (not-refined)
Wavelength: 0.96861 0.96861 ( 0.00000)
Detector distance: 303.81868 303.81868 ( 0.00000)
* (standard deviation)

Total calculation time: 542.00 seconds
Finished: Tuesday 14. February 2017 10:53:18

2017 prime tutorial

2017-02-14T18:41:12Z

Mona: /* Generating input file */

2017 prime tutorial

2017-02-14T18:38:56Z

Mona: /* Generating input file */

2017 prime tutorial

2017-02-14T18:19:30Z

Mona: /* Generating input file */

2017 prime tutorial

2017-02-14T18:18:20Z

Mona: /* Generating input file */

2017 prime tutorial

2017-02-14T18:17:35Z

Mona: /* Generating input file */

2017 prime tutorial

2017-02-14T18:15:35Z

Mona: /* Generating input file */

2017 prime tutorial

2017-02-14T18:13:59Z

Mona:

2017 prime tutorial

2017-02-14T18:12:30Z

Mona:

2017 prime tutorial

2017-02-14T18:12:08Z

Mona:

2017 prime tutorial

2017-02-14T18:01:07Z

Mona:

2017 prime tutorial

2017-02-14T18:00:03Z

Mona:

2017 prime tutorial

2017-02-14T17:53:34Z

Mona:

2017 prime tutorial

2017-02-14T17:43:55Z

Mona: Created page with "Post-refine and Merge Sample Data Set with PRIME (2017 Tutorial)"

Post-refine and Merge Sample Data Set with PRIME (2017 Tutorial)

2017 Tutorials

2017-02-14T17:39:29Z

Mona: /* 10:15 am: Tutorials 2 */

= Feb 16th =

9:00am: Session 1
Nick Sauter: Welcome and "Trials and tribulations merging still image data"
Aaron Brewster: "Metrology and non-isomorphism: hidden challenges in still image data reduction"
Axel Brunger: "Data processing of XFEL data from a limited number of crystals"
10:20am: Break
10:40am: Session 2
Art Lyubimov: "IOTA: Integration optimization, triage and analysis tool for XFEL data processing"
Monarin Uervirojnangkoorn: "Up and Running with Prime"
Jacques-Philippe Colletier: "Mosquito larvicide BinAB revealed by de novo phasing with an X-ray laser"
James Holton: "What if? Using at-scale image simulations to optimize data processing algorithms"
Noon: Working Lunch - Roundtable discussion of data processing challenges
1:00pm: Session 3
Graeme Winter and Richard Gildea: "DIALS - new methods for processing X-ray diffraction data"
James Parkhurst: "Robust background modelling in the presence of outliers in DIALS"
Jan Kern: TBD
Franklin Fuller: "Drop-by-Drop Transient Serial Crystallography of Metalloenzymes at an X-ray free electron laser"
Iris Young: "Room temperature studies of the oxygen-evolving complex of photosystem II using an X-ray free electron laser (XFEL)"
2:40pm: Break
3:00pm: Session 4
Aina Cohen: TBD
Rahel Woldeyes: "Using X-ray Free Electron lasers to visualize solvent in the M2 proton channel"
Danny Axford: "Highly efficient serial data collection from high-density fixed targets"
Christoph Mueller-Dieckmann: "Serial Synchrotron Crystallography at the ESRF using a high viscosity extruder"
Allen Orville: TBD
= Feb 17th =

== 9am: Tutorials 1 ==
Aaron Brewster and Iris Young: cctbx.xfel
Break: 10:00 am
== 10:15 am: Tutorials 2 ==
* Aaron Brewster and James Parkhurst: dials.stills_process
* Art Lyubimov: IOTA
* Monarin Uervirojnangkoorn: [[2017_prime_tutorial | PRIME]]
* Nick Sauter: [[2017_cxi_merge_tutorial | cxi.merge]]

== 12:15pm-2:30pm: Round table discussion. ==
Topics will include:
Is the current software meeting needs?
What are essential/timely avenues most useful for near-term development (first half of 2017)?
Where should we be going next (longer term future)?
Time resolved experiments?
Synchrotron serial crystallography?
== 2:30-4pm: Breakout sessions: ==
users work with developers and instructors on their own data. Hands-on walkthroughs and data analysis.

Cctbx.prime

2016-08-31T22:28:09Z

Mona: /* Solving indexing ambiguity (New) */

== Prime: '''p'''ost-'''r'''ef'''i'''nement and '''me'''rging ==
With the latest update, prime can be used to process data on multiple nodes (on a queuing system). At the moment, only LSF (bsub) is supported. See documentation below for more information on how to use the queuing system.

This major update replaces prime.postrefine with

<pre>
prime.run
</pre>

For auto mode, you can still use prime.run with your parameter phil file like before. For manual mode, the available sub commands in prime are:

<pre>
prime.genref #generates a reference set from given integration results
prime.postrefine #refines all images
prime.merge #merges all refined results for an mtz file
</pre>

You can choose to run these commands independently (ideally in the above order) using the same phil file. See [https://commons.wikimedia.org/wiki/File:Prime_flowcharts.tif "PRIME flowchart"]. This will give you the freedom to change something (e.g. set of parameters to refine, resolution cut-off, etc.) at different stages of the post-refinement and merging. See running prime in manual mode for more detail.

Step-by-step guidelines to post-refine and merge XFEL diffraction images. For more detail and citation, see
"Enabling X-ray Free Electron Laser Crystallography for Challenging Biological Systems from a Limited Number of Crystals"
[http://elifesciences.org/content/4/e05421 "DOI: http://dx.doi.org/10.7554/eLife.05421"]

== Prime is gui-ed ==
Thanks to Dr. Lyubimov, PRIME is also available as a Graphic User Interface program. Try it by running

<pre>
prime
</pre>

Click to see [https://commons.wikimedia.org/wiki/File:PRIME_main.png "PRIME main gui"] and [https://commons.wikimedia.org/wiki/File:PRIME_advanced_options.png "Advanced options"]

== Getting started ==
'''Generating input phil file''' 
Like most programs developed under ''cctbx'' framework, ''prime'' reads in input .phil file, which stores all the parameters needed to run post-refinement and merging steps. To generate the template .phil file, do the dry run by calling

<pre>
$ prime.run
</pre>

An example of the template .phil file:

<pre>
data = None
run_no = None
title = None
scale {
d_min = 0.1
d_max = 99
sigma_min = 1.5
}
...
</pre>

You can save the content of the output to any file name - in this tutorial, let's save it to thermolysin.phil.

'''First look at your phil file''' 
To run prime, set the required parameters to match with your experiments (you can leave other parameters with their default values - or just delete them from you .phil file). The most interesting parameters are shown below:

<pre>
data = /path/to/your/integarion/result/pickle_files
run_no = 001
title = First trial for thermolysin
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
}
postref {
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
allparams {
flag_on = False
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
}
merge {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
target_unit_cell = 93.99,93.99,130.87,90,90,120
target_space_group = P 61 2 2
pixel_size_mm = 0.102
</pre>

You should pay attention to d_min and d_max for the refinement and merging parameters. If you use IOTA to integrate the images, IOTA will output .phil file for prime that has the optimal resolution range. If not, a few trial-and-error runs may be required to get the best resolution range for your dataset. Use merging statistics output by prime and check the values of CC1/2 and I/sigI to find out your optimal resolution range.

Cell parameters (target_unit_cell and target_space_group) are required to run prime. Target cell parameter is used to remove some outlier images by controlling uc_tolerance parameter (the default value of tolerate range is 3% different). Space group parameter is used in removing outliers and merging with the given symmetry.

Don't forget also to change your pixel size in millimeters. Check what your detector is and note down its pixel size.

'''Running post-refinement in automatic mode''' 
Once you have the input .phil file, you can run ''prime'' by calling
<pre>
prime.run thermolysin.phil
</pre>
''Prime'' will post-refine and merge for reflection sets using three (default value) macrocycles. At the end of the run, you can obtain merging statistics in the last cycle - all other cycle statistics are also available in log.txt.

An example of merging statistics:
<pre>
Summary for 001/postref_cycle_1_merge.mtz
Bin Resolution Range Completeness <N_obs> |Rsplit CC1/2 N_ind |CCanom N_ind| <I/sigI> 
-------------------------------------------------------------------------------------------------------------
02 5.70 - 4.52 100.0 1055 / 1055 65.89 16.02 89.15 1055 0.00 0 20.17 2101.97
03 4.52 - 3.95 100.0 1032 / 1032 61.53 14.48 92.03 1032 0.00 0 20.39 2529.90
04 3.95 - 3.59 100.0 1016 / 1016 54.15 15.61 90.13 1016 0.00 0 16.69 1971.43
05 3.59 - 3.33 100.0 1004 / 1004 42.67 17.66 89.23 1004 0.00 0 14.21 1502.14
06 3.33 - 3.14 100.0 1013 / 1013 32.77 20.40 84.26 1013 0.00 0 11.76 1077.60
07 3.14 - 2.98 100.0 995 / 995 27.36 23.00 78.72 995 0.00 0 11.58 935.37
08 2.98 - 2.85 100.0 1006 / 1006 23.57 22.63 82.26 1006 0.00 0 10.56 722.62
09 2.85 - 2.74 100.0 986 / 986 16.64 28.51 72.90 985 0.00 0 10.01 591.56
10 2.74 - 2.65 99.9 989 / 990 12.41 31.35 72.95 987 0.00 0 9.91 515.07
11 2.65 - 2.56 99.7 979 / 982 9.35 37.14 65.31 970 0.00 0 9.31 438.96
12 2.56 - 2.49 98.0 979 / 999 6.06 45.98 45.37 930 0.00 0 9.45 390.05
13 2.49 - 2.42 95.1 931 / 979 4.46 50.68 34.20 834 0.00 0 8.93 334.80
14 2.42 - 2.37 91.7 896 / 977 3.35 55.66 37.15 729 0.00 0 9.27 320.17
15 2.37 - 2.31 83.9 829 / 988 2.61 56.92 43.21 600 0.00 0 9.60 296.67
16 2.31 - 2.26 72.4 702 / 969 1.97 65.81 26.89 386 0.00 0 10.29 284.39
17 2.26 - 2.22 59.1 582 / 985 1.75 64.72 31.28 275 0.00 0 9.87 284.06
18 2.22 - 2.18 52.9 513 / 970 1.51 71.27 16.86 188 0.00 0 8.93 215.31
19 2.18 - 2.14 35.7 349 / 978 1.32 62.26 68.25 90 0.00 0 8.22 199.09
20 2.14 - 2.10 23.1 227 / 981 1.20 92.14 -9.20 42 0.00 0 8.59 224.44
-------------------------------------------------------------------------------------------------------------
TOTAL 85.9 17224 / 20046 27.11 21.11 92.07 15305 0.00 0 12.87 999.53
-------------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 1809
No. bad cc frames: 153
No. bad G frames) : 0
No. bad unit cell frames: 5
No. bad gamma_e frames: 0
No. bad SE: 0
No. observations: 466997

</pre>

== Solving indexing ambiguity (New) ==
* SOFTWARE UPDATE REQUIRED *
With the latest version (Aug 31, 2016), you can solve the indexing ambiguity problem directly in prime. The Brehm & Diederichs algorithms ([http://dx.doi.org/10.1107/S1399004713025431 "doi:10.1107/S1399004713025431"]) have been implemented with bootstrap capability to handle large dataset.
 
'''Merohedral Twinning''' 
For merohedral twinning (27 space groups e.g. P6), the indexing choices will be determined automatically in prime. Use this default setting in your .phil file,

<pre>
indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
n_sample_frames = 300
n_selected_frames = 100
}
</pre>

The n_sample_frames parameter indicates no. of images that will be used for the calculation of the scoring function. After that, only n_selected_images will be used in the B&D algorithms. This saves a lot of computing time since only the selected images will be used for the determination of the ambiguity. You can change these two parameters to fit with your experiments. The default values are 300 and 100 (give 300 - use 100).

'''Pseudo-Merohedral Twinning''' 
For pseudo-merohedral twinning, due to different possibilities for the indexing choice, prime doesn't determine these choices automatically. If you suspect that you may have pseudo twinning (b and c are similar, beta angle is almost 90 degree but not quite), you have an option to force prime to determine the ambiguity according to your choices.
<pre>
indexing_ambiguity {
mode = Forced
index_basis_in = None
assigned_basis = -h,l,k
assigned_basis = -k, l, h
n_sample_frames = 300
n_selected_frames = 100
}
</pre>

When you set indexing_ambiguity.mode to Forced, you can assign indexing choices according to your problem. In this example, two more choices (-h, l, k and -k, l, h) were assigned as the indexing choice.

'''Reusing the solution''' 
At the end of the run, your solution pickle is saved to your_run_no/index_ambiguity/solution_pickle.pickle. If you don't want to spend time solving the ambiguity again in the next run, you can reuse this solution pickle by setting these parameters:
<pre>
indexing_ambiguity {
mode = Auto
index_basis_in = your_run_no/index_ambiguity/solution_pickle.pickle
}
</pre>

This will bypass the indexing ambiguity module. Prime will use the solution file to perform normal post-refinement and merging.

'''Using an external reference set''' 
To use another isomorphous dataset (e.g. from a synchrotron experiment) as a reference set to solve the ambiguity, you can specify an mtz file as part of these parameters:
<pre>
indexing_ambiguity {
mode = Auto
index_basis_in = path/to/your/mtz/file.mtz
}
</pre>
Again, you can choose to do Auto or Forced (with a list of assigned_basis parameters) depending on your problem.

== More detail with input parameters ==
Now that you have your first trial merged data set, you can explore different parameter settings to merge or to obtain the Bijvoet pairs (I+/I-) for your anomalous data set.

'''Anomalous data:'''
<pre>
target_anomalous_flag = True
</pre>
In the last cycle, prime will output a reflection set with I+ and I-.

'''Number of micro- and macrocycles'''
<pre>
n_postref_cycle = 3
n_postref_sub_cycle = 1
</pre>

'''Number of bins for merging statistics'''
<pre>
n_bins = 20
</pre>

== Help with input parameters ==

Most input parameters are self-explained. However, you can run -h switch to view help information for each parameter.
<pre>
prime.run -h
</pre>

== Running in manual mode ==
With the same phil file, you can run prime manually. This gives you more freedom in terms of parameter settings at different stages (generating reference set, post-refining images, and merging) or at different cycle of post-refinement.

'''Example A''': I want to generate a reference set then post-refine all the images on the '''scale factors only''' for '''three cycles''' then refine '''all parameters''' in the '''4th cycle'''. To do this, you can follow these steps:

To generate a reference set,
<pre>
prime.genref prime.phil
</pre>

To post-refine on scale factors only, modify your .phil file so that all parameters are turned ''off''.
<pre>
...
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
}
postref {
residual_threshold = 5
residual_threshold_xy = 5
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
allparams {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 3
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To refine all parameters one more cycle, update your .phil file again (flag_on = True)
<pre>
...
allparams {
flag_on = True
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 1
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To obtain the final merged mtz, run
<pre>
prime.merge prime.phil
</pre>

== Running on multiple nodes ==
For LCLS users (or other users with LSF bsub), you can use psana (or your) queuing system to parallelize the entire process. For example, if you want to run your job on 100 nodes using psanq, you can specify:
<pre>
queue {
mode = bsub
qname = psanaq
n_nodes = 100
}
timeout_seconds = 300
</pre>
Prime will divide all the images into 100 batches and submit them to different nodes. It will wait until all images in every batches are done before returning to the merging step (or the exit step in the manual mode). You can control timeout_seconds parameter to tell prime how long it should wait for all the image batches to finish. Usually, this timeout parameter is not used (all images should return before 300 seconds) but in case, you need to wait longer or shorter, you can modify this parameter.

Cctbx.prime

2016-08-31T22:27:44Z

Mona: /* Solving indexing ambiguity (New) */

== Prime: '''p'''ost-'''r'''ef'''i'''nement and '''me'''rging ==
With the latest update, prime can be used to process data on multiple nodes (on a queuing system). At the moment, only LSF (bsub) is supported. See documentation below for more information on how to use the queuing system.

This major update replaces prime.postrefine with

<pre>
prime.run
</pre>

For auto mode, you can still use prime.run with your parameter phil file like before. For manual mode, the available sub commands in prime are:

<pre>
prime.genref #generates a reference set from given integration results
prime.postrefine #refines all images
prime.merge #merges all refined results for an mtz file
</pre>

You can choose to run these commands independently (ideally in the above order) using the same phil file. See [https://commons.wikimedia.org/wiki/File:Prime_flowcharts.tif "PRIME flowchart"]. This will give you the freedom to change something (e.g. set of parameters to refine, resolution cut-off, etc.) at different stages of the post-refinement and merging. See running prime in manual mode for more detail.

Step-by-step guidelines to post-refine and merge XFEL diffraction images. For more detail and citation, see
"Enabling X-ray Free Electron Laser Crystallography for Challenging Biological Systems from a Limited Number of Crystals"
[http://elifesciences.org/content/4/e05421 "DOI: http://dx.doi.org/10.7554/eLife.05421"]

== Prime is gui-ed ==
Thanks to Dr. Lyubimov, PRIME is also available as a Graphic User Interface program. Try it by running

<pre>
prime
</pre>

Click to see [https://commons.wikimedia.org/wiki/File:PRIME_main.png "PRIME main gui"] and [https://commons.wikimedia.org/wiki/File:PRIME_advanced_options.png "Advanced options"]

== Getting started ==
'''Generating input phil file''' 
Like most programs developed under ''cctbx'' framework, ''prime'' reads in input .phil file, which stores all the parameters needed to run post-refinement and merging steps. To generate the template .phil file, do the dry run by calling

<pre>
$ prime.run
</pre>

An example of the template .phil file:

<pre>
data = None
run_no = None
title = None
scale {
d_min = 0.1
d_max = 99
sigma_min = 1.5
}
...
</pre>

You can save the content of the output to any file name - in this tutorial, let's save it to thermolysin.phil.

'''First look at your phil file''' 
To run prime, set the required parameters to match with your experiments (you can leave other parameters with their default values - or just delete them from you .phil file). The most interesting parameters are shown below:

<pre>
data = /path/to/your/integarion/result/pickle_files
run_no = 001
title = First trial for thermolysin
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
}
postref {
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
allparams {
flag_on = False
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
}
merge {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
target_unit_cell = 93.99,93.99,130.87,90,90,120
target_space_group = P 61 2 2
pixel_size_mm = 0.102
</pre>

You should pay attention to d_min and d_max for the refinement and merging parameters. If you use IOTA to integrate the images, IOTA will output .phil file for prime that has the optimal resolution range. If not, a few trial-and-error runs may be required to get the best resolution range for your dataset. Use merging statistics output by prime and check the values of CC1/2 and I/sigI to find out your optimal resolution range.

Cell parameters (target_unit_cell and target_space_group) are required to run prime. Target cell parameter is used to remove some outlier images by controlling uc_tolerance parameter (the default value of tolerate range is 3% different). Space group parameter is used in removing outliers and merging with the given symmetry.

Don't forget also to change your pixel size in millimeters. Check what your detector is and note down its pixel size.

'''Running post-refinement in automatic mode''' 
Once you have the input .phil file, you can run ''prime'' by calling
<pre>
prime.run thermolysin.phil
</pre>
''Prime'' will post-refine and merge for reflection sets using three (default value) macrocycles. At the end of the run, you can obtain merging statistics in the last cycle - all other cycle statistics are also available in log.txt.

An example of merging statistics:
<pre>
Summary for 001/postref_cycle_1_merge.mtz
Bin Resolution Range Completeness <N_obs> |Rsplit CC1/2 N_ind |CCanom N_ind| <I/sigI> 
-------------------------------------------------------------------------------------------------------------
02 5.70 - 4.52 100.0 1055 / 1055 65.89 16.02 89.15 1055 0.00 0 20.17 2101.97
03 4.52 - 3.95 100.0 1032 / 1032 61.53 14.48 92.03 1032 0.00 0 20.39 2529.90
04 3.95 - 3.59 100.0 1016 / 1016 54.15 15.61 90.13 1016 0.00 0 16.69 1971.43
05 3.59 - 3.33 100.0 1004 / 1004 42.67 17.66 89.23 1004 0.00 0 14.21 1502.14
06 3.33 - 3.14 100.0 1013 / 1013 32.77 20.40 84.26 1013 0.00 0 11.76 1077.60
07 3.14 - 2.98 100.0 995 / 995 27.36 23.00 78.72 995 0.00 0 11.58 935.37
08 2.98 - 2.85 100.0 1006 / 1006 23.57 22.63 82.26 1006 0.00 0 10.56 722.62
09 2.85 - 2.74 100.0 986 / 986 16.64 28.51 72.90 985 0.00 0 10.01 591.56
10 2.74 - 2.65 99.9 989 / 990 12.41 31.35 72.95 987 0.00 0 9.91 515.07
11 2.65 - 2.56 99.7 979 / 982 9.35 37.14 65.31 970 0.00 0 9.31 438.96
12 2.56 - 2.49 98.0 979 / 999 6.06 45.98 45.37 930 0.00 0 9.45 390.05
13 2.49 - 2.42 95.1 931 / 979 4.46 50.68 34.20 834 0.00 0 8.93 334.80
14 2.42 - 2.37 91.7 896 / 977 3.35 55.66 37.15 729 0.00 0 9.27 320.17
15 2.37 - 2.31 83.9 829 / 988 2.61 56.92 43.21 600 0.00 0 9.60 296.67
16 2.31 - 2.26 72.4 702 / 969 1.97 65.81 26.89 386 0.00 0 10.29 284.39
17 2.26 - 2.22 59.1 582 / 985 1.75 64.72 31.28 275 0.00 0 9.87 284.06
18 2.22 - 2.18 52.9 513 / 970 1.51 71.27 16.86 188 0.00 0 8.93 215.31
19 2.18 - 2.14 35.7 349 / 978 1.32 62.26 68.25 90 0.00 0 8.22 199.09
20 2.14 - 2.10 23.1 227 / 981 1.20 92.14 -9.20 42 0.00 0 8.59 224.44
-------------------------------------------------------------------------------------------------------------
TOTAL 85.9 17224 / 20046 27.11 21.11 92.07 15305 0.00 0 12.87 999.53
-------------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 1809
No. bad cc frames: 153
No. bad G frames) : 0
No. bad unit cell frames: 5
No. bad gamma_e frames: 0
No. bad SE: 0
No. observations: 466997

</pre>

== Solving indexing ambiguity (New) ==
* SOFTWARE UPDATE REQUIRED *
With the latest version (Aug 31, 2016), you can solve the indexing ambiguity problem directly in prime. The Brehm & Diederichs algorithms ([http://dx.doi.org/10.1107/S1399004713025431, "doi:10.1107/S1399004713025431"]) have been implemented with bootstrap capability to handle large dataset.
 
'''Merohedral Twinning''' 
For merohedral twinning (27 space groups e.g. P6), the indexing choices will be determined automatically in prime. Use this default setting in your .phil file,

<pre>
indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
n_sample_frames = 300
n_selected_frames = 100
}
</pre>

The n_sample_frames parameter indicates no. of images that will be used for the calculation of the scoring function. After that, only n_selected_images will be used in the B&D algorithms. This saves a lot of computing time since only the selected images will be used for the determination of the ambiguity. You can change these two parameters to fit with your experiments. The default values are 300 and 100 (give 300 - use 100).

'''Pseudo-Merohedral Twinning''' 
For pseudo-merohedral twinning, due to different possibilities for the indexing choice, prime doesn't determine these choices automatically. If you suspect that you may have pseudo twinning (b and c are similar, beta angle is almost 90 degree but not quite), you have an option to force prime to determine the ambiguity according to your choices.
<pre>
indexing_ambiguity {
mode = Forced
index_basis_in = None
assigned_basis = -h,l,k
assigned_basis = -k, l, h
n_sample_frames = 300
n_selected_frames = 100
}
</pre>

When you set indexing_ambiguity.mode to Forced, you can assign indexing choices according to your problem. In this example, two more choices (-h, l, k and -k, l, h) were assigned as the indexing choice.

'''Reusing the solution''' 
At the end of the run, your solution pickle is saved to your_run_no/index_ambiguity/solution_pickle.pickle. If you don't want to spend time solving the ambiguity again in the next run, you can reuse this solution pickle by setting these parameters:
<pre>
indexing_ambiguity {
mode = Auto
index_basis_in = your_run_no/index_ambiguity/solution_pickle.pickle
}
</pre>

This will bypass the indexing ambiguity module. Prime will use the solution file to perform normal post-refinement and merging.

'''Using an external reference set''' 
To use another isomorphous dataset (e.g. from a synchrotron experiment) as a reference set to solve the ambiguity, you can specify an mtz file as part of these parameters:
<pre>
indexing_ambiguity {
mode = Auto
index_basis_in = path/to/your/mtz/file.mtz
}
</pre>
Again, you can choose to do Auto or Forced (with a list of assigned_basis parameters) depending on your problem.

== More detail with input parameters ==
Now that you have your first trial merged data set, you can explore different parameter settings to merge or to obtain the Bijvoet pairs (I+/I-) for your anomalous data set.

'''Anomalous data:'''
<pre>
target_anomalous_flag = True
</pre>
In the last cycle, prime will output a reflection set with I+ and I-.

'''Number of micro- and macrocycles'''
<pre>
n_postref_cycle = 3
n_postref_sub_cycle = 1
</pre>

'''Number of bins for merging statistics'''
<pre>
n_bins = 20
</pre>

== Help with input parameters ==

Most input parameters are self-explained. However, you can run -h switch to view help information for each parameter.
<pre>
prime.run -h
</pre>

== Running in manual mode ==
With the same phil file, you can run prime manually. This gives you more freedom in terms of parameter settings at different stages (generating reference set, post-refining images, and merging) or at different cycle of post-refinement.

'''Example A''': I want to generate a reference set then post-refine all the images on the '''scale factors only''' for '''three cycles''' then refine '''all parameters''' in the '''4th cycle'''. To do this, you can follow these steps:

To generate a reference set,
<pre>
prime.genref prime.phil
</pre>

To post-refine on scale factors only, modify your .phil file so that all parameters are turned ''off''.
<pre>
...
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
}
postref {
residual_threshold = 5
residual_threshold_xy = 5
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
allparams {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 3
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To refine all parameters one more cycle, update your .phil file again (flag_on = True)
<pre>
...
allparams {
flag_on = True
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 1
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To obtain the final merged mtz, run
<pre>
prime.merge prime.phil
</pre>

== Running on multiple nodes ==
For LCLS users (or other users with LSF bsub), you can use psana (or your) queuing system to parallelize the entire process. For example, if you want to run your job on 100 nodes using psanq, you can specify:
<pre>
queue {
mode = bsub
qname = psanaq
n_nodes = 100
}
timeout_seconds = 300
</pre>
Prime will divide all the images into 100 batches and submit them to different nodes. It will wait until all images in every batches are done before returning to the merging step (or the exit step in the manual mode). You can control timeout_seconds parameter to tell prime how long it should wait for all the image batches to finish. Usually, this timeout parameter is not used (all images should return before 300 seconds) but in case, you need to wait longer or shorter, you can modify this parameter.

Cctbx.prime

2016-08-31T22:25:12Z

Mona: /* Solving indexing ambiguity (New) */

== Prime: '''p'''ost-'''r'''ef'''i'''nement and '''me'''rging ==
With the latest update, prime can be used to process data on multiple nodes (on a queuing system). At the moment, only LSF (bsub) is supported. See documentation below for more information on how to use the queuing system.

This major update replaces prime.postrefine with

<pre>
prime.run
</pre>

For auto mode, you can still use prime.run with your parameter phil file like before. For manual mode, the available sub commands in prime are:

<pre>
prime.genref #generates a reference set from given integration results
prime.postrefine #refines all images
prime.merge #merges all refined results for an mtz file
</pre>

You can choose to run these commands independently (ideally in the above order) using the same phil file. See [https://commons.wikimedia.org/wiki/File:Prime_flowcharts.tif "PRIME flowchart"]. This will give you the freedom to change something (e.g. set of parameters to refine, resolution cut-off, etc.) at different stages of the post-refinement and merging. See running prime in manual mode for more detail.

Step-by-step guidelines to post-refine and merge XFEL diffraction images. For more detail and citation, see
"Enabling X-ray Free Electron Laser Crystallography for Challenging Biological Systems from a Limited Number of Crystals"
[http://elifesciences.org/content/4/e05421 "DOI: http://dx.doi.org/10.7554/eLife.05421"]

== Prime is gui-ed ==
Thanks to Dr. Lyubimov, PRIME is also available as a Graphic User Interface program. Try it by running

<pre>
prime
</pre>

Click to see [https://commons.wikimedia.org/wiki/File:PRIME_main.png "PRIME main gui"] and [https://commons.wikimedia.org/wiki/File:PRIME_advanced_options.png "Advanced options"]

== Getting started ==
'''Generating input phil file''' 
Like most programs developed under ''cctbx'' framework, ''prime'' reads in input .phil file, which stores all the parameters needed to run post-refinement and merging steps. To generate the template .phil file, do the dry run by calling

<pre>
$ prime.run
</pre>

An example of the template .phil file:

<pre>
data = None
run_no = None
title = None
scale {
d_min = 0.1
d_max = 99
sigma_min = 1.5
}
...
</pre>

You can save the content of the output to any file name - in this tutorial, let's save it to thermolysin.phil.

'''First look at your phil file''' 
To run prime, set the required parameters to match with your experiments (you can leave other parameters with their default values - or just delete them from you .phil file). The most interesting parameters are shown below:

<pre>
data = /path/to/your/integarion/result/pickle_files
run_no = 001
title = First trial for thermolysin
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
}
postref {
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
allparams {
flag_on = False
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
}
merge {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
target_unit_cell = 93.99,93.99,130.87,90,90,120
target_space_group = P 61 2 2
pixel_size_mm = 0.102
</pre>

You should pay attention to d_min and d_max for the refinement and merging parameters. If you use IOTA to integrate the images, IOTA will output .phil file for prime that has the optimal resolution range. If not, a few trial-and-error runs may be required to get the best resolution range for your dataset. Use merging statistics output by prime and check the values of CC1/2 and I/sigI to find out your optimal resolution range.

Cell parameters (target_unit_cell and target_space_group) are required to run prime. Target cell parameter is used to remove some outlier images by controlling uc_tolerance parameter (the default value of tolerate range is 3% different). Space group parameter is used in removing outliers and merging with the given symmetry.

Don't forget also to change your pixel size in millimeters. Check what your detector is and note down its pixel size.

'''Running post-refinement in automatic mode''' 
Once you have the input .phil file, you can run ''prime'' by calling
<pre>
prime.run thermolysin.phil
</pre>
''Prime'' will post-refine and merge for reflection sets using three (default value) macrocycles. At the end of the run, you can obtain merging statistics in the last cycle - all other cycle statistics are also available in log.txt.

An example of merging statistics:
<pre>
Summary for 001/postref_cycle_1_merge.mtz
Bin Resolution Range Completeness <N_obs> |Rsplit CC1/2 N_ind |CCanom N_ind| <I/sigI> 
-------------------------------------------------------------------------------------------------------------
02 5.70 - 4.52 100.0 1055 / 1055 65.89 16.02 89.15 1055 0.00 0 20.17 2101.97
03 4.52 - 3.95 100.0 1032 / 1032 61.53 14.48 92.03 1032 0.00 0 20.39 2529.90
04 3.95 - 3.59 100.0 1016 / 1016 54.15 15.61 90.13 1016 0.00 0 16.69 1971.43
05 3.59 - 3.33 100.0 1004 / 1004 42.67 17.66 89.23 1004 0.00 0 14.21 1502.14
06 3.33 - 3.14 100.0 1013 / 1013 32.77 20.40 84.26 1013 0.00 0 11.76 1077.60
07 3.14 - 2.98 100.0 995 / 995 27.36 23.00 78.72 995 0.00 0 11.58 935.37
08 2.98 - 2.85 100.0 1006 / 1006 23.57 22.63 82.26 1006 0.00 0 10.56 722.62
09 2.85 - 2.74 100.0 986 / 986 16.64 28.51 72.90 985 0.00 0 10.01 591.56
10 2.74 - 2.65 99.9 989 / 990 12.41 31.35 72.95 987 0.00 0 9.91 515.07
11 2.65 - 2.56 99.7 979 / 982 9.35 37.14 65.31 970 0.00 0 9.31 438.96
12 2.56 - 2.49 98.0 979 / 999 6.06 45.98 45.37 930 0.00 0 9.45 390.05
13 2.49 - 2.42 95.1 931 / 979 4.46 50.68 34.20 834 0.00 0 8.93 334.80
14 2.42 - 2.37 91.7 896 / 977 3.35 55.66 37.15 729 0.00 0 9.27 320.17
15 2.37 - 2.31 83.9 829 / 988 2.61 56.92 43.21 600 0.00 0 9.60 296.67
16 2.31 - 2.26 72.4 702 / 969 1.97 65.81 26.89 386 0.00 0 10.29 284.39
17 2.26 - 2.22 59.1 582 / 985 1.75 64.72 31.28 275 0.00 0 9.87 284.06
18 2.22 - 2.18 52.9 513 / 970 1.51 71.27 16.86 188 0.00 0 8.93 215.31
19 2.18 - 2.14 35.7 349 / 978 1.32 62.26 68.25 90 0.00 0 8.22 199.09
20 2.14 - 2.10 23.1 227 / 981 1.20 92.14 -9.20 42 0.00 0 8.59 224.44
-------------------------------------------------------------------------------------------------------------
TOTAL 85.9 17224 / 20046 27.11 21.11 92.07 15305 0.00 0 12.87 999.53
-------------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 1809
No. bad cc frames: 153
No. bad G frames) : 0
No. bad unit cell frames: 5
No. bad gamma_e frames: 0
No. bad SE: 0
No. observations: 466997

</pre>

== Solving indexing ambiguity (New) ==
* SOFTWARE UPDATE REQUIRED *
With the latest version (Aug 31, 2016), you can solve the indexing ambiguity problem directly in prime. The Brehm & Diederichs algorithms (doi:10.1107/S1399004713025431) have been implemented with bootstrap capability to handle large dataset.
 
'''Merohedral Twinning''' 
For merohedral twinning (27 space groups e.g. P6), the indexing choices will be determined automatically in prime. Use this default setting in your .phil file,

<pre>
indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
n_sample_frames = 300
n_selected_frames = 100
}
</pre>

The n_sample_frames parameter indicates no. of images that will be used for the calculation of the scoring function. After that, only n_selected_images will be used in the B&D algorithms. This saves a lot of computing time since only the selected images will be used for the determination of the ambiguity. You can change these two parameters to fit with your experiments. The default values are 300 and 100 (give 300 - use 100).

'''Pseudo-Merohedral Twinning''' 
For pseudo-merohedral twinning, due to different possibilities for the indexing choice, prime doesn't determine these choices automatically. If you suspect that you may have pseudo twinning (b and c are similar, beta angle is almost 90 degree but not quite), you have an option to force prime to determine the ambiguity according to your choices.
<pre>
indexing_ambiguity {
mode = Forced
index_basis_in = None
assigned_basis = -h,l,k
assigned_basis = -k, l, h
n_sample_frames = 300
n_selected_frames = 100
}
</pre>

When you set indexing_ambiguity.mode to Forced, you can assign indexing choices according to your problem. In this example, two more choices (-h, l, k and -k, l, h) were assigned as the indexing choice.

'''Reusing the solution''' 
At the end of the run, your solution pickle is saved to your_run_no/index_ambiguity/solution_pickle.pickle. If you don't want to spend time solving the ambiguity again in the next run, you can reuse this solution pickle by setting these parameters:
<pre>
indexing_ambiguity {
mode = Auto
index_basis_in = your_run_no/index_ambiguity/solution_pickle.pickle
}
</pre>

This will bypass the indexing ambiguity module. Prime will use the solution file to perform normal post-refinement and merging.

'''Using an external reference set''' 
To use another isomorphous dataset (e.g. from a synchrotron experiment) as a reference set to solve the ambiguity, you can specify an mtz file as part of these parameters:
<pre>
indexing_ambiguity {
mode = Auto
index_basis_in = path/to/your/mtz/file.mtz
}
</pre>
Again, you can choose to do Auto or Forced (with a list of assigned_basis parameters) depending on your problem.

== More detail with input parameters ==
Now that you have your first trial merged data set, you can explore different parameter settings to merge or to obtain the Bijvoet pairs (I+/I-) for your anomalous data set.

'''Anomalous data:'''
<pre>
target_anomalous_flag = True
</pre>
In the last cycle, prime will output a reflection set with I+ and I-.

'''Number of micro- and macrocycles'''
<pre>
n_postref_cycle = 3
n_postref_sub_cycle = 1
</pre>

'''Number of bins for merging statistics'''
<pre>
n_bins = 20
</pre>

== Help with input parameters ==

Most input parameters are self-explained. However, you can run -h switch to view help information for each parameter.
<pre>
prime.run -h
</pre>

== Running in manual mode ==
With the same phil file, you can run prime manually. This gives you more freedom in terms of parameter settings at different stages (generating reference set, post-refining images, and merging) or at different cycle of post-refinement.

'''Example A''': I want to generate a reference set then post-refine all the images on the '''scale factors only''' for '''three cycles''' then refine '''all parameters''' in the '''4th cycle'''. To do this, you can follow these steps:

To generate a reference set,
<pre>
prime.genref prime.phil
</pre>

To post-refine on scale factors only, modify your .phil file so that all parameters are turned ''off''.
<pre>
...
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
}
postref {
residual_threshold = 5
residual_threshold_xy = 5
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
allparams {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 3
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To refine all parameters one more cycle, update your .phil file again (flag_on = True)
<pre>
...
allparams {
flag_on = True
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 1
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To obtain the final merged mtz, run
<pre>
prime.merge prime.phil
</pre>

== Running on multiple nodes ==
For LCLS users (or other users with LSF bsub), you can use psana (or your) queuing system to parallelize the entire process. For example, if you want to run your job on 100 nodes using psanq, you can specify:
<pre>
queue {
mode = bsub
qname = psanaq
n_nodes = 100
}
timeout_seconds = 300
</pre>
Prime will divide all the images into 100 batches and submit them to different nodes. It will wait until all images in every batches are done before returning to the merging step (or the exit step in the manual mode). You can control timeout_seconds parameter to tell prime how long it should wait for all the image batches to finish. Usually, this timeout parameter is not used (all images should return before 300 seconds) but in case, you need to wait longer or shorter, you can modify this parameter.

Cctbx.prime

2016-08-31T21:41:57Z

Mona: /* Solving indexing ambiguity (New) */

== Prime: '''p'''ost-'''r'''ef'''i'''nement and '''me'''rging ==
With the latest update, prime can be used to process data on multiple nodes (on a queuing system). At the moment, only LSF (bsub) is supported. See documentation below for more information on how to use the queuing system.

This major update replaces prime.postrefine with

<pre>
prime.run
</pre>

For auto mode, you can still use prime.run with your parameter phil file like before. For manual mode, the available sub commands in prime are:

<pre>
prime.genref #generates a reference set from given integration results
prime.postrefine #refines all images
prime.merge #merges all refined results for an mtz file
</pre>

You can choose to run these commands independently (ideally in the above order) using the same phil file. See [https://commons.wikimedia.org/wiki/File:Prime_flowcharts.tif "PRIME flowchart"]. This will give you the freedom to change something (e.g. set of parameters to refine, resolution cut-off, etc.) at different stages of the post-refinement and merging. See running prime in manual mode for more detail.

Step-by-step guidelines to post-refine and merge XFEL diffraction images. For more detail and citation, see
"Enabling X-ray Free Electron Laser Crystallography for Challenging Biological Systems from a Limited Number of Crystals"
[http://elifesciences.org/content/4/e05421 "DOI: http://dx.doi.org/10.7554/eLife.05421"]

== Prime is gui-ed ==
Thanks to Dr. Lyubimov, PRIME is also available as a Graphic User Interface program. Try it by running

<pre>
prime
</pre>

Click to see [https://commons.wikimedia.org/wiki/File:PRIME_main.png "PRIME main gui"] and [https://commons.wikimedia.org/wiki/File:PRIME_advanced_options.png "Advanced options"]

== Getting started ==
'''Generating input phil file''' 
Like most programs developed under ''cctbx'' framework, ''prime'' reads in input .phil file, which stores all the parameters needed to run post-refinement and merging steps. To generate the template .phil file, do the dry run by calling

<pre>
$ prime.run
</pre>

An example of the template .phil file:

<pre>
data = None
run_no = None
title = None
scale {
d_min = 0.1
d_max = 99
sigma_min = 1.5
}
...
</pre>

You can save the content of the output to any file name - in this tutorial, let's save it to thermolysin.phil.

'''First look at your phil file''' 
To run prime, set the required parameters to match with your experiments (you can leave other parameters with their default values - or just delete them from you .phil file). The most interesting parameters are shown below:

<pre>
data = /path/to/your/integarion/result/pickle_files
run_no = 001
title = First trial for thermolysin
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
}
postref {
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
allparams {
flag_on = False
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
}
merge {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
target_unit_cell = 93.99,93.99,130.87,90,90,120
target_space_group = P 61 2 2
pixel_size_mm = 0.102
</pre>

You should pay attention to d_min and d_max for the refinement and merging parameters. If you use IOTA to integrate the images, IOTA will output .phil file for prime that has the optimal resolution range. If not, a few trial-and-error runs may be required to get the best resolution range for your dataset. Use merging statistics output by prime and check the values of CC1/2 and I/sigI to find out your optimal resolution range.

Cell parameters (target_unit_cell and target_space_group) are required to run prime. Target cell parameter is used to remove some outlier images by controlling uc_tolerance parameter (the default value of tolerate range is 3% different). Space group parameter is used in removing outliers and merging with the given symmetry.

Don't forget also to change your pixel size in millimeters. Check what your detector is and note down its pixel size.

'''Running post-refinement in automatic mode''' 
Once you have the input .phil file, you can run ''prime'' by calling
<pre>
prime.run thermolysin.phil
</pre>
''Prime'' will post-refine and merge for reflection sets using three (default value) macrocycles. At the end of the run, you can obtain merging statistics in the last cycle - all other cycle statistics are also available in log.txt.

An example of merging statistics:
<pre>
Summary for 001/postref_cycle_1_merge.mtz
Bin Resolution Range Completeness <N_obs> |Rsplit CC1/2 N_ind |CCanom N_ind| <I/sigI> 
-------------------------------------------------------------------------------------------------------------
02 5.70 - 4.52 100.0 1055 / 1055 65.89 16.02 89.15 1055 0.00 0 20.17 2101.97
03 4.52 - 3.95 100.0 1032 / 1032 61.53 14.48 92.03 1032 0.00 0 20.39 2529.90
04 3.95 - 3.59 100.0 1016 / 1016 54.15 15.61 90.13 1016 0.00 0 16.69 1971.43
05 3.59 - 3.33 100.0 1004 / 1004 42.67 17.66 89.23 1004 0.00 0 14.21 1502.14
06 3.33 - 3.14 100.0 1013 / 1013 32.77 20.40 84.26 1013 0.00 0 11.76 1077.60
07 3.14 - 2.98 100.0 995 / 995 27.36 23.00 78.72 995 0.00 0 11.58 935.37
08 2.98 - 2.85 100.0 1006 / 1006 23.57 22.63 82.26 1006 0.00 0 10.56 722.62
09 2.85 - 2.74 100.0 986 / 986 16.64 28.51 72.90 985 0.00 0 10.01 591.56
10 2.74 - 2.65 99.9 989 / 990 12.41 31.35 72.95 987 0.00 0 9.91 515.07
11 2.65 - 2.56 99.7 979 / 982 9.35 37.14 65.31 970 0.00 0 9.31 438.96
12 2.56 - 2.49 98.0 979 / 999 6.06 45.98 45.37 930 0.00 0 9.45 390.05
13 2.49 - 2.42 95.1 931 / 979 4.46 50.68 34.20 834 0.00 0 8.93 334.80
14 2.42 - 2.37 91.7 896 / 977 3.35 55.66 37.15 729 0.00 0 9.27 320.17
15 2.37 - 2.31 83.9 829 / 988 2.61 56.92 43.21 600 0.00 0 9.60 296.67
16 2.31 - 2.26 72.4 702 / 969 1.97 65.81 26.89 386 0.00 0 10.29 284.39
17 2.26 - 2.22 59.1 582 / 985 1.75 64.72 31.28 275 0.00 0 9.87 284.06
18 2.22 - 2.18 52.9 513 / 970 1.51 71.27 16.86 188 0.00 0 8.93 215.31
19 2.18 - 2.14 35.7 349 / 978 1.32 62.26 68.25 90 0.00 0 8.22 199.09
20 2.14 - 2.10 23.1 227 / 981 1.20 92.14 -9.20 42 0.00 0 8.59 224.44
-------------------------------------------------------------------------------------------------------------
TOTAL 85.9 17224 / 20046 27.11 21.11 92.07 15305 0.00 0 12.87 999.53
-------------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 1809
No. bad cc frames: 153
No. bad G frames) : 0
No. bad unit cell frames: 5
No. bad gamma_e frames: 0
No. bad SE: 0
No. observations: 466997

</pre>

== Solving indexing ambiguity (New) ==
* SOFTWARE UPDATE REQUIRED *
With the latest version (Aug 31, 2016), you can solve the indexing ambiguity problem directly in prime. The Brehm & Diederichs algorithms (doi:10.1107/S1399004713025431) have been implemented with bootstrap capability to handle large dataset.
'''Merohedral Twinning''' 
For merohedral twinning (27 space groups e.g. P6), the indexing choices will be determined automatically in prime. Use this default setting in your .phil file,

<pre>
indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
n_sample_frames = 300
n_selected_frames = 100
}
</pre>

The n_sample_frames parameter indicates no. of images that will be used for the calculation of the scoring function. After that, only n_selected_images will be used in the B&D algorithms. This saves a lot of computing time since only the selected images will be used for the determination of the ambiguity. You can change these two parameters to fit with your experiments. The default values are 300 and 100 (give 300 - use 100).

'''Pseudo-Merohedral Twinning''' 
For pseudo-merohedral twinning, due to different possibilities for the indexing choice, prime doesn't determine these choices automatically. If you suspect that you may have pseudo twinning (b and c are similar, beta angle is almost 90 degree but not quite), you have an option to force prime to determine the ambiguity according to your choices.
<pre>
indexing_ambiguity {
mode = Forced
index_basis_in = None
assigned_basis = -h,l,k
assigned_basis = -k, l, h
n_sample_frames = 300
n_selected_frames = 100
}
</pre>

When you set indexing_ambiguity.mode to Forced, you can assign indexing choices according to your problem. In this example, two more choices (-h, l, k and -k, l, h) were assigned as the indexing choice.

'''Reusing the solution''' 
At the end of the run, your solution pickle is saved to your_run_no/index_ambiguity/solution_pickle.pickle. If you don't want to spend time solving the ambiguity again in the next run, you can reuse this solution pickle by setting these parameters:
<pre>
indexing_ambiguity {
mode = Auto
index_basis_in = your_run_no/index_ambiguity/solution_pickle.pickle
}
</pre>

This will bypass the indexing ambiguity module. Prime will use the solution file to perform normal post-refinement and merging.

'''Using an external reference set''' 
To use another isomorphous dataset (e.g. from a synchrotron experiment) as a reference set to solve the ambiguity, you can specify an mtz file as part of these parameters:
<pre>
indexing_ambiguity {
mode = Auto
index_basis_in = path/to/your/mtz/file.mtz
}
</pre>
Again, you can choose to do Auto or Forced (with a list of assigned_basis parameters) depending on your problem.

== More detail with input parameters ==
Now that you have your first trial merged data set, you can explore different parameter settings to merge or to obtain the Bijvoet pairs (I+/I-) for your anomalous data set.

'''Anomalous data:'''
<pre>
target_anomalous_flag = True
</pre>
In the last cycle, prime will output a reflection set with I+ and I-.

'''Number of micro- and macrocycles'''
<pre>
n_postref_cycle = 3
n_postref_sub_cycle = 1
</pre>

'''Number of bins for merging statistics'''
<pre>
n_bins = 20
</pre>

== Help with input parameters ==

Most input parameters are self-explained. However, you can run -h switch to view help information for each parameter.
<pre>
prime.run -h
</pre>

== Running in manual mode ==
With the same phil file, you can run prime manually. This gives you more freedom in terms of parameter settings at different stages (generating reference set, post-refining images, and merging) or at different cycle of post-refinement.

'''Example A''': I want to generate a reference set then post-refine all the images on the '''scale factors only''' for '''three cycles''' then refine '''all parameters''' in the '''4th cycle'''. To do this, you can follow these steps:

To generate a reference set,
<pre>
prime.genref prime.phil
</pre>

To post-refine on scale factors only, modify your .phil file so that all parameters are turned ''off''.
<pre>
...
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
}
postref {
residual_threshold = 5
residual_threshold_xy = 5
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
allparams {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 3
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To refine all parameters one more cycle, update your .phil file again (flag_on = True)
<pre>
...
allparams {
flag_on = True
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 1
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To obtain the final merged mtz, run
<pre>
prime.merge prime.phil
</pre>

== Running on multiple nodes ==
For LCLS users (or other users with LSF bsub), you can use psana (or your) queuing system to parallelize the entire process. For example, if you want to run your job on 100 nodes using psanq, you can specify:
<pre>
queue {
mode = bsub
qname = psanaq
n_nodes = 100
}
timeout_seconds = 300
</pre>
Prime will divide all the images into 100 batches and submit them to different nodes. It will wait until all images in every batches are done before returning to the merging step (or the exit step in the manual mode). You can control timeout_seconds parameter to tell prime how long it should wait for all the image batches to finish. Usually, this timeout parameter is not used (all images should return before 300 seconds) but in case, you need to wait longer or shorter, you can modify this parameter.

Cctbx.prime

2016-08-31T20:05:40Z

Mona: /* Solving indexing ambiguity (New) */

== Prime: '''p'''ost-'''r'''ef'''i'''nement and '''me'''rging ==
With the latest update, prime can be used to process data on multiple nodes (on a queuing system). At the moment, only LSF (bsub) is supported. See documentation below for more information on how to use the queuing system.

This major update replaces prime.postrefine with

<pre>
prime.run
</pre>

For auto mode, you can still use prime.run with your parameter phil file like before. For manual mode, the available sub commands in prime are:

<pre>
prime.genref #generates a reference set from given integration results
prime.postrefine #refines all images
prime.merge #merges all refined results for an mtz file
</pre>

You can choose to run these commands independently (ideally in the above order) using the same phil file. See [https://commons.wikimedia.org/wiki/File:Prime_flowcharts.tif "PRIME flowchart"]. This will give you the freedom to change something (e.g. set of parameters to refine, resolution cut-off, etc.) at different stages of the post-refinement and merging. See running prime in manual mode for more detail.

Step-by-step guidelines to post-refine and merge XFEL diffraction images. For more detail and citation, see
"Enabling X-ray Free Electron Laser Crystallography for Challenging Biological Systems from a Limited Number of Crystals"
[http://elifesciences.org/content/4/e05421 "DOI: http://dx.doi.org/10.7554/eLife.05421"]

== Prime is gui-ed ==
Thanks to Dr. Lyubimov, PRIME is also available as a Graphic User Interface program. Try it by running

<pre>
prime
</pre>

Click to see [https://commons.wikimedia.org/wiki/File:PRIME_main.png "PRIME main gui"] and [https://commons.wikimedia.org/wiki/File:PRIME_advanced_options.png "Advanced options"]

== Getting started ==
'''Generating input phil file''' 
Like most programs developed under ''cctbx'' framework, ''prime'' reads in input .phil file, which stores all the parameters needed to run post-refinement and merging steps. To generate the template .phil file, do the dry run by calling

<pre>
$ prime.run
</pre>

An example of the template .phil file:

<pre>
data = None
run_no = None
title = None
scale {
d_min = 0.1
d_max = 99
sigma_min = 1.5
}
...
</pre>

You can save the content of the output to any file name - in this tutorial, let's save it to thermolysin.phil.

'''First look at your phil file''' 
To run prime, set the required parameters to match with your experiments (you can leave other parameters with their default values - or just delete them from you .phil file). The most interesting parameters are shown below:

<pre>
data = /path/to/your/integarion/result/pickle_files
run_no = 001
title = First trial for thermolysin
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
}
postref {
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
allparams {
flag_on = False
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
}
merge {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
target_unit_cell = 93.99,93.99,130.87,90,90,120
target_space_group = P 61 2 2
pixel_size_mm = 0.102
</pre>

You should pay attention to d_min and d_max for the refinement and merging parameters. If you use IOTA to integrate the images, IOTA will output .phil file for prime that has the optimal resolution range. If not, a few trial-and-error runs may be required to get the best resolution range for your dataset. Use merging statistics output by prime and check the values of CC1/2 and I/sigI to find out your optimal resolution range.

Cell parameters (target_unit_cell and target_space_group) are required to run prime. Target cell parameter is used to remove some outlier images by controlling uc_tolerance parameter (the default value of tolerate range is 3% different). Space group parameter is used in removing outliers and merging with the given symmetry.

Don't forget also to change your pixel size in millimeters. Check what your detector is and note down its pixel size.

'''Running post-refinement in automatic mode''' 
Once you have the input .phil file, you can run ''prime'' by calling
<pre>
prime.run thermolysin.phil
</pre>
''Prime'' will post-refine and merge for reflection sets using three (default value) macrocycles. At the end of the run, you can obtain merging statistics in the last cycle - all other cycle statistics are also available in log.txt.

An example of merging statistics:
<pre>
Summary for 001/postref_cycle_1_merge.mtz
Bin Resolution Range Completeness <N_obs> |Rsplit CC1/2 N_ind |CCanom N_ind| <I/sigI> 
-------------------------------------------------------------------------------------------------------------
02 5.70 - 4.52 100.0 1055 / 1055 65.89 16.02 89.15 1055 0.00 0 20.17 2101.97
03 4.52 - 3.95 100.0 1032 / 1032 61.53 14.48 92.03 1032 0.00 0 20.39 2529.90
04 3.95 - 3.59 100.0 1016 / 1016 54.15 15.61 90.13 1016 0.00 0 16.69 1971.43
05 3.59 - 3.33 100.0 1004 / 1004 42.67 17.66 89.23 1004 0.00 0 14.21 1502.14
06 3.33 - 3.14 100.0 1013 / 1013 32.77 20.40 84.26 1013 0.00 0 11.76 1077.60
07 3.14 - 2.98 100.0 995 / 995 27.36 23.00 78.72 995 0.00 0 11.58 935.37
08 2.98 - 2.85 100.0 1006 / 1006 23.57 22.63 82.26 1006 0.00 0 10.56 722.62
09 2.85 - 2.74 100.0 986 / 986 16.64 28.51 72.90 985 0.00 0 10.01 591.56
10 2.74 - 2.65 99.9 989 / 990 12.41 31.35 72.95 987 0.00 0 9.91 515.07
11 2.65 - 2.56 99.7 979 / 982 9.35 37.14 65.31 970 0.00 0 9.31 438.96
12 2.56 - 2.49 98.0 979 / 999 6.06 45.98 45.37 930 0.00 0 9.45 390.05
13 2.49 - 2.42 95.1 931 / 979 4.46 50.68 34.20 834 0.00 0 8.93 334.80
14 2.42 - 2.37 91.7 896 / 977 3.35 55.66 37.15 729 0.00 0 9.27 320.17
15 2.37 - 2.31 83.9 829 / 988 2.61 56.92 43.21 600 0.00 0 9.60 296.67
16 2.31 - 2.26 72.4 702 / 969 1.97 65.81 26.89 386 0.00 0 10.29 284.39
17 2.26 - 2.22 59.1 582 / 985 1.75 64.72 31.28 275 0.00 0 9.87 284.06
18 2.22 - 2.18 52.9 513 / 970 1.51 71.27 16.86 188 0.00 0 8.93 215.31
19 2.18 - 2.14 35.7 349 / 978 1.32 62.26 68.25 90 0.00 0 8.22 199.09
20 2.14 - 2.10 23.1 227 / 981 1.20 92.14 -9.20 42 0.00 0 8.59 224.44
-------------------------------------------------------------------------------------------------------------
TOTAL 85.9 17224 / 20046 27.11 21.11 92.07 15305 0.00 0 12.87 999.53
-------------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 1809
No. bad cc frames: 153
No. bad G frames) : 0
No. bad unit cell frames: 5
No. bad gamma_e frames: 0
No. bad SE: 0
No. observations: 466997

</pre>

== Solving indexing ambiguity (New) ==
* SOFTWARE UPDATE REQUIRED *
With the latest version (Aug 31, 2016), you can solve the indexing ambiguity problem directly in prime. The Brehm & Diederichs algorithms (doi:10.1107/S1399004713025431) have been implemented with bootstrap capability to handle large dataset.
'''Merohedral Twinning''' 
For merohedral twinning (27 space groups e.g. P6), the indexing choices will be determined automatically in prime. Use this default setting in your .phil file,

<pre>
indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
n_sample_frames = 300
n_selected_frames = 100
}
</pre>

The n_sample_frames parameter indicates no. of images that will be used for the calculation of the scoring function. After that, only n_selected_images will be used in the B&D algorithms. This saves a lot of computing time since only the selected images will be used for the determination of the ambiguity. You can change these two parameters to fit with your experiments. The default values are 300 and 100 (give 300 - use 100).

'''Pseudo-Merohedral Twinning''' 
For pseudo-merohedral twinning, due to different possibilities for the indexing choice, prime doesn't determine these choices automatically. If you suspect that you may have pseudo twinning (b and c are similar, beta angle is almost 90 degree but not quite), you have an option to force prime to determine the ambiguity according to your choices.
<pre>
indexing_ambiguity {
mode = Forced
index_basis_in = None
assigned_basis = -h,l,k
assigned_basis = -k, l, h
n_sample_frames = 300
n_selected_frames = 100
}
</pre>

When you set indexing_ambiguity.mode to Forced, you can assign indexing choices according to your problem. In this example, two more choices (-h, l, k and -k, l, h) were assigned as the indexing choice.

'''Reusing the solution''' 
At the end of the run, your solution pickle is saved to your_run_no/index_ambiguity/solution_pickle.pickle. If you don't want to spend time solving the ambiguity again in the next run, you can reuse this solution pickle by setting these parameters:
<pre>
indexing_ambiguity {
mode = Auto
index_basis_in = your_run_no/index_ambiguity/solution_pickle.pickle
}
</pre>

This will bypass the indexing ambiguity module. Prime will use the solution file to perform normal post-refinement and merging.

'''Using an external reference set''' 
To use another isomorphous dataset (e.g. from a synchrotron experiment), you can specify an mtz file as part of these parameters:
<pre>
indexing_ambiguity {
mode = Auto
index_basis_in = path/to/your/mtz/file.mtz
}
</pre>
Again, you can choose to do Auto or Forced (with a list of assigned_basis parameters) depending on your problem.

== More detail with input parameters ==
Now that you have your first trial merged data set, you can explore different parameter settings to merge or to obtain the Bijvoet pairs (I+/I-) for your anomalous data set.

'''Anomalous data:'''
<pre>
target_anomalous_flag = True
</pre>
In the last cycle, prime will output a reflection set with I+ and I-.

'''Number of micro- and macrocycles'''
<pre>
n_postref_cycle = 3
n_postref_sub_cycle = 1
</pre>

'''Number of bins for merging statistics'''
<pre>
n_bins = 20
</pre>

== Help with input parameters ==

Most input parameters are self-explained. However, you can run -h switch to view help information for each parameter.
<pre>
prime.run -h
</pre>

== Running in manual mode ==
With the same phil file, you can run prime manually. This gives you more freedom in terms of parameter settings at different stages (generating reference set, post-refining images, and merging) or at different cycle of post-refinement.

'''Example A''': I want to generate a reference set then post-refine all the images on the '''scale factors only''' for '''three cycles''' then refine '''all parameters''' in the '''4th cycle'''. To do this, you can follow these steps:

To generate a reference set,
<pre>
prime.genref prime.phil
</pre>

To post-refine on scale factors only, modify your .phil file so that all parameters are turned ''off''.
<pre>
...
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
}
postref {
residual_threshold = 5
residual_threshold_xy = 5
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
allparams {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 3
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To refine all parameters one more cycle, update your .phil file again (flag_on = True)
<pre>
...
allparams {
flag_on = True
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 1
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To obtain the final merged mtz, run
<pre>
prime.merge prime.phil
</pre>

== Running on multiple nodes ==
For LCLS users (or other users with LSF bsub), you can use psana (or your) queuing system to parallelize the entire process. For example, if you want to run your job on 100 nodes using psanq, you can specify:
<pre>
queue {
mode = bsub
qname = psanaq
n_nodes = 100
}
timeout_seconds = 300
</pre>
Prime will divide all the images into 100 batches and submit them to different nodes. It will wait until all images in every batches are done before returning to the merging step (or the exit step in the manual mode). You can control timeout_seconds parameter to tell prime how long it should wait for all the image batches to finish. Usually, this timeout parameter is not used (all images should return before 300 seconds) but in case, you need to wait longer or shorter, you can modify this parameter.

Cctbx.prime

2016-08-31T20:01:01Z

Mona: /* Solving indexing ambiguity (New) */

== Prime: '''p'''ost-'''r'''ef'''i'''nement and '''me'''rging ==
With the latest update, prime can be used to process data on multiple nodes (on a queuing system). At the moment, only LSF (bsub) is supported. See documentation below for more information on how to use the queuing system.

This major update replaces prime.postrefine with

<pre>
prime.run
</pre>

For auto mode, you can still use prime.run with your parameter phil file like before. For manual mode, the available sub commands in prime are:

<pre>
prime.genref #generates a reference set from given integration results
prime.postrefine #refines all images
prime.merge #merges all refined results for an mtz file
</pre>

You can choose to run these commands independently (ideally in the above order) using the same phil file. See [https://commons.wikimedia.org/wiki/File:Prime_flowcharts.tif "PRIME flowchart"]. This will give you the freedom to change something (e.g. set of parameters to refine, resolution cut-off, etc.) at different stages of the post-refinement and merging. See running prime in manual mode for more detail.

Step-by-step guidelines to post-refine and merge XFEL diffraction images. For more detail and citation, see
"Enabling X-ray Free Electron Laser Crystallography for Challenging Biological Systems from a Limited Number of Crystals"
[http://elifesciences.org/content/4/e05421 "DOI: http://dx.doi.org/10.7554/eLife.05421"]

== Prime is gui-ed ==
Thanks to Dr. Lyubimov, PRIME is also available as a Graphic User Interface program. Try it by running

<pre>
prime
</pre>

Click to see [https://commons.wikimedia.org/wiki/File:PRIME_main.png "PRIME main gui"] and [https://commons.wikimedia.org/wiki/File:PRIME_advanced_options.png "Advanced options"]

== Getting started ==
'''Generating input phil file''' 
Like most programs developed under ''cctbx'' framework, ''prime'' reads in input .phil file, which stores all the parameters needed to run post-refinement and merging steps. To generate the template .phil file, do the dry run by calling

<pre>
$ prime.run
</pre>

An example of the template .phil file:

<pre>
data = None
run_no = None
title = None
scale {
d_min = 0.1
d_max = 99
sigma_min = 1.5
}
...
</pre>

You can save the content of the output to any file name - in this tutorial, let's save it to thermolysin.phil.

'''First look at your phil file''' 
To run prime, set the required parameters to match with your experiments (you can leave other parameters with their default values - or just delete them from you .phil file). The most interesting parameters are shown below:

<pre>
data = /path/to/your/integarion/result/pickle_files
run_no = 001
title = First trial for thermolysin
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
}
postref {
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
allparams {
flag_on = False
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
}
merge {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
target_unit_cell = 93.99,93.99,130.87,90,90,120
target_space_group = P 61 2 2
pixel_size_mm = 0.102
</pre>

You should pay attention to d_min and d_max for the refinement and merging parameters. If you use IOTA to integrate the images, IOTA will output .phil file for prime that has the optimal resolution range. If not, a few trial-and-error runs may be required to get the best resolution range for your dataset. Use merging statistics output by prime and check the values of CC1/2 and I/sigI to find out your optimal resolution range.

Cell parameters (target_unit_cell and target_space_group) are required to run prime. Target cell parameter is used to remove some outlier images by controlling uc_tolerance parameter (the default value of tolerate range is 3% different). Space group parameter is used in removing outliers and merging with the given symmetry.

Don't forget also to change your pixel size in millimeters. Check what your detector is and note down its pixel size.

'''Running post-refinement in automatic mode''' 
Once you have the input .phil file, you can run ''prime'' by calling
<pre>
prime.run thermolysin.phil
</pre>
''Prime'' will post-refine and merge for reflection sets using three (default value) macrocycles. At the end of the run, you can obtain merging statistics in the last cycle - all other cycle statistics are also available in log.txt.

An example of merging statistics:
<pre>
Summary for 001/postref_cycle_1_merge.mtz
Bin Resolution Range Completeness <N_obs> |Rsplit CC1/2 N_ind |CCanom N_ind| <I/sigI> 
-------------------------------------------------------------------------------------------------------------
02 5.70 - 4.52 100.0 1055 / 1055 65.89 16.02 89.15 1055 0.00 0 20.17 2101.97
03 4.52 - 3.95 100.0 1032 / 1032 61.53 14.48 92.03 1032 0.00 0 20.39 2529.90
04 3.95 - 3.59 100.0 1016 / 1016 54.15 15.61 90.13 1016 0.00 0 16.69 1971.43
05 3.59 - 3.33 100.0 1004 / 1004 42.67 17.66 89.23 1004 0.00 0 14.21 1502.14
06 3.33 - 3.14 100.0 1013 / 1013 32.77 20.40 84.26 1013 0.00 0 11.76 1077.60
07 3.14 - 2.98 100.0 995 / 995 27.36 23.00 78.72 995 0.00 0 11.58 935.37
08 2.98 - 2.85 100.0 1006 / 1006 23.57 22.63 82.26 1006 0.00 0 10.56 722.62
09 2.85 - 2.74 100.0 986 / 986 16.64 28.51 72.90 985 0.00 0 10.01 591.56
10 2.74 - 2.65 99.9 989 / 990 12.41 31.35 72.95 987 0.00 0 9.91 515.07
11 2.65 - 2.56 99.7 979 / 982 9.35 37.14 65.31 970 0.00 0 9.31 438.96
12 2.56 - 2.49 98.0 979 / 999 6.06 45.98 45.37 930 0.00 0 9.45 390.05
13 2.49 - 2.42 95.1 931 / 979 4.46 50.68 34.20 834 0.00 0 8.93 334.80
14 2.42 - 2.37 91.7 896 / 977 3.35 55.66 37.15 729 0.00 0 9.27 320.17
15 2.37 - 2.31 83.9 829 / 988 2.61 56.92 43.21 600 0.00 0 9.60 296.67
16 2.31 - 2.26 72.4 702 / 969 1.97 65.81 26.89 386 0.00 0 10.29 284.39
17 2.26 - 2.22 59.1 582 / 985 1.75 64.72 31.28 275 0.00 0 9.87 284.06
18 2.22 - 2.18 52.9 513 / 970 1.51 71.27 16.86 188 0.00 0 8.93 215.31
19 2.18 - 2.14 35.7 349 / 978 1.32 62.26 68.25 90 0.00 0 8.22 199.09
20 2.14 - 2.10 23.1 227 / 981 1.20 92.14 -9.20 42 0.00 0 8.59 224.44
-------------------------------------------------------------------------------------------------------------
TOTAL 85.9 17224 / 20046 27.11 21.11 92.07 15305 0.00 0 12.87 999.53
-------------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 1809
No. bad cc frames: 153
No. bad G frames) : 0
No. bad unit cell frames: 5
No. bad gamma_e frames: 0
No. bad SE: 0
No. observations: 466997

</pre>

== Solving indexing ambiguity (New) ==
* SOFTWARE UPDATE REQUIRED *
With the latest version (Aug 31, 2016), you can solve the indexing ambiguity problem directly in prime. The Brehm & Diederichs algorithms (doi:10.1107/S1399004713025431) have been implemented with bootstrap capability to handle large dataset.

For merohedral twining group, the indexing choices will be determined automatically in prime. Use this default setting in your .phil file,

<pre>
indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
n_sample_frames = 300
n_selected_frames = 100
}
</pre>

The n_sample_frames parameter indicates no. of images that will be used for the calculation of the scoring function. After that, only n_selected_images will be used in the B&D algorithms. This saves a lot of computing time since only the selected images will be used for the determination of the ambiguity. You can change these two parameters to fit with your experiments. The default values are 300 and 100 (give 300 - use 100).

For pseudo-merohedral twinning, due to different possibilities for the indexing choice, prime doesn't determine these choices automatically. If you suspect that you may have pseudo twinning (b and c are similar, beta angle is almost 90 degree but not quite), you have an option to force prime to determine the ambiguity according to your choices.
<pre>
indexing_ambiguity {
mode = Forced
index_basis_in = None
assigned_basis = -h,l,k
assigned_basis = -k, l, h
n_sample_frames = 300
n_selected_frames = 100
}
</pre>

When you set indexing_ambiguity.mode to Forced, you can assign indexing choices according to your problem. In this example, two more choices (-h, l, k and -k, l, h) were assigned as the indexing choice.

At the end of the run, your solution pickle is saved to your_run_no/index_ambiguity/solution_pickle.pickle. If you don't want to spend time solving the ambiguity again in the next run, you can reuse this solution pickle by setting these parameters:
<pre>
indexing_ambiguity {
mode = Auto
index_basis_in = your_run_no/index_ambiguity/solution_pickle.pickle
}
</pre>

This will bypass the indexing ambiguity module. Prime will use the solution file to perform normal post-refinement and merging.

To use another isomorphous dataset (e.g. from synchrotron experiment), you can specify the mtz file as part of these parameters:
<pre>
indexing_ambiguity {
mode = Auto
index_basis_in = path/to/your/mtz/file.mtz
}
</pre>
Again, you can choose to do Auto or Forced (with a list of assigned_basis parameters) depending on your problem.

== More detail with input parameters ==
Now that you have your first trial merged data set, you can explore different parameter settings to merge or to obtain the Bijvoet pairs (I+/I-) for your anomalous data set.

'''Anomalous data:'''
<pre>
target_anomalous_flag = True
</pre>
In the last cycle, prime will output a reflection set with I+ and I-.

'''Number of micro- and macrocycles'''
<pre>
n_postref_cycle = 3
n_postref_sub_cycle = 1
</pre>

'''Number of bins for merging statistics'''
<pre>
n_bins = 20
</pre>

== Help with input parameters ==

Most input parameters are self-explained. However, you can run -h switch to view help information for each parameter.
<pre>
prime.run -h
</pre>

== Running in manual mode ==
With the same phil file, you can run prime manually. This gives you more freedom in terms of parameter settings at different stages (generating reference set, post-refining images, and merging) or at different cycle of post-refinement.

'''Example A''': I want to generate a reference set then post-refine all the images on the '''scale factors only''' for '''three cycles''' then refine '''all parameters''' in the '''4th cycle'''. To do this, you can follow these steps:

To generate a reference set,
<pre>
prime.genref prime.phil
</pre>

To post-refine on scale factors only, modify your .phil file so that all parameters are turned ''off''.
<pre>
...
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
}
postref {
residual_threshold = 5
residual_threshold_xy = 5
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
allparams {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 3
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To refine all parameters one more cycle, update your .phil file again (flag_on = True)
<pre>
...
allparams {
flag_on = True
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 1
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To obtain the final merged mtz, run
<pre>
prime.merge prime.phil
</pre>

== Running on multiple nodes ==
For LCLS users (or other users with LSF bsub), you can use psana (or your) queuing system to parallelize the entire process. For example, if you want to run your job on 100 nodes using psanq, you can specify:
<pre>
queue {
mode = bsub
qname = psanaq
n_nodes = 100
}
timeout_seconds = 300
</pre>
Prime will divide all the images into 100 batches and submit them to different nodes. It will wait until all images in every batches are done before returning to the merging step (or the exit step in the manual mode). You can control timeout_seconds parameter to tell prime how long it should wait for all the image batches to finish. Usually, this timeout parameter is not used (all images should return before 300 seconds) but in case, you need to wait longer or shorter, you can modify this parameter.

Cctbx.prime

2016-08-31T20:00:10Z

Mona: /* Solving indexing ambiguity (New) */

== Prime: '''p'''ost-'''r'''ef'''i'''nement and '''me'''rging ==
With the latest update, prime can be used to process data on multiple nodes (on a queuing system). At the moment, only LSF (bsub) is supported. See documentation below for more information on how to use the queuing system.

This major update replaces prime.postrefine with

<pre>
prime.run
</pre>

For auto mode, you can still use prime.run with your parameter phil file like before. For manual mode, the available sub commands in prime are:

<pre>
prime.genref #generates a reference set from given integration results
prime.postrefine #refines all images
prime.merge #merges all refined results for an mtz file
</pre>

You can choose to run these commands independently (ideally in the above order) using the same phil file. See [https://commons.wikimedia.org/wiki/File:Prime_flowcharts.tif "PRIME flowchart"]. This will give you the freedom to change something (e.g. set of parameters to refine, resolution cut-off, etc.) at different stages of the post-refinement and merging. See running prime in manual mode for more detail.

Step-by-step guidelines to post-refine and merge XFEL diffraction images. For more detail and citation, see
"Enabling X-ray Free Electron Laser Crystallography for Challenging Biological Systems from a Limited Number of Crystals"
[http://elifesciences.org/content/4/e05421 "DOI: http://dx.doi.org/10.7554/eLife.05421"]

== Prime is gui-ed ==
Thanks to Dr. Lyubimov, PRIME is also available as a Graphic User Interface program. Try it by running

<pre>
prime
</pre>

Click to see [https://commons.wikimedia.org/wiki/File:PRIME_main.png "PRIME main gui"] and [https://commons.wikimedia.org/wiki/File:PRIME_advanced_options.png "Advanced options"]

== Getting started ==
'''Generating input phil file''' 
Like most programs developed under ''cctbx'' framework, ''prime'' reads in input .phil file, which stores all the parameters needed to run post-refinement and merging steps. To generate the template .phil file, do the dry run by calling

<pre>
$ prime.run
</pre>

An example of the template .phil file:

<pre>
data = None
run_no = None
title = None
scale {
d_min = 0.1
d_max = 99
sigma_min = 1.5
}
...
</pre>

You can save the content of the output to any file name - in this tutorial, let's save it to thermolysin.phil.

'''First look at your phil file''' 
To run prime, set the required parameters to match with your experiments (you can leave other parameters with their default values - or just delete them from you .phil file). The most interesting parameters are shown below:

<pre>
data = /path/to/your/integarion/result/pickle_files
run_no = 001
title = First trial for thermolysin
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
}
postref {
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
allparams {
flag_on = False
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
}
merge {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
target_unit_cell = 93.99,93.99,130.87,90,90,120
target_space_group = P 61 2 2
pixel_size_mm = 0.102
</pre>

You should pay attention to d_min and d_max for the refinement and merging parameters. If you use IOTA to integrate the images, IOTA will output .phil file for prime that has the optimal resolution range. If not, a few trial-and-error runs may be required to get the best resolution range for your dataset. Use merging statistics output by prime and check the values of CC1/2 and I/sigI to find out your optimal resolution range.

Cell parameters (target_unit_cell and target_space_group) are required to run prime. Target cell parameter is used to remove some outlier images by controlling uc_tolerance parameter (the default value of tolerate range is 3% different). Space group parameter is used in removing outliers and merging with the given symmetry.

Don't forget also to change your pixel size in millimeters. Check what your detector is and note down its pixel size.

'''Running post-refinement in automatic mode''' 
Once you have the input .phil file, you can run ''prime'' by calling
<pre>
prime.run thermolysin.phil
</pre>
''Prime'' will post-refine and merge for reflection sets using three (default value) macrocycles. At the end of the run, you can obtain merging statistics in the last cycle - all other cycle statistics are also available in log.txt.

An example of merging statistics:
<pre>
Summary for 001/postref_cycle_1_merge.mtz
Bin Resolution Range Completeness <N_obs> |Rsplit CC1/2 N_ind |CCanom N_ind| <I/sigI> 
-------------------------------------------------------------------------------------------------------------
02 5.70 - 4.52 100.0 1055 / 1055 65.89 16.02 89.15 1055 0.00 0 20.17 2101.97
03 4.52 - 3.95 100.0 1032 / 1032 61.53 14.48 92.03 1032 0.00 0 20.39 2529.90
04 3.95 - 3.59 100.0 1016 / 1016 54.15 15.61 90.13 1016 0.00 0 16.69 1971.43
05 3.59 - 3.33 100.0 1004 / 1004 42.67 17.66 89.23 1004 0.00 0 14.21 1502.14
06 3.33 - 3.14 100.0 1013 / 1013 32.77 20.40 84.26 1013 0.00 0 11.76 1077.60
07 3.14 - 2.98 100.0 995 / 995 27.36 23.00 78.72 995 0.00 0 11.58 935.37
08 2.98 - 2.85 100.0 1006 / 1006 23.57 22.63 82.26 1006 0.00 0 10.56 722.62
09 2.85 - 2.74 100.0 986 / 986 16.64 28.51 72.90 985 0.00 0 10.01 591.56
10 2.74 - 2.65 99.9 989 / 990 12.41 31.35 72.95 987 0.00 0 9.91 515.07
11 2.65 - 2.56 99.7 979 / 982 9.35 37.14 65.31 970 0.00 0 9.31 438.96
12 2.56 - 2.49 98.0 979 / 999 6.06 45.98 45.37 930 0.00 0 9.45 390.05
13 2.49 - 2.42 95.1 931 / 979 4.46 50.68 34.20 834 0.00 0 8.93 334.80
14 2.42 - 2.37 91.7 896 / 977 3.35 55.66 37.15 729 0.00 0 9.27 320.17
15 2.37 - 2.31 83.9 829 / 988 2.61 56.92 43.21 600 0.00 0 9.60 296.67
16 2.31 - 2.26 72.4 702 / 969 1.97 65.81 26.89 386 0.00 0 10.29 284.39
17 2.26 - 2.22 59.1 582 / 985 1.75 64.72 31.28 275 0.00 0 9.87 284.06
18 2.22 - 2.18 52.9 513 / 970 1.51 71.27 16.86 188 0.00 0 8.93 215.31
19 2.18 - 2.14 35.7 349 / 978 1.32 62.26 68.25 90 0.00 0 8.22 199.09
20 2.14 - 2.10 23.1 227 / 981 1.20 92.14 -9.20 42 0.00 0 8.59 224.44
-------------------------------------------------------------------------------------------------------------
TOTAL 85.9 17224 / 20046 27.11 21.11 92.07 15305 0.00 0 12.87 999.53
-------------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 1809
No. bad cc frames: 153
No. bad G frames) : 0
No. bad unit cell frames: 5
No. bad gamma_e frames: 0
No. bad SE: 0
No. observations: 466997

</pre>

== Solving indexing ambiguity (New) ==
* SOFTWARE UPDATE REQUIRED *
With the latest version (Aug 31, 2016), you can solve the indexing ambiguity problem directly in prime. The Brehm & Diederichs algorithms (doi:10.1107/S1399004713025431) have been implemented with bootstrap capability to handle large dataset.
For merohedral twining group, the indexing choices will be determined automatically in prime. Use this default setting in your .phil file,

<pre>
indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
n_sample_frames = 300
n_selected_frames = 100
}
</pre>

The n_sample_frames parameter indicates no. of images that will be used for the calculation of the scoring function. After that, only n_selected_images will be used in the B&D algorithms. This saves a lot of computing time since only the selected images will be used for the determination of the ambiguity. You can change these two parameters to fit with your experiments. The default values are 300 and 100 (give 300 - use 100).

For pseudo-merohedral twinning, due to different possibilities for the indexing choice, prime doesn't determine these choices automatically. If you suspect that you may have pseudo twinning (b and c are similar, beta angle is almost 90 degree but not quite), you have an option to force prime to determine the ambiguity according to your choices.
<pre>
indexing_ambiguity {
mode = Forced
index_basis_in = None
assigned_basis = -h,l,k
assigned_basis = -k, l, h
n_sample_frames = 300
n_selected_frames = 100
}
</pre>

When you set indexing_ambiguity.mode to Forced, you can assign indexing choices according to your problem. In this example, two more choices (-h, l, k and -k, l, h) were assigned as the indexing choice.

At the end of the run, your solution pickle is saved to your_run_no/index_ambiguity/solution_pickle.pickle. If you don't want to spend time solving the ambiguity again in the next run, you can reuse this solution pickle by setting these parameters:
<pre>
indexing_ambiguity {
mode = Auto
index_basis_in = your_run_no/index_ambiguity/solution_pickle.pickle
}
</pre>

This will bypass the indexing ambiguity module. Prime will use the solution file to perform normal post-refinement and merging.

To use another isomorphous dataset (e.g. from synchrotron experiment), you can specify the mtz file as part of these parameters:
<pre>
indexing_ambiguity {
mode = Auto
index_basis_in = path/to/your/mtz/file.mtz
}
</pre>
Again, you can choose to do Auto or Forced (with a list of assigned_basis parameters) depending on your problem.

== More detail with input parameters ==
Now that you have your first trial merged data set, you can explore different parameter settings to merge or to obtain the Bijvoet pairs (I+/I-) for your anomalous data set.

'''Anomalous data:'''
<pre>
target_anomalous_flag = True
</pre>
In the last cycle, prime will output a reflection set with I+ and I-.

'''Number of micro- and macrocycles'''
<pre>
n_postref_cycle = 3
n_postref_sub_cycle = 1
</pre>

'''Number of bins for merging statistics'''
<pre>
n_bins = 20
</pre>

== Help with input parameters ==

Most input parameters are self-explained. However, you can run -h switch to view help information for each parameter.
<pre>
prime.run -h
</pre>

== Running in manual mode ==
With the same phil file, you can run prime manually. This gives you more freedom in terms of parameter settings at different stages (generating reference set, post-refining images, and merging) or at different cycle of post-refinement.

'''Example A''': I want to generate a reference set then post-refine all the images on the '''scale factors only''' for '''three cycles''' then refine '''all parameters''' in the '''4th cycle'''. To do this, you can follow these steps:

To generate a reference set,
<pre>
prime.genref prime.phil
</pre>

To post-refine on scale factors only, modify your .phil file so that all parameters are turned ''off''.
<pre>
...
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
}
postref {
residual_threshold = 5
residual_threshold_xy = 5
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
allparams {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 3
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To refine all parameters one more cycle, update your .phil file again (flag_on = True)
<pre>
...
allparams {
flag_on = True
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 1
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To obtain the final merged mtz, run
<pre>
prime.merge prime.phil
</pre>

== Running on multiple nodes ==
For LCLS users (or other users with LSF bsub), you can use psana (or your) queuing system to parallelize the entire process. For example, if you want to run your job on 100 nodes using psanq, you can specify:
<pre>
queue {
mode = bsub
qname = psanaq
n_nodes = 100
}
timeout_seconds = 300
</pre>
Prime will divide all the images into 100 batches and submit them to different nodes. It will wait until all images in every batches are done before returning to the merging step (or the exit step in the manual mode). You can control timeout_seconds parameter to tell prime how long it should wait for all the image batches to finish. Usually, this timeout parameter is not used (all images should return before 300 seconds) but in case, you need to wait longer or shorter, you can modify this parameter.

Cctbx.prime

2016-08-31T19:45:53Z

Mona:

== Prime: '''p'''ost-'''r'''ef'''i'''nement and '''me'''rging ==
With the latest update, prime can be used to process data on multiple nodes (on a queuing system). At the moment, only LSF (bsub) is supported. See documentation below for more information on how to use the queuing system.

This major update replaces prime.postrefine with

<pre>
prime.run
</pre>

For auto mode, you can still use prime.run with your parameter phil file like before. For manual mode, the available sub commands in prime are:

<pre>
prime.genref #generates a reference set from given integration results
prime.postrefine #refines all images
prime.merge #merges all refined results for an mtz file
</pre>

You can choose to run these commands independently (ideally in the above order) using the same phil file. See [https://commons.wikimedia.org/wiki/File:Prime_flowcharts.tif "PRIME flowchart"]. This will give you the freedom to change something (e.g. set of parameters to refine, resolution cut-off, etc.) at different stages of the post-refinement and merging. See running prime in manual mode for more detail.

Step-by-step guidelines to post-refine and merge XFEL diffraction images. For more detail and citation, see
"Enabling X-ray Free Electron Laser Crystallography for Challenging Biological Systems from a Limited Number of Crystals"
[http://elifesciences.org/content/4/e05421 "DOI: http://dx.doi.org/10.7554/eLife.05421"]

== Prime is gui-ed ==
Thanks to Dr. Lyubimov, PRIME is also available as a Graphic User Interface program. Try it by running

<pre>
prime
</pre>

Click to see [https://commons.wikimedia.org/wiki/File:PRIME_main.png "PRIME main gui"] and [https://commons.wikimedia.org/wiki/File:PRIME_advanced_options.png "Advanced options"]

== Getting started ==
'''Generating input phil file''' 
Like most programs developed under ''cctbx'' framework, ''prime'' reads in input .phil file, which stores all the parameters needed to run post-refinement and merging steps. To generate the template .phil file, do the dry run by calling

<pre>
$ prime.run
</pre>

An example of the template .phil file:

<pre>
data = None
run_no = None
title = None
scale {
d_min = 0.1
d_max = 99
sigma_min = 1.5
}
...
</pre>

You can save the content of the output to any file name - in this tutorial, let's save it to thermolysin.phil.

'''First look at your phil file''' 
To run prime, set the required parameters to match with your experiments (you can leave other parameters with their default values - or just delete them from you .phil file). The most interesting parameters are shown below:

<pre>
data = /path/to/your/integarion/result/pickle_files
run_no = 001
title = First trial for thermolysin
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
}
postref {
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
allparams {
flag_on = False
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
}
merge {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
target_unit_cell = 93.99,93.99,130.87,90,90,120
target_space_group = P 61 2 2
pixel_size_mm = 0.102
</pre>

You should pay attention to d_min and d_max for the refinement and merging parameters. If you use IOTA to integrate the images, IOTA will output .phil file for prime that has the optimal resolution range. If not, a few trial-and-error runs may be required to get the best resolution range for your dataset. Use merging statistics output by prime and check the values of CC1/2 and I/sigI to find out your optimal resolution range.

Cell parameters (target_unit_cell and target_space_group) are required to run prime. Target cell parameter is used to remove some outlier images by controlling uc_tolerance parameter (the default value of tolerate range is 3% different). Space group parameter is used in removing outliers and merging with the given symmetry.

Don't forget also to change your pixel size in millimeters. Check what your detector is and note down its pixel size.

'''Running post-refinement in automatic mode''' 
Once you have the input .phil file, you can run ''prime'' by calling
<pre>
prime.run thermolysin.phil
</pre>
''Prime'' will post-refine and merge for reflection sets using three (default value) macrocycles. At the end of the run, you can obtain merging statistics in the last cycle - all other cycle statistics are also available in log.txt.

An example of merging statistics:
<pre>
Summary for 001/postref_cycle_1_merge.mtz
Bin Resolution Range Completeness <N_obs> |Rsplit CC1/2 N_ind |CCanom N_ind| <I/sigI> 
-------------------------------------------------------------------------------------------------------------
02 5.70 - 4.52 100.0 1055 / 1055 65.89 16.02 89.15 1055 0.00 0 20.17 2101.97
03 4.52 - 3.95 100.0 1032 / 1032 61.53 14.48 92.03 1032 0.00 0 20.39 2529.90
04 3.95 - 3.59 100.0 1016 / 1016 54.15 15.61 90.13 1016 0.00 0 16.69 1971.43
05 3.59 - 3.33 100.0 1004 / 1004 42.67 17.66 89.23 1004 0.00 0 14.21 1502.14
06 3.33 - 3.14 100.0 1013 / 1013 32.77 20.40 84.26 1013 0.00 0 11.76 1077.60
07 3.14 - 2.98 100.0 995 / 995 27.36 23.00 78.72 995 0.00 0 11.58 935.37
08 2.98 - 2.85 100.0 1006 / 1006 23.57 22.63 82.26 1006 0.00 0 10.56 722.62
09 2.85 - 2.74 100.0 986 / 986 16.64 28.51 72.90 985 0.00 0 10.01 591.56
10 2.74 - 2.65 99.9 989 / 990 12.41 31.35 72.95 987 0.00 0 9.91 515.07
11 2.65 - 2.56 99.7 979 / 982 9.35 37.14 65.31 970 0.00 0 9.31 438.96
12 2.56 - 2.49 98.0 979 / 999 6.06 45.98 45.37 930 0.00 0 9.45 390.05
13 2.49 - 2.42 95.1 931 / 979 4.46 50.68 34.20 834 0.00 0 8.93 334.80
14 2.42 - 2.37 91.7 896 / 977 3.35 55.66 37.15 729 0.00 0 9.27 320.17
15 2.37 - 2.31 83.9 829 / 988 2.61 56.92 43.21 600 0.00 0 9.60 296.67
16 2.31 - 2.26 72.4 702 / 969 1.97 65.81 26.89 386 0.00 0 10.29 284.39
17 2.26 - 2.22 59.1 582 / 985 1.75 64.72 31.28 275 0.00 0 9.87 284.06
18 2.22 - 2.18 52.9 513 / 970 1.51 71.27 16.86 188 0.00 0 8.93 215.31
19 2.18 - 2.14 35.7 349 / 978 1.32 62.26 68.25 90 0.00 0 8.22 199.09
20 2.14 - 2.10 23.1 227 / 981 1.20 92.14 -9.20 42 0.00 0 8.59 224.44
-------------------------------------------------------------------------------------------------------------
TOTAL 85.9 17224 / 20046 27.11 21.11 92.07 15305 0.00 0 12.87 999.53
-------------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 1809
No. bad cc frames: 153
No. bad G frames) : 0
No. bad unit cell frames: 5
No. bad gamma_e frames: 0
No. bad SE: 0
No. observations: 466997

</pre>

== Solving indexing ambiguity (New) ==
With the latest version (Aug 31, 2016), you can now solve indexing ambiguity directly in prime. The Brehm & Diederichs algorhtms (doi:10.1107/S1399004713025431) have been implemented with boostrap capability to handle large dataset.
For merohedral twining group, the indexing choices will be determined automatically in prime. Use this default settings in your .phil input parameter file,

<pre>
indexing_ambiguity {
mode = Auto
index_basis_in = None
assigned_basis = None
n_sample_frames = 300
n_selected_frames = 100
}
</pre>

The n_sample_frames parameter indicates no. of images that will be used for the calculation of the scoring function. After that, only n_selected_images will be used in the B&D algorithms. This saves a lot of computing time since only the good images will be used for the determination of the ambiguity. You can change these two parameters to fit with your experiments. The default values are 300 and 100.

For pseudo-merohedral twinning, due to different possibilities for the indexing choice, prime doesn't determine these choices automatically. If you suspect that you may have pseudo twinning (b and c are similar, beta angle is almost 90 degree but not quite), you have an option to force prime to determine the ambiguity according to your choices.
<pre>

</pre>

== More detail with input parameters ==
Now that you have your first trial merged data set, you can explore different parameter settings to merge or to obtain the Bijvoet pairs (I+/I-) for your anomalous data set.

'''Anomalous data:'''
<pre>
target_anomalous_flag = True
</pre>
In the last cycle, prime will output a reflection set with I+ and I-.

'''Number of micro- and macrocycles'''
<pre>
n_postref_cycle = 3
n_postref_sub_cycle = 1
</pre>

'''Number of bins for merging statistics'''
<pre>
n_bins = 20
</pre>

== Help with input parameters ==

Most input parameters are self-explained. However, you can run -h switch to view help information for each parameter.
<pre>
prime.run -h
</pre>

== Running in manual mode ==
With the same phil file, you can run prime manually. This gives you more freedom in terms of parameter settings at different stages (generating reference set, post-refining images, and merging) or at different cycle of post-refinement.

'''Example A''': I want to generate a reference set then post-refine all the images on the '''scale factors only''' for '''three cycles''' then refine '''all parameters''' in the '''4th cycle'''. To do this, you can follow these steps:

To generate a reference set,
<pre>
prime.genref prime.phil
</pre>

To post-refine on scale factors only, modify your .phil file so that all parameters are turned ''off''.
<pre>
...
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
}
postref {
residual_threshold = 5
residual_threshold_xy = 5
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
allparams {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 3
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To refine all parameters one more cycle, update your .phil file again (flag_on = True)
<pre>
...
allparams {
flag_on = True
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 1
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To obtain the final merged mtz, run
<pre>
prime.merge prime.phil
</pre>

== Running on multiple nodes ==
For LCLS users (or other users with LSF bsub), you can use psana (or your) queuing system to parallelize the entire process. For example, if you want to run your job on 100 nodes using psanq, you can specify:
<pre>
queue {
mode = bsub
qname = psanaq
n_nodes = 100
}
timeout_seconds = 300
</pre>
Prime will divide all the images into 100 batches and submit them to different nodes. It will wait until all images in every batches are done before returning to the merging step (or the exit step in the manual mode). You can control timeout_seconds parameter to tell prime how long it should wait for all the image batches to finish. Usually, this timeout parameter is not used (all images should return before 300 seconds) but in case, you need to wait longer or shorter, you can modify this parameter.

Cctbx.prime

2016-08-31T19:31:00Z

Mona: /* Prime is gui-ed (new!) */

== Prime: '''p'''ost-'''r'''ef'''i'''nement and '''me'''rging ==
With the latest update, prime can be used to process data on multiple nodes (on a queuing system). At the moment, only LSF (bsub) is supported. See documentation below for more information on how to use the queuing system.

This major update replaces prime.postrefine with

<pre>
prime.run
</pre>

For auto mode, you can still use prime.run with your parameter phil file like before. For manual mode, the available sub commands in prime are:

<pre>
prime.genref #generates a reference set from given integration results
prime.postrefine #refines all images
prime.merge #merges all refined results for an mtz file
</pre>

You can choose to run these commands independently (ideally in the above order) using the same phil file. See [https://commons.wikimedia.org/wiki/File:Prime_flowcharts.tif "PRIME flowchart"]. This will give you the freedom to change something (e.g. set of parameters to refine, resolution cut-off, etc.) at different stages of the post-refinement and merging. See running prime in manual mode for more detail.

Step-by-step guidelines to post-refine and merge XFEL diffraction images. For more detail and citation, see
"Enabling X-ray Free Electron Laser Crystallography for Challenging Biological Systems from a Limited Number of Crystals"
[http://elifesciences.org/content/4/e05421 "DOI: http://dx.doi.org/10.7554/eLife.05421"]

== Prime is gui-ed ==
Thanks to Dr. Lyubimov, PRIME is also available as a Graphic User Interface program. Try it by running

<pre>
prime
</pre>

Click to see [https://commons.wikimedia.org/wiki/File:PRIME_main.png "PRIME main gui"] and [https://commons.wikimedia.org/wiki/File:PRIME_advanced_options.png "Advanced options"]

== Getting started ==
'''Generating input phil file''' 
Like most programs developed under ''cctbx'' framework, ''prime'' reads in input .phil file, which stores all the parameters needed to run post-refinement and merging steps. To generate the template .phil file, do the dry run by calling

<pre>
$ prime.run
</pre>

An example of the template .phil file:

<pre>
data = None
run_no = None
title = None
scale {
d_min = 0.1
d_max = 99
sigma_min = 1.5
}
...
</pre>

You can save the content of the output to any file name - in this tutorial, let's save it to thermolysin.phil.

'''First look at your phil file''' 
To run prime, set the required parameters to match with your experiments (you can leave other parameters with their default values - or just delete them from you .phil file). The most interesting parameters are shown below:

<pre>
data = /path/to/your/integarion/result/pickle_files
run_no = 001
title = First trial for thermolysin
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
}
postref {
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
allparams {
flag_on = False
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
}
merge {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
target_unit_cell = 93.99,93.99,130.87,90,90,120
target_space_group = P 61 2 2
pixel_size_mm = 0.102
</pre>

You should pay attention to d_min and d_max for the refinement and merging parameters. If you use IOTA to integrate the images, IOTA will output .phil file for prime that has the optimal resolution range. If not, a few trial-and-error runs may be required to get the best resolution range for your dataset. Use merging statistics output by prime and check the values of CC1/2 and I/sigI to find out your optimal resolution range.

Cell parameters (target_unit_cell and target_space_group) are required to run prime. Target cell parameter is used to remove some outlier images by controlling uc_tolerance parameter (the default value of tolerate range is 3% different). Space group parameter is used in removing outliers and merging with the given symmetry.

Don't forget also to change your pixel size in millimeters. Check what your detector is and note down its pixel size.

'''Running post-refinement in automatic mode''' 
Once you have the input .phil file, you can run ''prime'' by calling
<pre>
prime.run thermolysin.phil
</pre>
''Prime'' will post-refine and merge for reflection sets using three (default value) macrocycles. At the end of the run, you can obtain merging statistics in the last cycle - all other cycle statistics are also available in log.txt.

An example of merging statistics:
<pre>
Summary for 001/postref_cycle_1_merge.mtz
Bin Resolution Range Completeness <N_obs> |Rsplit CC1/2 N_ind |CCanom N_ind| <I/sigI> 
-------------------------------------------------------------------------------------------------------------
02 5.70 - 4.52 100.0 1055 / 1055 65.89 16.02 89.15 1055 0.00 0 20.17 2101.97
03 4.52 - 3.95 100.0 1032 / 1032 61.53 14.48 92.03 1032 0.00 0 20.39 2529.90
04 3.95 - 3.59 100.0 1016 / 1016 54.15 15.61 90.13 1016 0.00 0 16.69 1971.43
05 3.59 - 3.33 100.0 1004 / 1004 42.67 17.66 89.23 1004 0.00 0 14.21 1502.14
06 3.33 - 3.14 100.0 1013 / 1013 32.77 20.40 84.26 1013 0.00 0 11.76 1077.60
07 3.14 - 2.98 100.0 995 / 995 27.36 23.00 78.72 995 0.00 0 11.58 935.37
08 2.98 - 2.85 100.0 1006 / 1006 23.57 22.63 82.26 1006 0.00 0 10.56 722.62
09 2.85 - 2.74 100.0 986 / 986 16.64 28.51 72.90 985 0.00 0 10.01 591.56
10 2.74 - 2.65 99.9 989 / 990 12.41 31.35 72.95 987 0.00 0 9.91 515.07
11 2.65 - 2.56 99.7 979 / 982 9.35 37.14 65.31 970 0.00 0 9.31 438.96
12 2.56 - 2.49 98.0 979 / 999 6.06 45.98 45.37 930 0.00 0 9.45 390.05
13 2.49 - 2.42 95.1 931 / 979 4.46 50.68 34.20 834 0.00 0 8.93 334.80
14 2.42 - 2.37 91.7 896 / 977 3.35 55.66 37.15 729 0.00 0 9.27 320.17
15 2.37 - 2.31 83.9 829 / 988 2.61 56.92 43.21 600 0.00 0 9.60 296.67
16 2.31 - 2.26 72.4 702 / 969 1.97 65.81 26.89 386 0.00 0 10.29 284.39
17 2.26 - 2.22 59.1 582 / 985 1.75 64.72 31.28 275 0.00 0 9.87 284.06
18 2.22 - 2.18 52.9 513 / 970 1.51 71.27 16.86 188 0.00 0 8.93 215.31
19 2.18 - 2.14 35.7 349 / 978 1.32 62.26 68.25 90 0.00 0 8.22 199.09
20 2.14 - 2.10 23.1 227 / 981 1.20 92.14 -9.20 42 0.00 0 8.59 224.44
-------------------------------------------------------------------------------------------------------------
TOTAL 85.9 17224 / 20046 27.11 21.11 92.07 15305 0.00 0 12.87 999.53
-------------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 1809
No. bad cc frames: 153
No. bad G frames) : 0
No. bad unit cell frames: 5
No. bad gamma_e frames: 0
No. bad SE: 0
No. observations: 466997

</pre>

== More detail with input parameters ==
Now that you have your first trial merged data set, you can explore different parameter settings to merge or to obtain the Bijvoet pairs (I+/I-) for your anomalous data set.

'''Anomalous data:'''
<pre>
target_anomalous_flag = True
</pre>
In the last cycle, prime will output a reflection set with I+ and I-.

'''Indexing ambiguity'''

For space groups with indexing ambiguity, use the solutions from cctbx.xfel (see Tutorial for resolving indexing ambiguity) to merge the data set.
<pre>
indexing_ambiguity {
flag_on = True
index_basis_in = /path/to/solution/pickle_file.pickle
}
</pre>

'''Number of micro- and macrocycles'''
<pre>
n_postref_cycle = 3
n_postref_sub_cycle = 3
</pre>

'''Number of bins for merging statistics'''
<pre>
n_bins = 20
</pre>

== Help with input parameters ==

Most input parameters are self-explained. However, you can run -h switch to view help information for each parameter.
<pre>
prime.run -h
</pre>

== Running in manual mode ==
With the same phil file, you can run prime manually. This gives you more freedom in terms of parameter settings at different stages (generating reference set, post-refining images, and merging) or at different cycle of post-refinement.

'''Example A''': I want to generate a reference set then post-refine all the images on the '''scale factors only''' for '''three cycles''' then refine '''all parameters''' in the '''4th cycle'''. To do this, you can follow these steps:

To generate a reference set,
<pre>
prime.genref prime.phil
</pre>

To post-refine on scale factors only, modify your .phil file so that all parameters are turned ''off''.
<pre>
...
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
}
postref {
residual_threshold = 5
residual_threshold_xy = 5
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
allparams {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 3
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To refine all parameters one more cycle, update your .phil file again (flag_on = True)
<pre>
...
allparams {
flag_on = True
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 1
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To obtain the final merged mtz, run
<pre>
prime.merge prime.phil
</pre>

== Running on multiple nodes ==
For LCLS users (or other users with LSF bsub), you can use psana (or your) queuing system to parallelize the entire process. For example, if you want to run your job on 100 nodes using psanq, you can specify:
<pre>
queue {
mode = bsub
qname = psanaq
n_nodes = 100
}
timeout_seconds = 300
</pre>
Prime will divide all the images into 100 batches and submit them to different nodes. It will wait until all images in every batches are done before returning to the merging step (or the exit step in the manual mode). You can control timeout_seconds parameter to tell prime how long it should wait for all the image batches to finish. Usually, this timeout parameter is not used (all images should return before 300 seconds) but in case, you need to wait longer or shorter, you can modify this parameter.

Cctbx.prime

2016-08-02T01:06:12Z

Mona:

== Prime: '''p'''ost-'''r'''ef'''i'''nement and '''me'''rging ==
With the latest update, prime can be used to process data on multiple nodes (on a queuing system). At the moment, only LSF (bsub) is supported. See documentation below for more information on how to use the queuing system.

This major update replaces prime.postrefine with

<pre>
prime.run
</pre>

For auto mode, you can still use prime.run with your parameter phil file like before. For manual mode, the available sub commands in prime are:

<pre>
prime.genref #generates a reference set from given integration results
prime.postrefine #refines all images
prime.merge #merges all refined results for an mtz file
</pre>

You can choose to run these commands independently (ideally in the above order) using the same phil file. See [https://commons.wikimedia.org/wiki/File:Prime_flowcharts.tif "PRIME flowchart"]. This will give you the freedom to change something (e.g. set of parameters to refine, resolution cut-off, etc.) at different stages of the post-refinement and merging. See running prime in manual mode for more detail.

Step-by-step guidelines to post-refine and merge XFEL diffraction images. For more detail and citation, see
"Enabling X-ray Free Electron Laser Crystallography for Challenging Biological Systems from a Limited Number of Crystals"
[http://elifesciences.org/content/4/e05421 "DOI: http://dx.doi.org/10.7554/eLife.05421"]

== Prime is gui-ed (new!)==
Thanks to Dr. Lyubimov, PRIME is also available as a Graphic User Interface program. Try it by running

<pre>
prime
</pre>

Click to see [https://commons.wikimedia.org/wiki/File:PRIME_main.png "PRIME main gui"] and [https://commons.wikimedia.org/wiki/File:PRIME_advanced_options.png "Advanced options"]

== Getting started ==
'''Generating input phil file''' 
Like most programs developed under ''cctbx'' framework, ''prime'' reads in input .phil file, which stores all the parameters needed to run post-refinement and merging steps. To generate the template .phil file, do the dry run by calling

<pre>
$ prime.run
</pre>

An example of the template .phil file:

<pre>
data = None
run_no = None
title = None
scale {
d_min = 0.1
d_max = 99
sigma_min = 1.5
}
...
</pre>

You can save the content of the output to any file name - in this tutorial, let's save it to thermolysin.phil.

'''First look at your phil file''' 
To run prime, set the required parameters to match with your experiments (you can leave other parameters with their default values - or just delete them from you .phil file). The most interesting parameters are shown below:

<pre>
data = /path/to/your/integarion/result/pickle_files
run_no = 001
title = First trial for thermolysin
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
}
postref {
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
allparams {
flag_on = False
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
}
merge {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
target_unit_cell = 93.99,93.99,130.87,90,90,120
target_space_group = P 61 2 2
pixel_size_mm = 0.102
</pre>

You should pay attention to d_min and d_max for the refinement and merging parameters. If you use IOTA to integrate the images, IOTA will output .phil file for prime that has the optimal resolution range. If not, a few trial-and-error runs may be required to get the best resolution range for your dataset. Use merging statistics output by prime and check the values of CC1/2 and I/sigI to find out your optimal resolution range.

Cell parameters (target_unit_cell and target_space_group) are required to run prime. Target cell parameter is used to remove some outlier images by controlling uc_tolerance parameter (the default value of tolerate range is 3% different). Space group parameter is used in removing outliers and merging with the given symmetry.

Don't forget also to change your pixel size in millimeters. Check what your detector is and note down its pixel size.

'''Running post-refinement in automatic mode''' 
Once you have the input .phil file, you can run ''prime'' by calling
<pre>
prime.run thermolysin.phil
</pre>
''Prime'' will post-refine and merge for reflection sets using three (default value) macrocycles. At the end of the run, you can obtain merging statistics in the last cycle - all other cycle statistics are also available in log.txt.

An example of merging statistics:
<pre>
Summary for 001/postref_cycle_1_merge.mtz
Bin Resolution Range Completeness <N_obs> |Rsplit CC1/2 N_ind |CCanom N_ind| <I/sigI> 
-------------------------------------------------------------------------------------------------------------
02 5.70 - 4.52 100.0 1055 / 1055 65.89 16.02 89.15 1055 0.00 0 20.17 2101.97
03 4.52 - 3.95 100.0 1032 / 1032 61.53 14.48 92.03 1032 0.00 0 20.39 2529.90
04 3.95 - 3.59 100.0 1016 / 1016 54.15 15.61 90.13 1016 0.00 0 16.69 1971.43
05 3.59 - 3.33 100.0 1004 / 1004 42.67 17.66 89.23 1004 0.00 0 14.21 1502.14
06 3.33 - 3.14 100.0 1013 / 1013 32.77 20.40 84.26 1013 0.00 0 11.76 1077.60
07 3.14 - 2.98 100.0 995 / 995 27.36 23.00 78.72 995 0.00 0 11.58 935.37
08 2.98 - 2.85 100.0 1006 / 1006 23.57 22.63 82.26 1006 0.00 0 10.56 722.62
09 2.85 - 2.74 100.0 986 / 986 16.64 28.51 72.90 985 0.00 0 10.01 591.56
10 2.74 - 2.65 99.9 989 / 990 12.41 31.35 72.95 987 0.00 0 9.91 515.07
11 2.65 - 2.56 99.7 979 / 982 9.35 37.14 65.31 970 0.00 0 9.31 438.96
12 2.56 - 2.49 98.0 979 / 999 6.06 45.98 45.37 930 0.00 0 9.45 390.05
13 2.49 - 2.42 95.1 931 / 979 4.46 50.68 34.20 834 0.00 0 8.93 334.80
14 2.42 - 2.37 91.7 896 / 977 3.35 55.66 37.15 729 0.00 0 9.27 320.17
15 2.37 - 2.31 83.9 829 / 988 2.61 56.92 43.21 600 0.00 0 9.60 296.67
16 2.31 - 2.26 72.4 702 / 969 1.97 65.81 26.89 386 0.00 0 10.29 284.39
17 2.26 - 2.22 59.1 582 / 985 1.75 64.72 31.28 275 0.00 0 9.87 284.06
18 2.22 - 2.18 52.9 513 / 970 1.51 71.27 16.86 188 0.00 0 8.93 215.31
19 2.18 - 2.14 35.7 349 / 978 1.32 62.26 68.25 90 0.00 0 8.22 199.09
20 2.14 - 2.10 23.1 227 / 981 1.20 92.14 -9.20 42 0.00 0 8.59 224.44
-------------------------------------------------------------------------------------------------------------
TOTAL 85.9 17224 / 20046 27.11 21.11 92.07 15305 0.00 0 12.87 999.53
-------------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 1809
No. bad cc frames: 153
No. bad G frames) : 0
No. bad unit cell frames: 5
No. bad gamma_e frames: 0
No. bad SE: 0
No. observations: 466997

</pre>

== More detail with input parameters ==
Now that you have your first trial merged data set, you can explore different parameter settings to merge or to obtain the Bijvoet pairs (I+/I-) for your anomalous data set.

'''Anomalous data:'''
<pre>
target_anomalous_flag = True
</pre>
In the last cycle, prime will output a reflection set with I+ and I-.

'''Indexing ambiguity'''

For space groups with indexing ambiguity, use the solutions from cctbx.xfel (see Tutorial for resolving indexing ambiguity) to merge the data set.
<pre>
indexing_ambiguity {
flag_on = True
index_basis_in = /path/to/solution/pickle_file.pickle
}
</pre>

'''Number of micro- and macrocycles'''
<pre>
n_postref_cycle = 3
n_postref_sub_cycle = 3
</pre>

'''Number of bins for merging statistics'''
<pre>
n_bins = 20
</pre>

== Help with input parameters ==

Most input parameters are self-explained. However, you can run -h switch to view help information for each parameter.
<pre>
prime.run -h
</pre>

== Running in manual mode ==
With the same phil file, you can run prime manually. This gives you more freedom in terms of parameter settings at different stages (generating reference set, post-refining images, and merging) or at different cycle of post-refinement.

'''Example A''': I want to generate a reference set then post-refine all the images on the '''scale factors only''' for '''three cycles''' then refine '''all parameters''' in the '''4th cycle'''. To do this, you can follow these steps:

To generate a reference set,
<pre>
prime.genref prime.phil
</pre>

To post-refine on scale factors only, modify your .phil file so that all parameters are turned ''off''.
<pre>
...
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
}
postref {
residual_threshold = 5
residual_threshold_xy = 5
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
allparams {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 3
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To refine all parameters one more cycle, update your .phil file again (flag_on = True)
<pre>
...
allparams {
flag_on = True
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 1
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To obtain the final merged mtz, run
<pre>
prime.merge prime.phil
</pre>

== Running on multiple nodes ==
For LCLS users (or other users with LSF bsub), you can use psana (or your) queuing system to parallelize the entire process. For example, if you want to run your job on 100 nodes using psanq, you can specify:
<pre>
queue {
mode = bsub
qname = psanaq
n_nodes = 100
}
timeout_seconds = 300
</pre>
Prime will divide all the images into 100 batches and submit them to different nodes. It will wait until all images in every batches are done before returning to the merging step (or the exit step in the manual mode). You can control timeout_seconds parameter to tell prime how long it should wait for all the image batches to finish. Usually, this timeout parameter is not used (all images should return before 300 seconds) but in case, you need to wait longer or shorter, you can modify this parameter.

Cctbx.prime

2016-08-02T01:05:46Z

Mona: /* Prime: post-refinement and merging */

== Prime: '''p'''ost-'''r'''ef'''i'''nement and '''me'''rging ==
With the latest update, prime can be used to process data on multiple nodes (on a queuing system). At the moment, only LSF (bsub) is supported. See documentation below for more information how to use the queuing system.

This major update replaces prime.postrefine with

<pre>
prime.run
</pre>

For auto mode, you can still use prime.run with your parameter phil file like before. For manual mode, the available sub commands in prime are:

<pre>
prime.genref #generates a reference set from given integration results
prime.postrefine #refines all images
prime.merge #merges all refined results for an mtz file
</pre>

You can choose to run these commands independently (ideally in the above order) using the same phil file. See [https://commons.wikimedia.org/wiki/File:Prime_flowcharts.tif "PRIME flowchart"]. This will give you the freedom to change something (e.g. set of parameters to refine, resolution cut-off, etc.) at different stages of the post-refinement and merging. See running prime in manual mode for more detail.

Step-by-step guidelines to post-refine and merge XFEL diffraction images. For more detail and citation, see
"Enabling X-ray Free Electron Laser Crystallography for Challenging Biological Systems from a Limited Number of Crystals"
[http://elifesciences.org/content/4/e05421 "DOI: http://dx.doi.org/10.7554/eLife.05421"]

== Prime is gui-ed (new!)==
Thanks to Dr. Lyubimov, PRIME is also available as a Graphic User Interface program. Try it by running

<pre>
prime
</pre>

Click to see [https://commons.wikimedia.org/wiki/File:PRIME_main.png "PRIME main gui"] and [https://commons.wikimedia.org/wiki/File:PRIME_advanced_options.png "Advanced options"]

== Getting started ==
'''Generating input phil file''' 
Like most programs developed under ''cctbx'' framework, ''prime'' reads in input .phil file, which stores all the parameters needed to run post-refinement and merging steps. To generate the template .phil file, do the dry run by calling

<pre>
$ prime.run
</pre>

An example of the template .phil file:

<pre>
data = None
run_no = None
title = None
scale {
d_min = 0.1
d_max = 99
sigma_min = 1.5
}
...
</pre>

You can save the content of the output to any file name - in this tutorial, let's save it to thermolysin.phil.

'''First look at your phil file''' 
To run prime, set the required parameters to match with your experiments (you can leave other parameters with their default values - or just delete them from you .phil file). The most interesting parameters are shown below:

<pre>
data = /path/to/your/integarion/result/pickle_files
run_no = 001
title = First trial for thermolysin
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
}
postref {
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
allparams {
flag_on = False
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
}
merge {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
target_unit_cell = 93.99,93.99,130.87,90,90,120
target_space_group = P 61 2 2
pixel_size_mm = 0.102
</pre>

You should pay attention to d_min and d_max for the refinement and merging parameters. If you use IOTA to integrate the images, IOTA will output .phil file for prime that has the optimal resolution range. If not, a few trial-and-error runs may be required to get the best resolution range for your dataset. Use merging statistics output by prime and check the values of CC1/2 and I/sigI to find out your optimal resolution range.

Cell parameters (target_unit_cell and target_space_group) are required to run prime. Target cell parameter is used to remove some outlier images by controlling uc_tolerance parameter (the default value of tolerate range is 3% different). Space group parameter is used in removing outliers and merging with the given symmetry.

Don't forget also to change your pixel size in millimeters. Check what your detector is and note down its pixel size.

'''Running post-refinement in automatic mode''' 
Once you have the input .phil file, you can run ''prime'' by calling
<pre>
prime.run thermolysin.phil
</pre>
''Prime'' will post-refine and merge for reflection sets using three (default value) macrocycles. At the end of the run, you can obtain merging statistics in the last cycle - all other cycle statistics are also available in log.txt.

An example of merging statistics:
<pre>
Summary for 001/postref_cycle_1_merge.mtz
Bin Resolution Range Completeness <N_obs> |Rsplit CC1/2 N_ind |CCanom N_ind| <I/sigI> 
-------------------------------------------------------------------------------------------------------------
02 5.70 - 4.52 100.0 1055 / 1055 65.89 16.02 89.15 1055 0.00 0 20.17 2101.97
03 4.52 - 3.95 100.0 1032 / 1032 61.53 14.48 92.03 1032 0.00 0 20.39 2529.90
04 3.95 - 3.59 100.0 1016 / 1016 54.15 15.61 90.13 1016 0.00 0 16.69 1971.43
05 3.59 - 3.33 100.0 1004 / 1004 42.67 17.66 89.23 1004 0.00 0 14.21 1502.14
06 3.33 - 3.14 100.0 1013 / 1013 32.77 20.40 84.26 1013 0.00 0 11.76 1077.60
07 3.14 - 2.98 100.0 995 / 995 27.36 23.00 78.72 995 0.00 0 11.58 935.37
08 2.98 - 2.85 100.0 1006 / 1006 23.57 22.63 82.26 1006 0.00 0 10.56 722.62
09 2.85 - 2.74 100.0 986 / 986 16.64 28.51 72.90 985 0.00 0 10.01 591.56
10 2.74 - 2.65 99.9 989 / 990 12.41 31.35 72.95 987 0.00 0 9.91 515.07
11 2.65 - 2.56 99.7 979 / 982 9.35 37.14 65.31 970 0.00 0 9.31 438.96
12 2.56 - 2.49 98.0 979 / 999 6.06 45.98 45.37 930 0.00 0 9.45 390.05
13 2.49 - 2.42 95.1 931 / 979 4.46 50.68 34.20 834 0.00 0 8.93 334.80
14 2.42 - 2.37 91.7 896 / 977 3.35 55.66 37.15 729 0.00 0 9.27 320.17
15 2.37 - 2.31 83.9 829 / 988 2.61 56.92 43.21 600 0.00 0 9.60 296.67
16 2.31 - 2.26 72.4 702 / 969 1.97 65.81 26.89 386 0.00 0 10.29 284.39
17 2.26 - 2.22 59.1 582 / 985 1.75 64.72 31.28 275 0.00 0 9.87 284.06
18 2.22 - 2.18 52.9 513 / 970 1.51 71.27 16.86 188 0.00 0 8.93 215.31
19 2.18 - 2.14 35.7 349 / 978 1.32 62.26 68.25 90 0.00 0 8.22 199.09
20 2.14 - 2.10 23.1 227 / 981 1.20 92.14 -9.20 42 0.00 0 8.59 224.44
-------------------------------------------------------------------------------------------------------------
TOTAL 85.9 17224 / 20046 27.11 21.11 92.07 15305 0.00 0 12.87 999.53
-------------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 1809
No. bad cc frames: 153
No. bad G frames) : 0
No. bad unit cell frames: 5
No. bad gamma_e frames: 0
No. bad SE: 0
No. observations: 466997

</pre>

== More detail with input parameters ==
Now that you have your first trial merged data set, you can explore different parameter settings to merge or to obtain the Bijvoet pairs (I+/I-) for your anomalous data set.

'''Anomalous data:'''
<pre>
target_anomalous_flag = True
</pre>
In the last cycle, prime will output a reflection set with I+ and I-.

'''Indexing ambiguity'''

For space groups with indexing ambiguity, use the solutions from cctbx.xfel (see Tutorial for resolving indexing ambiguity) to merge the data set.
<pre>
indexing_ambiguity {
flag_on = True
index_basis_in = /path/to/solution/pickle_file.pickle
}
</pre>

'''Number of micro- and macrocycles'''
<pre>
n_postref_cycle = 3
n_postref_sub_cycle = 3
</pre>

'''Number of bins for merging statistics'''
<pre>
n_bins = 20
</pre>

== Help with input parameters ==

Most input parameters are self-explained. However, you can run -h switch to view help information for each parameter.
<pre>
prime.run -h
</pre>

== Running in manual mode ==
With the same phil file, you can run prime manually. This gives you more freedom in terms of parameter settings at different stages (generating reference set, post-refining images, and merging) or at different cycle of post-refinement.

'''Example A''': I want to generate a reference set then post-refine all the images on the '''scale factors only''' for '''three cycles''' then refine '''all parameters''' in the '''4th cycle'''. To do this, you can follow these steps:

To generate a reference set,
<pre>
prime.genref prime.phil
</pre>

To post-refine on scale factors only, modify your .phil file so that all parameters are turned ''off''.
<pre>
...
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
}
postref {
residual_threshold = 5
residual_threshold_xy = 5
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
allparams {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 3
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To refine all parameters one more cycle, update your .phil file again (flag_on = True)
<pre>
...
allparams {
flag_on = True
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 1
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To obtain the final merged mtz, run
<pre>
prime.merge prime.phil
</pre>

== Running on multiple nodes ==
For LCLS users (or other users with LSF bsub), you can use psana (or your) queuing system to parallelize the entire process. For example, if you want to run your job on 100 nodes using psanq, you can specify:
<pre>
queue {
mode = bsub
qname = psanaq
n_nodes = 100
}
timeout_seconds = 300
</pre>
Prime will divide all the images into 100 batches and submit them to different nodes. It will wait until all images in every batches are done before returning to the merging step (or the exit step in the manual mode). You can control timeout_seconds parameter to tell prime how long it should wait for all the image batches to finish. Usually, this timeout parameter is not used (all images should return before 300 seconds) but in case, you need to wait longer or shorter, you can modify this parameter.

Cctbx.prime

2016-08-02T00:34:15Z

Mona: /* Prime: post-refinement and merging */

== Prime: '''p'''ost-'''r'''ef'''i'''nement and '''me'''rging ==
With the latest update, prime can be used to process data on multiple nodes (on queuing system). At the moment, only LSF (bsub) is supported. See documentation below for more information how to use the queuing system.

This major update replaces prime.postrefine with

<pre>
prime.run
</pre>

For auto mode, you can still use prime.run with your parameter phil file like before. For manual mode, the available sub commands in prime are:

<pre>
prime.genref #generates a reference set from given integration results
prime.postrefine #refines all images
prime.merge #merges all refined results for an mtz file
</pre>

You can choose to run these commands independently (ideally in the above order) using the same phil file. See [https://commons.wikimedia.org/wiki/File:Prime_flowcharts.tif "PRIME flowchart"]. This will give you the freedom to change something (e.g. set of parameters to refine, resolution cut-off, etc.) at different stages of the post-refinement and merging. See running prime in manual mode for more detail.

Step-by-step guidelines to post-refine and merge XFEL diffraction images. For more detail and citation, see
"Enabling X-ray Free Electron Laser Crystallography for Challenging Biological Systems from a Limited Number of Crystals"
[http://elifesciences.org/content/4/e05421 "DOI: http://dx.doi.org/10.7554/eLife.05421"]

== Prime is gui-ed (new!)==
Thanks to Dr. Lyubimov, PRIME is also available as a Graphic User Interface program. Try it by running

<pre>
prime
</pre>

Click to see [https://commons.wikimedia.org/wiki/File:PRIME_main.png "PRIME main gui"] and [https://commons.wikimedia.org/wiki/File:PRIME_advanced_options.png "Advanced options"]

== Getting started ==
'''Generating input phil file''' 
Like most programs developed under ''cctbx'' framework, ''prime'' reads in input .phil file, which stores all the parameters needed to run post-refinement and merging steps. To generate the template .phil file, do the dry run by calling

<pre>
$ prime.run
</pre>

An example of the template .phil file:

<pre>
data = None
run_no = None
title = None
scale {
d_min = 0.1
d_max = 99
sigma_min = 1.5
}
...
</pre>

You can save the content of the output to any file name - in this tutorial, let's save it to thermolysin.phil.

'''First look at your phil file''' 
To run prime, set the required parameters to match with your experiments (you can leave other parameters with their default values - or just delete them from you .phil file). The most interesting parameters are shown below:

<pre>
data = /path/to/your/integarion/result/pickle_files
run_no = 001
title = First trial for thermolysin
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
}
postref {
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
allparams {
flag_on = False
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
}
merge {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
target_unit_cell = 93.99,93.99,130.87,90,90,120
target_space_group = P 61 2 2
pixel_size_mm = 0.102
</pre>

You should pay attention to d_min and d_max for the refinement and merging parameters. If you use IOTA to integrate the images, IOTA will output .phil file for prime that has the optimal resolution range. If not, a few trial-and-error runs may be required to get the best resolution range for your dataset. Use merging statistics output by prime and check the values of CC1/2 and I/sigI to find out your optimal resolution range.

Cell parameters (target_unit_cell and target_space_group) are required to run prime. Target cell parameter is used to remove some outlier images by controlling uc_tolerance parameter (the default value of tolerate range is 3% different). Space group parameter is used in removing outliers and merging with the given symmetry.

Don't forget also to change your pixel size in millimeters. Check what your detector is and note down its pixel size.

'''Running post-refinement in automatic mode''' 
Once you have the input .phil file, you can run ''prime'' by calling
<pre>
prime.run thermolysin.phil
</pre>
''Prime'' will post-refine and merge for reflection sets using three (default value) macrocycles. At the end of the run, you can obtain merging statistics in the last cycle - all other cycle statistics are also available in log.txt.

An example of merging statistics:
<pre>
Summary for 001/postref_cycle_1_merge.mtz
Bin Resolution Range Completeness <N_obs> |Rsplit CC1/2 N_ind |CCanom N_ind| <I/sigI> 
-------------------------------------------------------------------------------------------------------------
02 5.70 - 4.52 100.0 1055 / 1055 65.89 16.02 89.15 1055 0.00 0 20.17 2101.97
03 4.52 - 3.95 100.0 1032 / 1032 61.53 14.48 92.03 1032 0.00 0 20.39 2529.90
04 3.95 - 3.59 100.0 1016 / 1016 54.15 15.61 90.13 1016 0.00 0 16.69 1971.43
05 3.59 - 3.33 100.0 1004 / 1004 42.67 17.66 89.23 1004 0.00 0 14.21 1502.14
06 3.33 - 3.14 100.0 1013 / 1013 32.77 20.40 84.26 1013 0.00 0 11.76 1077.60
07 3.14 - 2.98 100.0 995 / 995 27.36 23.00 78.72 995 0.00 0 11.58 935.37
08 2.98 - 2.85 100.0 1006 / 1006 23.57 22.63 82.26 1006 0.00 0 10.56 722.62
09 2.85 - 2.74 100.0 986 / 986 16.64 28.51 72.90 985 0.00 0 10.01 591.56
10 2.74 - 2.65 99.9 989 / 990 12.41 31.35 72.95 987 0.00 0 9.91 515.07
11 2.65 - 2.56 99.7 979 / 982 9.35 37.14 65.31 970 0.00 0 9.31 438.96
12 2.56 - 2.49 98.0 979 / 999 6.06 45.98 45.37 930 0.00 0 9.45 390.05
13 2.49 - 2.42 95.1 931 / 979 4.46 50.68 34.20 834 0.00 0 8.93 334.80
14 2.42 - 2.37 91.7 896 / 977 3.35 55.66 37.15 729 0.00 0 9.27 320.17
15 2.37 - 2.31 83.9 829 / 988 2.61 56.92 43.21 600 0.00 0 9.60 296.67
16 2.31 - 2.26 72.4 702 / 969 1.97 65.81 26.89 386 0.00 0 10.29 284.39
17 2.26 - 2.22 59.1 582 / 985 1.75 64.72 31.28 275 0.00 0 9.87 284.06
18 2.22 - 2.18 52.9 513 / 970 1.51 71.27 16.86 188 0.00 0 8.93 215.31
19 2.18 - 2.14 35.7 349 / 978 1.32 62.26 68.25 90 0.00 0 8.22 199.09
20 2.14 - 2.10 23.1 227 / 981 1.20 92.14 -9.20 42 0.00 0 8.59 224.44
-------------------------------------------------------------------------------------------------------------
TOTAL 85.9 17224 / 20046 27.11 21.11 92.07 15305 0.00 0 12.87 999.53
-------------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 1809
No. bad cc frames: 153
No. bad G frames) : 0
No. bad unit cell frames: 5
No. bad gamma_e frames: 0
No. bad SE: 0
No. observations: 466997

</pre>

== More detail with input parameters ==
Now that you have your first trial merged data set, you can explore different parameter settings to merge or to obtain the Bijvoet pairs (I+/I-) for your anomalous data set.

'''Anomalous data:'''
<pre>
target_anomalous_flag = True
</pre>
In the last cycle, prime will output a reflection set with I+ and I-.

'''Indexing ambiguity'''

For space groups with indexing ambiguity, use the solutions from cctbx.xfel (see Tutorial for resolving indexing ambiguity) to merge the data set.
<pre>
indexing_ambiguity {
flag_on = True
index_basis_in = /path/to/solution/pickle_file.pickle
}
</pre>

'''Number of micro- and macrocycles'''
<pre>
n_postref_cycle = 3
n_postref_sub_cycle = 3
</pre>

'''Number of bins for merging statistics'''
<pre>
n_bins = 20
</pre>

== Help with input parameters ==

Most input parameters are self-explained. However, you can run -h switch to view help information for each parameter.
<pre>
prime.run -h
</pre>

== Running in manual mode ==
With the same phil file, you can run prime manually. This gives you more freedom in terms of parameter settings at different stages (generating reference set, post-refining images, and merging) or at different cycle of post-refinement.

'''Example A''': I want to generate a reference set then post-refine all the images on the '''scale factors only''' for '''three cycles''' then refine '''all parameters''' in the '''4th cycle'''. To do this, you can follow these steps:

To generate a reference set,
<pre>
prime.genref prime.phil
</pre>

To post-refine on scale factors only, modify your .phil file so that all parameters are turned ''off''.
<pre>
...
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
}
postref {
residual_threshold = 5
residual_threshold_xy = 5
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
allparams {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 3
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To refine all parameters one more cycle, update your .phil file again (flag_on = True)
<pre>
...
allparams {
flag_on = True
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 1
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To obtain the final merged mtz, run
<pre>
prime.merge prime.phil
</pre>

== Running on multiple nodes ==
For LCLS users (or other users with LSF bsub), you can use psana (or your) queuing system to parallelize the entire process. For example, if you want to run your job on 100 nodes using psanq, you can specify:
<pre>
queue {
mode = bsub
qname = psanaq
n_nodes = 100
}
timeout_seconds = 300
</pre>
Prime will divide all the images into 100 batches and submit them to different nodes. It will wait until all images in every batches are done before returning to the merging step (or the exit step in the manual mode). You can control timeout_seconds parameter to tell prime how long it should wait for all the image batches to finish. Usually, this timeout parameter is not used (all images should return before 300 seconds) but in case, you need to wait longer or shorter, you can modify this parameter.

Cctbx.prime

2016-08-02T00:30:25Z

Mona:

== Prime: '''p'''ost-'''r'''ef'''i'''nement and '''me'''rging ==
With the latest update, prime can be used to process data on multiple nodes (on queuing system). At the moment, only LSF (bsub) is supported. See documentation below for more information how to use the queuing system.

This major update replaces prime.postrefine with

<pre>
prime.run
</pre>

For auto mode, you can still use prime.run with your parameter phil file like before. For manual mode, the available sub commands in prime are:

<pre>
prime.genref #generates a reference set from given integration results
prime.postrefine #refines all images
prime.merge #merges all refined results for an mtz file
</pre>

You can choose to run these commands independently (ideally in the above order) using the same phil file. See [https://commons.wikimedia.org/wiki/File:Prime_flowcharts.tif "PRIME flowchart"]. This will you a freedom to change something (e.g. set of parameters to refine, resolution cut-off, etc.) at different stages of the post-refinement and merging. See running prime in manual mode for more detail.

Step-by-step guidelines to post-refine and merge XFEL diffraction images. For more detail and citation, see
"Enabling X-ray Free Electron Laser Crystallography for Challenging Biological Systems from a Limited Number of Crystals"
[http://elifesciences.org/content/4/e05421 "DOI: http://dx.doi.org/10.7554/eLife.05421"]

== Prime is gui-ed (new!)==
Thanks to Dr. Lyubimov, PRIME is also available as a Graphic User Interface program. Try it by running

<pre>
prime
</pre>

Click to see [https://commons.wikimedia.org/wiki/File:PRIME_main.png "PRIME main gui"] and [https://commons.wikimedia.org/wiki/File:PRIME_advanced_options.png "Advanced options"]

== Getting started ==
'''Generating input phil file''' 
Like most programs developed under ''cctbx'' framework, ''prime'' reads in input .phil file, which stores all the parameters needed to run post-refinement and merging steps. To generate the template .phil file, do the dry run by calling

<pre>
$ prime.run
</pre>

An example of the template .phil file:

<pre>
data = None
run_no = None
title = None
scale {
d_min = 0.1
d_max = 99
sigma_min = 1.5
}
...
</pre>

You can save the content of the output to any file name - in this tutorial, let's save it to thermolysin.phil.

'''First look at your phil file''' 
To run prime, set the required parameters to match with your experiments (you can leave other parameters with their default values - or just delete them from you .phil file). The most interesting parameters are shown below:

<pre>
data = /path/to/your/integarion/result/pickle_files
run_no = 001
title = First trial for thermolysin
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
}
postref {
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
allparams {
flag_on = False
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
}
merge {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
target_unit_cell = 93.99,93.99,130.87,90,90,120
target_space_group = P 61 2 2
pixel_size_mm = 0.102
</pre>

You should pay attention to d_min and d_max for the refinement and merging parameters. If you use IOTA to integrate the images, IOTA will output .phil file for prime that has the optimal resolution range. If not, a few trial-and-error runs may be required to get the best resolution range for your dataset. Use merging statistics output by prime and check the values of CC1/2 and I/sigI to find out your optimal resolution range.

Cell parameters (target_unit_cell and target_space_group) are required to run prime. Target cell parameter is used to remove some outlier images by controlling uc_tolerance parameter (the default value of tolerate range is 3% different). Space group parameter is used in removing outliers and merging with the given symmetry.

Don't forget also to change your pixel size in millimeters. Check what your detector is and note down its pixel size.

'''Running post-refinement in automatic mode''' 
Once you have the input .phil file, you can run ''prime'' by calling
<pre>
prime.run thermolysin.phil
</pre>
''Prime'' will post-refine and merge for reflection sets using three (default value) macrocycles. At the end of the run, you can obtain merging statistics in the last cycle - all other cycle statistics are also available in log.txt.

An example of merging statistics:
<pre>
Summary for 001/postref_cycle_1_merge.mtz
Bin Resolution Range Completeness <N_obs> |Rsplit CC1/2 N_ind |CCanom N_ind| <I/sigI> 
-------------------------------------------------------------------------------------------------------------
02 5.70 - 4.52 100.0 1055 / 1055 65.89 16.02 89.15 1055 0.00 0 20.17 2101.97
03 4.52 - 3.95 100.0 1032 / 1032 61.53 14.48 92.03 1032 0.00 0 20.39 2529.90
04 3.95 - 3.59 100.0 1016 / 1016 54.15 15.61 90.13 1016 0.00 0 16.69 1971.43
05 3.59 - 3.33 100.0 1004 / 1004 42.67 17.66 89.23 1004 0.00 0 14.21 1502.14
06 3.33 - 3.14 100.0 1013 / 1013 32.77 20.40 84.26 1013 0.00 0 11.76 1077.60
07 3.14 - 2.98 100.0 995 / 995 27.36 23.00 78.72 995 0.00 0 11.58 935.37
08 2.98 - 2.85 100.0 1006 / 1006 23.57 22.63 82.26 1006 0.00 0 10.56 722.62
09 2.85 - 2.74 100.0 986 / 986 16.64 28.51 72.90 985 0.00 0 10.01 591.56
10 2.74 - 2.65 99.9 989 / 990 12.41 31.35 72.95 987 0.00 0 9.91 515.07
11 2.65 - 2.56 99.7 979 / 982 9.35 37.14 65.31 970 0.00 0 9.31 438.96
12 2.56 - 2.49 98.0 979 / 999 6.06 45.98 45.37 930 0.00 0 9.45 390.05
13 2.49 - 2.42 95.1 931 / 979 4.46 50.68 34.20 834 0.00 0 8.93 334.80
14 2.42 - 2.37 91.7 896 / 977 3.35 55.66 37.15 729 0.00 0 9.27 320.17
15 2.37 - 2.31 83.9 829 / 988 2.61 56.92 43.21 600 0.00 0 9.60 296.67
16 2.31 - 2.26 72.4 702 / 969 1.97 65.81 26.89 386 0.00 0 10.29 284.39
17 2.26 - 2.22 59.1 582 / 985 1.75 64.72 31.28 275 0.00 0 9.87 284.06
18 2.22 - 2.18 52.9 513 / 970 1.51 71.27 16.86 188 0.00 0 8.93 215.31
19 2.18 - 2.14 35.7 349 / 978 1.32 62.26 68.25 90 0.00 0 8.22 199.09
20 2.14 - 2.10 23.1 227 / 981 1.20 92.14 -9.20 42 0.00 0 8.59 224.44
-------------------------------------------------------------------------------------------------------------
TOTAL 85.9 17224 / 20046 27.11 21.11 92.07 15305 0.00 0 12.87 999.53
-------------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 1809
No. bad cc frames: 153
No. bad G frames) : 0
No. bad unit cell frames: 5
No. bad gamma_e frames: 0
No. bad SE: 0
No. observations: 466997

</pre>

== More detail with input parameters ==
Now that you have your first trial merged data set, you can explore different parameter settings to merge or to obtain the Bijvoet pairs (I+/I-) for your anomalous data set.

'''Anomalous data:'''
<pre>
target_anomalous_flag = True
</pre>
In the last cycle, prime will output a reflection set with I+ and I-.

'''Indexing ambiguity'''

For space groups with indexing ambiguity, use the solutions from cctbx.xfel (see Tutorial for resolving indexing ambiguity) to merge the data set.
<pre>
indexing_ambiguity {
flag_on = True
index_basis_in = /path/to/solution/pickle_file.pickle
}
</pre>

'''Number of micro- and macrocycles'''
<pre>
n_postref_cycle = 3
n_postref_sub_cycle = 3
</pre>

'''Number of bins for merging statistics'''
<pre>
n_bins = 20
</pre>

== Help with input parameters ==

Most input parameters are self-explained. However, you can run -h switch to view help information for each parameter.
<pre>
prime.run -h
</pre>

== Running in manual mode ==
With the same phil file, you can run prime manually. This gives you more freedom in terms of parameter settings at different stages (generating reference set, post-refining images, and merging) or at different cycle of post-refinement.

'''Example A''': I want to generate a reference set then post-refine all the images on the '''scale factors only''' for '''three cycles''' then refine '''all parameters''' in the '''4th cycle'''. To do this, you can follow these steps:

To generate a reference set,
<pre>
prime.genref prime.phil
</pre>

To post-refine on scale factors only, modify your .phil file so that all parameters are turned ''off''.
<pre>
...
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
}
postref {
residual_threshold = 5
residual_threshold_xy = 5
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
allparams {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 3
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To refine all parameters one more cycle, update your .phil file again (flag_on = True)
<pre>
...
allparams {
flag_on = True
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 1
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To obtain the final merged mtz, run
<pre>
prime.merge prime.phil
</pre>

== Running on multiple nodes ==
For LCLS users (or other users with LSF bsub), you can use psana (or your) queuing system to parallelize the entire process. For example, if you want to run your job on 100 nodes using psanq, you can specify:
<pre>
queue {
mode = bsub
qname = psanaq
n_nodes = 100
}
timeout_seconds = 300
</pre>
Prime will divide all the images into 100 batches and submit them to different nodes. It will wait until all images in every batches are done before returning to the merging step (or the exit step in the manual mode). You can control timeout_seconds parameter to tell prime how long it should wait for all the image batches to finish. Usually, this timeout parameter is not used (all images should return before 300 seconds) but in case, you need to wait longer or shorter, you can modify this parameter.

Cctbx.prime

2016-08-02T00:17:56Z

Mona: /* Running in manual mode */

== Prime: '''p'''ost-'''r'''ef'''i'''nement and '''me'''rging ==
With the latest update, prime can be used to process data on multiple nodes (on queuing system). At the moment, only LSF (bsub) is supported. See documentation below for more information how to use the queuing system.

This major update replaces prime.postrefine with

<pre>
prime.run
</pre>

For auto mode, you can still use prime.run with your parameter phil file like before. For manual mode, the available sub commands in prime are:

<pre>
prime.genref #generates a reference set from given integration results
prime.postrefine #refines all images
prime.merge #merges all refined results for an mtz file
</pre>

You can choose to run these commands independently (ideally in the above order) using the same phil file. See [https://commons.wikimedia.org/wiki/File:Prime_flowcharts.tif "PRIME flowchart"]. This will you a freedom to change something (e.g. set of parameters to refine, resolution cut-off, etc.) at different stages of the post-refinement and merging. See running prime in manual mode for more detail.

Step-by-step guidelines to post-refine and merge XFEL diffraction images. For more detail and citation, see
"Enabling X-ray Free Electron Laser Crystallography for Challenging Biological Systems from a Limited Number of Crystals"
[http://elifesciences.org/content/4/e05421 "DOI: http://dx.doi.org/10.7554/eLife.05421"]

== Prime is gui-ed (new!)==
Thanks to Dr. Lyubimov, PRIME is also available as a Graphic User Interface program. Try it by running

<pre>
prime
</pre>

Click to see [https://commons.wikimedia.org/wiki/File:PRIME_main.png "PRIME main gui"] and [https://commons.wikimedia.org/wiki/File:PRIME_advanced_options.png "Advanced options"]

== Getting started ==
'''Generating input phil file''' 
Like most programs developed under ''cctbx'' framework, ''prime'' reads in input .phil file, which stores all the parameters needed to run post-refinement and merging steps. To generate the template .phil file, do the dry run by calling

<pre>
$ prime.run
</pre>

An example of the template .phil file:

<pre>
data = None
run_no = None
title = None
scale {
d_min = 0.1
d_max = 99
sigma_min = 1.5
}
...
</pre>

You can save the content of the output to any file name - in this tutorial, let's save it to thermolysin.phil.

'''First look at your phil file''' 
To run prime, set the required parameters to match with your experiments (you can leave other parameters with their default values - or just delete them from you .phil file). The most interesting parameters are shown below:

<pre>
data = /path/to/your/integarion/result/pickle_files
run_no = 001
title = First trial for thermolysin
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
}
postref {
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
allparams {
flag_on = False
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
}
merge {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
target_unit_cell = 93.99,93.99,130.87,90,90,120
target_space_group = P 61 2 2
pixel_size_mm = 0.102
</pre>

You should pay attention to d_min and d_max for the refinement and merging parameters. If you use IOTA to integrate the images, IOTA will output .phil file for prime that has the optimal resolution range. If not, a few trial-and-error runs may be required to get the best resolution range for your dataset. Use merging statistics output by prime and check the values of CC1/2 and I/sigI to find out your optimal resolution range.

Cell parameters (target_unit_cell and target_space_group) are required to run prime. Target cell parameter is used to remove some outlier images by controlling uc_tolerance parameter (the default value of tolerate range is 3% different). Space group parameter is used in removing outliers and merging with the given symmetry.

Don't forget also to change your pixel size in millimeters. Check what your detector is and note down its pixel size.

'''Running post-refinement in automatic mode''' 
Once you have the input .phil file, you can run ''prime'' by calling
<pre>
prime.run thermolysin.phil
</pre>
''Prime'' will post-refine and merge for reflection sets using three (default value) macrocycles. At the end of the run, you can obtain merging statistics in the last cycle - all other cycle statistics are also available in log.txt.

An example of merging statistics:
<pre>
Summary for 001/postref_cycle_1_merge.mtz
Bin Resolution Range Completeness <N_obs> |Rsplit CC1/2 N_ind |CCanom N_ind| <I/sigI> 
-------------------------------------------------------------------------------------------------------------
02 5.70 - 4.52 100.0 1055 / 1055 65.89 16.02 89.15 1055 0.00 0 20.17 2101.97
03 4.52 - 3.95 100.0 1032 / 1032 61.53 14.48 92.03 1032 0.00 0 20.39 2529.90
04 3.95 - 3.59 100.0 1016 / 1016 54.15 15.61 90.13 1016 0.00 0 16.69 1971.43
05 3.59 - 3.33 100.0 1004 / 1004 42.67 17.66 89.23 1004 0.00 0 14.21 1502.14
06 3.33 - 3.14 100.0 1013 / 1013 32.77 20.40 84.26 1013 0.00 0 11.76 1077.60
07 3.14 - 2.98 100.0 995 / 995 27.36 23.00 78.72 995 0.00 0 11.58 935.37
08 2.98 - 2.85 100.0 1006 / 1006 23.57 22.63 82.26 1006 0.00 0 10.56 722.62
09 2.85 - 2.74 100.0 986 / 986 16.64 28.51 72.90 985 0.00 0 10.01 591.56
10 2.74 - 2.65 99.9 989 / 990 12.41 31.35 72.95 987 0.00 0 9.91 515.07
11 2.65 - 2.56 99.7 979 / 982 9.35 37.14 65.31 970 0.00 0 9.31 438.96
12 2.56 - 2.49 98.0 979 / 999 6.06 45.98 45.37 930 0.00 0 9.45 390.05
13 2.49 - 2.42 95.1 931 / 979 4.46 50.68 34.20 834 0.00 0 8.93 334.80
14 2.42 - 2.37 91.7 896 / 977 3.35 55.66 37.15 729 0.00 0 9.27 320.17
15 2.37 - 2.31 83.9 829 / 988 2.61 56.92 43.21 600 0.00 0 9.60 296.67
16 2.31 - 2.26 72.4 702 / 969 1.97 65.81 26.89 386 0.00 0 10.29 284.39
17 2.26 - 2.22 59.1 582 / 985 1.75 64.72 31.28 275 0.00 0 9.87 284.06
18 2.22 - 2.18 52.9 513 / 970 1.51 71.27 16.86 188 0.00 0 8.93 215.31
19 2.18 - 2.14 35.7 349 / 978 1.32 62.26 68.25 90 0.00 0 8.22 199.09
20 2.14 - 2.10 23.1 227 / 981 1.20 92.14 -9.20 42 0.00 0 8.59 224.44
-------------------------------------------------------------------------------------------------------------
TOTAL 85.9 17224 / 20046 27.11 21.11 92.07 15305 0.00 0 12.87 999.53
-------------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 1809
No. bad cc frames: 153
No. bad G frames) : 0
No. bad unit cell frames: 5
No. bad gamma_e frames: 0
No. bad SE: 0
No. observations: 466997

</pre>

== More detail with input parameters ==
Now that you have your first trial merged data set, you can explore different parameter settings to merge or to obtain the Bijvoet pairs (I+/I-) for your anomalous data set.

'''Anomalous data:'''
<pre>
target_anomalous_flag = True
</pre>
In the last cycle, prime will output a reflection set with I+ and I-.

'''Indexing ambiguity'''

For space groups with indexing ambiguity, use the solutions from cctbx.xfel (see Tutorial for resolving indexing ambiguity) to merge the data set.
<pre>
indexing_ambiguity {
flag_on = True
index_basis_in = /path/to/solution/pickle_file.pickle
}
</pre>

'''Number of micro- and macrocycles'''
<pre>
n_postref_cycle = 3
n_postref_sub_cycle = 3
</pre>

'''Number of bins for merging statistics'''
<pre>
n_bins = 20
</pre>

== Help with input parameters ==

Most input parameters are self-explained. However, you can run -h switch to view help information for each parameter.
<pre>
prime.run -h
</pre>

== Running in manual mode ==
With the same phil file, you can run prime manually. This gives you more freedom in terms of parameter settings at different stages (generating reference set, post-refining images, and merging) or at different cycle of post-refinement.

'''Example A''': I want to generate a reference set then post-refine all the images on the '''scale factors only''' for '''three cycles''' then refine '''all parameters''' in the '''4th cycle'''. To do this, you can follow these steps:

To generate a reference set,
<pre>
prime.genref prime.phil
</pre>

To post-refine on scale factors only, modify your .phil file so that all parameters are turned ''off''.
<pre>
...
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
}
postref {
residual_threshold = 5
residual_threshold_xy = 5
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
allparams {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 3
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To refine all parameters one more cycle, update your .phil file again (flag_on = True)
<pre>
...
allparams {
flag_on = True
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 1
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To obtain the final merged mtz, run
<pre>
prime.merge prime.phil
</pre>

Cctbx.prime

2016-08-02T00:16:42Z

Mona: /* Running in manual mode */

== Prime: '''p'''ost-'''r'''ef'''i'''nement and '''me'''rging ==
With the latest update, prime can be used to process data on multiple nodes (on queuing system). At the moment, only LSF (bsub) is supported. See documentation below for more information how to use the queuing system.

This major update replaces prime.postrefine with

<pre>
prime.run
</pre>

For auto mode, you can still use prime.run with your parameter phil file like before. For manual mode, the available sub commands in prime are:

<pre>
prime.genref #generates a reference set from given integration results
prime.postrefine #refines all images
prime.merge #merges all refined results for an mtz file
</pre>

You can choose to run these commands independently (ideally in the above order) using the same phil file. See [https://commons.wikimedia.org/wiki/File:Prime_flowcharts.tif "PRIME flowchart"]. This will you a freedom to change something (e.g. set of parameters to refine, resolution cut-off, etc.) at different stages of the post-refinement and merging. See running prime in manual mode for more detail.

Step-by-step guidelines to post-refine and merge XFEL diffraction images. For more detail and citation, see
"Enabling X-ray Free Electron Laser Crystallography for Challenging Biological Systems from a Limited Number of Crystals"
[http://elifesciences.org/content/4/e05421 "DOI: http://dx.doi.org/10.7554/eLife.05421"]

== Prime is gui-ed (new!)==
Thanks to Dr. Lyubimov, PRIME is also available as a Graphic User Interface program. Try it by running

<pre>
prime
</pre>

Click to see [https://commons.wikimedia.org/wiki/File:PRIME_main.png "PRIME main gui"] and [https://commons.wikimedia.org/wiki/File:PRIME_advanced_options.png "Advanced options"]

== Getting started ==
'''Generating input phil file''' 
Like most programs developed under ''cctbx'' framework, ''prime'' reads in input .phil file, which stores all the parameters needed to run post-refinement and merging steps. To generate the template .phil file, do the dry run by calling

<pre>
$ prime.run
</pre>

An example of the template .phil file:

<pre>
data = None
run_no = None
title = None
scale {
d_min = 0.1
d_max = 99
sigma_min = 1.5
}
...
</pre>

You can save the content of the output to any file name - in this tutorial, let's save it to thermolysin.phil.

'''First look at your phil file''' 
To run prime, set the required parameters to match with your experiments (you can leave other parameters with their default values - or just delete them from you .phil file). The most interesting parameters are shown below:

<pre>
data = /path/to/your/integarion/result/pickle_files
run_no = 001
title = First trial for thermolysin
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
}
postref {
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
allparams {
flag_on = False
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
}
merge {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
target_unit_cell = 93.99,93.99,130.87,90,90,120
target_space_group = P 61 2 2
pixel_size_mm = 0.102
</pre>

You should pay attention to d_min and d_max for the refinement and merging parameters. If you use IOTA to integrate the images, IOTA will output .phil file for prime that has the optimal resolution range. If not, a few trial-and-error runs may be required to get the best resolution range for your dataset. Use merging statistics output by prime and check the values of CC1/2 and I/sigI to find out your optimal resolution range.

Cell parameters (target_unit_cell and target_space_group) are required to run prime. Target cell parameter is used to remove some outlier images by controlling uc_tolerance parameter (the default value of tolerate range is 3% different). Space group parameter is used in removing outliers and merging with the given symmetry.

Don't forget also to change your pixel size in millimeters. Check what your detector is and note down its pixel size.

'''Running post-refinement in automatic mode''' 
Once you have the input .phil file, you can run ''prime'' by calling
<pre>
prime.run thermolysin.phil
</pre>
''Prime'' will post-refine and merge for reflection sets using three (default value) macrocycles. At the end of the run, you can obtain merging statistics in the last cycle - all other cycle statistics are also available in log.txt.

An example of merging statistics:
<pre>
Summary for 001/postref_cycle_1_merge.mtz
Bin Resolution Range Completeness <N_obs> |Rsplit CC1/2 N_ind |CCanom N_ind| <I/sigI> 
-------------------------------------------------------------------------------------------------------------
02 5.70 - 4.52 100.0 1055 / 1055 65.89 16.02 89.15 1055 0.00 0 20.17 2101.97
03 4.52 - 3.95 100.0 1032 / 1032 61.53 14.48 92.03 1032 0.00 0 20.39 2529.90
04 3.95 - 3.59 100.0 1016 / 1016 54.15 15.61 90.13 1016 0.00 0 16.69 1971.43
05 3.59 - 3.33 100.0 1004 / 1004 42.67 17.66 89.23 1004 0.00 0 14.21 1502.14
06 3.33 - 3.14 100.0 1013 / 1013 32.77 20.40 84.26 1013 0.00 0 11.76 1077.60
07 3.14 - 2.98 100.0 995 / 995 27.36 23.00 78.72 995 0.00 0 11.58 935.37
08 2.98 - 2.85 100.0 1006 / 1006 23.57 22.63 82.26 1006 0.00 0 10.56 722.62
09 2.85 - 2.74 100.0 986 / 986 16.64 28.51 72.90 985 0.00 0 10.01 591.56
10 2.74 - 2.65 99.9 989 / 990 12.41 31.35 72.95 987 0.00 0 9.91 515.07
11 2.65 - 2.56 99.7 979 / 982 9.35 37.14 65.31 970 0.00 0 9.31 438.96
12 2.56 - 2.49 98.0 979 / 999 6.06 45.98 45.37 930 0.00 0 9.45 390.05
13 2.49 - 2.42 95.1 931 / 979 4.46 50.68 34.20 834 0.00 0 8.93 334.80
14 2.42 - 2.37 91.7 896 / 977 3.35 55.66 37.15 729 0.00 0 9.27 320.17
15 2.37 - 2.31 83.9 829 / 988 2.61 56.92 43.21 600 0.00 0 9.60 296.67
16 2.31 - 2.26 72.4 702 / 969 1.97 65.81 26.89 386 0.00 0 10.29 284.39
17 2.26 - 2.22 59.1 582 / 985 1.75 64.72 31.28 275 0.00 0 9.87 284.06
18 2.22 - 2.18 52.9 513 / 970 1.51 71.27 16.86 188 0.00 0 8.93 215.31
19 2.18 - 2.14 35.7 349 / 978 1.32 62.26 68.25 90 0.00 0 8.22 199.09
20 2.14 - 2.10 23.1 227 / 981 1.20 92.14 -9.20 42 0.00 0 8.59 224.44
-------------------------------------------------------------------------------------------------------------
TOTAL 85.9 17224 / 20046 27.11 21.11 92.07 15305 0.00 0 12.87 999.53
-------------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 1809
No. bad cc frames: 153
No. bad G frames) : 0
No. bad unit cell frames: 5
No. bad gamma_e frames: 0
No. bad SE: 0
No. observations: 466997

</pre>

== More detail with input parameters ==
Now that you have your first trial merged data set, you can explore different parameter settings to merge or to obtain the Bijvoet pairs (I+/I-) for your anomalous data set.

'''Anomalous data:'''
<pre>
target_anomalous_flag = True
</pre>
In the last cycle, prime will output a reflection set with I+ and I-.

'''Indexing ambiguity'''

For space groups with indexing ambiguity, use the solutions from cctbx.xfel (see Tutorial for resolving indexing ambiguity) to merge the data set.
<pre>
indexing_ambiguity {
flag_on = True
index_basis_in = /path/to/solution/pickle_file.pickle
}
</pre>

'''Number of micro- and macrocycles'''
<pre>
n_postref_cycle = 3
n_postref_sub_cycle = 3
</pre>

'''Number of bins for merging statistics'''
<pre>
n_bins = 20
</pre>

== Help with input parameters ==

Most input parameters are self-explained. However, you can run -h switch to view help information for each parameter.
<pre>
prime.run -h
</pre>

== Running in manual mode ==
With the same phil file, you can run prime manually. This gives you more freedom in terms of parameter settings at different stages (generating reference set, post-refining images, and merging) or at different cycle of post-refinement.

'''Example A''': I want to generate a reference set then post-refine all the images on the '''scale factors only''' for '''three cycles''' then refine '''all parameters''' in the '''4th cycle'''. To do this, you can follow these steps:

To generate a reference set,
<pre>
prime.genref prime.phil
</pre>

To post-refine on scale factors only, modify your .phil file so that all parameters are turned ''off''.
<pre>
...
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
}
postref {
residual_threshold = 5
residual_threshold_xy = 5
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
allparams {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 3
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To refine all parameters one more cycle, update your .phil file again
<pre>
...
allparams {
flag_on = '''True'''
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 1
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To obtain the final merged mtz, run
<pre>
prime.merge prime.phil
</pre>

Cctbx.prime

2016-08-02T00:16:08Z

Mona: /* Running in manual mode */

== Prime: '''p'''ost-'''r'''ef'''i'''nement and '''me'''rging ==
With the latest update, prime can be used to process data on multiple nodes (on queuing system). At the moment, only LSF (bsub) is supported. See documentation below for more information how to use the queuing system.

This major update replaces prime.postrefine with

<pre>
prime.run
</pre>

For auto mode, you can still use prime.run with your parameter phil file like before. For manual mode, the available sub commands in prime are:

<pre>
prime.genref #generates a reference set from given integration results
prime.postrefine #refines all images
prime.merge #merges all refined results for an mtz file
</pre>

You can choose to run these commands independently (ideally in the above order) using the same phil file. See [https://commons.wikimedia.org/wiki/File:Prime_flowcharts.tif "PRIME flowchart"]. This will you a freedom to change something (e.g. set of parameters to refine, resolution cut-off, etc.) at different stages of the post-refinement and merging. See running prime in manual mode for more detail.

Step-by-step guidelines to post-refine and merge XFEL diffraction images. For more detail and citation, see
"Enabling X-ray Free Electron Laser Crystallography for Challenging Biological Systems from a Limited Number of Crystals"
[http://elifesciences.org/content/4/e05421 "DOI: http://dx.doi.org/10.7554/eLife.05421"]

== Prime is gui-ed (new!)==
Thanks to Dr. Lyubimov, PRIME is also available as a Graphic User Interface program. Try it by running

<pre>
prime
</pre>

Click to see [https://commons.wikimedia.org/wiki/File:PRIME_main.png "PRIME main gui"] and [https://commons.wikimedia.org/wiki/File:PRIME_advanced_options.png "Advanced options"]

== Getting started ==
'''Generating input phil file''' 
Like most programs developed under ''cctbx'' framework, ''prime'' reads in input .phil file, which stores all the parameters needed to run post-refinement and merging steps. To generate the template .phil file, do the dry run by calling

<pre>
$ prime.run
</pre>

An example of the template .phil file:

<pre>
data = None
run_no = None
title = None
scale {
d_min = 0.1
d_max = 99
sigma_min = 1.5
}
...
</pre>

You can save the content of the output to any file name - in this tutorial, let's save it to thermolysin.phil.

'''First look at your phil file''' 
To run prime, set the required parameters to match with your experiments (you can leave other parameters with their default values - or just delete them from you .phil file). The most interesting parameters are shown below:

<pre>
data = /path/to/your/integarion/result/pickle_files
run_no = 001
title = First trial for thermolysin
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
}
postref {
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
allparams {
flag_on = False
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
}
merge {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
target_unit_cell = 93.99,93.99,130.87,90,90,120
target_space_group = P 61 2 2
pixel_size_mm = 0.102
</pre>

You should pay attention to d_min and d_max for the refinement and merging parameters. If you use IOTA to integrate the images, IOTA will output .phil file for prime that has the optimal resolution range. If not, a few trial-and-error runs may be required to get the best resolution range for your dataset. Use merging statistics output by prime and check the values of CC1/2 and I/sigI to find out your optimal resolution range.

Cell parameters (target_unit_cell and target_space_group) are required to run prime. Target cell parameter is used to remove some outlier images by controlling uc_tolerance parameter (the default value of tolerate range is 3% different). Space group parameter is used in removing outliers and merging with the given symmetry.

Don't forget also to change your pixel size in millimeters. Check what your detector is and note down its pixel size.

'''Running post-refinement in automatic mode''' 
Once you have the input .phil file, you can run ''prime'' by calling
<pre>
prime.run thermolysin.phil
</pre>
''Prime'' will post-refine and merge for reflection sets using three (default value) macrocycles. At the end of the run, you can obtain merging statistics in the last cycle - all other cycle statistics are also available in log.txt.

An example of merging statistics:
<pre>
Summary for 001/postref_cycle_1_merge.mtz
Bin Resolution Range Completeness <N_obs> |Rsplit CC1/2 N_ind |CCanom N_ind| <I/sigI> 
-------------------------------------------------------------------------------------------------------------
02 5.70 - 4.52 100.0 1055 / 1055 65.89 16.02 89.15 1055 0.00 0 20.17 2101.97
03 4.52 - 3.95 100.0 1032 / 1032 61.53 14.48 92.03 1032 0.00 0 20.39 2529.90
04 3.95 - 3.59 100.0 1016 / 1016 54.15 15.61 90.13 1016 0.00 0 16.69 1971.43
05 3.59 - 3.33 100.0 1004 / 1004 42.67 17.66 89.23 1004 0.00 0 14.21 1502.14
06 3.33 - 3.14 100.0 1013 / 1013 32.77 20.40 84.26 1013 0.00 0 11.76 1077.60
07 3.14 - 2.98 100.0 995 / 995 27.36 23.00 78.72 995 0.00 0 11.58 935.37
08 2.98 - 2.85 100.0 1006 / 1006 23.57 22.63 82.26 1006 0.00 0 10.56 722.62
09 2.85 - 2.74 100.0 986 / 986 16.64 28.51 72.90 985 0.00 0 10.01 591.56
10 2.74 - 2.65 99.9 989 / 990 12.41 31.35 72.95 987 0.00 0 9.91 515.07
11 2.65 - 2.56 99.7 979 / 982 9.35 37.14 65.31 970 0.00 0 9.31 438.96
12 2.56 - 2.49 98.0 979 / 999 6.06 45.98 45.37 930 0.00 0 9.45 390.05
13 2.49 - 2.42 95.1 931 / 979 4.46 50.68 34.20 834 0.00 0 8.93 334.80
14 2.42 - 2.37 91.7 896 / 977 3.35 55.66 37.15 729 0.00 0 9.27 320.17
15 2.37 - 2.31 83.9 829 / 988 2.61 56.92 43.21 600 0.00 0 9.60 296.67
16 2.31 - 2.26 72.4 702 / 969 1.97 65.81 26.89 386 0.00 0 10.29 284.39
17 2.26 - 2.22 59.1 582 / 985 1.75 64.72 31.28 275 0.00 0 9.87 284.06
18 2.22 - 2.18 52.9 513 / 970 1.51 71.27 16.86 188 0.00 0 8.93 215.31
19 2.18 - 2.14 35.7 349 / 978 1.32 62.26 68.25 90 0.00 0 8.22 199.09
20 2.14 - 2.10 23.1 227 / 981 1.20 92.14 -9.20 42 0.00 0 8.59 224.44
-------------------------------------------------------------------------------------------------------------
TOTAL 85.9 17224 / 20046 27.11 21.11 92.07 15305 0.00 0 12.87 999.53
-------------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 1809
No. bad cc frames: 153
No. bad G frames) : 0
No. bad unit cell frames: 5
No. bad gamma_e frames: 0
No. bad SE: 0
No. observations: 466997

</pre>

== More detail with input parameters ==
Now that you have your first trial merged data set, you can explore different parameter settings to merge or to obtain the Bijvoet pairs (I+/I-) for your anomalous data set.

'''Anomalous data:'''
<pre>
target_anomalous_flag = True
</pre>
In the last cycle, prime will output a reflection set with I+ and I-.

'''Indexing ambiguity'''

For space groups with indexing ambiguity, use the solutions from cctbx.xfel (see Tutorial for resolving indexing ambiguity) to merge the data set.
<pre>
indexing_ambiguity {
flag_on = True
index_basis_in = /path/to/solution/pickle_file.pickle
}
</pre>

'''Number of micro- and macrocycles'''
<pre>
n_postref_cycle = 3
n_postref_sub_cycle = 3
</pre>

'''Number of bins for merging statistics'''
<pre>
n_bins = 20
</pre>

== Help with input parameters ==

Most input parameters are self-explained. However, you can run -h switch to view help information for each parameter.
<pre>
prime.run -h
</pre>

== Running in manual mode ==
With the same phil file, you can run prime manually. This gives you more freedom in terms of parameter settings at different stages (generating reference set, post-refining images, and merging) or at different cycle of post-refinement.

'''Example A''': I want to generate a reference set then post-refine all the images on the '''scale factors only''' for '''three cycles''' then refine all parameters in the 4th cycle. To do this, you can follow these steps:

To generate a reference set,
<pre>
prime.genref prime.phil
</pre>

To post-refine on scale factors only, modify your .phil file so that all parameters are turned ''off''.
<pre>
...
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
}
postref {
residual_threshold = 5
residual_threshold_xy = 5
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
allparams {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 3
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To refine all parameters one more cycle, update your .phil file again
<pre>
...
allparams {
flag_on = '''True'''
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 1
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To obtain the final merged mtz, run
<pre>
prime.merge prime.phil
</pre>

Cctbx.prime

2016-08-02T00:15:23Z

Mona: /* Running in manual mode */

== Prime: '''p'''ost-'''r'''ef'''i'''nement and '''me'''rging ==
With the latest update, prime can be used to process data on multiple nodes (on queuing system). At the moment, only LSF (bsub) is supported. See documentation below for more information how to use the queuing system.

This major update replaces prime.postrefine with

<pre>
prime.run
</pre>

For auto mode, you can still use prime.run with your parameter phil file like before. For manual mode, the available sub commands in prime are:

<pre>
prime.genref #generates a reference set from given integration results
prime.postrefine #refines all images
prime.merge #merges all refined results for an mtz file
</pre>

You can choose to run these commands independently (ideally in the above order) using the same phil file. See [https://commons.wikimedia.org/wiki/File:Prime_flowcharts.tif "PRIME flowchart"]. This will you a freedom to change something (e.g. set of parameters to refine, resolution cut-off, etc.) at different stages of the post-refinement and merging. See running prime in manual mode for more detail.

Step-by-step guidelines to post-refine and merge XFEL diffraction images. For more detail and citation, see
"Enabling X-ray Free Electron Laser Crystallography for Challenging Biological Systems from a Limited Number of Crystals"
[http://elifesciences.org/content/4/e05421 "DOI: http://dx.doi.org/10.7554/eLife.05421"]

== Prime is gui-ed (new!)==
Thanks to Dr. Lyubimov, PRIME is also available as a Graphic User Interface program. Try it by running

<pre>
prime
</pre>

Click to see [https://commons.wikimedia.org/wiki/File:PRIME_main.png "PRIME main gui"] and [https://commons.wikimedia.org/wiki/File:PRIME_advanced_options.png "Advanced options"]

== Getting started ==
'''Generating input phil file''' 
Like most programs developed under ''cctbx'' framework, ''prime'' reads in input .phil file, which stores all the parameters needed to run post-refinement and merging steps. To generate the template .phil file, do the dry run by calling

<pre>
$ prime.run
</pre>

An example of the template .phil file:

<pre>
data = None
run_no = None
title = None
scale {
d_min = 0.1
d_max = 99
sigma_min = 1.5
}
...
</pre>

You can save the content of the output to any file name - in this tutorial, let's save it to thermolysin.phil.

'''First look at your phil file''' 
To run prime, set the required parameters to match with your experiments (you can leave other parameters with their default values - or just delete them from you .phil file). The most interesting parameters are shown below:

<pre>
data = /path/to/your/integarion/result/pickle_files
run_no = 001
title = First trial for thermolysin
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
}
postref {
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
allparams {
flag_on = False
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
}
merge {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
target_unit_cell = 93.99,93.99,130.87,90,90,120
target_space_group = P 61 2 2
pixel_size_mm = 0.102
</pre>

You should pay attention to d_min and d_max for the refinement and merging parameters. If you use IOTA to integrate the images, IOTA will output .phil file for prime that has the optimal resolution range. If not, a few trial-and-error runs may be required to get the best resolution range for your dataset. Use merging statistics output by prime and check the values of CC1/2 and I/sigI to find out your optimal resolution range.

Cell parameters (target_unit_cell and target_space_group) are required to run prime. Target cell parameter is used to remove some outlier images by controlling uc_tolerance parameter (the default value of tolerate range is 3% different). Space group parameter is used in removing outliers and merging with the given symmetry.

Don't forget also to change your pixel size in millimeters. Check what your detector is and note down its pixel size.

'''Running post-refinement in automatic mode''' 
Once you have the input .phil file, you can run ''prime'' by calling
<pre>
prime.run thermolysin.phil
</pre>
''Prime'' will post-refine and merge for reflection sets using three (default value) macrocycles. At the end of the run, you can obtain merging statistics in the last cycle - all other cycle statistics are also available in log.txt.

An example of merging statistics:
<pre>
Summary for 001/postref_cycle_1_merge.mtz
Bin Resolution Range Completeness <N_obs> |Rsplit CC1/2 N_ind |CCanom N_ind| <I/sigI> 
-------------------------------------------------------------------------------------------------------------
02 5.70 - 4.52 100.0 1055 / 1055 65.89 16.02 89.15 1055 0.00 0 20.17 2101.97
03 4.52 - 3.95 100.0 1032 / 1032 61.53 14.48 92.03 1032 0.00 0 20.39 2529.90
04 3.95 - 3.59 100.0 1016 / 1016 54.15 15.61 90.13 1016 0.00 0 16.69 1971.43
05 3.59 - 3.33 100.0 1004 / 1004 42.67 17.66 89.23 1004 0.00 0 14.21 1502.14
06 3.33 - 3.14 100.0 1013 / 1013 32.77 20.40 84.26 1013 0.00 0 11.76 1077.60
07 3.14 - 2.98 100.0 995 / 995 27.36 23.00 78.72 995 0.00 0 11.58 935.37
08 2.98 - 2.85 100.0 1006 / 1006 23.57 22.63 82.26 1006 0.00 0 10.56 722.62
09 2.85 - 2.74 100.0 986 / 986 16.64 28.51 72.90 985 0.00 0 10.01 591.56
10 2.74 - 2.65 99.9 989 / 990 12.41 31.35 72.95 987 0.00 0 9.91 515.07
11 2.65 - 2.56 99.7 979 / 982 9.35 37.14 65.31 970 0.00 0 9.31 438.96
12 2.56 - 2.49 98.0 979 / 999 6.06 45.98 45.37 930 0.00 0 9.45 390.05
13 2.49 - 2.42 95.1 931 / 979 4.46 50.68 34.20 834 0.00 0 8.93 334.80
14 2.42 - 2.37 91.7 896 / 977 3.35 55.66 37.15 729 0.00 0 9.27 320.17
15 2.37 - 2.31 83.9 829 / 988 2.61 56.92 43.21 600 0.00 0 9.60 296.67
16 2.31 - 2.26 72.4 702 / 969 1.97 65.81 26.89 386 0.00 0 10.29 284.39
17 2.26 - 2.22 59.1 582 / 985 1.75 64.72 31.28 275 0.00 0 9.87 284.06
18 2.22 - 2.18 52.9 513 / 970 1.51 71.27 16.86 188 0.00 0 8.93 215.31
19 2.18 - 2.14 35.7 349 / 978 1.32 62.26 68.25 90 0.00 0 8.22 199.09
20 2.14 - 2.10 23.1 227 / 981 1.20 92.14 -9.20 42 0.00 0 8.59 224.44
-------------------------------------------------------------------------------------------------------------
TOTAL 85.9 17224 / 20046 27.11 21.11 92.07 15305 0.00 0 12.87 999.53
-------------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 1809
No. bad cc frames: 153
No. bad G frames) : 0
No. bad unit cell frames: 5
No. bad gamma_e frames: 0
No. bad SE: 0
No. observations: 466997

</pre>

== More detail with input parameters ==
Now that you have your first trial merged data set, you can explore different parameter settings to merge or to obtain the Bijvoet pairs (I+/I-) for your anomalous data set.

'''Anomalous data:'''
<pre>
target_anomalous_flag = True
</pre>
In the last cycle, prime will output a reflection set with I+ and I-.

'''Indexing ambiguity'''

For space groups with indexing ambiguity, use the solutions from cctbx.xfel (see Tutorial for resolving indexing ambiguity) to merge the data set.
<pre>
indexing_ambiguity {
flag_on = True
index_basis_in = /path/to/solution/pickle_file.pickle
}
</pre>

'''Number of micro- and macrocycles'''
<pre>
n_postref_cycle = 3
n_postref_sub_cycle = 3
</pre>

'''Number of bins for merging statistics'''
<pre>
n_bins = 20
</pre>

== Help with input parameters ==

Most input parameters are self-explained. However, you can run -h switch to view help information for each parameter.
<pre>
prime.run -h
</pre>

== Running in manual mode ==
With the same phil file, you can run prime manually. This gives you more freedom in terms of parameter settings at different stages (generating reference set, post-refining images, and merging) or at different cycle of post-refinement.

'''Example A''': I want to generate a reference set then post-refine all the images on the '''scale factors only''' for '''three cycles''' then refine all parameters in the 4th cycle. You'd run

To generate a reference set,
<pre>
prime.genref prime.phil
</pre>

To post-refine on scale factors only, modify your .phil file so that all parameters are turned ''off''.
<pre>
...
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
}
postref {
residual_threshold = 5
residual_threshold_xy = 5
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
allparams {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 3
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To refine all parameters one more cycle, update your .phil file again
<pre>
...
allparams {
flag_on = '''True'''
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 1
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To obtain the final merged mtz, run
<pre>
prime.merge prime.phil
</pre>

Cctbx.prime

2016-08-02T00:14:38Z

Mona: /* Running in manual mode */

== Prime: '''p'''ost-'''r'''ef'''i'''nement and '''me'''rging ==
With the latest update, prime can be used to process data on multiple nodes (on queuing system). At the moment, only LSF (bsub) is supported. See documentation below for more information how to use the queuing system.

This major update replaces prime.postrefine with

<pre>
prime.run
</pre>

For auto mode, you can still use prime.run with your parameter phil file like before. For manual mode, the available sub commands in prime are:

<pre>
prime.genref #generates a reference set from given integration results
prime.postrefine #refines all images
prime.merge #merges all refined results for an mtz file
</pre>

You can choose to run these commands independently (ideally in the above order) using the same phil file. See [https://commons.wikimedia.org/wiki/File:Prime_flowcharts.tif "PRIME flowchart"]. This will you a freedom to change something (e.g. set of parameters to refine, resolution cut-off, etc.) at different stages of the post-refinement and merging. See running prime in manual mode for more detail.

Step-by-step guidelines to post-refine and merge XFEL diffraction images. For more detail and citation, see
"Enabling X-ray Free Electron Laser Crystallography for Challenging Biological Systems from a Limited Number of Crystals"
[http://elifesciences.org/content/4/e05421 "DOI: http://dx.doi.org/10.7554/eLife.05421"]

== Prime is gui-ed (new!)==
Thanks to Dr. Lyubimov, PRIME is also available as a Graphic User Interface program. Try it by running

<pre>
prime
</pre>

Click to see [https://commons.wikimedia.org/wiki/File:PRIME_main.png "PRIME main gui"] and [https://commons.wikimedia.org/wiki/File:PRIME_advanced_options.png "Advanced options"]

== Getting started ==
'''Generating input phil file''' 
Like most programs developed under ''cctbx'' framework, ''prime'' reads in input .phil file, which stores all the parameters needed to run post-refinement and merging steps. To generate the template .phil file, do the dry run by calling

<pre>
$ prime.run
</pre>

An example of the template .phil file:

<pre>
data = None
run_no = None
title = None
scale {
d_min = 0.1
d_max = 99
sigma_min = 1.5
}
...
</pre>

You can save the content of the output to any file name - in this tutorial, let's save it to thermolysin.phil.

'''First look at your phil file''' 
To run prime, set the required parameters to match with your experiments (you can leave other parameters with their default values - or just delete them from you .phil file). The most interesting parameters are shown below:

<pre>
data = /path/to/your/integarion/result/pickle_files
run_no = 001
title = First trial for thermolysin
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
}
postref {
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
allparams {
flag_on = False
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
}
merge {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
target_unit_cell = 93.99,93.99,130.87,90,90,120
target_space_group = P 61 2 2
pixel_size_mm = 0.102
</pre>

You should pay attention to d_min and d_max for the refinement and merging parameters. If you use IOTA to integrate the images, IOTA will output .phil file for prime that has the optimal resolution range. If not, a few trial-and-error runs may be required to get the best resolution range for your dataset. Use merging statistics output by prime and check the values of CC1/2 and I/sigI to find out your optimal resolution range.

Cell parameters (target_unit_cell and target_space_group) are required to run prime. Target cell parameter is used to remove some outlier images by controlling uc_tolerance parameter (the default value of tolerate range is 3% different). Space group parameter is used in removing outliers and merging with the given symmetry.

Don't forget also to change your pixel size in millimeters. Check what your detector is and note down its pixel size.

'''Running post-refinement in automatic mode''' 
Once you have the input .phil file, you can run ''prime'' by calling
<pre>
prime.run thermolysin.phil
</pre>
''Prime'' will post-refine and merge for reflection sets using three (default value) macrocycles. At the end of the run, you can obtain merging statistics in the last cycle - all other cycle statistics are also available in log.txt.

An example of merging statistics:
<pre>
Summary for 001/postref_cycle_1_merge.mtz
Bin Resolution Range Completeness <N_obs> |Rsplit CC1/2 N_ind |CCanom N_ind| <I/sigI> 
-------------------------------------------------------------------------------------------------------------
02 5.70 - 4.52 100.0 1055 / 1055 65.89 16.02 89.15 1055 0.00 0 20.17 2101.97
03 4.52 - 3.95 100.0 1032 / 1032 61.53 14.48 92.03 1032 0.00 0 20.39 2529.90
04 3.95 - 3.59 100.0 1016 / 1016 54.15 15.61 90.13 1016 0.00 0 16.69 1971.43
05 3.59 - 3.33 100.0 1004 / 1004 42.67 17.66 89.23 1004 0.00 0 14.21 1502.14
06 3.33 - 3.14 100.0 1013 / 1013 32.77 20.40 84.26 1013 0.00 0 11.76 1077.60
07 3.14 - 2.98 100.0 995 / 995 27.36 23.00 78.72 995 0.00 0 11.58 935.37
08 2.98 - 2.85 100.0 1006 / 1006 23.57 22.63 82.26 1006 0.00 0 10.56 722.62
09 2.85 - 2.74 100.0 986 / 986 16.64 28.51 72.90 985 0.00 0 10.01 591.56
10 2.74 - 2.65 99.9 989 / 990 12.41 31.35 72.95 987 0.00 0 9.91 515.07
11 2.65 - 2.56 99.7 979 / 982 9.35 37.14 65.31 970 0.00 0 9.31 438.96
12 2.56 - 2.49 98.0 979 / 999 6.06 45.98 45.37 930 0.00 0 9.45 390.05
13 2.49 - 2.42 95.1 931 / 979 4.46 50.68 34.20 834 0.00 0 8.93 334.80
14 2.42 - 2.37 91.7 896 / 977 3.35 55.66 37.15 729 0.00 0 9.27 320.17
15 2.37 - 2.31 83.9 829 / 988 2.61 56.92 43.21 600 0.00 0 9.60 296.67
16 2.31 - 2.26 72.4 702 / 969 1.97 65.81 26.89 386 0.00 0 10.29 284.39
17 2.26 - 2.22 59.1 582 / 985 1.75 64.72 31.28 275 0.00 0 9.87 284.06
18 2.22 - 2.18 52.9 513 / 970 1.51 71.27 16.86 188 0.00 0 8.93 215.31
19 2.18 - 2.14 35.7 349 / 978 1.32 62.26 68.25 90 0.00 0 8.22 199.09
20 2.14 - 2.10 23.1 227 / 981 1.20 92.14 -9.20 42 0.00 0 8.59 224.44
-------------------------------------------------------------------------------------------------------------
TOTAL 85.9 17224 / 20046 27.11 21.11 92.07 15305 0.00 0 12.87 999.53
-------------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 1809
No. bad cc frames: 153
No. bad G frames) : 0
No. bad unit cell frames: 5
No. bad gamma_e frames: 0
No. bad SE: 0
No. observations: 466997

</pre>

== More detail with input parameters ==
Now that you have your first trial merged data set, you can explore different parameter settings to merge or to obtain the Bijvoet pairs (I+/I-) for your anomalous data set.

'''Anomalous data:'''
<pre>
target_anomalous_flag = True
</pre>
In the last cycle, prime will output a reflection set with I+ and I-.

'''Indexing ambiguity'''

For space groups with indexing ambiguity, use the solutions from cctbx.xfel (see Tutorial for resolving indexing ambiguity) to merge the data set.
<pre>
indexing_ambiguity {
flag_on = True
index_basis_in = /path/to/solution/pickle_file.pickle
}
</pre>

'''Number of micro- and macrocycles'''
<pre>
n_postref_cycle = 3
n_postref_sub_cycle = 3
</pre>

'''Number of bins for merging statistics'''
<pre>
n_bins = 20
</pre>

== Help with input parameters ==

Most input parameters are self-explained. However, you can run -h switch to view help information for each parameter.
<pre>
prime.run -h
</pre>

== Running in manual mode ==
With the same phil file, you can run prime manually. This gives you more freedom in terms of parameter settings at different stages (generating reference set, post-refining images, and merging) or at different cycle of post-refinement.

Example A: I want to generate a reference set then post-refine all the images on the '''scale factors only''' for '''three cycles''' then refine all parameters in the 4th cycle. You'd run

To generate a reference set,
<pre>
prime.genref prime.phil
</pre>

To post-refine on scale factors only, modify your .phil file so that all parameters are turned ''off''.
<pre>
...
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
}
postref {
residual_threshold = 5
residual_threshold_xy = 5
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
allparams {
flag_on = False
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 3
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To refine all parameters one more cycle, update your .phil file again
<pre>
...
allparams {
flag_on = '''True'''
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 1
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To obtain the final merged mtz, run
<pre>
prime.merge prime.phil
</pre>

Cctbx.prime

2016-08-02T00:13:39Z

Mona: /* Running in manual mode */

== Prime: '''p'''ost-'''r'''ef'''i'''nement and '''me'''rging ==
With the latest update, prime can be used to process data on multiple nodes (on queuing system). At the moment, only LSF (bsub) is supported. See documentation below for more information how to use the queuing system.

This major update replaces prime.postrefine with

<pre>
prime.run
</pre>

For auto mode, you can still use prime.run with your parameter phil file like before. For manual mode, the available sub commands in prime are:

<pre>
prime.genref #generates a reference set from given integration results
prime.postrefine #refines all images
prime.merge #merges all refined results for an mtz file
</pre>

You can choose to run these commands independently (ideally in the above order) using the same phil file. See [https://commons.wikimedia.org/wiki/File:Prime_flowcharts.tif "PRIME flowchart"]. This will you a freedom to change something (e.g. set of parameters to refine, resolution cut-off, etc.) at different stages of the post-refinement and merging. See running prime in manual mode for more detail.

Step-by-step guidelines to post-refine and merge XFEL diffraction images. For more detail and citation, see
"Enabling X-ray Free Electron Laser Crystallography for Challenging Biological Systems from a Limited Number of Crystals"
[http://elifesciences.org/content/4/e05421 "DOI: http://dx.doi.org/10.7554/eLife.05421"]

== Prime is gui-ed (new!)==
Thanks to Dr. Lyubimov, PRIME is also available as a Graphic User Interface program. Try it by running

<pre>
prime
</pre>

Click to see [https://commons.wikimedia.org/wiki/File:PRIME_main.png "PRIME main gui"] and [https://commons.wikimedia.org/wiki/File:PRIME_advanced_options.png "Advanced options"]

== Getting started ==
'''Generating input phil file''' 
Like most programs developed under ''cctbx'' framework, ''prime'' reads in input .phil file, which stores all the parameters needed to run post-refinement and merging steps. To generate the template .phil file, do the dry run by calling

<pre>
$ prime.run
</pre>

An example of the template .phil file:

<pre>
data = None
run_no = None
title = None
scale {
d_min = 0.1
d_max = 99
sigma_min = 1.5
}
...
</pre>

You can save the content of the output to any file name - in this tutorial, let's save it to thermolysin.phil.

'''First look at your phil file''' 
To run prime, set the required parameters to match with your experiments (you can leave other parameters with their default values - or just delete them from you .phil file). The most interesting parameters are shown below:

<pre>
data = /path/to/your/integarion/result/pickle_files
run_no = 001
title = First trial for thermolysin
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
}
postref {
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
allparams {
flag_on = False
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
}
merge {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
target_unit_cell = 93.99,93.99,130.87,90,90,120
target_space_group = P 61 2 2
pixel_size_mm = 0.102
</pre>

You should pay attention to d_min and d_max for the refinement and merging parameters. If you use IOTA to integrate the images, IOTA will output .phil file for prime that has the optimal resolution range. If not, a few trial-and-error runs may be required to get the best resolution range for your dataset. Use merging statistics output by prime and check the values of CC1/2 and I/sigI to find out your optimal resolution range.

Cell parameters (target_unit_cell and target_space_group) are required to run prime. Target cell parameter is used to remove some outlier images by controlling uc_tolerance parameter (the default value of tolerate range is 3% different). Space group parameter is used in removing outliers and merging with the given symmetry.

Don't forget also to change your pixel size in millimeters. Check what your detector is and note down its pixel size.

'''Running post-refinement in automatic mode''' 
Once you have the input .phil file, you can run ''prime'' by calling
<pre>
prime.run thermolysin.phil
</pre>
''Prime'' will post-refine and merge for reflection sets using three (default value) macrocycles. At the end of the run, you can obtain merging statistics in the last cycle - all other cycle statistics are also available in log.txt.

An example of merging statistics:
<pre>
Summary for 001/postref_cycle_1_merge.mtz
Bin Resolution Range Completeness <N_obs> |Rsplit CC1/2 N_ind |CCanom N_ind| <I/sigI> 
-------------------------------------------------------------------------------------------------------------
02 5.70 - 4.52 100.0 1055 / 1055 65.89 16.02 89.15 1055 0.00 0 20.17 2101.97
03 4.52 - 3.95 100.0 1032 / 1032 61.53 14.48 92.03 1032 0.00 0 20.39 2529.90
04 3.95 - 3.59 100.0 1016 / 1016 54.15 15.61 90.13 1016 0.00 0 16.69 1971.43
05 3.59 - 3.33 100.0 1004 / 1004 42.67 17.66 89.23 1004 0.00 0 14.21 1502.14
06 3.33 - 3.14 100.0 1013 / 1013 32.77 20.40 84.26 1013 0.00 0 11.76 1077.60
07 3.14 - 2.98 100.0 995 / 995 27.36 23.00 78.72 995 0.00 0 11.58 935.37
08 2.98 - 2.85 100.0 1006 / 1006 23.57 22.63 82.26 1006 0.00 0 10.56 722.62
09 2.85 - 2.74 100.0 986 / 986 16.64 28.51 72.90 985 0.00 0 10.01 591.56
10 2.74 - 2.65 99.9 989 / 990 12.41 31.35 72.95 987 0.00 0 9.91 515.07
11 2.65 - 2.56 99.7 979 / 982 9.35 37.14 65.31 970 0.00 0 9.31 438.96
12 2.56 - 2.49 98.0 979 / 999 6.06 45.98 45.37 930 0.00 0 9.45 390.05
13 2.49 - 2.42 95.1 931 / 979 4.46 50.68 34.20 834 0.00 0 8.93 334.80
14 2.42 - 2.37 91.7 896 / 977 3.35 55.66 37.15 729 0.00 0 9.27 320.17
15 2.37 - 2.31 83.9 829 / 988 2.61 56.92 43.21 600 0.00 0 9.60 296.67
16 2.31 - 2.26 72.4 702 / 969 1.97 65.81 26.89 386 0.00 0 10.29 284.39
17 2.26 - 2.22 59.1 582 / 985 1.75 64.72 31.28 275 0.00 0 9.87 284.06
18 2.22 - 2.18 52.9 513 / 970 1.51 71.27 16.86 188 0.00 0 8.93 215.31
19 2.18 - 2.14 35.7 349 / 978 1.32 62.26 68.25 90 0.00 0 8.22 199.09
20 2.14 - 2.10 23.1 227 / 981 1.20 92.14 -9.20 42 0.00 0 8.59 224.44
-------------------------------------------------------------------------------------------------------------
TOTAL 85.9 17224 / 20046 27.11 21.11 92.07 15305 0.00 0 12.87 999.53
-------------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 1809
No. bad cc frames: 153
No. bad G frames) : 0
No. bad unit cell frames: 5
No. bad gamma_e frames: 0
No. bad SE: 0
No. observations: 466997

</pre>

== More detail with input parameters ==
Now that you have your first trial merged data set, you can explore different parameter settings to merge or to obtain the Bijvoet pairs (I+/I-) for your anomalous data set.

'''Anomalous data:'''
<pre>
target_anomalous_flag = True
</pre>
In the last cycle, prime will output a reflection set with I+ and I-.

'''Indexing ambiguity'''

For space groups with indexing ambiguity, use the solutions from cctbx.xfel (see Tutorial for resolving indexing ambiguity) to merge the data set.
<pre>
indexing_ambiguity {
flag_on = True
index_basis_in = /path/to/solution/pickle_file.pickle
}
</pre>

'''Number of micro- and macrocycles'''
<pre>
n_postref_cycle = 3
n_postref_sub_cycle = 3
</pre>

'''Number of bins for merging statistics'''
<pre>
n_bins = 20
</pre>

== Help with input parameters ==

Most input parameters are self-explained. However, you can run -h switch to view help information for each parameter.
<pre>
prime.run -h
</pre>

== Running in manual mode ==
With the same phil file, you can run prime manually. This gives you more freedom in terms of parameter settings at different stages (generating reference set, post-refining images, and merging) or at different cycle of post-refinement.

Example A: I want to generate a reference set then post-refine all the images on the '''scale factors only''' for '''three cycles''' then refine all parameters in the 4th cycle. You'd run

To generate a reference set,
<pre>
prime.genref prime.phil
</pre>

To post-refine on scale factors only, modify your .phil file so that all parameters are turned ''off''.
<pre>
...
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
}
postref {
residual_threshold = 5
residual_threshold_xy = 5
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = '''False'''
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = '''False'''
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = '''False'''
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
allparams {
flag_on = '''False'''
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 3
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To refine all parameters one more cycle, update your .phil file again
<pre>
...
allparams {
flag_on = '''True'''
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 1
...
</pre>
Then run,
<pre>
prime.postrefine prime.phil
</pre>
To obtain the final merged mtz, run
<pre>
prime.merge prime.phil
</pre>

Cctbx.prime

2016-08-02T00:10:02Z

Mona: /* Running in manual mode */

== Prime: '''p'''ost-'''r'''ef'''i'''nement and '''me'''rging ==
With the latest update, prime can be used to process data on multiple nodes (on queuing system). At the moment, only LSF (bsub) is supported. See documentation below for more information how to use the queuing system.

This major update replaces prime.postrefine with

<pre>
prime.run
</pre>

For auto mode, you can still use prime.run with your parameter phil file like before. For manual mode, the available sub commands in prime are:

<pre>
prime.genref #generates a reference set from given integration results
prime.postrefine #refines all images
prime.merge #merges all refined results for an mtz file
</pre>

You can choose to run these commands independently (ideally in the above order) using the same phil file. See [https://commons.wikimedia.org/wiki/File:Prime_flowcharts.tif "PRIME flowchart"]. This will you a freedom to change something (e.g. set of parameters to refine, resolution cut-off, etc.) at different stages of the post-refinement and merging. See running prime in manual mode for more detail.

Step-by-step guidelines to post-refine and merge XFEL diffraction images. For more detail and citation, see
"Enabling X-ray Free Electron Laser Crystallography for Challenging Biological Systems from a Limited Number of Crystals"
[http://elifesciences.org/content/4/e05421 "DOI: http://dx.doi.org/10.7554/eLife.05421"]

== Prime is gui-ed (new!)==
Thanks to Dr. Lyubimov, PRIME is also available as a Graphic User Interface program. Try it by running

<pre>
prime
</pre>

Click to see [https://commons.wikimedia.org/wiki/File:PRIME_main.png "PRIME main gui"] and [https://commons.wikimedia.org/wiki/File:PRIME_advanced_options.png "Advanced options"]

== Getting started ==
'''Generating input phil file''' 
Like most programs developed under ''cctbx'' framework, ''prime'' reads in input .phil file, which stores all the parameters needed to run post-refinement and merging steps. To generate the template .phil file, do the dry run by calling

<pre>
$ prime.run
</pre>

An example of the template .phil file:

<pre>
data = None
run_no = None
title = None
scale {
d_min = 0.1
d_max = 99
sigma_min = 1.5
}
...
</pre>

You can save the content of the output to any file name - in this tutorial, let's save it to thermolysin.phil.

'''First look at your phil file''' 
To run prime, set the required parameters to match with your experiments (you can leave other parameters with their default values - or just delete them from you .phil file). The most interesting parameters are shown below:

<pre>
data = /path/to/your/integarion/result/pickle_files
run_no = 001
title = First trial for thermolysin
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
}
postref {
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
allparams {
flag_on = False
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
}
merge {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
target_unit_cell = 93.99,93.99,130.87,90,90,120
target_space_group = P 61 2 2
pixel_size_mm = 0.102
</pre>

You should pay attention to d_min and d_max for the refinement and merging parameters. If you use IOTA to integrate the images, IOTA will output .phil file for prime that has the optimal resolution range. If not, a few trial-and-error runs may be required to get the best resolution range for your dataset. Use merging statistics output by prime and check the values of CC1/2 and I/sigI to find out your optimal resolution range.

Cell parameters (target_unit_cell and target_space_group) are required to run prime. Target cell parameter is used to remove some outlier images by controlling uc_tolerance parameter (the default value of tolerate range is 3% different). Space group parameter is used in removing outliers and merging with the given symmetry.

Don't forget also to change your pixel size in millimeters. Check what your detector is and note down its pixel size.

'''Running post-refinement in automatic mode''' 
Once you have the input .phil file, you can run ''prime'' by calling
<pre>
prime.run thermolysin.phil
</pre>
''Prime'' will post-refine and merge for reflection sets using three (default value) macrocycles. At the end of the run, you can obtain merging statistics in the last cycle - all other cycle statistics are also available in log.txt.

An example of merging statistics:
<pre>
Summary for 001/postref_cycle_1_merge.mtz
Bin Resolution Range Completeness <N_obs> |Rsplit CC1/2 N_ind |CCanom N_ind| <I/sigI> 
-------------------------------------------------------------------------------------------------------------
02 5.70 - 4.52 100.0 1055 / 1055 65.89 16.02 89.15 1055 0.00 0 20.17 2101.97
03 4.52 - 3.95 100.0 1032 / 1032 61.53 14.48 92.03 1032 0.00 0 20.39 2529.90
04 3.95 - 3.59 100.0 1016 / 1016 54.15 15.61 90.13 1016 0.00 0 16.69 1971.43
05 3.59 - 3.33 100.0 1004 / 1004 42.67 17.66 89.23 1004 0.00 0 14.21 1502.14
06 3.33 - 3.14 100.0 1013 / 1013 32.77 20.40 84.26 1013 0.00 0 11.76 1077.60
07 3.14 - 2.98 100.0 995 / 995 27.36 23.00 78.72 995 0.00 0 11.58 935.37
08 2.98 - 2.85 100.0 1006 / 1006 23.57 22.63 82.26 1006 0.00 0 10.56 722.62
09 2.85 - 2.74 100.0 986 / 986 16.64 28.51 72.90 985 0.00 0 10.01 591.56
10 2.74 - 2.65 99.9 989 / 990 12.41 31.35 72.95 987 0.00 0 9.91 515.07
11 2.65 - 2.56 99.7 979 / 982 9.35 37.14 65.31 970 0.00 0 9.31 438.96
12 2.56 - 2.49 98.0 979 / 999 6.06 45.98 45.37 930 0.00 0 9.45 390.05
13 2.49 - 2.42 95.1 931 / 979 4.46 50.68 34.20 834 0.00 0 8.93 334.80
14 2.42 - 2.37 91.7 896 / 977 3.35 55.66 37.15 729 0.00 0 9.27 320.17
15 2.37 - 2.31 83.9 829 / 988 2.61 56.92 43.21 600 0.00 0 9.60 296.67
16 2.31 - 2.26 72.4 702 / 969 1.97 65.81 26.89 386 0.00 0 10.29 284.39
17 2.26 - 2.22 59.1 582 / 985 1.75 64.72 31.28 275 0.00 0 9.87 284.06
18 2.22 - 2.18 52.9 513 / 970 1.51 71.27 16.86 188 0.00 0 8.93 215.31
19 2.18 - 2.14 35.7 349 / 978 1.32 62.26 68.25 90 0.00 0 8.22 199.09
20 2.14 - 2.10 23.1 227 / 981 1.20 92.14 -9.20 42 0.00 0 8.59 224.44
-------------------------------------------------------------------------------------------------------------
TOTAL 85.9 17224 / 20046 27.11 21.11 92.07 15305 0.00 0 12.87 999.53
-------------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 1809
No. bad cc frames: 153
No. bad G frames) : 0
No. bad unit cell frames: 5
No. bad gamma_e frames: 0
No. bad SE: 0
No. observations: 466997

</pre>

== More detail with input parameters ==
Now that you have your first trial merged data set, you can explore different parameter settings to merge or to obtain the Bijvoet pairs (I+/I-) for your anomalous data set.

'''Anomalous data:'''
<pre>
target_anomalous_flag = True
</pre>
In the last cycle, prime will output a reflection set with I+ and I-.

'''Indexing ambiguity'''

For space groups with indexing ambiguity, use the solutions from cctbx.xfel (see Tutorial for resolving indexing ambiguity) to merge the data set.
<pre>
indexing_ambiguity {
flag_on = True
index_basis_in = /path/to/solution/pickle_file.pickle
}
</pre>

'''Number of micro- and macrocycles'''
<pre>
n_postref_cycle = 3
n_postref_sub_cycle = 3
</pre>

'''Number of bins for merging statistics'''
<pre>
n_bins = 20
</pre>

== Help with input parameters ==

Most input parameters are self-explained. However, you can run -h switch to view help information for each parameter.
<pre>
prime.run -h
</pre>

== Running in manual mode ==
With the same phil file, you can run prime manually. This gives you more freedom in terms of parameter settings at different stages (generating reference set, post-refining images, and merging) or at different cycle of post-refinement.

Example A: I want to generate a reference set the post-refine all the images on the scale factors only for three cycles then refine all parameters in the 4th cycle. You'd run

For generating a reference set,
<pre>
prime.genref prime.phil
</pre>

For post-refinement on scale factors only, modify your .phil file so that all parameters are turned ''off''.
<pre>
...
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
}
postref {
residual_threshold = 5
residual_threshold_xy = 5
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = '''False'''
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = '''False'''
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = '''False'''
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
allparams {
flag_on = '''False'''
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 3
...
</pre>

Cctbx.prime

2016-08-02T00:09:27Z

Mona:

== Prime: '''p'''ost-'''r'''ef'''i'''nement and '''me'''rging ==
With the latest update, prime can be used to process data on multiple nodes (on queuing system). At the moment, only LSF (bsub) is supported. See documentation below for more information how to use the queuing system.

This major update replaces prime.postrefine with

<pre>
prime.run
</pre>

For auto mode, you can still use prime.run with your parameter phil file like before. For manual mode, the available sub commands in prime are:

<pre>
prime.genref #generates a reference set from given integration results
prime.postrefine #refines all images
prime.merge #merges all refined results for an mtz file
</pre>

You can choose to run these commands independently (ideally in the above order) using the same phil file. See [https://commons.wikimedia.org/wiki/File:Prime_flowcharts.tif "PRIME flowchart"]. This will you a freedom to change something (e.g. set of parameters to refine, resolution cut-off, etc.) at different stages of the post-refinement and merging. See running prime in manual mode for more detail.

Step-by-step guidelines to post-refine and merge XFEL diffraction images. For more detail and citation, see
"Enabling X-ray Free Electron Laser Crystallography for Challenging Biological Systems from a Limited Number of Crystals"
[http://elifesciences.org/content/4/e05421 "DOI: http://dx.doi.org/10.7554/eLife.05421"]

== Prime is gui-ed (new!)==
Thanks to Dr. Lyubimov, PRIME is also available as a Graphic User Interface program. Try it by running

<pre>
prime
</pre>

Click to see [https://commons.wikimedia.org/wiki/File:PRIME_main.png "PRIME main gui"] and [https://commons.wikimedia.org/wiki/File:PRIME_advanced_options.png "Advanced options"]

== Getting started ==
'''Generating input phil file''' 
Like most programs developed under ''cctbx'' framework, ''prime'' reads in input .phil file, which stores all the parameters needed to run post-refinement and merging steps. To generate the template .phil file, do the dry run by calling

<pre>
$ prime.run
</pre>

An example of the template .phil file:

<pre>
data = None
run_no = None
title = None
scale {
d_min = 0.1
d_max = 99
sigma_min = 1.5
}
...
</pre>

You can save the content of the output to any file name - in this tutorial, let's save it to thermolysin.phil.

'''First look at your phil file''' 
To run prime, set the required parameters to match with your experiments (you can leave other parameters with their default values - or just delete them from you .phil file). The most interesting parameters are shown below:

<pre>
data = /path/to/your/integarion/result/pickle_files
run_no = 001
title = First trial for thermolysin
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
}
postref {
scale {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = True
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
allparams {
flag_on = False
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
}
merge {
d_min = 2.1
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 3
}
target_unit_cell = 93.99,93.99,130.87,90,90,120
target_space_group = P 61 2 2
pixel_size_mm = 0.102
</pre>

You should pay attention to d_min and d_max for the refinement and merging parameters. If you use IOTA to integrate the images, IOTA will output .phil file for prime that has the optimal resolution range. If not, a few trial-and-error runs may be required to get the best resolution range for your dataset. Use merging statistics output by prime and check the values of CC1/2 and I/sigI to find out your optimal resolution range.

Cell parameters (target_unit_cell and target_space_group) are required to run prime. Target cell parameter is used to remove some outlier images by controlling uc_tolerance parameter (the default value of tolerate range is 3% different). Space group parameter is used in removing outliers and merging with the given symmetry.

Don't forget also to change your pixel size in millimeters. Check what your detector is and note down its pixel size.

'''Running post-refinement in automatic mode''' 
Once you have the input .phil file, you can run ''prime'' by calling
<pre>
prime.run thermolysin.phil
</pre>
''Prime'' will post-refine and merge for reflection sets using three (default value) macrocycles. At the end of the run, you can obtain merging statistics in the last cycle - all other cycle statistics are also available in log.txt.

An example of merging statistics:
<pre>
Summary for 001/postref_cycle_1_merge.mtz
Bin Resolution Range Completeness <N_obs> |Rsplit CC1/2 N_ind |CCanom N_ind| <I/sigI> 
-------------------------------------------------------------------------------------------------------------
02 5.70 - 4.52 100.0 1055 / 1055 65.89 16.02 89.15 1055 0.00 0 20.17 2101.97
03 4.52 - 3.95 100.0 1032 / 1032 61.53 14.48 92.03 1032 0.00 0 20.39 2529.90
04 3.95 - 3.59 100.0 1016 / 1016 54.15 15.61 90.13 1016 0.00 0 16.69 1971.43
05 3.59 - 3.33 100.0 1004 / 1004 42.67 17.66 89.23 1004 0.00 0 14.21 1502.14
06 3.33 - 3.14 100.0 1013 / 1013 32.77 20.40 84.26 1013 0.00 0 11.76 1077.60
07 3.14 - 2.98 100.0 995 / 995 27.36 23.00 78.72 995 0.00 0 11.58 935.37
08 2.98 - 2.85 100.0 1006 / 1006 23.57 22.63 82.26 1006 0.00 0 10.56 722.62
09 2.85 - 2.74 100.0 986 / 986 16.64 28.51 72.90 985 0.00 0 10.01 591.56
10 2.74 - 2.65 99.9 989 / 990 12.41 31.35 72.95 987 0.00 0 9.91 515.07
11 2.65 - 2.56 99.7 979 / 982 9.35 37.14 65.31 970 0.00 0 9.31 438.96
12 2.56 - 2.49 98.0 979 / 999 6.06 45.98 45.37 930 0.00 0 9.45 390.05
13 2.49 - 2.42 95.1 931 / 979 4.46 50.68 34.20 834 0.00 0 8.93 334.80
14 2.42 - 2.37 91.7 896 / 977 3.35 55.66 37.15 729 0.00 0 9.27 320.17
15 2.37 - 2.31 83.9 829 / 988 2.61 56.92 43.21 600 0.00 0 9.60 296.67
16 2.31 - 2.26 72.4 702 / 969 1.97 65.81 26.89 386 0.00 0 10.29 284.39
17 2.26 - 2.22 59.1 582 / 985 1.75 64.72 31.28 275 0.00 0 9.87 284.06
18 2.22 - 2.18 52.9 513 / 970 1.51 71.27 16.86 188 0.00 0 8.93 215.31
19 2.18 - 2.14 35.7 349 / 978 1.32 62.26 68.25 90 0.00 0 8.22 199.09
20 2.14 - 2.10 23.1 227 / 981 1.20 92.14 -9.20 42 0.00 0 8.59 224.44
-------------------------------------------------------------------------------------------------------------
TOTAL 85.9 17224 / 20046 27.11 21.11 92.07 15305 0.00 0 12.87 999.53
-------------------------------------------------------------------------------------------------------------

Summary of refinement and merging
No. good frames: 1809
No. bad cc frames: 153
No. bad G frames) : 0
No. bad unit cell frames: 5
No. bad gamma_e frames: 0
No. bad SE: 0
No. observations: 466997

</pre>

== More detail with input parameters ==
Now that you have your first trial merged data set, you can explore different parameter settings to merge or to obtain the Bijvoet pairs (I+/I-) for your anomalous data set.

'''Anomalous data:'''
<pre>
target_anomalous_flag = True
</pre>
In the last cycle, prime will output a reflection set with I+ and I-.

'''Indexing ambiguity'''

For space groups with indexing ambiguity, use the solutions from cctbx.xfel (see Tutorial for resolving indexing ambiguity) to merge the data set.
<pre>
indexing_ambiguity {
flag_on = True
index_basis_in = /path/to/solution/pickle_file.pickle
}
</pre>

'''Number of micro- and macrocycles'''
<pre>
n_postref_cycle = 3
n_postref_sub_cycle = 3
</pre>

'''Number of bins for merging statistics'''
<pre>
n_bins = 20
</pre>

== Help with input parameters ==

Most input parameters are self-explained. However, you can run -h switch to view help information for each parameter.
<pre>
prime.run -h
</pre>

== Running in manual mode ==
With the same phil file, you can run prime manually. This gives you more freedom in terms of parameter settings at different stages (generating reference set, post-refining images, and merging) or at different cycle of post-refinement.

Example A: I want to generate a reference set the post-refine all the images on the scale factors only for three cycles then refine all parameters in the 4th cycle. You'd run

For generating a reference set,
<pre>
prime.genref prime.phil
</pre>

For post-refinement on scale factors only, modify your .phil file so that all parameters are turned ''off''.
</pre>
...
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
}
postref {
residual_threshold = 5
residual_threshold_xy = 5
scale {
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
crystal_orientation {
flag_on = '''False'''
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
reflecting_range {
flag_on = '''False'''
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
}
unit_cell {
flag_on = '''False'''
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
allparams {
flag_on = '''False'''
d_min = 2.5
d_max = 45
sigma_min = 1.5
partiality_min = 0.1
uc_tolerance = 5
}
}
...
n_postref_cycle = 3
...
</pre>