We first tested a number of propensity scale methods on the Pellequer data set 14. For every scale and window size, a ROC-curve and area under it, the A_roc-value, was calculated as a measure of the prediction accuracy. 1000 bootstrap samples were drawn from the predictions in order to estimate the standard error of the A_roc-value, . The best scale was found to be the one by Levitt 8 (window size of 11, A_roc = 0.658 ± 0.013). This method with will be denoted Levitt. The second best scale is the scale by Parker et al. 6 (window size 9, A_roc = 0.654 ± 0.013), denoted Parker. The other scales, that were tested, did not perform as well as the scales by Parker et al. 6 and Levitt 8.

Performing a permutation experiment 1000 times, we estimated the P-value for the hypothesis that a method performs like a random model, where the alternative hypothesis is that it performs better than a random model. The resulting P-values for Parker and Levitt were both below 0.1%.

Experiments were conducted in which hidden Markov models (HMMs) were used for the prediction of the location of linear B-cell epitopes. The methods were build from positive windows extracted from the AntiJen data set. The HMMs were tested on the Pellequer data set to find the optimal parameters. Different sizes of the extracted peptide windows, different weights of pseudo count correction for estimating the amino acid frequencies and different sizes of the smoothing window were tested. For the best method, the size of the extracted windows was found to be 5, the size of the smoothing window was 9 and the pseudo-count correction was 10⁷. The performance of the method on the Pellequer data set was A_roc = 0.663 ± 0.012. This method with these parameters will be denoted HMM.

In order to make more accurate predictions, the hidden Markov model (HMM) was combined with one of the two best propensity scale methods (Parker and Levitt). The combinations were done as weighted sums of normalized prediction values. The sum of the weights on the two methods was kept equal to one and different weight-pairs were tested. The Pellequer data set was used to optimize the parameter values. The combination methods with the highest A_roc-values were chosen for further comparisons and are shown in Table 1. The combinational method with the highest A_roc-value is denoted BepiPred and it is the candidate method for predicting linear B-cell epitopes in this paper. It is a combination of HMM and Parker.

To make an unbiased validation of the methods, tests were performed on an independent data set, the HIV data set. The results are shown in Table 2. BepiPred is again seen to be the best method. ROC-curves for the selected methods are shown in Figure 1, and chosen values are given in Table 3.

Paired t-tests were performed for the predictions on the HIV data set to determine if one method had a prediction accuracy that was significant higher than another. Table 4 shows that BepiPred was found to be significantly better than all other tested methods, and that HMM was not significantly better than Parker.

A data set was used for the tests and optimization of the methods. Since this dataset was unavailable in an electronic form it was recreated by Lund et al. 17. The epitope annotations were taken from Pellequer et al. 14 and references herein. An exception was the sequence of scorpion neurotoxin, in which the data was taken from 18. This data set, denoted the Pellequer data set, contains 14 protein sequences and 83 epitopes. The epitope density is 0.34.

A second data set was used to train and build the hidden Markov model. This data set was extracted from the AntiJen database, formerly JenPep 13http://www.jenner.ac.uk/AntiJen. This data set, denoted the AntiJen data set, consists of 127 protein sequences, and the epitope density is 0.08. The proteins of this data set are not fully annotated, and the annotation for the non-epitope stretches is not known.

A separate data set was made allowing an unbiased validation of the methods. It consists of epitopes found in the proteins of HIV taken from the HIV Molecular Immunology Database of the Los Alamos National Laboratory 19http://www.hiv.lanl.gov. The epitopes in this data set are overlapping to some degree. Therefore a procedure for determining more accurate borders of the minimal epitopes was applied to the epitopes. If a smaller epitope was contained as part of a larger epitope, the larger epitope was discarded from the data set. Two of the sequences had no assigned epitopes and were therefore discarded from the data set. The HIV data set consists of 10 protein sequences and the epitope density is 0.38.

The propensity scale methods assign a propensity value to every amino acid of the query protein sequence. Fluctuations are reduced by applying a running mean window. In the N- and C- termini we used asymmetric windows to avoid discarding prediction examples. The scales used in this study are based on antigenicity 20, hydrophilicity 6, inverted hydrophobicity 2122, accessibility 9 and secondary structure 78.

Let i = (i₁, i₂, ..., i_w) denote a sequence of amino acids, which has been extracted from a protein sequence. Let j denote the position in this window, j = 1...w. On basis of i, the hidden Markov model predicts if the center position of the window is annotated as part of an epitope. In the N- and C-termini, parts of the extracted windows are exceeding the terminals. For these residues, the character 'X' is used, which does not count when the hidden Markov model is used for the predictions. The prediction score for a window is given by

which is the log odds of the residue at the center position of the window is being part of an epitope (Epitope model) as opposed to if it is occurring by chance (Random model).

To construct the Random model, background frequencies of the Swiss-Prot database 23, q_i, is used. For the Epitope model, p_i,j is the effective amino acid probability of having amino acid i at position j according to the model.

To calculate the values of p_i,j, all windows, for which their center position is annotated as part of an epitope, are extracted from atraining data set. Again, if an extracted window exceeds the N or C terminal, the character 'X' is used, which does not count when calculating the parameters.

These extracted peptide windows form a matrix of aligned peptides of the width w. From this alignment, p_i,j is calculated as the pseudo count corrected probability of occurrence of amino acid i in column j, estimated as in 24. To make the pseudo count correction, pseudo count frequencies, g_i,j, are calculated. They are given by

where p_k,j is the observed frequency of amino acid k in column j of the alignment 25. The variable b_i,k is the Blosum 62 substitution matrix frequency, e.g. the frequency of which i is aligned to k 26.

To give an example of using (2), let the window size, w = 1. The model is then only covering residues, which are annotated as being part of linear B-cell epitopes. If the observed peptides consists of the following single amino acid sequences L and V, with the frequencies p_L,1 = 0.5 and p_V,1 = 0.5, then the pseudo-count frequency for e.g. I is given by

The effective amino acid frequencies are calculated as a weighted average of the observed frequency and the pseudo count frequency,

Here, α is the effective number of sequences in the alignment - 1, and β is the pseudo count correction 25, which is also called the weight on low counts. To finish the calculation example, let β be very large as it is in this work. Then p_I,1 ≈ g_I,1 = 0.14.

Note that we shall use the term hidden Markov model throughout this work to refer to the weight matrix generated using (1). The parameters of the ungapped Markov model are calculated using a so-called Gibbs sampler, written by Nielsen et al. 24.

The result of applying (1) is a prediction score for every residue of the query sequence. To reduce fluctuations, a smoothing window is applied to every position. It is made asymmetric in the N- and C- termini in order to conserve prediction examples.

The result of applying a prediction method to a data set is a set of prediction examples, x = (x₁, x₂, ...,x_N). Let n denote the residue number. Every x_n consists of a target value and a predicted value. If the residue is annotated as part of an epitope, the target value is 1, zero otherwise. If asymmetric smoothing windows are used in the N- and C- termini, the variable N is equal to the number of residues in the data set.

According to a variable threshold, the prediction examples are classified as positives or negatives, and according to the target values, the predictions can be true or false. The predictions can be either true positives (TP), true negatives (TN), false positives (FP) or false negatives (FN).

The prediction accuracy is measured by constructing Receiver Operational Characteristics, ROC, curves 27. For every value of the threshold, the true positive proportion, TP/(TP+FN), and the false positive proportion, FP/(FP+TN), is calculated. A ROC-curve is constructed by plotting the false positive proportion against the true positive proportion for all values of the threshold. It is therefore a non-parametric measure.

The sensitivity is equal to the true positive proportion, and the specificity, given by TN/(FP+TN), is equal to 1 – the false positive proportion. In this way, a ROC-curve is displaying the trade-off between the sensitivity and the specificity for all possible thresholds. A good method has a high true positive proportion when it has a low false positive proportion. A such model has a high sensitivity and a high specificity. The performance of the method is measured as the area under the curve, the A_roc-value. For a random prediction, the true positive proportion is equal to the false positive proportion for every value of the threshold. Then A_roc = 0.5. For a perfect method, A_roc = 1.

Bootstrapping is used to estimate the standard error of the A_roc-value, as a measure of the uncertainty of the A_roc-value 28. The relation between the standard error and the standard deviation, s, is that se = , where r is the number of repeats of the underlying experiment 29.

Bootstrapping is a method for generating pseudo-replica (bootstrap samples) of the predictions, denoted x*, which deviate a little from x. The bootstrap sample, x* = , is defined as a random sample of size N, drawn with replacement from x. Some of the prediction examples from x may appear zero times, some one time, some twice etc. Drawing a bootstrap sample can in other words be done by copying randomly chosen prediction examples, x_n, from x into x*. In this way, some variation from x is introduced into x*.

Totally B bootstrap samples are drawn. Let x^*b denote the b'th bootstrap sample. The prediction accuracy of x^*b is calculated as .

The result of the bootstrap experiment is x^*1, x^*2,...,x^*B and hence . The standard error of the original A_roc-value is given by

where is the expected value of , given by 28. Note the similarity to the way the standard deviation is calculated. approaches the original A_roc-value as B gets large.

A paired t-test is performed in order to determine if one method is more accurate than another. The H₀-hypothesis for this test is that two means are equal, μ₁ = μ₂. Instead of μ, and hence A_roc is used. The starting point is the performance measures of the two methods, A_roc,M1 and A_roc,M2, where M1 denotes method 1. By bootstrapping we have the vectors and . Every bootstrap pair are drawn identically for every b, making the two A_roc-values paired.

The H₀-hypothesis is therefore A_roc,M1 = A_roc,M2 and the alternative hypothesis A_roc,M1 > A_roc,M2. The test statistic t is given by

The paired difference of the b'th bootstrap samples, D^b, is given by

The variable is calculated as the expected value of D^b, and is calculated using (4) but replacing with D^b. The test statistic is following a t-distribution with m = B - 1 degrees of freedom, which approaches the normal distribution for m > 30, then t ≈ z. The P-value for the test is then given by 1 - F(z), where F(z) is the cumulative normal distribution. See 29 for more information about the paired t-test.

When testing the H₀-hypothesis that a method performs like a random model, a permutation experiment can be made. The alternative hypothesis is that the method is performing better than a random model. From the predictions of the method, x, the target values are permuted to result in a new prediction set, x^perm,p. This is done for p = 1...p_max. For every p, the prediction accuracy is calculated as . The P-value for the H₀-hypothesis is calculated as the proportion of times for which > A_roc.