Continuous Distributed Representation of Biological Sequences for Deep Proteomics and Genomics — TEI-tagged view

Ehsaneddin AsgaripersName, Mohammad R. K. MofradpersName (2015date). “Continuous Distributed Representation of Biological Sequences for Deep Proteomics and Genomics.” PLOS ONEtitle·j, BerkeleyplaceName.

Object of study — Proteinterm·key and gene sequencesterm·key treated as text, and whether a word-embedding trained on them captures biophysical and biochemical structure.

Research question — Can the distributed word-representation idea from language be used to embed biological sequencesterm·key in a way that captures their biochemical and structural properties?

Contribution — Introduces ProtVecterm·mth, applying the word2vecterm·key skip-gramterm·mth model to k-mersterm·dom of protein sequencesterm·dom to learn continuous embeddingsterm·key that capture biophysical and biochemical properties, importing NLP word-embedding techniques into bioinformatics.

Full text — tagged

The reference list is not part of the text we hold for this paper. Our copy was extracted from the publisher's page and the bibliography did not survive that extraction, so the article below ends before its references. Nothing has been removed by us.

Abstract We introduce a new representationterm·key and feature extractionterm·mth method for biological sequencesterm·key. Named bio-vectors (BioVecterm·mth) to refer to biological sequencesterm·key in general with protein-vectors (ProtVecterm·mth) for proteinsterm·key (amino-acid sequencesterm·key) and gene-vectors (GeneVecterm·mth) for gene sequencesterm·key, this representationterm·key can be widely used in applications of deep learningterm·ai in proteomics and genomics. In the present paper, we focus on protein-vectors that can be utilized in a wide array of bioinformatics investigations such as family classificationterm·key, proteinterm·key visualization, structure prediction, disordered proteinterm·dom identification, and protein-protein interaction prediction. In this method, we adopt artificial neural network approaches and represent a protein sequenceterm·dom with a single dense n-dimensional vectorterm·key. To evaluate this method, we apply it in classificationterm·key of 324,018 protein sequencesterm·dom obtained from Swiss-Protterm·dom belonging to 7,027 proteinterm·key families, where an average family classification accuracyterm·var of 93%±0.06% is obtained, outperforming existing family classificationterm·key methods. In addition, we use ProtVecterm·mth representationterm·key to predict disordered proteinsterm·dom from structured proteinsterm·key. Two databases of disordered sequencesterm·key are used: the DisProtterm·dom database as well as a database featuring the disordered regions of nucleoporins rich with phenylalanine-glycine repeats (FG-Nupsterm·dom). Using support vector machineterm·mth classifiers, FG-Nupterm·dom sequencesterm·key are distinguished from structured protein sequencesterm·dom found in Proteinterm·key Data Bank (PDB) with a 99.8% accuracy, and unstructured DisProtterm·dom sequencesterm·key are differentiated from structured DisProtterm·dom sequencesterm·key with 100.0% accuracy. These results indicate that by only providing sequenceterm·key data for various proteinsterm·key into this model, accurate information about proteinterm·key structure can be determined. Importantly, this model needs to be trained only once and can then be applied to extract a comprehensive set of information regarding proteinsterm·key of interest. Moreover, this representationterm·key can be considered as pre-trainingterm·mth for various applications of deep learningterm·ai in bioinformatics. The related data is available at Life Language Processing Website: http://llp.berkeley.edu and Harvard Dataverse: http://dx.doi.org/10.7910/DVN/JMFHTN .

Introduction Nature uses certain languages to describe biological sequencesterm·key such as DNA, RNA, and proteinsterm·key. Much like humans adopt languages to communicate, biological organisms use sophisticated languages to convey information within and between cells. Inspired by this conceptual analogyterm, we adopt existing methods in natural language processing (NLP) to gain a deeper understanding of the “language of lifeterm” with the ultimate goal to discover functions encoded within biological sequencesterm·key [ 1 – 4 ].

Feature extractionterm·mth is an important step in data analysis, machine learning and NLP. It refers to finding an interpretable representationterm·key of data for machines that can increase performance of learning algorithms. Even the most sophisticated algorithms would perform poorly if inappropriate features are used, while simple methods can potentially perform well when they are fed with the appropriate features. Feature extractionterm·mth can be done manually or in an unsupervised fashion. In this paper, we propose an unsupervised data-driven distributed representationterm·mth for biological sequencesterm·key. This method, called bio-vectors (BioVecterm·mth) in general and more specifically protein-vectors (ProtVecterm·mth) for proteinsterm·key, can be applied to a wide range of problems in bioinformatics, such as proteinterm·key visualization, protein familyterm·dom classificationterm·key, structure prediction, domain extraction, and interaction prediction. In this approach, each biological sequenceterm·key is embedded in an n-dimensional vectorterm·key that characterizes biophysical and biochemical properties of sequencesterm·key using neural networks. In the following, we first explain how this method works and how it is trained from 546,790 sequencesterm·key of Swiss-Protterm·dom database. Subsequently, we will analyze the biophysical and the biochemical properties of this representationterm·key qualitatively and quantitatively. To further evaluate this feature extractionterm·mth method, we apply it in classificationterm·key of 7,027 proteinterm·key families of 324,018 protein sequencesterm·dom in Swiss-Protterm·dom. In the next step, we use this approach for visualization and characterization of two categories of disordered sequencesterm·key: the DisProtterm·dom database as well as a database of disordered regions of phenylalanine-glycine nucleoporins (FG-Nupsterm·dom). Finally, we classify these proteinterm·key families using support vector machineterm·mth (SVM) classifiers [ 5 ]. As a key advantage of the proposed method, the embeddingterm·key needs to be trained only once and then may be used to encode biological sequencesterm·key in a given problem. The related data and future updated will be available at: http://llp.berkeley.edu and Harvard Dataverse: http://dx.doi.org/10.7910/DVN/JMFHTN .

Distributed Representationterm·mth Distributed representationterm·mth has proved one of the most successful approaches in machine learning [ 6 – 10 ]. The main idea in this approach is encoding and storing information about an item within a system through establishing its interactions with other members. Distributed representationterm·mth was originally inspired by the structure of human memory, where the items are stored in a “content-addressableterm” fashion. Content-based storing allows for efficiently recalling items from partial descriptionsterm. Since the content-addressableterm items and their properties are stored within a close proximity, such a system provides a viable infrastructure to generalize features attributed to an item.

Continuous vectorterm·key representationterm·key, as a distributed representationterm·mth for words, has been recently established in natural language processing (NLP) as an efficient way to represent semantic/syntactic units with many applications. In this model, each word is embedded in a vectorterm·key in an n-dimensional space. Similar words have close vectorsterm·key, where similarity is defined in terms of both syntax and semantic. The basic idea behind training such vectorsterm·key is that the meaning of a word is characterized by its contextterm, i.e. neighboring words. Thus, words and their contextsterm are considered to be positive training samplesterm. Such vectorsterm·key can be trained using large amounts of textual data in a variety of ways, e.g. neural network architectures like the Skip-gramterm·mth model [ 10 ].

Interesting patterns have been observed by training word vectorsterm·key using Skip-gramterm·mth in natural language. Words with similar vectorterm·key representationsterm·key show multiple degrees of similarity. For instance, K i n g → - M a n → + W o m a n → resembles the closest vectorterm·key to the word Q u e e n → [ 11 ].

In this work, we seek unique patterns in biological sequencesterm·key to facilitate biophysical and biochemical interpretations. We show how Skip-gramterm·mth can be used to train a distributed representationterm·mth for biological sequencesterm·key over a large set of sequencesterm·key, and establish physical and chemical interpretations for such representationsterm·key. We propose this as a general-purpose representationterm for protein sequencesterm·dom that can be used in a wide range of bioinformatics problems, including protein familyterm·dom classificationterm·key, proteinterm·key interaction prediction, structure prediction, motif extraction, proteinterm·key visualization, and domain identification. To illustrate, we specifically tackle visualization and protein familyterm·dom classificationterm·key problems.

Protein Familyterm·dom Classificationterm·key A protein familyterm·dom is a set of proteinsterm·key that are evolutionarily related, typically involving similar structures or functions. The large gap between the number of known sequencesterm·key versus the amount of known functional information about sequencesterm·key has motivated family (function) identification methods based on primary sequencesterm·var [ 12 – 14 ]. Protein Familyterm·dom Database (Pfamterm·dom) is a widely used source for proteinterm·key families [ 15 ]. In Pfamterm·dom, a family can be classified as a “family”, “domain”, “repeat”, or “motif”. In this study, we utilize ProtVecterm·mth to classify proteinterm·key families in Swiss-Protterm·dom using the information provided by Pfamterm·dom database and we obtain a high classificationterm·key accuracy.

Protein familyterm·dom classificationterm·key based on the primary structures (sequencesterm·key) has been performed using classifiers such as support vector machineterm·mth classifier (SVM) [ 16 – 18 ]. Besides the primary sequenceterm·var, the existing methods typically require extensive information for feature extractionterm, e.g. hydrophobicityterm·dom, normalized Van der Waals volume, polarity, polarizability, charge, surface tension, secondary structure and solvent accessibility. The reported accuracies of a previous study on family classificationterm·key have been in the range of 69.1–99.6% for 54 proteinterm·key families [ 16 ]. In another study, researchers used motifs from proteinterm·key interactions for detecting Structural Classificationterm·key of Proteinsterm·key (SCOP) [ 19 ] families for 368 proteinsterm·key, and obtained a classificationterm·key accuracy of 75% at sensitivityterm of 10% [ 20 ]. In contrast, our proposed approach is trained based solely on primary sequenceterm·var information, yet achieving high accuracy when applied in classificationsterm·key of proteinterm·key families.

Disordered Proteinsterm·dom Proteinsterm·key can be fully or partially unstructured, i.e. lacking a secondary or ordered tertiary three-dimensional structure. Due to their abundance and the critical roles they play in cell biology, disordered proteinsterm·dom are considered to be an important class of proteinsterm·key [ 21 ]. Several studies have focused on disordered peptides and their functional analysis in recent years [ 22 – 24 ].

In the present work, we introduce ProtVecterm·mth for the visualization and characterization of two categories of disordered proteinsterm·dom: DisProtterm·dom database as well as a database of disordered regions of phenylalanine-glycine nucleoporins (FG-Nupsterm·dom) [ 25 ].

DisProtterm·dom is a database of experimentally identified disordered proteinsterm·dom that categorizes disordered and ordered regions of a collection of proteinsterm·key [ 26 ]. DisProtterm·dom Release 6.02 consists of 694 proteinsterm·key presenting 1539 disordered, and 95 ordered regions. FG-Nupsterm·dom dataset is a collection of FG-Nupsterm·dom disordered sequencesterm·key [ 27 ]. Nucleoporins form the nuclear pore complex (NPC), the sole gateway for bidirectional transport of cargo between the cytoplasm and the nucleus in eukaryotic cells [ 28 ]. Since FG-Nupsterm·dom are mostly computationally identified, only 10 sequencesterm·key out of 1,138 disordered sequencesterm·key exist in Swiss-Protterm·dom. A recent study on features of FG-Nupsterm·dom versus DisProtterm·dom showed biophysical differences between FG-Nupsterm·dom and average DisProtterm·dom sequencesterm·key [ 29 ].

We further propose using protein-vectors for the visualization of biological sequencesterm·key. Simplicity and biophysical interpretations encoded within ProtVecterm·mth distinguishes this method from the previous work [ 30 , 31 ]. As an example, we use ProtVecterm·mth for the visualization of FG-Nupsterm·dom, DisProtterm·dom, and structured PDB proteinsterm·key. This visualization confirms the results obtained [ 29 ] on the biophysical features of FG-Nupsterm·dom and typical disordered proteinsterm·dom. Furthermore, we employ ProtVecterm·mth to classify FG-Nupsterm·dom versus random PDB sequencesterm·key as well as DisProtterm·dom disordered regions versus disport ordered regions.

Methods Protein-Spaceterm Construction Our goal is to construct a distributed representationterm·mth of biological sequencesterm·key. In the training process of word embeddingterm·mth in NLP, a large corpus of sentences should be fed into the training algorithm to ensure sufficient contextsterm are observed. Similarly, a large corpus is needed to train distributed representationterm·mth of biological sequencesterm·key. We use Swiss-Protterm·dom as a rich proteinterm·key database, which consists of 546,790 manually annotated and reviewed sequencesterm·key.

The next step in training distributed representationsterm·mth is to break the sequencesterm·key into sub sequencesterm (i.e. biological wordsterm·def). The simplest and most common technique in bioinformatics to study sequencesterm·key involves fixed-length overlapping n-gramsterm·mth [ 32 – 34 ]. However, instead of using n-gramsterm·mth directly in feature extractionterm·mth, we utilize n-gramterm·mth modeling for training a general purpose distributed representationterm·mth of sequencesterm·key. This so-called embeddingterm·key model needs to be trained only once and may then be adopted in feature extractionterm·mth part of specific problems.

In n-gramterm·mth modeling of proteinterm·key informatics, usually an overlapping window of 3 to 6 residues is used. Instead of taking overlapping windows, we generate 3 lists of shifted non-overlapping wordsterm, as shown in Fig 1 . Evaluating K-nearest neighbors in a 2xfold cross-validationterm·stat for different window sizesterm·var, embedding vector sizesterm·var and overlapping versus non-overlapping n-gramsterm·mth showed a more consistent embeddingterm·key training for a window sizeterm·var of 3 and the mentioned splitting.

Fig 1. Protein sequenceterm·dom splitting.

In order to prepare the training data, each protein sequenceterm·dom will be represented as three sequencesterm (1, 2, 3) of 3-gramsterm·mth.

The same procedure is applied on all 546,790 sequencesterm·key in Swiss-Protterm·dom, thus at the end we obtain a corpus consisting of 546,790 × 3 = 1,640,370 sequences of 3-gramsterm (3-gramterm·mth is a “biological” word consisting of 3 amino acids). The next step is training the embeddingterm·key based on such data through a Skip-gramterm·mth neural network. In training word vectorterm·key representationsterm·key, Skip-gramterm·mth attempts to maximize the probability of observed word sequencesterm·key (contextsterm). In other words, for a given training sequenceterm·key of words we would like to find their corresponding n-dimensional vectorsterm·key maximizing the following average log probability function. Such a constraint allows similar words to assume a similar representationterm·key in this space.

argmax v , v 1 N ∑ i = 1 N ∑ - c ≤ j ≤ c , j ≠ 0 log p ( w i + j | w i ) p ( w i + j | w i ) = exp ( v ′ w i + j T v w i ) ∑ k = 1 W exp ( v ′ w k T v w i ) , (1) where N is the length of the training sequenceterm·key, 2 c is the window sizeterm·var we consider as the contextterm, w i is the center of the window, W is the number of words in the dictionary and v w and v w ′ are input and output n-dimensional representationsterm·key of word w , respectively. The probability p ( w i + j | w i ) is defined using a softmax function. Hierarchical softmax or negative samplingterm are efficient approximations of such a softmax function. In the implementation we use (Word2Vecterm·key) [ 10 ] negative samplingterm has been utilized, which is considered as the state-of-the-art for training word vectorterm·key representationterm·key. Negative samplingterm uses the following objective function in the calculation of the word vectorsterm·key: arg max θ ∏ ( w , c ) ∈ D p ( D = 1 | c , w ; θ ) ∏ ( w , c ) ∈ D ′ p ( D = 0 | c , w ; θ ) , (2) where D is the set of all word and contextterm pairs (w, c) existing in the training data (positive samplesterm) and D ′ is a randomly generated set of incorrect (w, c) pairsterm (negative samplesterm).

p ( D = 1| w , c ; θ ) is the probability that (w, c) pair came from the training data and p ( D = 0| w , c ; θ ) is the probability that ( w , c ) did not come from the training data. The term p ( D = 1| c , w ; θ ) can be defined using a sigmoid function on the word vectorsterm·key: p ( D = 1 | w , c ; θ ) = 1 1 + e - v c · v w , where the parameters θ are the word vectorsterm·key we train within the optimization framework: v c and v w ∈ R d are vectorterm·key representationsterm·key for the contextterm c and the word w respectively [ 35 ]. In Eq (2) , the positive samplesterm maximize the probabilities of the observed (w, c) pairs in the training data, while the negative samplesterm prevent all vectorsterm·key from having the same value by disallowing some incorrect (w, c) pairsterm. To train the embeddingterm·key vectorsterm·key, we consider a vectorterm·key size of 100 and a contextterm size of 25. Thus each 3-gramterm·mth is presented as a vector of size 100term.

Protein-Spaceterm Analysis To qualitatively analyze the distribution of various biophysical and biochemical properties within the training space, we project all 3-gramterm·mth embeddingsterm·key from 100-dimensional space to a 2D space using Stochastic Neighbor Embeddingterm·key [ 36 ]. Mass, volume, polarity, hydrophobicityterm·dom, charge, and van der Waals volume properties were analyzed. The data is adopted from [ 37 ]. In addition, to quantitatively measure the continuity of these propertiesterm in the protein-spaceterm, the best Lipschitz constantterm, i.e. the smallest k satisfying is calculated: d f ( f p r o p ( w 1 ) , f p r o p ( w 2 ) ) ≤ k × d w ( w 1 , w 2 ) , (3) where f is the scale of one of the properties of a given 3-gramsterm·mth (e.g., average mass, hydrophobicityterm·dom, etc.), d is the distance metric, d f is the absolute value of score differences, and d w is Euclidian distance between two 3-gramsterm·mth w 1 and w 2 . The Lipschitzterm·mth constant is calculated for the aforementioned properties.

Protein Familyterm·dom Classificationterm·key To evaluate the strength of the proposed representationterm·key, we set up a classificationterm·key task on proteinterm·key families. Family information of 324,018 protein sequencesterm·dom in Swiss-Protterm·dom is extracted from the Protein Familyterm·dom (Pfamterm·dom) database, resulting in a total of 7,027 distinct families for Swiss-Protterm·dom sequencesterm·key. Each sequenceterm·key is represented as the summation of the vector representation of overlapping 3-gramsterm. Thus, each sequenceterm·key is presented as a vector of size 100term. For each family type, the same number of instances from Swiss-Protterm·dom are selected randomly to form the negative examplesterm. Support vector machineterm·mth classifiers are used to evaluate the strength of ProtVecterm·mth in the classificationterm·key of proteinterm·key families through 10 × fold cross-validationsterm. We perform the classificationterm·key over 7,027 proteinterm·key families consisting of 324,018 sequencesterm·key. For the evaluation we report specificityterm (true negative rate), sensitivityterm (true positive rate), and the accuracy of family classificationsterm·key.

S e n s i t i v i t y = T P r a t e = T P T P + F N S p e c i f i c i t y = 1 - F P r a t e = T N F P + T N A c c u r a c y = T P + T N P + N Visualization and Classificationterm·key of Disordered Proteinsterm·dom Two databases of disordered proteinsterm·dom are used for disordered proteinterm·dom prediction: DisProtterm·dom database (694 sequencesterm·key) and FG-Nupsterm·dom dataset (1,138 sequencesterm·key).

FG-Nupsterm·dom Characterization To distinguish the characteristics of FG-Nupsterm·dom, a collection of 1,138 FG-Nupsterm·dom and two random sets of 1,138 structured proteinsterm·key from Proteinterm·key Data Bank (PDB) [ 38 ] are compared. Since PDB sequencesterm·key on average have a shorter length than disordered proteinsterm·dom, the two sets are selected from PDB in such a way that they have an average length of 900 residues, the same as the average length of the disordered proteinterm·dom dataset. For visualization purposes, the ProtVecterm·mth is reduced from 100 dimensions to a 2D space using Stochastic Neighbor Embeddingterm·key [ 36 ].

We quantitatively evaluate how ProtVecterm·mth can be used to distinguish between FG-Nupsterm·dom versus typical PDB sequencesterm·key using a support vector machineterm·mth binary classifier. The positive examples were the aforementioned 1,138 disordered FG-Nupsterm·dom proteinsterm·key and the negative examplesterm (again 1,138 sequencesterm·key) are selected randomly from PDB with the same average length of disordered sequencesterm·key (≈ 900 residues). We present each protein sequenceterm·dom as a summation of its ProtVecsterm·mth of all 3-gramsterm·mth. Since the average length of structured proteinsterm·key is shorter than FG-Nupsterm·dom, and to avoid trivial cases, the PDB sequencesterm·key are selected in a way to maintain the same average length.

DisProtterm·dom Characterization To distinguish the characteristics of DisProtterm·dom sequencesterm·key, we use DisProtterm·dom Release 6.02, consisting of 694 proteinsterm·key presenting 1539 disordered and 95 ordered regions, and perform the same experiment as for FG-Nupsterm·dom with DisProtterm·dom sequencesterm·key.

Results Protein-Spaceterm Analysis Although the protein-spaceterm is trained based on only the primary sequencesterm·var of proteinsterm·key, it offers several interesting biochemical and biophysical implications. In order to study these features, we visualized the distribution of different criteria, including mass, volume, polarity, hydrophobicityterm·dom, charge, and van der Waals volume in this space. To do so, for each 3-gramterm·mth we conducted qualitative and quantitative analyses as described below.

Qualitative Analysis In order to visualize the distribution of the aforementioned properties, we projected all 3-gramterm·mth embeddingsterm·key from 100-dimensional space to a 2D space using Stochastic Neighbor Embeddingterm·key (t-SNEterm·mth) [ 36 ]. In the diagrams presented in Fig 2 , each point represents a 3-gramterm·mth and is colored according to its scale in each property. Interestingly, as can be seen in the figure, 3-gramsterm·mth with the same biophysical and biochemical properties were grouped together. This observation suggests that the proposed embeddingterm·key not only encodes protein sequencesterm·dom in an efficient way that proved useful for classificationterm·key purposes, but also reveals some important physical and chemical patterns in protein sequencesterm·dom.

Fig 2. Normalized distributions of biochemical and biophysical properties in protein-spaceterm.

In these plots, each point represents a 3-gramterm·mth (a word of three residues) and the colors indicate the scale for each property. Data points in these plots are projected from a 100-dimensional space a 2D space using t-SNEterm·mth. As it is shown words with similar properties are automatically clustered together meaning that the properties are smoothly distributed in this space.

Quantitative Analysis Although Fig 2 illustrates the smoothness of protein-spaceterm with respect to different physical and chemical meanings, we required a quantitative approach to measure the continuity of these propertiesterm in the protein-spaceterm. To do so, we calculated the best Lipschitz constantterm. For all 6 properties presented in Fig 2 , we calculated the minimum k . To evaluate this result we made an artificial space called “scrambled spaceterm·mth” by randomly shuffling the labels of 3-gramsterm in the 100 dimensional space. Table 1 contains the values of Libschitz constants for protein-spaceterm versus the “scrambled spaceterm·mth” with respect to different properties and also their ratio.

Table 1. Using Lipschitzterm·mth number to evaluate the continuity of ProtVecterm·mth with respect to biophysical and biochemical properties.

Lipschitzterm·mth Number Property protein-Spaceterm The scrambled spaceterm·mth Ratio Mass 0.3137 0.6605 0.4750 Volume 0.3742 0.6699 0.5586 Van Der Waal Volume 0.3629 0.6431 0.5643 Polarity 0.4757 1.2551 0.3790 Hydrophobicityterm·dom 0.608 1.448 0.4203 Charge 0.8733 1.3620 0.6412 Average 0.50 1.01 0.51 Normally if k = 1 the function is called a short mapterm, and if 0 ≤ k < 1 the function is called a contractionterm. The results suggest that the protein-spaceterm is on average 2-times smoother in terms of physical and chemical propertiesterm than a random spaceterm. This quantitative result supports our qualitative observation of the space structure in Fig 2 , and suggests that our training space encodes, 3-gramsterm·mth in an informative manner.

Protein Familyterm·dom Classificationterm·key In order to evaluate the strength of ProtVecterm·mth, we performed classificationsterm·key of 7,027 proteinterm·key families and obtained a weighted averageterm·stat accuracy of 93 ± 0.06%, which exhibits a more reliable result than the existing methods. In contrast to the existing methods, our proposed approach is trained based on primary sequence information aloneterm.

Table 2 shows the sensitivityterm, specificityterm, and the accuracy for the most frequent families in Swiss-Protterm·dom. These results suggest that structural features of proteinsterm·key can be accurately predicted from the primary sequenceterm·var information solely. The results for all 7,027 families can be found in Supplementary Information, see S1 File . The average accuracy for the first 1,000 (261,149 sequencesterm·key), 2,000 (293,957 sequencesterm·key), 3,000 (308,292 sequencesterm·key), and 4,000 (316,135 sequencesterm·key) frequent families were respectively 94% ± 0.05%,93% ± 0.05%, 92% ± 0.06%, and 91% ± 0.08%. To compute the overall accuracy for all 7,026 families, we calculated the weighted averageterm·stat accuracy, because for the families with number of instances less than 10, the validation set are not statistically sufficient and they should have less contribution in the overall accuracy. The weighted accuracy of all 7,027 families (weighted based on the number of instances) was 93% ± 0.06%.

Table 2. Performance of protein familyterm·dom classificationterm·key using SVM and ProtVecterm·mth over some of the most frequent families in Swiss-Protterm·dom.

Families are sorted with respect to their frequency in Swiss-Protterm·dom.

Training instances Classificationterm·key Result Family name # of positive sequencesterm·key # of negative sequencesterm·key Specificityterm Sensitivityterm Accuracy 50S ribosome-binding GTPase 3,084 3,084 0.95 0.93 0.94 Helicase conserved C-terminal domain 2,518 2,518 0.83 0.80 0.82 ATP synthase alpha-beta family, nucleotide-binding domain 2,387 2,387 0.98 0.97 0.97 7 transmembrane receptor (rhodopsin family) 1,820 1,820 0.95 0.96 0.95 Amino acid kinase family 1,750 1,750 0.91 0.92 0.91 ATPase family associated with various cellular activities (AAA) 1711 1711 0.92 0.90 0.91 tRNA synthetases class I (I, L, M and V) 1,634 1,634 0.97 0.97 0.97 tRNA synthetases class II (D, K and N) 1,419 1,419 0.88 0.83 0.85 Major Facilitator Superfamily 1,303 1,303 0.95 0.97 0.96 Hsp70 proteinterm·key 1,272 1,272 0.97 0.97 0.97 NADH-Ubiquinone-plastoquinone (complex I), various chains 1,251 1,251 0.97 0.97 0.97 Histidine biosynthesis proteinterm·key 1,248 1,248 0.96 0.97 0.97 TCP-1-cpn60 chaperonin family 1,246 1,246 0.95 0.96 0.95 EPSP synthase (3-phosphoshikimate 1-carboxyvinyltransferase) 1,207 1,207 0.96 0.96 0.96 Aldehyde dehydrogenase family 1,200 1,200 0.93 0.94 0.94 Shikimate–quinate 5-dehydrogenase 1,128 1,128 0.87 0.89 0.88 GHMP kinases N terminal domain 1,120 1,120 0.88 0.92 0.90 Ribosomal proteinterm·key S2 1,083 1,083 0.95 0.96 0.95 Ribosomal proteinterm·key S4–S9 N-terminal domain 1,072 1,072 0.95 0.97 0.96 Ribosomal proteinterm·key L16p-L10e 1,053 1,053 0.95 0.96 0.96 KOW motif 1,047 1,047 0.93 0.95 0.94 Uncharacterized protein familyterm·dom UPF0004 1,044 1,044 0.95 0.97 0.96 Ribosomal proteinterm·key S12-S23 1,016 1,016 0.94 0.98 0.96 GHMP kinases C terminal 1,011 1,011 0.88 0.92 0.90 Ribosomal proteinterm·key S14p-S29e 997 997 0.93 0.98 0.95 Ribosomal proteinterm·key S11 980 980 0.96 0.98 0.97 UvrB-uvrC motif 968 968 0.94 0.96 0.95 Ribosomal proteinterm·key L33 958 958 0.96 0.98 0.97 BRCA1 C Terminus (BRCT) domain 956 956 0.94 0.95 0.95 RF-1 domain 950 950 0.93 0.97 0.95 Ankyrin repeats (3 copies) 944 944 0.89 0.88 0.88 Ribosomal proteinterm·key L20 932 932 0.96 0.99 0.97 RNA polymerase beta subunit 912 912 0.94 0.97 0.95 Ribosomal proteinterm·key S18 908 908 0.93 0.97 0.95 ATP synthase B-B CF(0) 900 900 0.92 0.94 0.93 Peptidase family M20-M25-M40 889 889 0.92 0.93 0.93 Ribosomal proteinterm·key L18e-L15 887 887 0.93 0.96 0.95 Glucose inhibited division proteinterm·key A 886 886 0.95 0.96 0.95 NADH-ubiquinone-plastoquinone oxidoreductase chain 4L 885 885 0.94 0.97 0.96 lactate-malate dehydrogenase, NAD binding domain 880 880 0.92 0.94 0.93 HD domain 879 879 0.93 0.93 0.93 Ribosomal proteinterm·key S10p-S20e 873 873 0.95 0.97 0.96 Pyridoxal-phosphate dependent enzyme 870 870 0.91 0.91 0.91 Ribosomal L18p-L5e family 860 860 0.93 0.96 0.94 Ribosomal proteinterm·key L3 855 855 0.94 0.97 0.96 tRNA synthetases class I (M) 843 843 0.94 0.96 0.95 UbiA prenyltransferase family 841 841 0.94 0.95 0.95 Ribosomal proteinterm·key L4–L1 family 841 841 0.94 0.95 0.95 Ribosomal proteinterm·key S16 840 840 0.93 0.97 0.95 Ribosomal proteinterm·key S13-S18 840 840 0.94 0.97 0.95 MraW methylase family 837 837 0.95 0.98 0.96 Ribosomal L32p protein familyterm·dom 825 825 0.94 0.97 0.95 Elongation factor TS 819 819 0.94 0.97 0.96 Tetrahydrofolate dehydrogenase-cyclohydrolase, catalytic domain 817 817 0.94 0.96 0.95 ATP synthase delta (OSCP) subunit 813 813 0.93 0.96 0.94 tRNA synthetases class I (C) catalytic domain 812 812 0.95 0.97 0.96 SecA Wing and Scaffold domain 805 805 0.95 0.97 0.96 Ribonuclease HII 795 795 0.93 0.94 0.93 Ribosomal proteinterm·key L31 795 795 0.97 0.99 0.98 Ribosomal L27 proteinterm·key 794 794 0.98 0.99 0.99 IPP transferase 794 794 0.93 0.95 0.94 GTP-binding proteinterm·key LepA C-terminus 793 793 0.96 0.98 0.97 Ribosomal proteinterm·key L17 791 791 0.92 0.96 0.94 Ribosomal proteinterm·key L23 790 790 0.91 0.96 0.94 Ribosomal proteinterm·key L10 781 781 0.90 0.92 0.91 Ribosomal proteinterm·key L19 780 780 0.94 0.97 0.95 Ribosomal proteinterm·key S20 774 774 0.95 0.97 0.96 Ribosomal proteinterm·key L35 769 769 0.93 0.97 0.95 Phosphoglucomutase-phosphomannomutase, C-terminal domain 768 768 0.92 0.96 0.94 AMP-binding enzyme 767 767 0.87 0.89 0.88 Ribosomal prokaryotic L21 proteinterm·key 766 766 0.93 0.96 0.95 tRNA methyl transferase 759 759 0.94 0.96 0.95 Ribosomal L29 proteinterm·key 757 757 0.95 0.97 0.96 Glycosyl transferase family, a-b domain 754 754 0.90 0.91 0.91 Translation initiation factor IF-2, N-terminal region 750 750 0.96 0.98 0.97 Ribosomal L28 family 749 749 0.93 0.98 0.95 Glycosyl transferase family 4 739 739 0.96 0.98 0.97 tRNA synthetases class I (R) 736 736 0.93 0.96 0.95 Bacterial trigger factor proteinterm·key (TF) C-terminus 733 733 0.95 0.96 0.95 For the first 1,000 families 261,149 261,149 0.92 0.95 0.94 For the first 2,000 families 293,957 293,957 0.90 0.96 0.93 For the first 3,000 families 308,292 308,292 0.89 0.96 0.92 For the first 4,000 families 316,135 316,135 0.87 0.96 0.91 Weighted averageterm·stat for all 7,027 families 324,018 324,018 0.91 0.95 0.93 Disordered Proteinsterm·dom Visualization and Classificationterm·key Due to the functional importance of disordered proteinsterm·dom, prediction of unstructured regions of disordered proteinsterm·dom and determining the sequenceterm·key patterns featured in disordered regions is a critical problem in proteinterm·key bioinformatics. We evaluated the ability of ProtVecterm·mth to characterize and discern disordered proteinterm·dom sequencesterm·key from structured sequencesterm·key.

FG-Nupsterm·dom Characterization In this case study, we used the FG-Nupsterm·dom collection of 1,138 disordered proteinsterm·dom containing disorder regions with a fraction of at least one third of the sequenceterm·key length. For comparison purposes, we also collected two sets of structured proteinsterm·key from Proteinterm·key Data Bank (PDB).

In order to visualize each dataset, we reduced the dimensionality of the protein-spaceterm using Stochastic Neighbor Embeddingterm·key [ 36 , 39 ] and then generated the 2D histogram of all overlapping 3-gramsterm·mth occurring in each dataset. As shown in Fig 3 (see column (b)), the two random sets from structured proteinsterm·key had nearly identical patterns. However, the FG-Nupsterm·dom dataset exhibits a substantially different pattern. To amplify the characteristic of disordered sequencesterm·key we have also examined the histogram of disordered regions of FG-Nupsterm·dom (see Fig 3 , column (a)).

Fig 3. Visualization of protein sequencesterm·dom using ProtVecterm·mth can characterize FGNUPsterm·dom versus Disport disordered sequencesterm·key and structured sequencesterm·key.

Column (a) compares FG Nup sequencesterm·key 2D histogram (at the bottom) with 2D histogram of FG Nup disordered regions (on top). Column (b) compares 2D histogram two random sets of structured sequencesterm·key with the same average length as the FG-Nupsterm·dom. Column (c) compares between 2D histogram of DisProtterm·dom sequencesterm·key (at the bottom) and 2D histogram of DisProtterm·dom disordered regions (on top).

In the next step, we quantitatively evaluated how ProtVecterm·mth can be used to distinguish between FG-Nupsterm·dom versus typical PDB sequencesterm·key using a support vector machineterm·mth binary classificationterm·key. The positive examples were the above mentioned 1,138 disordered FG-Nupsterm·dom proteinsterm·key and negative examplesterm (again 1,138 sequencesterm·key) were selected randomly from PDB with the same average length of disordered sequencesterm·key (≈ 900 residues). We represented each protein sequenceterm·dom as a summation of its ProtVecsterm·mth of all 3-gramsterm·mth. Since on average the length of structured proteinsterm·key were shorter than FG-Nupsterm·dom, in order to avoid trivial cases, the PDB sequencesterm·key were selected in such a way as to maintain the same average length. But still, an accuracy of 99.81% was obtained with high sensitivityterm and specificityterm ( Table 3 ). The distribution of the classified proteinsterm·key in a 2D space is shown in Fig 4 .

Table 3. The performance of FG-Nupsterm·dom disordered proteinterm·dom classificationterm·key in a 10xFold cross-validationterm·stat using SVM.

Sensitivityterm Specificityterm Accuracy 0.9987 0.9974 0.9981 Fig 4. Classificationterm·key of FG-Nupsterm·dom versus PDB structured sequencesterm·key.

In this figure, each point presents a proteinterm·key projected into a 2D space.

DisProtterm·dom characterization In this part, we used DisProtterm·dom consisting of 694 proteinsterm·key presenting 1539 disordered, and 95 ordered regions. We performed the same analysis as we did for FG-Nupsterm·dom with DisProtterm·dom sequencesterm·key (see Fig 3 column (c)). Since the size of DisProtterm·dom was relatively small compared to that of the FG-Nupsterm·dom, the scales of columns (a),(b) were not comparable with column (c) (see Fig 3 ). The visualization of disordered regions of DisProtterm·dom sequencesterm·key ( Fig 3 column (c), on top) revealed a different characteristic than FG-Nupsterm·dom disordered regions ( Fig 3 column (a), on top). A visual comparison between Figs 3 and 2 suggest that the FG-Nupsterm·dom have a significantly higher amount of hydrophobic residues and less polar residues in their disordered regions than the experimentally identified disordered proteinsterm·dom in DisProtterm·dom [ 27 , 29 ]. Additionally, the DisProtterm·dom disordered regions versus DisProtterm·dom ordered regions can be classified with 100% accuracy respectively using SVM and ProtVecterm·mth.

Conclusions An unsupervised data-driven distributed representationterm·mth, called ProtVecterm·mth, was proposed for application of machine learning approaches in biological sequencesterm·key. By training this representationterm·key solely on protein sequencesterm·dom, our feature extractionterm·mth approach was able to capture a diverse range of meaningful physical and chemical propertiesterm. We demonstrated that ProtVecterm·mth can be used as an informative and dense representationterm for biological sequencesterm·key in protein familyterm·dom classificationterm·key, and obtained an average family classification accuracyterm·var of 93%.

We further proposed ProtVecterm·mth as a powerful approach for proteinterm·key data visualization and showed the utility of this approach by providing an example in characterization of disordered proteinterm·dom sequencesterm·key vs. structured protein sequencesterm·dom. Our results suggest that ProtVecterm·mth can characterize protein sequencesterm·dom in terms of biochemical and biophysical interpretations of the underlying patterns. In addition, this dense representationterm·key of sequencesterm·key can help to discriminate between various categories of sequencesterm·key, e.g. disordered proteinsterm·dom. Furthermore, we demonstrated that ProtVecterm·mth was able to identify disordered sequencesterm·key with an accuracy of nearly 100%. The related data is available at: http://llp.berkeley.edu and Harvard Dataverse: http://dx.doi.org/10.7910/DVN/JMFHTN .

Another advantage of this method is that embeddingsterm·key could be trained once and then used to encode biological sequencesterm·key in any given problem. In general, machine learning approaches in bioinformatics can widely benefit from bio-vectors (ProtVecterm·mth and GeneVecterm·mth) representationterm·key. This representationterm·key can be considered as pre-trainingterm·mth for various applications of deep learningterm·ai in bioinformatics. In particular, ProtVecterm·mth can be used in proteinterm·key interaction predictions, structure prediction, and proteinterm·key data visualization.

Supporting Information S1 File. The results of family classificationterm·key task for all 7,027 families. (XLSX) Click here for additional data file. (316.7KB, xlsx) Acknowledgments Fruitful discussions with Kiavash Garakani, Mohammad Soheilypour, Zeinab Jahed, Mohaddeseh Peyro, Hengameh Shams and other members of the Molecular Cell Biomechanics Lab at the University of California Berkeley are gratefully acknowledged.

Data Availability All relevant data are within the paper and its Supporting Information files. Our web-based tools and trained data are available at Life Language Processing website: http://llp.berkeley.edu and through the Dataverse database: http://dx.doi.org/10.7910/DVN/JMFHTN , and will be regularly updated for calculation/classificationterm·key of ProtVecsterm·mth as well as visualization of biological sequencesterm·key.

Funding Statement Financial support from National Science Foundation through a CAREER Award (CBET-0955291) is gratefully acknowledged. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.