Organizers:Krallinger/Valencia

PPI Task Introduction

The aim of this task is to promote the development of automated systems that are able to extract biologically relevant information directly from the literature, in this case related to protein-protein interaction (PPI) annotation information. The resulting text mining tools should be able to improve access to this information for database curators, experimental biologists as well as bioinformaticians.

In order to reinforce the construction of practically usable applications, we will specially encourage the implementation of online systems that produce results in predefined prediction formats and with processing time constraints to facilitate direct comparison and integration of automatically generated results. The posed PPI tasks are directly inspired by the needs of biologists and database curators, following user demands that are based on the general steps underlying the PPI annotation workflow. These tasks cover (1) the selection of relevant articles from PubMed (Article Classification Task - ACT) and (2) linking of article to important experimental methods, i.e. interaction detection methods (Interaction Method Task - IMT).

ACT-BC-III (Article Classification Task – BioCreative III)

A common user need is to determine for a given collection of articles, e.g. defined as a list of PubMed records, derived from a keyword search (e.g. using a particular disease term) or a journal of interest which records are actually PPI relevant (contain descriptions indicating that this article is relevant for PPI article curation). This is as well the case when considering a list of articles related to a protein of interest for which all known interactions need to be extracted, or when some sort of periodic literature curation process is followed, and biologists want to determine which articles for a given time span (e.g. a month) are curation relevant.

To promote the development of such system, participating teams will be provided with a collection of recent PubMed records derived from a standard PubMed query limited in terms of: (a) time (time span of max. a single month) and (b) availability (only considering records with both abstracts and links to free full text articles). The upper boundary of such a query is a collection around 15,000 records. Participating systems will be asked to carry out a binary classification for whether an article contains protein interaction descriptions. The format of the results will be similar to the ACT task of BCII.5. Note that participating systems can also use information from the corresponding full text articles in case their pipeline is able to get access to them, but from the evaluation perspective, only information from the abstracts will be considered.

Evaluation

Evaluation will be based on comparison between automatically generated results and manual examination of a set of PubMed records. A similar set up will be followed as in case of BCII.5 for the evaluation of the systems. Additionally, we plan to evaluate how much time can be saved by using the automatic predictions as compared to unassisted manual classification. Therefore, the amount of time spent for manually labeling the abstracts will be recorded. One can then evaluate systems in terms of how long it would have taken to review the ranked list of articles submitted by participating systems in order to classify all the relevant records. The manual classification will be based on predefined curation guidelines that are refined through feedback of professional biocurators. To determine the difficulty and consistency of this task, an Inter-annotator agreement (IAA) study will be carried out.

ACT results have to be reported as four tab-separated columns:

1. Article identifier.
2. Classification result (0 for negative, 1 for positive hits).
3. Unique rank of that classification result in the range [1..Nc], where Nc is the total number of hits for negative (c=0) or positive (c=1) results.
4. Confidence for that classification in the range ]0..1], i.e., excluding zero-confidence.

IMT-BCIII (Interaction Method Task – BioCreative III)

A crucial aspect for the correct annotation of experimentally determined protein interactions is to determine the technique described in the article to support a given interaction. Experimental techniques or qualifiers are also relevant for other annotations, such as Gene Ontology (evidence codes). This is also important to correctly associate the article to controlled vocabulary terms relevant for biology. This task will be similar in essence to the Interaction Method Subtask of BioCreative II.

In case of protein-protein interaction annotation, efforts have been made to develop a controlled vocabulary about interaction detection methods in order to standardize the terminology important to serve as experimental evidence support. A considerable amount of curation work is devoted to the manual extraction of the experimental evidence supporting protein interaction pairs described in articles. For this task, we will ask participants to provide, for each full text article, a ranked list of interaction detection methods, defined by their corresponding unique concept identifier from the PSI-MI ontology. To browse this ontology, please refer to: http://www.ebi.ac.uk/ontology-lookup/browse.do?ontName=MI

To download the MI ontology please refer to the MI obo download file: http://psidev.sourceforge.net/mi/rel25/data/psi-mi25.obo

IMT results are to be returned in six tab-separated columns, consisting of:

1. Article identifier
2. Interaction Detection Method MI identifier
3. Term (as defined in the MI ontology)
4. Evidence string (max 500 characters) derived from the full text paper
5. Unique rank in the range [1..N], where N is the total number of hits for that article.
6. Confidence for that concept in the range ]0..1], i.e., excluding zero-confidence.

The provided training set will consist of over 1000 full text articles with their corresponding interaction detection method identifiers. The test set will consist in over 100 full text articles for which interaction detection need to be returned by participating systems. Evaluation criteria will be similar to the INT task of BioCreative II.5.

Participation

Same as for BCII.5, with both online and offline participation allowed, however online systems for all the tasks are preferred. We may require this time at least one online system submission per team.