Quantification annotation track

For the pre-conference part we make the current version available of the QuantML annotation scheme and markup language, as developed as a basis for the ISO project, and documented in the technical report Semantic Annotation of Quantification in Natural Language (TiCC Technical Report 2020-2, Tilburg University). Participants in the shared task, the 'Quantification Annotation Challenge', are asked to annotate as many sentences as possible of the Quantification Challenge test suite: (.docx file), using QuantML, and to submit a short project note (up t0 4 pages) with their observations on the adequacy of the QuantML scheme and representations, the annotation guidelines, problems encountered in annotating the test sentences or related examples, other properties that should be included, and relations to recently proposed representations of quantification, such as those of the Groningen MeaningBank or AMR annnotations or frame representations. The aim of the shared task is to identify strenghts and weaknesses of the QuantML scheme, to signal problematic issues, to get an impression of annotator agareement, and to make comparisons to other recently proposed representations of quantification phenomena. Questions to consider include the following:

• Can you annotate the test sentence in a satisfactory manner using the QuantML scheme? What problems do you encounter in trying to do so?
• Do you think the sentence should rather be annotated in a different way? Why? How?
• How does the annotation compare to other representations of quantification and scope, for instance with AMR annotation?
• Are the current annotation guidelines adequate for indicating how to make the annotation? How could they be improved?
• What implications and possibilities do you see for the design of an annotation tool for quantification?

Please submit your observations and annotations (in a separate file) before May 1st (extended deadline) on the ISA-17 submission page. The project notes will be reviewed for presentation during the on-site part of the workshop.

Relevant background publications are:

• Bunt, Harry, James Pustejovsky, and Kiyong Lee. "Towards an ISO Standard for the Annotation of Quantification." Proceedings of LREC 2018.
• Bunt, Harry. "An Annotation Scheme for Quantification."Proceedings of IWCS 2019.

Visual representation track

For the pre-conference on-line work, we supply a new VoxML annotation interface and batch of images of objects and people participating in activities and events. Follow this link to the page for the VoxML/Voxicon simulation and reasoning platform, where you find the images to be annotated, using a vocabulary of objects, attributes, actions, and relations. Accompanying the collection of images you also find guidelines for how annotation should be performed, using an annotation template. Participants in the shared task of this track are asked to annotate as many of the images as they can using the provided tool, and to submit a short project note covering the adequacy of the annotation guidelines, of the functionality of the annotation tool, and any properties in the raw image data while annotating, and optional deeper investigations into other things that would be interesting to annotate (other data, properties that should be included, problems pertaining to standardisation, ects.). These project notes will be reviewed for presentations at the on-side portion of the workshop. For any questions please contact the track coordinator James Pustejovsky (jamesp@brandeis.edu).

Background document: the LREC 2016 paper VoxML: A Visualization Modeling Language by James Pustejovsky and Nikhil Krishnaswamy.

For both tracks, annotators should attempt to follow the guidelines and stay within the provided vocabulary where possible, but are permitted to go outside if required. In this case, we ask that they summarily justify their additions and alternatives in their project note. The track organisers will provide feedback on submitted annotations, observations and analyses during the on-site portion of the workshop. This will be followed by a discussion of the entire annotation effort, lessons learned (problems, gaps, suitability of results for ML, automated generations, etc.), and the adequacy and expressiveness of the representation language.Deadline for submission of project note and accompanying annotations is April 21.