Human-in-the-loop is a model of interaction where a machine process and one or more humans have an iterative interaction. In this paradigm the user has the ability to heavily influence the outcome of the process by providing feedback to the system as well as the opportunity to grab different perspectives about the underlying domain and understand the step by step machine process leading to a certain outcome. Amongst the current major concerns in Artificial Intelligence research are being able to explain and understand the results as well as avoiding bias in the underlying data that might lead to unfair or unethical conclusions. Typically, computers are fast and accurate in processing vast amounts of data. People, however, are creative and bring in their perspectives and interpretation power. Bringing humans and machines together creates a natural symbiosis for accurate interpretation of data at scale. The goal of this workshop is to bring together researchers and practitioners in various areas of AI (i.e., Machine Learning, NLP, Computational Advertising, etc.) to explore new pathways of the humanintheloop paradigm.
The proceedings of the workshop will be published online (open access) and through ACM Digital Library, as a companion volume of The Web Conf. Papers must be submitted in PDF according to the new ACM format published in ACM guidelines, selecting the generic “sigconf” sample. Submissions should not exceed 8 pages including any diagrams or appendices and references. The PDF files must have all non-standard fonts embedded. Submissions must be self-contained and in English. Please submit your contributions to EasyChair and select "Augmenting Intelligence with Humans-in-the-Loop Workshop".
The old paradigm of computing - where machines do something for the humans - has changed: more and more humans and machines are working with and for each other, in a partnership. We can see the effectiveness of this paradigm in many areas, ranging from human computation (where humans do some of the computation in place of the machines), computer-supported cooperative work, social computing, web search, recommender systems, computer-mediated communication, to name a few. The HumL workshop focused on research questions around the design of web information systems that keep humans in the loop. The three main categories of topics covered in this edition of the workshop were human factors, data collections and applications. The paper presentations covered a wide range of topics related to the efficient and effective combination of the strong sides of both machine and crowd computation. Overall the workshop presented a nicely diverse set of use cases and domains where these topics were studied. Empirical results were provided and discussed with respect to (1) methods for data quality ensurance and labeling task efficiency, (2) the role of gamification elements for improving crowd perfomance as well as the role of quantum mathematics to simulate human behavior. The workshop was opened in the morning by a very interesting invited talk by Praveen Paritosh from Google Research, talking about the missing science of knowledge curation. Praveen discussed the use of the right incentives to motivate human contribution to the creation of knowledge resources and intruduced three types of data for training purposes: found data, bought data and vested data and argued on the long term value of producing high quality vested data resources. Amrapali Zaveri discussed findings about optimal amount of workers and tasks per worker in a parallelized and decentralized crowdsourcing setting. They introduce a two-staged statistical guideline, CrowdED, for optimal crowdsourcing experimental design in order to a-priori estimate optimal worker and task assignments to obtain maximum accuracy on all tasks. Anastasia Sofou presented their work on making Cultural Heritage more accessible and reusable. They introduce WITH, an aggregation platform that provides enhanced services and enables human-computer collaboration for data annotations and enrichment. Specifically, the system combines machine intelligence from image and text analysis and human intelligence through gamified crowdsourcing. Giorgio Maria Di Nunzio described experiments for the classification of newswires and systematic medical reviews, where authors use gamification techniques to transform labelling tasks into an interactive learning process. Wei Sun presented results of field experiments on 169 television advertisements and 2334 participants, with a dynamic Bayesian network (DBN)-based system assisting human designers produce effective advertisements with predictable outcomes (e.g. linking advertisement content, viewers emotional responses and effectiveness metrics for ad avoidance, sharing and influence on purchase). Steve Welch presented their study on the tradeoff for machine learning model performance in terms of batch size of the requested annotation labels and the training time in the domain of real-time domain specific relation extraction. The afternoon session was opened by Elena Simperl from the University of Southampton, who generated lots of interest with her research on how humans and bots contribute together to the development of the Wikidata knowledge graph. Chris Welty outlined a new and more general approach to recognizing context, grounded in a fairly simple intuition: is far more appropriate to model human behavior in human in the loop systems using notions from quantum mechanics, such as superpositions of states. Maria Teresa Pazienza presented a hybrind human and machine computation framework for ontology learning which combines machine extraction of triples from heterogeneous sources and human validation of these. Rafael Zequeira discussed the results from a speech quality assessment study in the context of data reliability in crowdsourcing. Specifically, the authors focus on how ``trapping questions'' or ``outlier detection'' assure reliable results, and show that only a combination of both techniques provides quality improvement.