the second international workshop on Augmenting Intelligence with Humans­-in-­the-­Loop
co-located with ISWC 2018
Workshop date: 9 October 2018
(HumL workshop series)

Human­-in-­the-­loop is a model of interaction where a machine process and one or more humans have an iterative interaction. In this paradigm the user has the ability to heavily influence the outcome of the process by providing feedback to the system as well as the opportunity to grab different perspectives about the underlying domain and understand the step by step machine process leading to a certain outcome. Amongst the current major concerns in Artificial Intelligence research are being able to explain and understand the results as well as avoiding bias in the underlying data that might lead to unfair or unethical conclusions. Typically, computers are fast and accurate in processing vast amounts of data. People, however, are creative and bring in their perspectives and interpretation power. Bringing humans and machines together creates a natural symbiosis for accurate interpretation of data at scale. The goal of this workshop is to bring together researchers and practitioners in various areas of AI (i.e., Machine Learning, NLP, Computational Advertising, etc.) to explore new pathways of the human­in­the­loop paradigm.

Keynote Speaker

Alyssa Simpson Rochwerger
Figure Eight

Reality check: How businesses are using Human In the Loop processes to drive real value

Artificial Intelligence - everyone is talking about it but who is actually doing it (and generating business results). This session takes an industry by industry perspective on true AI adoption, disambiguating the hype from the reality, the theoretical from the practical and the research labs from Return Of Investment. Alyssa will showcase companies getting actual real value from leveraging AI and discuss ideas around how any company, from SMB to enterprise, can use AI within their own business and industry.


Important Dates

All dates are 23:59 Hawaii Time
  • Abstract submission: asap before paper deadline
  • Paper submission deadline: 1 June 2018
  • Author notification: 27 June 2018
  • Early-bird registrations deadline: 29 June 2018
  • Final version deadline: 24 July 2018
  • Workshop date: 9 October 2018

Call for Contributions


  • Human Factors:
    • Human­computer cooperative work
    • Mobile crowdsourcing applications
    • Human Factors in Crowdsourcing
    • Social computing
    • Ethics of Crowdsourcing
    • Gamification techniques
  • Data Collection:
    • Data annotations task design
    • Data collection for specific domains (e.g. with privacy constraints)
    • Data privacy
    • Multi­linguality aspects
  • Machine Learning:
    • Dealing with sparse and noisy annotated data
    • Crowdsourcing for Active Learning
    • Statistics and learning theory
  • Applications:
    • Healthcare
    • NLP technologies
    • Translation
    • Data quality control
    • Sentiment analysis

All submissions must be written in English. We accept the following formats of submissions:

  • Full paper with a maximum of 12 pages including references.
  • Short paper with a maximum of 6 pages including references.

Two formats are possible for the submission: PDF and HTML. PDF submissions must be formatted according to the information for LNCS Authors ( We would like to encourage you to submit your paper as HTML, in which case you need to submit a zip archive containing an HTML file and all used resources. If you are new to HTML submission these are good places to start:

In order to check if your HTML submission is compliant with the page limit constraint, please use one of the LNCS layouts and printing/storing it as PDF. Please submit your contributions electronically in PDF or HTML format to EasyChair
Accepted papers will be published online via CEUR-WS.


Lora Aroyo VU University Amsterdam

Gianluca Demartini University of Queensland, Australia

Anna Lisa Gentile IBM Research Almaden

Chris Welty Google

Program Committee

  • Michele Catasta, Stanford University
  • Irene Celino, CEFRIEL
  • Alessandro Checco, University of Sheffield
  • Anni Coden, IBM Research
  • Philippe Cudre-Mauroux, University of Fribourg
  • Djellel E. Difallah, NYU Center for Data Science
  • Anca Dumitrache, VU University Amsterdam
  • Giorgio Maria Di Nunzio, University of Padova
  • Ujwal Gadiraju, L3S Research Center
  • Daniel F. Gruhl, IBM Research
  • Oana Inel, VU University Amsterdam
  • Ismini Lourentzou, University of Illinois at Urbana - Champaign
  • Silvio Peroni, University of Bologna
  • Marta Sabou, Vienna University of Technology
  • Cristina Sarasua, University of Zurich
  • Maja Vukovic, IBM Research
  • Jie Yang, University of Fribourg
  • Amrapali Zaveri, Maastricht University

Contact Us