HumBL@WWW2019

the third international workshop on Augmenting Intelligence with Bias-Aware Humans­-in-­the-­Loop
co-located with TheWebConf (WWW2019)
(HumL workshop series)

Human-in-the-loop is a model of interaction where a machine process and one or more humans have an iterative interaction. In this paradigm the user has the ability to heavily influence the outcome of the process by providing feedback to the system as well as the opportunity to grab different perspectives about the underlying domain and understand the step by step machine process leading to a certain outcome. Amongst the current major concerns in Artificial Intelligence research are being able to explain and understand the results as well as avoiding bias in the underlying data that might lead to unfair or unethical conclusions. Typically, computers are fast and accurate in processing vast amounts of data. People, however, are creative and bring in their perspectives and interpretation power. Bringing humans and machines together creates a natural symbiosis for accurate interpretation of data.

Crowdsourcing has become a successful method to obtain the human computation needed to augment algorithms and perform high quality data management. Humans, though, have various cognitive biases that influence the way they interpret statements, make decisions and remember information. If we use crowdsourcing to generate ground truth, it is important to identify existing biases among crowdsourcing contributors and analyze the effects that their biases may produce. At the same time, having access to a potentially large number of people can give us the opportunity to handle the biases in existing data and systems.

The goal of this workshop is to bring together researchers and practitioners in various areas of AI (i.e., Machine Learning, NLP, Computational Advertising, etc.) to explore new pathways of the human-in-the-loop paradigm. We aim to analyze both existing biases in crowdsourcing, and explore various methods to manage bias via crowdsourcing. We would like to discuss different types of biases, measures and methods to track bias, as well as methodologies to prevent and mitigate different types of bias. We will provide a framework for discussion among scholars, practitioners and other interested parties, including crowd workers, requesters and crowdsourcing platform managers.

Joint Keynote Series

Program

May 13th 2019, 9am, room RegencyA + RegencyB

Important Dates

Call for Contributions

Topics

• Human Factors:
• Human­­-computer cooperative work
• Mobile crowdsourcing applications
• Human Factors in Crowdsourcing
• Social computing
• Ethics of Crowdsourcing
• Gamification techniques
• Data Collection:
• Data annotations task design
• Data collection for specific domains (e.g. with privacy constraints)
• Data privacy
• Multi­-linguality aspects
• Machine Learning:
• Dealing with sparse and noisy annotated data
• Crowdsourcing for Active Learning
• Statistics and learning theory
• Applications:
• Healthcare
• NLP technologies
• Translation
• Data quality control
• Sentiment analysis
• Bias in Crowdsourcing:
• Contributor and crowd worker sampling bias during the recruitment
• Effect of cultural, gender and ethnic biases
• Effect of worker training and past experiences
• Effect of worker expertise vs interest
• Bias in experts vs bias in crowdsourcing
• Bias in outsourcing vs bias in crowdsourcing
• Sources of bias in crowdsourcing: task selection, experience, devices, reward, etc.
• Taxonomies and categorizations of different biases in crowdsourcing
• Task assignment/recommendation for reducing bias
• Effect of worker engagement on bias
• Responsibility and ethics in crowdsourcing and bias management
• Preventing bias in crowdsourcing
• Creating awareness of cognitive biases among crowdsourcing agents
• Crowdsourcing for Bias Management:
• Identifying new types of cognitive bias in data or content using crowdsourcing
• Measuring bias in data or content using crowdsourcing
• Removing bias in data or content using crowdsourcing
• Presenting bias information to end users to create awareness
• Ethics of data collection for bias management
• Dealing with algorithmic bias using crowdsourcing
• Fake news detection with crowdsourcing
• Diversification of sources by means of crowdsourcing
• Provenance and traceability in crowdsourcing
• Long-term crowd engagement
• Generating benchmarks for bias management through crowdsourcing
  

Authors can submit four types of papers:

• short papers (up to 6 pages in length), plus unlimited pages for references
• full papers (up to 10 pages in length), plus unlimited pages for references
• position papers (up to 4 pages in length), plus unlimited pages for references
• demo papers (up to 4 pages in length), plus unlimited pages for references
Page limits include diagrams and appendices. All submissions must be written in English.
The proceedings of the workshops will be published jointly with the conference proceedings, therefore submissions should be formatted according to the formatting instructions in the General Guidelines for the WebConference and must be submitted in PDF according to the ACM format published in the ACM guidelines, selecting the generic “sigconf” sample. The PDF files must have all non-standard fonts embedded.
Please submit your contributions to EasyChair.

Organization

Program Committee