Competition

AutoML Challenge 2018

Overview

Machine learning has achieved great success in online advertising, recommender system, financial market analysis, computer vision, linguistics, bioinformatics and many other fields, but this achievements crucially depend on human machine-learning experts. In almost all these successful machine learning applications, human experts are involved in all machine learning stages including: transforming real world problems into machine learning tasks, collecting data, doing feature engineering, selecting or designing the model architecture, tuning model’s hyper-parameters, evaluating model’s performance, deploying the machine learning system in online systems and so on. As the complexity of these tasks is often beyond non-experts, the rapid growth of machine learning applications has created a demand for off-the-shelf machine learning methods that can be used easily and without expert knowledge. We call the resulting research area that targets progressive automation of machine learning AutoML (Automatic Machine Learning).
The goal of PAKDD 2018 data competition (AutoML Challenge 2018) is to solve real world classification problems without any human intervention.
There are two phases in the competition:

  • The Development phase is a phase with result or code submission, in this phase competitors can improve their AutoML methods by tweaking them on some public datasets we will provide, and run their methods on their own systems (without limitations on computational resources), finally submit only their results to our server, code submission in this phase will NOT be evaluated.
  • The AutoML phase is a phase with code submission: the code submitted will be executed automatically on the challenge servers to train and test learning machines in limited time (a budget of computing time will be assigned to each participant), using new datasets that competitors have never seen before. If the code fails, the participants can re-submit ONE code patch to fix bugs.

Joining the AutoML phase is more appreciated, but a small fraction of the prizes can be won by just joining the Development phase.

Please visit the official competition website for further information https://www.4paradigm.com/competition/pakdd2018

Prizes (sponsored by 4paradigm)

AutoML Phase
– First place: USD 3000 (+$600 Travel Grant)
– Second place: USD 1500 (+$600 Travel Grant)
– Third place: USD 750 (+$600 Travel Grant)
* A fraction of the prize amount might be used as travel grant to attend the conference and workshop.

Evaluation

Performance of submitted results and code will be evaluated by measuring their AUC on testing datasets. The final score will be the average AUCs on all testing datasets and a ranking will be generated from such results.
For the AutoML phase, there is also a time budget for each dataset, the testing result should be presented by the learner within the time budget, if not, the AUC will be set to 0.5.

Dissemination

Top ranked participants will be invited to a workshop collocated with PAKDD 2018 to describe their methods and findings. Winners of prizes are expected to attend.
Also, organizers are making arrangements for the possible publication of a book chapter or article written jointly by organizers and the participants with the best solutions. Details TBA.

Participants

– The competition will be run in the CodaLab competition platform.
– The competition is open for all interested researchers, specialists and students. Members of the Contest Organizing Committee cannot participate.
– Participants may submit solutions as teams made up of one or more persons.
– Each team needs to designate a leader responsible for communication with the Organizers.
– One person can only be part of one team.
– A winner of the competition is chosen on the basis of the final evaluation results. In the case of draws in the evaluation scores, time of the submission will be taken into account.
– Each team is obligated to provide a short report (fact sheet) describing their final solution.
– By enrolling to this competition you grant the organizers rights to process your submissions for the purpose of evaluation and post-competition research.

More detailed terms and conditions will be released in the competition main page: https://www.4paradigm.com/competition/pakdd2018.

Timeline

Competition begins: November 20, 2017 November 27, 2017
Development phase ends: March 5, 2018 (tentative)
AutoML phase ends: March 15, 2018
Deadline to send reports: March 20, 2018 (tentative)
Reports from selected teams: April 15, 2018
Main Conference begins: June 03, 2018

Committee

– Isabelle Guyon, Université Paris-Saclay & ChaLearn, Paris, France (Adviser)
– Wei-Wei Tu, the Fourth Paradigm Inc., Beijing, China (Co-Chair)
– Hugo Jair Escalante, INAOE (Mexico), ChaLearn (USA) (Co-Chair)
– Yuqiang Chen, the Fourth Paradigm Inc., Beijing, China
– Guangchuan Shi, the Fourth Paradigm Inc., Beijing, China
– Hai Wang, the Fourth Paradigm Inc., Beijing, China

Sponsorship

The Fourth Paradigm Inc.(https://www.4paradigm.com/) is the main sponsor.
ChaLearn(http://www.chalearn.org/) and CodaLab(http://codalab.org/) is the platform and data provider.

About

Previous AutoML Challenges can be found in https://competitions.codalab.org/competitions/2321.
AutoML workshops can be found in http://automl.chalearn.org/.
Springer Series on Challenges in Machine Learning can be found in http://www.springer.com/series/15602.
Detailed papers about previous AutoML Challenges:
– I. Guyon et al. A Brief Review of the ChaLearn AutoML Challenge: Any-time Any-dataset Learning Without Human Intervention. ICML W 2016.
– I. Guyon et al. Design of the 2015 ChaLearn AutoML challenge. IJCNN 2015.

More details can be found in the competition main page: https://www.4paradigm.com/competition/pakdd2018