| --- |
| license: apache-2.0 |
| language: |
| - en |
| metrics: |
| - accuracy |
| library_name: transformers |
| pipeline_tag: text-classification |
| tags: |
| - customer-service-tickets |
| - github-issues |
| - bart-large-mnli |
| - zero-shot-classification |
| - NLP |
| widget: |
| - text: "Sign up form is not working" |
| example_title: "Example 1" |
| - text: "json and yaml support" |
| example_title: "Example 2" |
| - text: "fullscreen and tabs media key don't do what they should" |
| example_title: "Example 2" |
| --- |
| |
| # GitHub issues classifier (using zero shot classification) |
|
|
| Predicts wether a statement is a feature request, issue/bug or question |
|
|
| This model was trained using the [**Zero-shot classifier distillation**](https://github.com/huggingface/transformers/tree/main/examples/research_projects/zero-shot-distillation) method |
| with the [BART-large-mnli](https://huggingface.co/facebook/bart-large-mnli) model as teacher model, to train a classifier on Github issues from the [Github Issues Prediction dataset](https://www.kaggle.com/datasets/anmolkumar/github-bugs-prediction) |
|
|
| ## Labels |
|
|
| As per the dataset Kaggle competition, the classifier predicts wether an issue is a bug, feature or question. After playing around with different labels pre-training I've used a different mapping |
| of labels that yielded better predictions (see notebook [here](https://www.kaggle.com/code/antoinemacia/zero-shot-classifier-for-bug-analysis/edit) for details), labels being |
|
|
| * issue |
| * feature request |
| * question |
|
|
|
|
| ## Training data |
|
|
| * 15k of Github issues titles ("unlabeled_titles_simple.txt") |
| * Hypothesis used: "This request is a {}" |
| * Teacher model used: valhalla/distilbart-mnli-12-1 |
| * Studend model used: distilbert-base-uncased |
|
|
| ## Results |
|
|
| Agreement of student and teacher predictions: **94.82%** |
|
|
| See [this notebook](https://www.kaggle.com/code/antoinemacia/zero-shot-classifier-for-bug-analysis/edit) for more info on feature engineering choice made |
|
|
|
|
| ## How to train using your own dataset |
| * Download training dataset from https://www.kaggle.com/datasets/anmolkumar/github-bugs-prediction |
| * Modify and run convert.py, updating the paths to convert to a CSV |
| * Run distill.py with the csv file (see [here](https://github.com/huggingface/transformers/tree/main/examples/research_projects/zero-shot-distillation) for more info) |
|
|
| ## Acknowledgements |
|
|
| * Joe Davison and his article on [Zero-Shot Learning in Modern NLP](https://joeddav.github.io/blog/2020/05/29/ZSL.html) |
| * Jeremy Howard, fast.ai and his notebook [Iterate like a grandmaster](https://www.kaggle.com/code/antoinemacia/iterate-like-a-grandmaster) |
