---
marp: true
theme: uncover
color: #eee
colorSecondary: #333
backgroundColor: #111
paginate: true
---

<style>
li  {
  font-size: 0.8em;
}
</style>

# Tester son Modèle
---
## 1. À la main

--- 
## Outils d'introspection:
  - [What If Tool](https://pair-code.github.io/what-if-tool/) - see [Iris demo](https://pair-code.github.io/what-if-tool/demos/iris.html)
  - [TensorFlow Playground](http://playground.tensorflow.org/)
  - [TensorFlow Embedding Projector](https://projector.tensorflow.org/)

---
## 2. Pour éviter des régressions

<!-- 
Comparer aux précédentes métriques
Comparer aux précédentes datas
-->
---
## De manière continue
<!-- 
In your CI!
 -->

---
### Quand la data change
> The tricky part is making the new dataset “compliant” with what the model expects.
> ~Jacek Podlewski, [Quora](https://www.quora.com/How-can-I-test-my-trained-model-on-a-completely-new-dataset)

---
### Quand le modèle évolue
---
### Dans ta CI/CD
![](https://air.imag.fr/images/5/55/Demo_2.PNG)

_CircleCI, a continuous integration platform_

- Ref: [CircleCI ML Guide](https://circleci.com/blog/increase-reliability-in-data-science-and-machine-learning-projects-with-circleci/)
- Ref: [Valohai's Colecting User Feedback](https://valohai.com/blog/cicd-for-machine-learning/)


---

# Évaluer un modèle

Sur les métriques visées

--- 

Sur ses biais éventuels

- [TDS: 3 common strategies to measure bias in NLP models](https://towardsdatascience.com/3-common-strategies-to-measure-bias-in-nlp-models-2022-b948a671d257)

- Par perturbation/permutation: e.g. [Reducing Sentiment Bias in Language Models
via Counterfactual Evaluation](https://arxiv.org/pdf/1911.03064.pdf)
  - [CHECK LIST!](https://aclanthology.org/2020.acl-main.442.pdf)
    - https://github.com/marcotcr/checklist

---

### Adversarial perturbations 

![bg right:40% fit](https://i0.wp.com/bdtechtalks.com/wp-content/uploads/2020/11/adversarial-js.jpg?resize=1024%2C568&ssl=1)

- [See Christophm's Interpretable Machine Learning: Something is Wrong With my Dog](https://christophm.github.io/interpretable-ml-book/adversarial.html)

- Pour Aller plus loin : [Adversarial Robustness - Theory and Practice](https://adversarial-ml-tutorial.org/)


---

## Avec des users différents

![bg right fit](https://images.unsplash.com/photo-1587440871875-191322ee64b0?ixlib=rb-4.0.3&ixid=MnwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8&auto=format&fit=crop&w=1771&q=80)
<!-- Image credit: UX Indonesia, Unsplash.com -->

- [Case Study by _dscout_: User Research for ML Systems](https://dscout.com/people-nerds/user-research-for-machine-learning)

---

![bg right fit](https://www.boxuk.com/wp-content/uploads/2019/09/blog-post-teaser-guerrilla-testing-square.jpg)
<!-- Image credit: BoxUK | https://www.boxuk.com/insight/unboxing-guerrilla-usability-testing/ -->

### Avec du 
### Guerrilla Testing
- [How to do Guerrilla Testing](https://medium.com/one-more-thing-studio/how-to-do-guerrilla-testing-right-f36beb186ca)
- [Designing a Chatbot Personality](https://chatbotsmagazine.com/designing-a-chatbots-personality-52dcf1f4df7d)