Tag Archive: model evaluation

Hello my AI friends!

Today, I would like to share with you skrobot!

skrobot is an open-source Python module I have created at Medoid AI for automating Machine Learning (ML) tasks. It is built on top of scikit-learn framework and follows object-oriented design (OOD) for ML task creation, execution, and reproducibility. Multiple ML tasks can be combined together to implement an experiment. It also provides seamless tracking and logging of experiments (e.g. saving experiments’ parameters).

It can help Data Scientists and Machine Learning Engineers:

  • to keep track of modelling experiments / tasks
  • to automate the repetitive (and boring) stuff when designing ML modelling pipelines
  • to spend more time on the things that truly matter when solving a problem

The current release of skrobot (1.0.13) supports the following ML tasks for binary classification problems:

  • Automated feature engineering / synthesis
  • Feature selection
  • Model selection (i.e. hyperparameters search)
  • Model evaluation
  • Model training
  • Model prediction

For more information you can check out the online documentation!

Lastly, many thanks to all contributors who helped to extend and support skrobot.

Stay safe!

Assume a train/validation/test split and an error metric for evaluating a machine learning model.

In case of high validation/test errors something is not working well and we can try to diagnose if it is a high-bias or high-variance problem.

When both the training and validation errors are high then we have a high-bias model (underfitting). When we have a high validation error and low training error we have a high-variance model (overfitting).

Also, there is a case where both validation and training errors are low but test error is high. This can happen either because we test the model in data from a different world (different data distribution) or we have overfitted to the hyperparameters on validation data, same as when overfitting the models’s parameters on training data.

Machine Learning practitioners always focus on finding in an unbiased way models where validation and training errors are low. When it happens, we say “this a good fit” and then we monitor the model on unseen test data to see how it performs in the real world.