Why Agile principles aren't suitable for data sciences projects? ❌

First things first, Agile principles are all right when correctly using proven frameworks.

Agile principles introduced the idea that the functional perimeter can change along the project's lifetime and that product's value should be carefully assessed to make sure not to build something the end-user doesn't want or doesn't need wasting time and money.

For any client/user, what is important is to control budget, time to market, and value delivered. And for too long, the functional scope was considered as a proxy for value.

Guess what? A better proxy for value is users' feedbacks. That's why Agile frameworks are built to focus on getting feedbacks regularly and challenge the product backlog accordingly. Having regular touchpoints with them - during sprint reviews - is the opportunity to add OR REMOVE features from the backlog.

Agile principles eliminate outdated tensions and sterile discussions over what the 'business' wrote on the 60-pages long requirements .docx 8 months ago. And we're all glad it did!

An additional benefit of Agile frameworks is that it offers the opportunity to use the scope as a pressure relief valve. When your project is over-constrained, you can discuss with the user which features not to deliver instead of skipping tests telling no one (quality drifting), missing delivery date (planning drifting), or doubling the size of your team (cost drifting πŸ’Έ).

As a matter of fact, in Scrum, whatever the team committed to, the increments delivered only contain features meeting the definition of done requirements. Therefore, increments delivered during a sprint are most of the time a subset of all user stories/features selected during the sprint planning. Meanwhile, the Scrum Master makes sure the team and sprints size (ie duration and unit cost of an iteration) aren’t changed under the product owner and other stakeholders' pressure (ie duration and unit cost of an iteration).

In a data science project, you often build three core components:

  • A β€œmodel” πŸ€– with capabilities to predict/classify/identify clusters or outliers...
  • Web services allowing applications to request inferences to the model
  • A solution (ex.: a web app πŸ‘©β€πŸ’») leveraging those web services to deliver value to users

The last two points are more of a classical software engineering nature and project managers may/should handle them according to the Agile principles.

In another hand, the β€œmodel” you want to implement has a unique feature. In that case, how can you use the functional perimeter as an adjustment variable when a project hazard occurs?

Don’t do Agile for a one-feature project! 🚨 Project managers should adopt specific project management strategies to make data science projects happen. I will cover this topic in a future blog post.


Want to know more about Data Science and Project Management applied to β€œdata”? Subscribe (it’s free!).

Paul-Mehdy M'Rabet

Paul-Mehdy M'Rabet

I help CIO and CDO from global manufacturers and industrial leaders to design ambitious roadmap their teams can actually deliver and I love it! I write about #ai, #data and #finance.