The Final Frontier? Testing in Production - Marcel Gehlen and Benjamin Hofmann

No tester wants to hear a developer say “It works on my machine!” because what it actually says is: “Since it worked on my development environment I assume it also works on your test environment hence you cannot possibly have found a bug."

We know this not to be true, yet make the same assumption between environments in a later stage: We test our software on test environments and assume that our test results carry over to production. We are actually not testing the software in the setting where our users are facing it.

To top it off, we spend a considerable amount of money trying to copy production. Managing test environments is often hard, complex and needs a lot of maintenance effort.

A lot of people are already using techniques, which take testing into production like Beta Testing, A/B Testing or Monitoring as Testing. We intend to push the envelope a little further and additionally move acceptance testing, automated checks or exploration to the production stage. To do so we need to take several things into consideration, such as making sure test data does not mess up productive data and analytics, as well as hiding untested features from customers.

In this talk you will learn about popular testing in production techniques. We also want to show you some strategies, which help tackling common constraints you will face and provide you with an approach to gradually shift your testing to production.

Key Takeaways:

• introduction into popular testing in production techniques like Beta Testing, A/B Testing or Monitoring as Testing
• strategies for tackling common constraints when trying to test in production
• an approach how you can gradually shift your testing from testing environments to production