Automatic regression tests with change detection in blue/green deployments

Our product is almost fully deployed in CI, however one remaining part that we have to wrap up is still the regression (update: characterization / golden master) testing: So considering we have a blue/green deployment, we would test the new (green) vs the old (blue) application and see where user queries would produce a different result. In case we do produce different results, the CI environment should be able to resolve this based on expectation: If the expectation is that there is a delta, it should take that into account. If there was a delta when there shouldn't have been, it should abort the CI and notify the DevOps engineer. What idiomatic and practicable solutions exist to tackle this problem? I'm sure it has already been solved...

Mar 13, 2025 - 19:55

Our product is almost fully deployed in CI, however one remaining part that we have to wrap up is still the regression (update: characterization / golden master) testing: So considering we have a blue/green deployment, we would test the new (green) vs the old (blue) application and see where user queries would produce a different result. In case we do produce different results, the CI environment should be able to resolve this based on expectation: If the expectation is that there is a delta, it should take that into account. If there was a delta when there shouldn't have been, it should abort the CI and notify the DevOps engineer.

What idiomatic and practicable solutions exist to tackle this problem? I'm sure it has already been solved...