Mutivariate Testing In The Wrong Hands!ByRaymond D. Matkowsky
I have recently read a blog, entitled “Why MVT doesn’t live up to the hype & isn’t worth significant investment.” The author goes on to dispute the value of multivariate (MVT) testing in favor of simple A/B testing. I agree with the author on certain points, but he fails to see that misuse is the source of the problem not the test itself.
First of all, there is nothing as bad as a productivity tool that makes you unproductive by giving you the wrong answer. I have been doing multivariate testing for forty plus years. An early employer sent me to a MVT all day class, once a week for six months to learn how to do the analysis. I object to these tools being sold as commodities. It takes training and experience to handle such an analysis properly. You do not plug in some numbers into a computer program and have it spit out your answer. I agree with the author of the blog when he says that you must have a hypothesis and a desired outcome behind each number. You need a trained person for that.
It is also a desired outcome of the test to sense how one particular variable interacts and to what extent with other factors. This is something that A/B testing does not do. For a real life example, suppose a concentration of catalyst is more effective at a lower temperature than the one being used. Or, a certain reaction will proceed as desired at one pH and not another. Sometimes it is a factor that is considered insignificant that plays a major role in the outcome. These observations have happened to me during my career.
The author of the blog also mentions that it is not unusual for a pre-packaged program to test 16, 32, or 72 versions at once. This demonstrates how the lack of knowledge and experience will result in gibberish. You should not test any more than four variables at once. Ideally, you should restrict your tests to three to obtain some meaningful results. View your system as a pie. You can slice this pie an infinite number of times, but sooner or later each slice will be equal to the thickness of your knife. Think of the knife as error. Your slices will fall into the range of error. Every possible test, even A/B, has an element of error to it. A/B doesn’t even try to measure this error, whereas MVT does.
True, MVT will take longer upfront than A/B, especially if you are first trying to formulate your hypothesis. However, in the long run MVT will save you much more time than a series of A/B tests.
If you are just going to check one factor against another, by all means, A/B testing is the quickest route. However, if you are going to check more than one, MVT is a better bet. Just make sure you have a knowledgeable person conducting the test.
Do you have any other suggestions, please share them with your fellow readers. Email me at rdm@datastats.com.
Copyright © 2015 Raymond D. Matkowsky
Raymond D. Matkowsky is the Chief Executive Officer of Data Stats, a consulting firm specializing in system or product improvement through mathematical and scientific modeling. He can be reached at rdm@datastats.com or through Data Stats’ web site at www.datastats.com |
