So we may also want to look at how long users spend in the calendar view to get a better sense. How many % of users click on the calendar view feature? - After users click on the feature, we know no more information if no further interactions are committed.We then look at the corresponding metrics to see if there’s a significant increase in these three key actions in the test group, as compared with control group.īesides that, we also want to know how users interact with the feature as they dig deeper into it. Therefore, we need to track how many docs users create, access and share, on average, every week for the control and test group, respectively. Access(and edit) existing docs ( +28.2%).The key actions users take within Paper and the usage increases we want to achieve are: The first thought that comes to our minds is the metrics that are directly related to the experiment hypothesis, which are the metrics that measure the user’s usage of Paper. We would randomly assign 50% of the users to the test group (feature-switch on), while the other 50% remaining as the control group (feature-switch off). To increase statistical power and reach statistical significance, we want to include all Paper users in this experiment and split them into two groups. Since not every Dropbox user has access to Paper, the user base is fairly small. We need to put those ambiguous words into clear, specific metrics to make sure that everyone is on the same page and is looking at the same direction.įor now, the Dropbox Paper beta is a web-only app that is only available in English. But what is success for this experiment? Who exactly are the “users”? How to define “significant”? How do we measure “increase their usage of Paper”? When we can confidently declare that user’s behavior has been “changed”? Our hypothesis is, by adding the calendar view for Paper feature, we can significantly change user’s behavior to increase their usage of Paper. But qualitative research is no less important than experimentation and normally is before quantitive analysis.) (I omitted the qualitative research since the emphasis of this article was on experimentation. I will put this lifecycle into perspective, giving the context of the calendar view feature experimentation.
0 Comments
Leave a Reply. |