Background

An instant messaging company wants to conduct an A/B test on how the change of providing a preview to the in-app share link will affect the click-through rate of the link. The users are randomly assigned into three variants: control and two treatment groups.

Description of Variants

There are three different groups in the experiment, two of which are treatment groups and the other one is a control group. Each group has 10000 users. The description of each variant is as follows:

• Treatment A: Provide a preview of the content along with the share link.
• Treatment B: Provide a preview of the content and a thumbnail along with the share link.
• Control: Only print the share link itself in plain text.

Compare Means on User Characteristics

You are going to test whether the following 16 user characteristics are different between any of two variants. For example, you will compare the user age between control and Treatment A, between Control and Treatment B, and between Treatment A and Treatment B. Their definitions are in the chart below.

Attr. Name	Definition	Format (Unit)
age	Age of each user	38 (years old)
wage	Annual salary of each user	(US$) 10000
eduLv	Highest education degree, such as high school diploma, etc.	4 (means UG)
regDate	Registry date of a user's account	DD-MM-YYYY
friendSum	The number of a user's friends on the day right before the experiment	135 (people)
followSum	The number of a user's followings on the day right before the experiment	87 (people)
groupSum	The number of chat groups a user participated in on the day right before the experiment	62 (groups)
tagSum	The number of tags a user intersted in on the day right before the experiment	19 (tags)
loginFreq	Weekly frequency of login before the experiment	12 (times/week)
actTime	The average active minutes of a user per week before the experiment	1294 (mins/week)
msgSent	The number of messages a user sent per week before the experiment	315 (msgs/week)
postFreq	The number of posts a user published per month before the experiment	19 (posts/mon)
shareFreq	The number of posts a user shared per month before the experiment	26 (posts/mon)
clickCount	The number of times a user viewed others' posts per month before the experiment	42 (times/mon)
likeCount	The number of times a user liked others' posts per month before the experiment	14 (times/mon)
cmtCount	The number of a user's comments to others' posts per month before the experiment	8 (cmts/mon)

Task 1

In this task, you will check whether there is significant difference in the median of the user characteristics ['wage','friendSum','followSum'] between any two variants. Summarize your results and try to interpret them.

Task 2

a. Would these 16 user characteristics be affected by the treatments? Give your answer and briefly explain the reason.

b. Apply t-test on those variables, report the p-value, condifence interval, and the alpha value you use, interpret your action and the result considering the Type I error.

c. Identify the variables that are significantly different between groups. Are those the violations of randomization? Give your answer and briefly explain the reason.