A Product Data Scientist’s Take on LinkedIn Games After 500 Days of Play

I-day streak on LinkedIn Games. Sure, LinkedIn additionally has video games, they usually’ve been round for over a 12 months. Infrequently, I discover new video games, design tweaks, and new options being rolled out. As a Information Scientist, I’ve all the time questioned what LinkedIn is attempting to attain with LinkedIn Video games and the way they’re testing the adjustments. 

With AI augmenting and even automating many coding and primary analytics duties, product sense and area experience turn into increasingly more vital for information scientists. Due to this fact, on this article, I’m utilizing LinkedIn Video games for example to indicate how a Product Information Scientist thinks and works. That is additionally the kind of psychological train I follow when getting ready for product case interviews.

I. What’s the aim of LinkedIn Video games

Step one in any product case is to grasp the product’s aim. Per LinkedIn, “Video games on LinkedIn are thinking-oriented video games that can assist you sharpen your minds, take a fast break, and have the chance to attach with one another and spark conversations.”

These video games are fast mind teasers, so that they do assist customers “sharpen minds and take a fast break” to some extent. However I consider the true intention hides behind the final half — “join with one another and spark conversations”. 

Why does this matter? LinkedIn generates most of its income from expertise options, promoting platform, and premium subscriptions. All of those depend on an lively person base — recruiters want a big pool of lively candidates, advertisers want focused impressions, and the worth of premium subscriptions will increase with the community measurement. Furthermore, the important thing to sustaining an lively person base is person engagement and interactions, which in the end result in increased retention.  

In information language, MAU (month-to-month lively customers) is without doubt one of the commonest metrics to measure the lively person base of a product. MAU in month X = lively customers in month X-1 + customers acquired/resurrected in month X - customers churned in month X. For LinkedIn, I consider LinkedIn Video games is a characteristic that’s designed to develop MAU by lowering the final element, “customers churned this month.”

II. How does LinkedIn Video games obtain this aim

Now that we’re clear in regards to the aim of enhancing retention, the subsequent query is, how does LinkedIn Video games obtain it? I believe there are two mechanisms — direct interactions with LinkedIn Video games and oblique engagement pushed by coming again to the platform. 

1. Direct interactions with LinkedIn Video games

On daily basis, LinkedIn publishes a sport put up and encourages customers to share their scores and suggestions. That is precisely what they meant by “assist you … join with one another and spark conversations.” Under is a screenshot I took at round 10 PM Pacific Time on 11/29 — round 22 hours after the every day Zip sport was refreshed. Yow will discover the entry level to this put up after ending a sport, or it would present up in your homepage. This put up had 1240 reactions and 1370 feedback. Many customers put up their scores and work together with one another. 

This type of social interplay is valued by many LinkedIn customers. Sharing your good sport rating can be just like sharing a small achievement, so it doesn’t work in opposition to the skilled social community picture of LinkedIn. Because of this, LinkedIn Video games creates a community impact that will increase retention. 

2. Oblique engagement from returning to the platform

In the meantime, there are completely folks like me who’re merely hooked on the video games however by no means share scores or touch upon the put up. No interplay doesn’t imply LinkedIn Video games doesn’t obtain the retention aim for this group of customers. The truth that it brings customers again on daily basis is already a powerful retention lever. 

Linked Video games achieves this by making a behavior loop. Let me put it within the Hooked Mannequin (Set off-> Motion -> Reward -> Funding) to unpack it:

1. Set off: Customers are prompted to return by exterior triggers like push notifications and homepage modules, and inside triggers equivalent to the need to take care of a streak.
2. Motion: The puzzles are simple to grasp, low-friction to enter, and fast to play.
3. Reward: Customers get a unique puzzle on daily basis, earn streak badges, and might compete with their connections.
4. Funding: Customers “make investments” by constructing a streak, getting connections to play, sharing outcomes, enhancing their leaderboard rank, and so on. On daily basis, customers construct up sunk effort, making it more durable to cease.

With this behavior loop, customers come again every day. So long as a person opens LinkedIn, there may be additionally an opportunity that they may take a look at different issues, like community updates, messages, job openings, and so on. These actions might all result in significant engagements outdoors of the Video games characteristic and improve general retention. 

III. Experimentations on LinkedIn Video games 

We lined the aim of LinkedIn Video games and the mechanisms behind it— LinkedIn Video games goals to enhance person retention by encouraging interplay on Video games content material and growing general product engagement. As a knowledge scientist, for those who work on this product, a key a part of your job will likely be collaborating with Product Managers, Designers, and Engineers to brainstorm initiatives and run experiments to measure the retention impression. And that is clearly taking place with LinkedIn Video games, as I’ve seen so many design adjustments over time. Let me stroll via some examples and talk about how information scientists will likely be concerned.

1. Entry factors to LinkedIn Video games

Proper now, you possibly can entry LinkedIn Video games via:

• Games hub
• Seek for video games on the LinkedIn Search bar
• My Community web page
• At present’s Video games part underneath LinkedIn Information in your Desktop homepage or Facet panel within the LinkedIn cell app
• Notifications

However this isn’t all the time the case. I keep in mind sooner or later the entry level on the My Community web page disappeared, and I needed to search within the app to search out the video games. However a couple of days later, it appeared once more. The situation of entry factors determines how simple it’s to search out the characteristic, for each new and returning customers. However extra entry factors aren’t all the time higher. Whereas extra entry factors improve visibility, every of them can create a contextual bias — customers who land on My Community would possibly behave in a different way than those that come via a notification— thus, totally different entry level has totally different impacts on engagement and retention. In different phrases, they may cannibalize one another. 

For instance, the My Community entry level sits beneath the invites and above the connection suggestions. When a person visits this web page to play the every day sport, inevitably, they may see the pending invites, and it’ll remind them to take motion — increasing connections is a crucial a part of making a person’s LinkedIn expertise significant and invaluable. In the meantime, in the event that they go to their homepage for the video games, they may as an alternative see different customers’ posts, and usually tend to work together with the posts.

Several types of interactions have totally different impacts on retention, and it’s exhausting to estimate the precise impression of eradicating/including an entry level with out operating an experiment. Now the duty is on the info scientists to design the experiment. 

Right here is the way it might look:

• Experiment design: management = present design, therapy = eradicating the entry level on My Community
• Randomization unit: user-level A/B testing, 50% customers will randomly see the management vs. the therapy design
• Major metric: 7-day retention charge — time window can differ based mostly on how shortly we need to measure the retention impression and any previous learnings. One caveat is that retention is a lagging metric, and LinkedIn Video games may need comparatively low site visitors in comparison with the remainder of the platform, which makes it troublesome to detect retention impression within the brief time period. In that case, the first metric would possibly have to shift to a number one indicator of retention, or information scientists would possibly have to depend on causal inference strategies to estimate the retention carry extra reliably.
• Secondary metrics: % customers performed a sport; % customers interacted with community posts; % customers added connections; Common periods per person; Common time spent per person
• Guardrail metric: common app/web site efficiency

Information scientists will work with the cross-functional group to align on metrics based mostly on the aim of the experiment, run energy evaluation to find out the experiment time size and scope, conduct implementation checks, and ultimately analyze the outcomes to make a name on the most effective mixture of entry factors. 

2. Notifications

A number of months again, I began receiving reminder notifications like “You’re on a xxx-day streak. Play xxx now to maintain it going”. Later, after ending the video games, there may be one other set of notifications saying “congrats on ending xxx”. 

Notifications could be annoying, however they’re very efficient in bringing customers again. For instance, Duolingo is legendary for its inventive and “psychologically manipulative” notifications (I’m on a 1735-day Duolingo streak by the way in which). Their early blog post described how Duolingo used multi-armed bandits to search out the best-performing notification.

Equally, optimizing notifications can have a big impact on LinkedIn Video games. Information scientists can run experiments to check:

1. When to ship the reminder notification — it may very well be throughout lunch break or after work time when customers usually tend to be accessible, or when the person often opens the app, and even once they performed the sport yesterday.
2. When to ship the congrats notification — the congrats notification may very well be used to carry a person again to the app and encourage them to put up their outcomes and work together with different gamers. Equally, we will take a look at sending it proper after ending the sport, or perhaps later within the day when extra customers have performed the sport.
3. The message textual content — ought to the tone be impartial or extra aggressive? How lengthy ought to it’s?
4. The CTA (name to motion) textual content — “Clear up now”? “Play now”? “Prolong your streak”? Completely different textual content on the button might result in totally different click-through charges.
5. Frequency — if a person doesn’t come again to play the sport after the primary notification, ought to we ship one other reminder?

Right here is only a brief listing off the highest of my head, but it surely’s already a number of totally different combos of notification designs. It’s completely doable that textual content A coupled with timing X is best than textual content B coupled with timing Y. Due to this fact, operating experiments for every determination one after the other is each inefficient and will result in a sub-optimal outcome. That’s the reason Duolingo talked about the multi-armed bandit framework above. It’s a framework to check a number of variations concurrently, and in contrast to conventional A/B exams, it hurries up the experiments by robotically diverting extra site visitors to the profitable arms based mostly on a reward perform and lowering the variety of arms within the take a look at shortly. Due to this fact, the multi-armed bandit may very well be very helpful to check LinkedIn Video games notifications. If you wish to be taught extra, here is one other article by Sew Repair on how they use multi-armed bandits of their experimentation platform.

So what’s the info scientist’s function right here? After all, they may brainstorm with the stakeholders to give you totally different variations, outline the reward capabilities (e.g., whether or not a person performs in the present day’s puzzle), run the multi-armed bandit setup, and interpret the outcomes. 

3. Recreation outcomes web page 

One other space the place I’ve seen many adjustments is the sport outcomes web page. After ending the sport, the person first lands on a outcomes abstract, with engaging stats playing cards like “On fireplace 500-day win streak!”, “High 95% All gamers”, and “Smarter than 90% of CEOs”. It additionally has a distinguished “Share” button that prompts you to share your outcomes as a put up or as a direct message to your connections.

After that, there’s a lengthy outcomes web page with seven main sections:

1. Header — how shortly the person solved the puzzle with copy and share CTAs.
2. Connection leaderboard — the place you rank amongst your connections. When you click on on “see full leaderboard”, there are CTAs to nudge connections who haven’t performed in the present day.
3. The “play one other sport” CTA asks you to discover totally different video games.
4. One other abstract panel with extra stats, together with all-time win charge, greatest rating, streak badges, and a push notification toggle.
5. Weekly business and faculty leaderboards with share choices.
6. Hyperlink to the every day sport put up, the place you possibly can react or remark instantly.
7. “Unlock this week’s bonus puzzle” by inviting your connections to play the sport. 

Do you see the sample? Each part has some CTAs to encourage sharing, engagement, or social interplay. Nonetheless, is the present sequence of playing cards the most effective sequence for the retention final result? Is there a greater UI for the outcomes stat playing cards with a better share charge? Do folks even care in regards to the rating of their employer and faculty? 

To reply these questions, a knowledge scientist would design experiments just like the one we mentioned within the entry factors part to measure the click-through charges, interactions, and general retention impression.

To take one step additional, totally different customers may need totally different preferences. For instance, 

• Person A needs to “showcase” their rating and the way a lot smarter they’re than the CEOs, so the present sequence works completely for them, as they’ll do it proper on the primary display. 
• Person B feels a powerful belonging to their group, so they may share the leaderboard, asking coworkers or classmates to affix the sport to enhance the rating of their firm or faculty. Due to this fact, displaying the leaderboards on prime will enhance their engagement. 
• Person C likes to share their puzzle suggestions and talk about with different gamers, then we should always transfer the put up up on the outcomes web page. 
• Person D merely enjoys the sport, and they might invite others to unlock extra puzzles if they’re supplied this feature. However with the present design, they may not even scroll down all the way in which to the underside and miss the “unlock this week’s bonus puzzle” card. 

This type of customized outcomes web page makes a variety of sense theoretically, however how one can make it work is one other sophisticated information science query. Information scientists might section customers based mostly on person profiles and previous actions — for instance, what number of connections they’ve, what number of of their connections have performed the video games, if the person usually posts or feedback, and so on. Then the info scientists might analyze the experiment outcomes by totally different person segments to establish which design works the most effective for every section and give you the personalization technique. To make the system even smarter, information scientists might construct a machine studying mannequin to foretell the cardboard format that maximizes engagement for every person. 

4. Nuance: Community impact

Final however not least, let me discuss an vital nuance for operating experiments on a social platform like LinkedIn — the community impact. A/B testing has a powerful Steady Unit Remedy Worth Assumption (SUTVA), which assumes a person person’s final result is set solely by the therapy they obtain, and isn’t affected by the therapies of different customers. Nonetheless, this doesn’t all the time maintain on social networks. 

Contemplating the LinkedIn Video games instance — Assume we modified the leaderboard UI and, in consequence, customers within the therapy group have a better likelihood to “nudge” their connections. And plenty of “nudged” customers — some within the management group — find yourself enjoying the sport too. This community impact biases the experiment outcome and dilutes the relative impression between therapy and management. LinkedIn has written about this actual problem and walked via how they detected the impression of this interference utilizing cluster-based experiments. Briefly, LinkedIn teams closely-connected customers into one cluster whereas minimizing interactions between clusters, then they use clusters because the randomization unit (customers in the identical cluster both all go to therapy or all go to regulate) to measure the impression with minimal interference. Due to this fact, for adjustments that might have a powerful community impact, the cluster-based experiment is an effective various. Although the trade-off is that cluster-based experiments usually cut back energy, so that they require cautious cluster building and energy evaluation.

LinkedIn Video games appear easy, however behind each button, notification, streak badge, and leaderboard, there may be seemingly a sequence of product hypotheses, experiments, choices, and information science work. 

After all, that is simply my mind train as a LinkedIn person, however I hope this text helps you higher perceive what Product Information Science appears to be like like in follow. When getting ready for product case interviews and even once you discover a brand new characteristic on a product you utilize frequently, you possibly can apply comparable psychological workouts to sharpen your product sense and turn into a stronger information scientist.