About the role
As an Experimentation & Analysis Manager, you'll use your combination of marketing expertise and analytical skills to run growth experiments, analyze the results, identify impactful growth opportunities, and educate growth teams on rigorous experimentation. You'll work with the Growth Strategy & Operations and Growth Data Science and Engineering teams to conceptualize and build the foundation for experimentation at Shopify.
We care about growing creative, collaborative people, and strong, capable teams. Your peers in Shopify's Growth Org have diverse backgrounds, skill sets, and experiences but share an established track record of driving significant business growth.
The Experimentation & Analysis Manager will be responsible for embedding within bottom-of-the-funnel Growth teams to lead all parts of the experimentation process, from hypothesis generation and experimental design to statistical analysis and internal education on rigorous experimentation practices. Your success will hinge on your ability to strategically design product and email experiments, rigorously analyze and present the results, and enable your colleagues through best practices and streamlined workflows.
You'll be doing things like:
- Strategize and lead the experimentation roadmaps for high-growth retention teams.
- Conduct quantitative modeling to understand incremental benefit and potential cannibalization from financial incentives.
- Collaborate with conversion rate optimizers in Growth to help them run smarter experiments, faster.
- Deep dive on experimentation workflows, educating and enabling your colleagues to run more rigorous experiments.
- Work, as part of a multidisciplinary team, on answering questions and unlocking growth through experimentation.
- Use your analytical skills to find new opportunities the company should be exploring and experimenting with.
- Influence the roadmaps of other teams and help craft the future direction of the company.
You'll need to have:
- Expertise in experimental design and analysis, including A/B, and full and partial factorial designs that control for threats to validity. Experience experimenting with small sample sizes and stratifying to ensure samples are balanced.
- Strong experience working with high-velocity experimentation programs. You’re familiar with the entire experimentation process, from ideation through to iteration. You’ve run experiments concurrently and successfully managed that complexity.
- Deep domain expertise, and understanding of the nuances of both conversion rate optimization and retention. Experience understanding and communicating long-term effects, and developing short-term proxy metrics.
- Experience with analytics tools, such as Excel, Mode, Tableau, Amplitude, and/or Google Analytics.
- Strong communication and leadership skills, including setting an experimentation strategy and designing experimentation workflows, managing stakeholders’ requests, etc.
- Experience with Bayesian statistics and mathematical concepts, such as sample size, credible intervals, etc.
- Experience working with different stakeholders across multiple disciplines, including data scientists, engineers, customer success, product managers, and marketers.
It would be wonderful if you also had:
- Experience working in SaaS or other technology companies. It’s useful to understand our industry and business model.
- Experience with SQL and/or Python. You may not be able to write complex queries from scratch, but you have a high-level understanding of databases and how to query them.
- Strong project management and prioritization skills with an approach that defaults to action. You can drive good decisions quickly and help other teams do the same.
Shopify is now permanently remote, and working towards a future that is digital by default. Learn more about what this can mean for you.
Our belief is that a strong commitment to diversity & inclusion enables us to truly make commerce better for everyone. We encourage applications from Indigenous peoples, racialized people, people with disabilities, people from gender and sexually diverse communities, and/or people with intersectional identities. Please take a look at our Sustainability Reports to learn more about Shopify’s commitments to our communities, and our planet.
At Shopify, we understand that experience comes in many forms. We’re dedicated to adding new perspectives to the team - so if your experience is this close to what we’re looking for, please consider applying.