Conducting A/B Testing On Sites

As UX researchers, we often encounter a common challenge: deciding whether one design truly outperforms another. Maybe one version of an interface feels faster or looks cleaner. But how do we know if those differences are meaningful - or just the result of chance? To answer that, we turn to statistical comparisons. When comparing numeric metrics like task time or SUS scores, one of the first decisions is whether you’re working with the same users across both designs or two separate groups. If it's the same users, a paired t-test helps isolate the design effect by removing between-subject variability. For independent groups, a two-sample t-test is appropriate, though it requires more participants to detect small effects due to added variability. Binary outcomes like task success or conversion are another common case. If different users are tested on each version, a two-proportion z-test is suitable. But when the same users attempt tasks under both designs, McNemar’s test allows you to evaluate whether the observed success rates differ in a meaningful way. Task time data in UX is often skewed, which violates assumptions of normality. A good workaround is to log-transform the data before calculating confidence intervals, and then back-transform the results to interpret them on the original scale. It gives you a more reliable estimate of the typical time range without being overly influenced by outliers. Statistical significance is only part of the story. Once you establish that a difference is real, the next question is: how big is the difference? For continuous metrics, Cohen’s d is the most common effect size measure, helping you interpret results beyond p-values. For binary data, metrics like risk difference, risk ratio, and odds ratio offer insight into how much more likely users are to succeed or convert with one design over another. Before interpreting any test results, it’s also important to check a few assumptions: are your groups independent, are the data roughly normal (or corrected for skew), and are variances reasonably equal across groups? Fortunately, most statistical tests are fairly robust, especially when sample sizes are balanced. If you're working in R, I’ve included code in the carousel. This walkthrough follows the frequentist approach to comparing designs. I’ll also be sharing a follow-up soon on how to tackle the same questions using Bayesian methods.

How we boosted test productivity by 50% 🚀 Test design was eating 30% of our time. Worse, we kept rewriting what we already had. Sound familiar? Our short release cycles made it tough. Most sprints had bug fixes, not big new features. Yet we still wrote new test cases every time. Why? Because no one could find the old ones. The search for test cases was taking hours. All our test cases? Dumped in folders. No structure. No reuse. No time to breathe. So we made one big change: 🧭 Built a functional test map of the app 📁 Organised cases in our test management tool ♻️ Set a 20% reusability goal per sprint 📚 Created a library of 250+ reusable cases The result? ✅ Standardised design ✅ Less rework ✅ 50% productivity gain in test creation If you're not reusing test design, you're wasting effort. Have you tried building a reusable test case library?

Don’t Focus Too Much On Writing More Tests Too Soon 📌 Prioritize Quality over Quantity - Make sure the tests you have (and this can even be just a single test) are useful, well-written and trustworthy. Make them part of your build pipeline. Make sure you know who needs to act when the test(s) should fail. Make sure you know who should write the next test. 📌 Test Coverage Analysis: Regularly assess the coverage of your tests to ensure they adequately exercise all parts of the codebase. Tools like code coverage analysis can help identify areas where additional testing is needed. 📌 Code Reviews for Tests: Just like code changes, tests should undergo thorough code reviews to ensure their quality and effectiveness. This helps catch any issues or oversights in the testing logic before they are integrated into the codebase. 📌 Parameterized and Data-Driven Tests: Incorporate parameterized and data-driven testing techniques to increase the versatility and comprehensiveness of your tests. This allows you to test a wider range of scenarios with minimal additional effort. 📌 Test Stability Monitoring: Monitor the stability of your tests over time to detect any flakiness or reliability issues. Continuous monitoring can help identify and address any recurring problems, ensuring the ongoing trustworthiness of your test suite. 📌 Test Environment Isolation: Ensure that tests are run in isolated environments to minimize interference from external factors. This helps maintain consistency and reliability in test results, regardless of changes in the development or deployment environment. 📌 Test Result Reporting: Implement robust reporting mechanisms for test results, including detailed logs and notifications. This enables quick identification and resolution of any failures, improving the responsiveness and reliability of the testing process. 📌 Regression Testing: Integrate regression testing into your workflow to detect unintended side effects of code changes. Automated regression tests help ensure that existing functionality remains intact as the codebase evolves, enhancing overall trust in the system. 📌 Periodic Review and Refinement: Regularly review and refine your testing strategy based on feedback and lessons learned from previous testing cycles. This iterative approach helps continually improve the effectiveness and trustworthiness of your testing process.

After mentoring 50+ QA professionals and collaborating across cross-functional teams, I’ve noticed a consistent pattern: Great testers don’t just find bugs faster — they identify patterns of failure faster. The biggest bottleneck isn’t just in writing test cases. It’s in the 10-15 minutes of uncertainty, thinking: What should I validate here? Which testing approach fits best? Here’s my Pattern Recognition Framework for QA Testing 1. Test Strategy Mapping Keywords:“new feature”, “undefined requirements”, “early lifecycle” Use when feature is still evolving — pair with Product/Dev, define scope, test ideas, and risks collaboratively. 2. Boundary Value & Equivalence Class Keywords: “numeric input”, “range validation”, “min/max”, “edge cases” Perfect for form fields, data constraints, and business rules. Spot breakpoints before users do. 3. Exploratory Testing Keywords: “new flow”, “UI revamp”, “unusual user behavior”, “random crashes” Ideal when specs are incomplete or fast feedback is required. Let intuition and product understanding lead. 4. Regression Testing Keywords: “old functionality”, “code refactor”, “hotfix deployment” Always triggered post-deployment or sprint-end. Automate for stability, manually validate for confidence. 5. API Testing (Contract + Behavior) Keywords: “REST API”, “status codes”, “response schema”, “integration bugs” Use when backend is decoupled. Postman, Postbot, REST Assured — pick your tool, validate deeply. 6. Performance & Load Keywords: “slowness”, “timeout”, “scaling issue”, “traffic spike” JMeter, k6, or BlazeMeter — simulate real user load and catch bottlenecks before production does. 7. Automation Feasibility Keywords: “repeated scenarios”, “stable UI/API”, “smoke/sanity” Use Selenium, Cypress, Playwright, or hybrid frameworks — focus on ROI, not just coverage. 8. Log & Debug Analysis Keywords: “not reproducible”, “backend errors”, “intermittent failures” Dig into logs, inspect API calls, use browser/network tools — find the hidden patterns others miss. 9. Security Testing Basics Keywords: “user data”, “auth issues”, “role-based access” Check if roles, tokens, and inputs are secure. Include OWASP mindset even in regular QA sprints. 10. Test Coverage Risk Matrix Keywords: “limited time”, “high-risk feature”, “critical path” Map test coverage against business risk. Choose wisely — not everything needs to be tested, but the right things must be. 11.Shift-Left Testing (Early Validation) Keywords: “user stories”, “acceptance criteria”, “BDD”, “grooming phase” Get involved from day one. Collaborate with product and devs to prevent defects, not just detect them. Why This Matters for QA Leaders? Faster bug detection = Higher release confidence Right testing approach = Less flakiness & rework Pattern recognition = Scalable, proactive QA culture When your team recognizes the right test strategy in 30 seconds instead of 10 minutes — that’s quality at speed, not just quality at scale

During my recent #interview for the Test Lead position at a leading product based company, I was asked to outline a test strategy for a call center flow that includes components such as language preference, issue types, new/existing plans, and agent availability. The interviewer emphasized the importance of identifying bottlenecks, particularly with a focus on API testing. 1. Understanding the Call Center Flow The call center flow consists of: - Language Preference: Allowing customers to select their preferred language. - Types of Issues: Handling various customer issues effectively (e.g., billing, technical support). - New/Existing Plan: Differentiating between new and existing customers. - Agent Availability: Ensuring calls are routed to available agents based on the above conditions. 2. Identifying Potential Bottlenecks I identified several key areas where bottlenecks could occur: - Language Handling: Quick switching between languages without delays. - Routing Logic: Efficient and responsive call routing based on issue types and customer status. - Agent Availability Checks: Real-time accuracy in reflecting agent availability. - Load Handling: System performance during peak call times when multiple calls are initiated. 3. Manual Testing Strategy To address the flow manually, I proposed: - Scenario Testing: Creating comprehensive test cases covering all possible user paths and edge cases. - Exploratory Testing: Conducting exploratory tests to uncover hidden issues, especially regarding language preferences and issue categorization. - User Acceptance Testing (UAT): Engaging real users to validate the flow against business requirements. 4. API Testing Strategy Given the interview's emphasis on API testing, I outlined a focused strategy: a. Identify Key APIs - Language Preference API - Issue Routing API - Customer Status API - Agent Availability API b. Automation of API Tests - Using tools like Postman and Library as RestAssured for automation. - Automating tests for various scenarios, including successful calls with different languages, correct routing for issue types, and validating customer status. c. Load and Performance Testing - Conducting load tests on APIs to assess performance under peak conditions, utilizing tools like JMeter or Gatling. d. Continuous Testing - Integrating API tests into the CI/CD pipeline to ensure rapid feedback and improve reliability with every code change. #Conclusion In summary, I emphasized the importance of identifying bottlenecks and implementing a robust API testing strategy to ensure a smooth and efficient call center flow. This dual approach of manual and automated testing not only mitigates risks but also enhances the overall user experience.

How are you managing your test environments in an optimized and cost effective way? I recently came to know about the concept of the Ephemeral Environment which has great potential to solve this problem. In this short post I am sharing my learnings from that exploration. Shared development and testing environments often lead to long wait times, inaccessible resources, and unpredictable service availability. These issues not only delay feedback cycles but also turn automated testing into a flaky and frustrating experience. 𝗪𝗵𝗮𝘁 𝗔𝗿𝗲 𝗘𝗽𝗵𝗲𝗺𝗲𝗿𝗮𝗹 𝗘𝗻𝘃𝗶𝗿𝗼𝗻𝗺𝗲𝗻𝘁𝘀? Ephemeral environments are short-lived, on-demand replicas of your application stack spun up for a specific purpose, such as testing a pull request or reviewing a feature branch. Those are disposable, isolated mini environments, tailor made for a task, and destroyed once that task is complete. They are often powered by Kubernetes-native technologies, making them inherently scalable, automated, and aligned with modern infrastructure practices. 𝗞𝗲𝘆 𝗖𝗵𝗮𝗿𝗮𝗰𝘁𝗲𝗿𝗶𝘀𝘁𝗶𝗰𝘀 𝗼𝗳 𝗘𝗽𝗵𝗲𝗺𝗲𝗿𝗮𝗹 𝗘𝗻𝘃𝗶𝗿𝗼𝗻𝗺𝗲𝗻𝘁𝘀 • 𝗔𝘂𝘁𝗼𝗺𝗮𝘁𝗲𝗱: Provisioned and destroyed automatically through CI/CD pipelines. • 𝗦𝗵𝗼𝗿𝘁-𝗟𝗶𝘃𝗲𝗱: Exist only as long as the task (e.g., a PR review or test run) requires. • 𝗣𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻-𝗟𝗶𝗸𝗲: Provide realistic conditions for accurate and meaningful testing. • 𝗖𝗼𝘀𝘁-𝗘𝗳𝗳𝗲𝗰𝘁𝗶𝘃𝗲: Optimize infrastructure usage by spinning up only what’s needed and when it’s needed. 𝗧𝘆𝗽𝗶𝗰𝗮𝗹 𝗨𝘀𝗲 𝗖𝗮𝘀𝗲𝘀 • Feature Branch Isolation • Bug Reproduction • Automation Testing • UAT Demos 𝗘𝘅𝗮𝗺𝗽𝗹𝗲 𝗪𝗼𝗿𝗸𝗳𝗹𝗼𝘄:  • A developer opens a pull request and pushes code. • The CI pipeline detects changes and spins up an environment. • Tests run in an isolated environment. • Peers review the changes while testing continues. • Upon merge of the PR, the environment is automatically torn down. 𝗜𝘀𝗼𝗹𝗮𝘁𝗶𝗼𝗻 𝗠𝗲𝗰𝗵𝗮𝗻𝗶𝘀𝗺 To maintain safety and predictability, ephemeral environments rely on robust isolation mechanisms: • 𝗥𝗲𝗾𝘂𝗲𝘀𝘁 𝗜𝘀𝗼𝗹𝗮𝘁𝗶𝗼𝗻: Traffic is tagged (often using headers or tenancy tokens) and routed only to the appropriate test environment ensuring test requests don’t interfere with live traffic. • 𝗗𝗮𝘁𝗮 𝗜𝘀𝗼𝗹𝗮𝘁𝗶𝗼𝗻: Techniques like test-specific accounts, Kafka topic tagging, or namespace-specific configuration ensure test data doesn't flood into production systems. 𝗧𝗼𝗼𝗹𝘀 𝗮𝗻𝗱 𝗘𝗰𝗼𝘀𝘆𝘀𝘁𝗲𝗺 A growing set of tools are emerging in this space: • 𝗖𝗼𝗺𝗺𝗲𝗿𝗰𝗶𝗮𝗹: Signadot, Okteto Cloud, Qovery etc • 𝗢𝗽𝗲𝗻 𝗦𝗼𝘂𝗿𝗰𝗲: Telepresence, Tilt 𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲𝘀 𝗼𝗳 𝗨𝘀𝗶𝗻𝗴 𝗘𝗽𝗵𝗲𝗺𝗲𝗿𝗮𝗹 𝗘𝗻𝘃𝗶𝗿𝗼𝗻𝗺𝗲𝗻𝘁𝘀 • Security and Access Control • Compliance and Governance • Resource Quota Management • Debug-ability and Observability How are you managing environment dependencies and test isolation in your workflow? Drop a comment and share your experience.

Selenium has been the gold standard for automated browser testing for good reasons: 🔹It's open-source and free 🔹 It has multi-browser and multi-platform support 🔹 A robust community backs it. But here’s the catch, it has: 🔹 Slower execution than modern tools 🔹 No built-in reporting or test management 🔹 A steep learning curve, especially for engineers outside testing. That last point is critical. Here is why: Recently, we discussed a report at work: The State of Front End Development by The House of Software. One stat stood out: Cypress ranked second. Playwright came in fifth. Selenium? Nowhere in the top 10. At first, this was surprising. Selenium has been a staple in browser testing for years. So why the low ranking? The answer was in the survey details. The respondents weren’t test professionals—they were 6,000 developers. These were front-end engineers focusing on unit and integration testing in modern workflows. While tools like Jest unsurprisingly topped the list, Playwright and Cypres' high rankings also confirmed what I’ve observed firsthand: their developer-friendly design makes them better suited to modern teams and workflows across the spectrum of the team, not just for testers. In my experience, here are some of the reasons why these tools are gaining traction amongst engineers: 🔹 Streamlined setup makes them easy to adopt. 🔹 Seamless CI/CD integration aligns with modern workflows. 🔹 Intuitive design and built-in debugging tools reduce friction for new users. On one project, I faced a common challenge: too many engineers, too few testers, too little time. Our solution? Make quality a shared responsibility across the team, including engineers on debugging and investigation of automation failures. Here’s what worked: 🔹 Lowering the Learning Curve: Playwright simplified onboarding for engineers by reducing friction. Its seamless integration with modern CI/CD pipelines and built-in features—such as tracing, screenshots, and debugging tools—eliminated the need for external libraries or frameworks 🔹 Collaborative Test Maintenance: While testers handled test creation during the sprint, maintaining those tests became a team-wide effort, fostering a culture of shared responsibility and collaboration. 🔹 Engineers took the lead in investigating E2E test automation failures: instead of waiting on testers, engineers analyzed issues with the tests, identified bugs, and resolved them. Sometimes, it’s not about the "best" tool—it’s about the tool your team will actually use. What has your experience been with getting engineers involved in the automation of checks within your teams?

Your P-value Is 0.03. That Does Not Mean You Are Right. Software will always give us an answer. A terrible data will produces clean looking results. The wrong statistical test will spits out a p-value. Even impossible numbers get processed without complaint. But here is the problem. Software does not think. We do. Before you celebrate that significant p-value, let's walk through this roadmap. 🚷 Stop 1: Check your data: Are there missing values you forgot about? Any impossible entries like age equals 200 years or negative blood pressure readings? Are your variables coded correctly so that "1" means what you think it means? Garbage in garbage out. No statistical test can fix bad data. 🚫 Stop 2: Check your assumptions. Is your data severely skewed when your test assumes normality? Are there extreme outliers pulling the results in one direction? Did you check whether your groups have equal variances? The wrong assumptions can make a real effect disappear or create a fake effect from nothing. 🔞 Stop 3: Check your sample size. Is your sample too small to detect anything meaningful? Is it large enough to represent the population you want to generalize to? A tiny sample can miss real effects. A huge sample can make trivial differences look important. 🚫 Stop 4: Check your statistical test. Does your test actually match your data type and research question? You cannot use a t-test for categorical outcomes. You cannot use chi-square for continuous variables. The wrong test gives you the wrong answer even with perfect data. 🚫 Stop 5: Check your interpretation. What is your effect size? Is it clinically meaningful or just statistically detectable? What does your confidence interval tell you about precision? Statistical significance is not the same as real world importance. 🚫 Stop 6: Check if it makes sense. Does your result align with existing knowledge? If your new blood pressure drug increases heart attacks, that should make you pause. If your intervention works too well, be suspicious. Extraordinary claims require extraordinary evidence. 🚫 Stop 7: Check for consistency. What happens if you run the analysis slightly differently? Do you get similar results with different but reasonable approaches? Are your findings robust or fragile? If small changes flip your conclusions, then there's a problem. The truth about research software: It is a powerful tool in the hands of someone who understands what they are doing. It is also a dangerous weapon in the hands of someone who does not. Your software will never tell you that your sample size is too small. It will never warn you that your assumptions are violated. It will never question whether your research question makes sense. That is our job. The biggest risk in research is not getting the wrong answer. It is being confident in the wrong answer. Check your work. Every step. Every time. That is statistical thinking.