The state of UI testing at Mixpanel

End-to-end tests

At Mixpanel, we’ve been writing UI tests for a long time. However, they haven’t always been easy to set up, write, and debug. When we first began testing the UI, we wrote tests in Python using the Selenium framework. In this setup, the Python tests interact with the browser through the API provided by Selenium. These Selenium commands are then sent to browser-specific drivers for controlling different browsers. These “end-to-end” tests required setting up a web server, database, and various supporting backend services, as well as populating these services with the data needed for the tests. These tests have the benefit of not just testing the UI, but also testing the integration between the backend services needed to render the UI. The intent was for these tests to mimic the experience of an end user visiting Mixpanel’s production website.


Figure 1. The setup of Mixpanel’s end-to-end tests.

However, these end-to-end tests also have a number of downsides:

  • Front-end developers have to learn the Selenium API, which is quite different from other tools used by front-end developers.
  • This setup introduced latency in several places: 1) latency from when a Selenium command is sent by the test to when it’s executed in the browser, 2) latency of network requests made by the browser to the web server. Having to take these variable delays into account made it harder to write tests that weren’t flaky.
  • There was overhead setting up all the backend services and populating them with the fixture data needed for each test.
  • Tests became harder to maintain/debug, since issues were not limited to just the front-end, and could be from any of the backend services in the stack.

End-to-end tests are currently used to test Mixpanel’s older reports. The components within these older reports were built with Backbone. Many of these components have dependencies on global state and intertwined dependencies with other components. This was partially due to lack of discipline but also because these components were written before JavaScript had the good module and bundler tooling (e.g. Webpack) that it currently has. These entangled dependencies made it hard to test individual components in isolation. This is one of the reasons end-to-end tests were used to test these reports – they required a web server to serve a production-like version of the website with all the necessary dependencies for the components being tested.

WCT tests

In the previous section, we saw how Mixpanel used to write UI tests, and some of their drawbacks. However, recent front-end developments at Mixpanel have allowed us to prefer a different approach to UI testing that solves the above-mentioned problems.

In the last 1-2 years, we have started using Web Components1 as the building blocks for Mixpanel’s newer reports. These reports have a top-level “application” custom element which is composed of other custom elements, all the way down to custom elements representing basic components like buttons and tooltips.

When we started using custom elements, we focused on creating components with well-defined attribute-based interfaces to pass information into the component, and event propagation for the component to communicate with the outside world. This contrasts with the entangled dependencies that exist in Mixpanel’s older Backbone components2. Creating modular custom elements has now made it possible to write more isolated/modular tests for individual custom elements like buttons and tooltips, while also being able to write higher-level tests for an entire report composed of many custom elements.

These new-style tests are written using the web-component-tester (WCT) browser testing framework, which came out of the Polymer3 project. Hence, we refer to them as WCT tests. While the end-to-end tests exercise the entire stack, WCT tests are strictly front-end-only UI tests. WCT tests are written in JavaScript, which runs on the same web page as the components they’re testing.

Figure 2. The WCT test running environment. The test code is being run in an iframe on the left. Information about success/failure of individual tests is output on the right side.

WCT tests address all the downsides of end-to-end tests mentioned earlier:

  • When writing WCT tests, developer can use the JavaScript DOM APIs and other JavaScript testing libraries like Sinon instead of needing to learn a new set of APIs (Selenium4).
  • WCT tests run faster and are less prone to race conditions since the test code runs directly on the web page, whereas the end-to-end tests have a layer of separation between test code and the web page.
  • WCT tests have no dependencies on a web server by mocking out requests to the server, and thus are much easier to setup, write, and maintain compared to end-to-end tests. (See the Mocking server responses section below for more details.)

Contrast the simplicity of the setup for these WCT tests shown in Figure 3 below with the setup for end-to-end tests from Figure 1.


Figure 3. The setup of WCT tests.

Guidelines when writing WCT tests

We use the WCT framework as an environment for running UI tests, as seen in Figure 2 earlier. However, the framework doesn’t enforce how tests should be written/structured. So we’ve come up with some of our guidelines when writing WCT tests which are discussed below.

Mocking server responses

As described earlier, Mixpanel’s end-to-end UI tests required setting up backend services. In contrast, WCT browser tests are front-end-only. Any server requests are stubbed to return mock responses. Our front-end code uses fetch to make network requests. At the beginning of each test, we use the Sinon mocking library to create a mock server that responds to fetch  requests that will be made by the test.

DOM helpers

Interacting with the DOM is a necessity for browser tests. To keep our code DRY, we created a small library of utilities that all tests should use when they need to interact with the DOM. Here are some of the utilities we have:

  • Helpers to wait during a test. For example, nextAnimationFrame is an async function that awaits till the next requestAnimationFrame. retryable and condition will wait until some condition is met. (They’re described in more detail in a later section.)
  • Helpers for interacting with DOM elements. For example, clickElement will click an element while sendInput will send text to an input element.
  • Helpers for querying element in the Shadow DOM, since our custom elements make use of the Shadow DOM. For example, queryShadowSelectors queries for the first matching element in the Shadow DOM, while queryShadowSelectorsAll queries for all matching elements (similar to querySelectorAll) in the Shadow DOM.

Element wrappers

When writing browser tests, a large portion of the test code will be for performing actions on components and querying the state of the component after these actions. Often, multiple tests perform similar interactions with the same component. To keep the test code DRY, we created the concept of “element wrappers”.

An element wrapper is a helper class that wraps an element5 in the DOM. They consist of the methods mentioned above that are needed by test code for performing actions and querying the DOM state of these elements.

Besides keeping the test code DRY, another benefit of element wrappers is that they are modular. They allow you to group all the possible interactions with a component in a single place. Similar to how custom elements can be composed of other custom elements, element wrappers can mirror this composability by providing helper methods that return element wrappers for child elements. These child element wrappers can then be used to interact with these child elements.

An example of an element wrapper is the Calendar element wrapper which wraps the <mp-calendar> custom element, which is used for picking dates from a calendar.


Figure 4. Screenshot of the <mp-calendar>  custom element.

Below is the implementation6 of the Calendar element wrapper.

The Calendar element wrapper provides methods like clickDate and clickNextMonthButton for performing actions on the <mp-calendar> custom element. It also provides query methods like isNextMonthButtonDisabled for querying the DOM state of the <mp-calendar> custom element.

Waiting without sleeping

Within our WCT tests, it’s often necessary to write asynchronous code that waits for some condition before continuing the test:

  • The Panel library we use for creating custom elements batches DOM updates to the next requestAnimationFrame by default for performance reasons. This means any time we perform an action on an element (e.g. clicking a button), the update to the DOM associated with the change happens asynchronously. Since a large portion of browser testing is triggering actions on the web page, needing to wait for the DOM to update is a common occurrence in our tests.
  • fetch requests (even though they’re mocked) are asynchronous.
  • Animations will delay a component from reaching its final state.

To deal with the abundance of asynchronous code in our WCT tests, we have opted to use async/await syntax introduced in ES2017. This allows the test code to be more readable by removing the excessive nesting associated with callbacks and (to a lesser extent) Promises.

An anti-pattern when you need to wait within a test is to sleep. However, this makes the test brittle and slows the test down because you end up sleeping longer than needed in most cases. Instead, the test should wait for some explicit conditions to be met to decide if it can continue execution. In this vein, we created some helper functions for this use case: retryable and condition. Both these functions take a function as input and will repeatedly execute it until some condition is met or a predefined timeout. retryable will continue to execute the function until it doesn’t throw an exception. condition will continue to execute the function until it returns a truthy value.

Below is a simplified version of a WCT test in our codebase that follows these guidelines. Comments have been added for explanation purposes.

UI testing within CI

The end-to-end and WCT tests are run on every pull request. They are also regularly run on master to catch any bad code that might have slipped through the cracks. WCT tests selectively run depending on the code change. For instance, if only backend changes are made, WCT tests will not run. If front-end changes are made to a single report, only the WCT tests for that report will run. The end-to-end tests in contrast are run for every pull request since virtually any code change (front-end or backend) could impact them.

The end-to-end tests are run in VMs that are set up with all the backend services needed to run them. The tests are run in Chrome on this VM using Xvfb7. In contrast, the WCT tests run on Sauce Labs, a platform for running automated browser tests that the WCT frameworks supports of the box. Sauce Labs itself allows configuring a list of browser environments to test on. Below is the wct.conf.js (WCT framework configuration file) we use to run our tests on Sauce Labs.

As you can see, we run our WCT tests on the latest version of Chrome, Firefox, Safari, and Edge.

Closing remarks

In this post, we looked at the different types tests we write to test the UI at Mixpanel. In the beginning, we wrote only end-to-end tests which exercise the entire stack. Despite them being ill-suited for the purpose, we used end-to-end tests for testing the UI for a long time because that’s all we had. However, because of better modularization of our front-end code, we are now able to write front-end-only WCT tests for this purpose. Nonetheless, the introduction of WCT tests don’t obviate the need for end-to-end tests, which still serve the important function of verifying high-level behavior across the stack.

Since WCT tests are easier and less time-consuming to write compared to end-to-end tests, developers have been much more receptive to writing them. The difference in adoption between test two types can be seen by taking a look at our codebase. We currently have almost 7x8 as many WCT tests as end-to-end tests, despite the fact that we’ve only been using WCT for a couple of years. Reducing the friction in writing and maintaining UI tests has therefore increased our regression coverage significantly, making for both happier users and happier front-end engineers.

  1. See https://engineering.mixpanel.com/2018/06/12/making-web-components-work/ to learn more about how we use Web Components.
  2. See https://engineering.mixpanel.com/2015/04/08/straightening-our-backbone-a-lesson-in-event-driven-ui-development/ for more discussion on this.
  3. Polymer is a library for writing web applications using Web Components. Although we use WCT for browser testing, we don’t use Polymer for creating Web Components. Instead, we use our own library Panel, which https://engineering.mixpanel.com/2018/06/12/making-web-components-work/ has more information about.
  4. The WCT framework internally uses Selenium for controlling the browser. However, tests written using the WCT framework never have to interact with Selenium itself.
  5. We primarily use element wrappers to wrap custom elements, but they can also wrap other HTML elements as well.
  6. Parts of the implementation have been removed to improve readability.
  7. Xvfb allows running GUI applications when there’s no physical display, which is the case with the VM that Selenium tests are run on.
  8. At the time of writing this article, we have ~60 end-to-end tests compared to ~400 WCT tests.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.