Browser Automation with Puppeteer

Automating browsers provide many benefits including faster execution of repetitive tasks, ability to parallelise workloads and improved test coverage for your website. Google recently announced Puppeteer, a new tool to assist with Chrome browser automation.

Code examples are included so you can follow along. If you are familiar with browser automation already, feel free to jump to the section titled “Puppeteer: A practical example” which includes more advanced usage of Puppeteer.

Here we will cover:

  1. A quick introduction to browser automation: How it’s typically done & use cases.

A quick introduction to browser automation, including use cases

Browser automation enables you to programmatically control a browser. For example, you could do some of the following:

  • Observe product pricing updates on an online store to discover the best time to purchase a particular product.

All modern browsers may be automated, including Chrome, Firefox, Edge & Safari. You can also automate mobile browsers. This post will briefly cover Chrome & Firefox.

For most use cases, to control a browser programmatically, you should use high level browser automation software. One popular choice is WebDriver, which we leverage for automating functional testing with Intern.

You are most likely reading this article in a browser. Notice that the browser has supporting elements for your use, such as a menu bar, address bar, and toolbar. These items are part of the GUI (Graphical User Interface). A headless browser has no GUI and no visual components. It runs as a process, and will expose a mechanism to enable outside interaction from source code or other software programs.

To run a version of Chrome (nightly Canary builds) in a headless environment, you can use one of the following commands:

Mac OS X:

/Applications/Google\ Chrome\\ Chrome\ Canary --headless --remote-debugging-port=9222 --disable-gpu


google-chrome --headless --remote-debugging-port=9222 --disable-gpu


chrome.exe --headless --remote-debugging-port=9222 --disable-gpu

The remote debugging flag enables you to use DevTools for some remote inspection of the headless browser tab.

In the past, you may have used PhantomJS to achieve your browser automation tasks, however Chrome headless runs faster and also consumes less memory.

To programmatically interact with this headless version of Chrome, you can send commands over the Chrome DevTools protocol. Using the DevTools protocol, you can do most things you would do in your usual browser DevTools. Here’s an example of communicating over the DevTools Protocol via WebSockets to retrieve the current page URL from the inspected page:

// Gist: You would use your own URL hereconst devtools = new WebSocket('ws://localhost:9222/devtools/page/69990451-aaab-4ef8-87b1-ea77b8101b2a');devtools.onmessage = ({data}) => {const {result: {result: {value}}} = JSON.parse(data);console.log('WebSocket Message Received: ', value)};devtools.send(JSON.stringify({id: 1,method: 'Runtime.evaluate',params: {expression: `'The current URL is: ' + location.href`}}));

As you can see, considering what it does, the code above is low-level and verbose. Puppeteer provides a more concise API to automate browser operations.


The automation code you write using the Puppeteer API actually makes calls over the DevTools Protocol API, which is exactly what’s covered in the previous section. Instead of having to craft WebSocket payloads, you can call APIs such as:


As shown in the Puppeteer documentation, you can run Puppeteer from Node.js code like this:

const puppeteer = require('puppeteer');(async () => {const browser = await puppeteer.launch();const page = await browser.newPage();await page.goto('');await page.screenshot({path: 'example.png'});browser.close();})();

The example above does the following:

  1. Launches Chrome.

Puppeteer includes code examples and API documentation.

Puppeteer: A practical example

You could call the previous example (taking a screenshot of a page) the ‘Hello world’ of browser automation! It is useful to have a more practical example, or an example which uses a wider set of APIs, for demonstrative purposes.

The SitePen contact page includes a contact form. After some debugging with DevTools, notice the form fields are customised from a CMS (Content Management System), the fields themselves are created after a client-side JavaScript request.

Firstly, consider the following hypothetical scenario: we have some browser automation tests which navigate to the SitePen contact page, fills in form fields and submits the form. Before we discuss a solution, let’s understand the potential problems.

In browser automation scenarios, what are the potential problems which can arise when allowing third party resources to download?

Tip: Notice that some of these problems are not unique to third party requests. Imagine your website fetches some comments for a blog post from your own server via Ajax, think about what problems listed below would also apply to that example.


Browser automation is notoriously slow. Often due to varying factors, like hard ‘sleeps’ rather than polling for changes, third-party scripts which are slow to download and slow to modify the page (which automation frameworks usually wait for), and many other aspects.

In this example: Network speed is a small issue.

As you might notice, the form itself appeared at the 1.58 second mark, which is almost half a second after the first paint. You might think saving a few hundred milliseconds is not a big deal, but consider that many companies run large suites of browser automation tests frequently throughout the day, as the codebase changes. Seconds add up!


Third party services can:

  • Go down for maintenance.

These issues can add to the frustration of working with your browser automation setup.


Imagine your automation scripts target elements on the page via CSS selectors. If these same elements can appear or disappear based on what returns from a CMS, your automation scripts will likely fail. Dynamic data (rather than static data) can be challenging to work with in automation scripts as your code must accommodate multiple scenarios. This can also increase the complexity of your automation codebase.

Notice how speed, reliability & consistency are three potential problems which you can alleviate by mocking resources to third-party services. Generally, your goal is to manipulate a resource to cater to your specific use case.

Notice how the form field labels include the text ‘intercepted’.

You can find a complete code example which creates the screenshot above, on GitHub. The code example on GitHub does the following:

  • Starts a static web server to serve the mock file.

Firefox Headless

Firefox also offers a headless mode. Additionally, Mozilla offers JavaScript and Java code examples for how to utilise Firefox’s headless mode. At the time of this writing, Firefox headless is only supported on Windows, with plans to support other platforms in the near future. There is a guide for connecting WebDriver to a headless version of Firefox here.


We covered a number of topics:

  • Headless Browsers: Chrome and Firefox are able to run without a GUI. There are more options out there for other browsers (e.g. Safari & Edge)

Leveraging these techniques for browser automation, we can develop solutions to solve problems, whether it’s for quality assurance, productivity enhancements or data aggregation.

Next steps

Do you need help creating a testing strategy for your next project? Contact us to discuss how we can help!

Get help from SitePen On-Demand Development, our fast and efficient solutions to JavaScript development problems of any size.

Have a question? We’re here to help! Get in touch and let’s see how we can work together.

Originally published at on October 4, 2017.



Modernizing Apps, Tools & Teams | | Twitter: @sitepen

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store