Thursday, November 21, 2019

Using authenticated proxy with Selenium / Packaging Chrome extensions with Ruby


Recently I've got the request to implement authenticated proxy support for our product test framework. The problem is that recent browsers do not allow the widely popular syntax and still ask you to manually enter credentials.

The next problem is that Selenium does not let you interact with these basic auth dialogs [1][2]. So how should one go about this?

Chrome allows you to do this with a custom extension that you can insert with selenium/watir.

One additional complication is that we can use a different proxy server each time. Thus extension needs to be packaged on the fly.

Chrome extension

This is the proxy extension as I use it. See it as an example for whatever you'll be trying to do. It consists of only 2 files you can put in an empty directory.


    "version": "0.0.1",
    "manifest_version": 2,
    "name": "Authenticated Proxy",
    "permissions": [
    "background": {
        "scripts": ["background.js"]


var config = {
  mode: "fixed_servers",
  rules: {
    singleProxy: {
      scheme: "<%= proxy_proto %>",
      host: "<%= proxy_host %>",
      port: parseInt(<%= proxy_port %>)
    bypassList: <%= proxy_bypass.split(/[ ,]/).delete_if(&:empty?).to_json %>

chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});

function callbackFn(details) {
  return {
    authCredentials: {
      username: "<%= proxy_user %>",
      password: "<%= proxy_pass %>"

  {urls: ["<all_urls>"]},

Protocol Buffers

As you can see on the web site, Protocol Buffers is a method of serializing structured data. For CRX3 (unlike CRX2) format it is part of the required header for the extension.

I decided to use ruby-protobuf project instead of the google ruby library because it appeared well maintained and pure ruby. I assume google ruby library will work well too.

The Packager

 A CRX v3 file would consist of:
  • Cr24 - ASCII 8bit magic string
  • 3 - protocol version in unsigned 32bit little endian
  • header length in bytes in unsigned 32bit little endian
  • header itself - the protobuf serialized object
    • crx3.proto - the protobuf descriptor
    • as a rule of thumb
      •  all lengths inside are given as unsigned 32bit little-endian integers
      • key files are inserted in PKCS#8 binary encoding (Ruby's key.to_der worked fine)
  • ZIP archive of the extension files

Generating protobuf stub

We need to install Google protobuf compiler protoc. You can save the protocol file in a directory where you want stub to live in. Then generate by

protoc --plugin=protoc-gen-ruby-protobuf=`ls ~/bin/protoc-gen-ruby` --ruby-protobuf_out=./ path/chrome_crx3/crx3.proto
This will create a file crx3.pb.rb in the same directory as the protocol file. All you need is to require 'path/crx3.pb.rb' wherever you want to use that format.

Actual packager

At this point the packager is straightforward to implement. Pasting the whole logic here.

We have one ::zip method to generate a ZIP archive in memory. If an ERB binding is provided by caller, any .erb files are processed. That's how the above background.js.erb works.

The method ::header_v3_extension generates the signature and constructs the whole file header.

Finally ::pack_extension just glues the two methods above to generate the final extension.


require 'erb'
require 'find'
require 'openssl'
require 'zip'

require_relative 'resource/chrome_crx3/crx3.pb.rb'

class ChromeExtension
  def self.gen_rsa_key(len=2048)

  #  @note file format spec pointers:
  def self.header_v3_extension(data, key: nil)
    key ||= gen_rsa_key()

    digest ='sha256')
    signed_data =
    signed_data.crx_id = digest.digest(key.public_key.to_der)[0...16]
    signed_data = signed_data.encode

    signature_data = "ASCII-8BIT")
    signature_data << "CRX3 SignedData\00"
    signature_data << [ signed_data.size ].pack("V")
    signature_data << signed_data
    signature_data << data

    signature = key.sign(digest, signature_data)

    proof =
    proof.public_key = key.public_key.to_der
    proof.signature = signature

    header_struct =
    header_struct.sha256_with_rsa = [proof]
    header_struct.signed_header_data = signed_data
    header_struct = header_struct.encode

    header = "ASCII-8BIT")
    header << "Cr24"
    header << [ 3 ].pack("V") # version
    header << [ header_struct.size ].pack("V")
    header << header_struct

    return header

  # @param file [String] to write result to
  # @param dir [String] to read extension from
  # @param key [OpenSSL::PKey]
  # @param crxv [String] version of CRX file to create
  # @param erb_binding [Binding] optional if you want to process ERB files
  # @return undefined
  def self.pack_extension(file:, dir:, key: nil, crxv: "v3", erb_binding: nil)
    zip = zip(dir: dir, erb_binding: erb_binding), 'wb') do |io|
      io.write self.send(:"header_#{crxv}_extension", zip, key: key)
      io.write zip

  # @param dir [String] to read extension from
  # @param erb_binding [Binding] optional if you want to process ERB files
  # @return [String] the zip file content
  def, erb_binding: nil)
    dir_prefix_len = dir.end_with?("/") ? dir.length : dir.length + 1
    zip =
    zip.set_encoding "ASCII-8BIT"
    Zip::OutputStream::write_buffer(zip) do |zio|
      Find.find(dir) do |file|
        if File.file? file
          if erb_binding && file.end_with?(".erb")
            erb = file)
            erb.location = file
            Kernel.puts erb.result(erb_binding)
    return zip.string

Using the packager

Packing the extension is as simple as:
require 'chrome_extension'

ChromeExtension.pack_extension(file: "/path/of/target/extension.crx", dir: "/path/of/proxy/extension")

Using the extension with Watir

proxy_proto, proxy_user, proxy_pass, proxy_host, proxy_port = <...>
chrome_caps =
chrome_caps.proxy ={http: "#{proxy_proto}://#{proxy_host}:#{proxy_port}", :ssl => "#{proxy_proto}://#{proxy_host}:#{proxy_port}")
# there is a bug in Watir where providing an object here results in an error 
# options =
# options.add_extension proxy_chrome_ext_file if proxy_chrome_ext_file
options = {}
options[:extensions] = [proxy_chrome_ext_file] if proxy_chrome_ext_file
browser = :chrome, desired_capabilities: chrome_caps, switches: chrome_switches, options: options

Bonus content - CRX2 method

  #  @note original crx2 format description
  def self.header_v2_extension(data, key: nil)
    key ||= gen_rsa_key()
    digest ='sha1')
    header = "ASCII-8BIT")

    # it is exactly same signature as `ssh_do_sign(data)` from net/ssh does
    signature = key.sign(digest, data)
    signature_length = signature.length
    pubkey_length = key.public_key.to_der.length

    header << "Cr24"
    header << [ 2 ].pack("V") # version
    header << [ pubkey_length ].pack("V")
    header << [ signature_length ].pack("V")
    header << key.public_key.to_der
    header << signature

    return header