Using authenticated proxy with Selenium / Packaging Chrome extensions with Ruby

Overview

Recently I've got the request to implement authenticated proxy support for our product test framework. The problem is that recent browsers do not allow the widely popular http://username:password@proxy.example.com syntax and still ask you to manually enter credentials.

The next problem is that Selenium does not let you interact with these basic auth dialogs [1][2]. So how should one go about this?

Chrome allows you to do this with a custom extension that you can insert with selenium/watir.


One additional complication is that we can use a different proxy server each time. Thus extension needs to be packaged on the fly.

Chrome extension

This is the proxy extension as I use it. See it as an example for whatever you'll be trying to do. It consists of only 2 files you can put in an empty directory.

manifest.json

{
    "version": "0.0.1",
    "manifest_version": 2,
    "name": "Authenticated Proxy",
    "permissions": [
        "<all_urls>",
        "proxy",
        "unlimitedStorage",
        "webRequest",
        "webRequestBlocking",
        "storage",
        "tabs"
    ],
    "background": {
        "scripts": ["background.js"]
    },
    "minimum_chrome_version":"23.0.0"
}

background.js.erb

var config = {
  mode: "fixed_servers",
  rules: {
    singleProxy: {
      scheme: "<%= proxy_proto %>",
      host: "<%= proxy_host %>",
      port: parseInt(<%= proxy_port %>)
    },
    bypassList: <%= proxy_bypass.split(/[ ,]/).delete_if(&:empty?).to_json %>
  }
};

chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});

function callbackFn(details) {
  return {
    authCredentials: {
      username: "<%= proxy_user %>",
      password: "<%= proxy_pass %>"
    }
  };
}

chrome.webRequest.onAuthRequired.addListener(
  callbackFn,
  {urls: ["<all_urls>"]},
  ['blocking']
);

Protocol Buffers

As you can see on the web site, Protocol Buffers is a method of serializing structured data. For CRX3 (unlike CRX2) format it is part of the required header for the extension.

I decided to use ruby-protobuf project instead of the google ruby library because it appeared well maintained and pure ruby. I assume google ruby library will work well too.

The Packager

 A CRX v3 file would consist of:

• Cr24 - ASCII 8bit magic string
• 3 - protocol version in unsigned 32bit little endian
• header length in bytes in unsigned 32bit little endian
• header itself - the protobuf serialized object
• crx3.proto - the protobuf descriptor
• as a rule of thumb
•  all lengths inside are given as unsigned 32bit little-endian integers
• key files are inserted in PKCS#8 binary encoding (Ruby's key.to_der worked fine)
• ZIP archive of the extension files

Generating protobuf stub

We need to install Google protobuf compiler protoc. You can save the protocol file in a directory where you want stub to live in. Then generate by

protoc --plugin=protoc-gen-ruby-protobuf=`ls ~/bin/protoc-gen-ruby` --ruby-protobuf_out=./ path/chrome_crx3/crx3.proto

This will create a file crx3.pb.rb in the same directory as the protocol file. All you need is to require 'path/crx3.pb.rb' wherever you want to use that format.

Actual packager

At this point the packager is straightforward to implement. Pasting the whole logic here.

We have one ::zip method to generate a ZIP archive in memory. If an ERB binding is provided by caller, any .erb files are processed. That's how the above background.js.erb works.

The method ::header_v3_extension generates the signature and constructs the whole file header.

Finally ::pack_extension just glues the two methods above to generate the final extension.

chrome_extension.rb

require 'erb'
require 'find'
require 'openssl'
require 'zip'

require_relative 'resource/chrome_crx3/crx3.pb.rb'

class ChromeExtension
  def self.gen_rsa_key(len=2048)
    OpenSSL::PKey::RSA.generate(len)
  end

  #  @note file format spec pointers:
  #    https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/977b9b99-2bb9-476b-992f-97a3e37bf20c%40chromium.org
  def self.header_v3_extension(data, key: nil)
    key ||= gen_rsa_key()

    digest = OpenSSL::Digest.new('sha256')
    signed_data = Crx_file::SignedData.new
    signed_data.crx_id = digest.digest(key.public_key.to_der)[0...16]
    signed_data = signed_data.encode

    signature_data = String.new(encoding: "ASCII-8BIT")
    signature_data << "CRX3 SignedData\00"
    signature_data << [ signed_data.size ].pack("V")
    signature_data << signed_data
    signature_data << data

    signature = key.sign(digest, signature_data)

    proof = Crx_file::AsymmetricKeyProof.new
    proof.public_key = key.public_key.to_der
    proof.signature = signature

    header_struct = Crx_file::CrxFileHeader.new
    header_struct.sha256_with_rsa = [proof]
    header_struct.signed_header_data = signed_data
    header_struct = header_struct.encode

    header = String.new(encoding: "ASCII-8BIT")
    header << "Cr24"
    header << [ 3 ].pack("V") # version
    header << [ header_struct.size ].pack("V")
    header << header_struct

    return header
  end

  # @param file [String] to write result to
  # @param dir [String] to read extension from
  # @param key [OpenSSL::PKey]
  # @param crxv [String] version of CRX file to create
  # @param erb_binding [Binding] optional if you want to process ERB files
  # @return undefined
  def self.pack_extension(file:, dir:, key: nil, crxv: "v3", erb_binding: nil)
    zip = zip(dir: dir, erb_binding: erb_binding)

    File.open(file, 'wb') do |io|
      io.write self.send(:"header_#{crxv}_extension", zip, key: key)
      io.write zip
    end
  end

  # @param dir [String] to read extension from
  # @param erb_binding [Binding] optional if you want to process ERB files
  # @return [String] the zip file content
  def self.zip(dir:, erb_binding: nil)
    dir_prefix_len = dir.end_with?("/") ? dir.length : dir.length + 1
    zip = StringIO.new
    zip.set_encoding "ASCII-8BIT"
    Zip::OutputStream::write_buffer(zip) do |zio|
      Find.find(dir) do |file|
        if File.file? file
          if erb_binding && file.end_with?(".erb")
            zio.put_next_entry(file[dir_prefix_len...-4])
            erb = ERB.new(File.read file)
            erb.location = file
            zio.write(erb.result(erb_binding))
            Kernel.puts erb.result(erb_binding)
          else
            zio.put_next_entry(file[dir_prefix_len..-1])
            zio.write(File.read(file))
          end
        end
      end
    end
    return zip.string
  end
end

Using the packager

Packing the extension is as simple as:

require 'chrome_extension'

ChromeExtension.pack_extension(file: "/path/of/target/extension.crx", dir: "/path/of/proxy/extension")

Using the extension with Watir

proxy_proto, proxy_user, proxy_pass, proxy_host, proxy_port = <...>
chrome_caps = Selenium::WebDriver::Remote::Capabilities.chrome()
chrome_caps.proxy = Selenium::WebDriver::Proxy.new({http: "#{proxy_proto}://#{proxy_host}:#{proxy_port}", :ssl => "#{proxy_proto}://#{proxy_host}:#{proxy_port}")
# there is a bug in Watir where providing an object here results in an error 
# options = Selenium::WebDriver::Chrome::Options.new
# options.add_extension proxy_chrome_ext_file if proxy_chrome_ext_file
options = {}
options[:extensions] = [proxy_chrome_ext_file] if proxy_chrome_ext_file
browser = Watir::Browser.new :chrome, desired_capabilities: chrome_caps, switches: chrome_switches, options: options

Bonus content - CRX2 method

  #  @note original crx2 format description https://web.archive.org/web/20180114090616/https://developer.chrome.com/extensions/crx
  def self.header_v2_extension(data, key: nil)
    key ||= gen_rsa_key()
    digest = OpenSSL::Digest.new('sha1')
    header = String.new(encoding: "ASCII-8BIT")

    # it is exactly same signature as `ssh_do_sign(data)` from net/ssh does
    signature = key.sign(digest, data)
    signature_length = signature.length
    pubkey_length = key.public_key.to_der.length

    header << "Cr24"
    header << [ 2 ].pack("V") # version
    header << [ pubkey_length ].pack("V")
    header << [ signature_length ].pack("V")
    header << key.public_key.to_der
    header << signature

    return header
  end

Credits

• https://botproxy.net/docs/how-to/setting-chromedriver-proxy-auth-with-selenium-using-python/
• A nice blog post about doing this with python, unfortunately it is using CRX v2 file format.
• You can see content of actual Chrome extension there as well.
• https://groups.google.com/a/chromium.org/d/topic/chromium-extensions/K3YIsNL_Et4/discussion
• discussion about CRX file format with links to other implementations
• https://github.com/pawliczka/CRX3-Creator
• python implementation of extension packager
• https://crx-checker.appspot.com
• CRX verifier for both versions