Overview
Recently I've got the request to implement authenticated proxy support for our product test framework. The problem is that recent browsers do not allow the widely popularhttp://username:password@proxy.example.com syntax and still ask you to manually enter credentials.The next problem is that Selenium does not let you interact with these basic auth dialogs [1][2]. So how should one go about this?
Chrome allows you to do this with a custom extension that you can insert with selenium/watir.
One additional complication is that we can use a different proxy server each time. Thus extension needs to be packaged on the fly.
Chrome extension
This is the proxy extension as I use it. See it as an example for whatever you'll be trying to do. It consists of only 2 files you can put in an empty directory.manifest.json
{
"version": "0.0.1",
"manifest_version": 2,
"name": "Authenticated Proxy",
"permissions": [
"<all_urls>",
"proxy",
"unlimitedStorage",
"webRequest",
"webRequestBlocking",
"storage",
"tabs"
],
"background": {
"scripts": ["background.js"]
},
"minimum_chrome_version":"23.0.0"
}background.js.erb
var config = {
mode: "fixed_servers",
rules: {
singleProxy: {
scheme: "<%= proxy_proto %>",
host: "<%= proxy_host %>",
port: parseInt(<%= proxy_port %>)
},
bypassList: <%= proxy_bypass.split(/[ ,]/).delete_if(&:empty?).to_json %>
}
};
chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});
function callbackFn(details) {
return {
authCredentials: {
username: "<%= proxy_user %>",
password: "<%= proxy_pass %>"
}
};
}
chrome.webRequest.onAuthRequired.addListener(
callbackFn,
{urls: ["<all_urls>"]},
['blocking']
);Protocol Buffers
As you can see on the web site, Protocol Buffers is a method of serializing structured data. For CRX3 (unlike CRX2) format it is part of the required header for the extension.I decided to use ruby-protobuf project instead of the google ruby library because it appeared well maintained and pure ruby. I assume google ruby library will work well too.
The Packager
A CRX v3 file would consist of:- Cr24 - ASCII 8bit magic string
- 3 - protocol version in unsigned 32bit little endian
- header length in bytes in unsigned 32bit little endian
- header itself - the protobuf serialized object
- crx3.proto - the protobuf descriptor
- as a rule of thumb
- all lengths inside are given as unsigned 32bit little-endian integers
- key files are inserted in PKCS#8 binary encoding (Ruby's
key.to_derworked fine) - ZIP archive of the extension files
Generating protobuf stub
We need to install Google protobuf compilerprotoc. You can save the protocol file in a directory where you want stub to live in. Then generate byprotoc --plugin=protoc-gen-ruby-protobuf=`ls ~/bin/protoc-gen-ruby` --ruby-protobuf_out=./ path/chrome_crx3/crx3.protocrx3.pb.rb in the same directory as the protocol file. All you need is to require 'path/crx3.pb.rb' wherever you want to use that format.Actual packager
At this point the packager is straightforward to implement. Pasting the whole logic here.We have one
::zip method to generate a ZIP archive in memory. If an ERB binding is provided by caller, any .erb files are processed. That's how the above background.js.erb works.The method
::header_v3_extension generates the signature and constructs the whole file header.Finally
::pack_extension just glues the two methods above to generate the final extension.chrome_extension.rb
require 'erb' require 'find' require 'openssl' require 'zip' require_relative 'resource/chrome_crx3/crx3.pb.rb' class ChromeExtension def self.gen_rsa_key(len=2048) OpenSSL::PKey::RSA.generate(len) end # @note file format spec pointers: # https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/977b9b99-2bb9-476b-992f-97a3e37bf20c%40chromium.org def self.header_v3_extension(data, key: nil) key ||= gen_rsa_key() digest = OpenSSL::Digest.new('sha256') signed_data = Crx_file::SignedData.new signed_data.crx_id = digest.digest(key.public_key.to_der)[0...16] signed_data = signed_data.encode signature_data = String.new(encoding: "ASCII-8BIT") signature_data << "CRX3 SignedData\00" signature_data << [ signed_data.size ].pack("V") signature_data << signed_datasignature_data << datasignature = key.sign(digest, signature_data) proof = Crx_file::AsymmetricKeyProof.new proof.public_key = key.public_key.to_der proof.signature = signature header_struct = Crx_file::CrxFileHeader.new header_struct.sha256_with_rsa = [proof] header_struct.signed_header_data = signed_data header_struct = header_struct.encode header = String.new(encoding: "ASCII-8BIT") header << "Cr24" header << [ 3 ].pack("V") # version header << [ header_struct.size ].pack("V") header << header_struct return header end # @param file [String] to write result to # @param dir [String] to read extension from # @param key [OpenSSL::PKey] # @param crxv [String] version of CRX file to create # @param erb_binding [Binding] optional if you want to process ERB files # @return undefined def self.pack_extension(file:, dir:, key: nil, crxv: "v3", erb_binding: nil) zip = zip(dir: dir, erb_binding: erb_binding) File.open(file, 'wb') do |io| io.write self.send(:"header_#{crxv}_extension", zip, key: key) io.write zip end end # @param dir [String] to read extension from # @param erb_binding [Binding] optional if you want to process ERB files # @return [String] the zip file content def self.zip(dir:, erb_binding: nil) dir_prefix_len = dir.end_with?("/") ? dir.length : dir.length + 1 zip = StringIO.new zip.set_encoding "ASCII-8BIT" Zip::OutputStream::write_buffer(zip) do |zio| Find.find(dir) do |file| if File.file? file if erb_binding && file.end_with?(".erb") zio.put_next_entry(file[dir_prefix_len...-4]) erb = ERB.new(File.read file) erb.location = file zio.write(erb.result(erb_binding)) Kernel.puts erb.result(erb_binding) else zio.put_next_entry(file[dir_prefix_len..-1]) zio.write(File.read(file)) end end end end return zip.string end end
Using the packager
Packing the extension is as simple as:require 'chrome_extension'
ChromeExtension.pack_extension(file: "/path/of/target/extension.crx", dir: "/path/of/proxy/extension")Using the extension with Watir
proxy_proto, proxy_user, proxy_pass, proxy_host, proxy_port = <...>
chrome_caps = Selenium::WebDriver::Remote::Capabilities.chrome()
chrome_caps.proxy = Selenium::WebDriver::Proxy.new({http: "#{proxy_proto}://#{proxy_host}:#{proxy_port}", :ssl => "#{proxy_proto}://#{proxy_host}:#{proxy_port}")
# there is a bug in Watir where providing an object here results in an error
# options = Selenium::WebDriver::Chrome::Options.new
# options.add_extension proxy_chrome_ext_file if proxy_chrome_ext_file
options = {}
options[:extensions] = [proxy_chrome_ext_file] if proxy_chrome_ext_file
browser = Watir::Browser.new :chrome, desired_capabilities: chrome_caps, switches: chrome_switches, options: optionsBonus content - CRX2 method
# @note original crx2 format description https://web.archive.org/web/20180114090616/https://developer.chrome.com/extensions/crx
def self.header_v2_extension(data, key: nil)
key ||= gen_rsa_key()
digest = OpenSSL::Digest.new('sha1')
header = String.new(encoding: "ASCII-8BIT")
# it is exactly same signature as `ssh_do_sign(data)` from net/ssh does
signature = key.sign(digest, data)
signature_length = signature.length
pubkey_length = key.public_key.to_der.length
header << "Cr24"
header << [ 2 ].pack("V") # version
header << [ pubkey_length ].pack("V")
header << [ signature_length ].pack("V")
header << key.public_key.to_der
header << signature
return header
endCredits
- https://botproxy.net/docs/how-to/setting-chromedriver-proxy-auth-with-selenium-using-python/
- A nice blog post about doing this with python, unfortunately it is using CRX v2 file format.
- You can see content of actual Chrome extension there as well.
- https://groups.google.com/a/chromium.org/d/topic/chromium-extensions/K3YIsNL_Et4/discussion
- discussion about CRX file format with links to other implementations
- https://github.com/pawliczka/CRX3-Creator
- python implementation of extension packager
- https://crx-checker.appspot.com
- CRX verifier for both versions
