Thursday, November 21, 2019

Using authenticated proxy with Selenium / Packaging Chrome extensions with Ruby


Recently I've got the request to implement authenticated proxy support for our product test framework. The problem is that recent browsers do not allow the widely popular syntax and still ask you to manually enter credentials.

The next problem is that Selenium does not let you interact with these basic auth dialogs [1][2]. So how should one go about this?

Chrome allows you to do this with a custom extension that you can insert with selenium/watir.

One additional complication is that we can use a different proxy server each time. Thus extension needs to be packaged on the fly.

Chrome extension

This is the proxy extension as I use it. See it as an example for whatever you'll be trying to do. It consists of only 2 files you can put in an empty directory.


    "version": "0.0.1",
    "manifest_version": 2,
    "name": "Authenticated Proxy",
    "permissions": [
    "background": {
        "scripts": ["background.js"]


var config = {
  mode: "fixed_servers",
  rules: {
    singleProxy: {
      scheme: "<%= proxy_proto %>",
      host: "<%= proxy_host %>",
      port: parseInt(<%= proxy_port %>)
    bypassList: <%= proxy_bypass.split(/[ ,]/).delete_if(&:empty?).to_json %>

chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});

function callbackFn(details) {
  return {
    authCredentials: {
      username: "<%= proxy_user %>",
      password: "<%= proxy_pass %>"

  {urls: ["<all_urls>"]},

Protocol Buffers

As you can see on the web site, Protocol Buffers is a method of serializing structured data. For CRX3 (unlike CRX2) format it is part of the required header for the extension.

I decided to use ruby-protobuf project instead of the google ruby library because it appeared well maintained and pure ruby. I assume google ruby library will work well too.

The Packager

 A CRX v3 file would consist of:
  • Cr24 - ASCII 8bit magic string
  • 3 - protocol version in unsigned 32bit little endian
  • header length in bytes in unsigned 32bit little endian
  • header itself - the protobuf serialized object
    • crx3.proto - the protobuf descriptor
    • as a rule of thumb
      •  all lengths inside are given as unsigned 32bit little-endian integers
      • key files are inserted in PKCS#8 binary encoding (Ruby's key.to_der worked fine)
  • ZIP archive of the extension files

Generating protobuf stub

We need to install Google protobuf compiler protoc. You can save the protocol file in a directory where you want stub to live in. Then generate by

protoc --plugin=protoc-gen-ruby-protobuf=`ls ~/bin/protoc-gen-ruby` --ruby-protobuf_out=./ path/chrome_crx3/crx3.proto
This will create a file crx3.pb.rb in the same directory as the protocol file. All you need is to require 'path/crx3.pb.rb' wherever you want to use that format.

Actual packager

At this point the packager is straightforward to implement. Pasting the whole logic here.

We have one ::zip method to generate a ZIP archive in memory. If an ERB binding is provided by caller, any .erb files are processed. That's how the above background.js.erb works.

The method ::header_v3_extension generates the signature and constructs the whole file header.

Finally ::pack_extension just glues the two methods above to generate the final extension.


require 'erb'
require 'find'
require 'openssl'
require 'zip'

require_relative 'resource/chrome_crx3/crx3.pb.rb'

class ChromeExtension
  def self.gen_rsa_key(len=2048)

  #  @note file format spec pointers:
  def self.header_v3_extension(data, key: nil)
    key ||= gen_rsa_key()

    digest ='sha256')
    signed_data =
    signed_data.crx_id = digest.digest(key.public_key.to_der)[0...16]
    signed_data = signed_data.encode

    signature_data = "ASCII-8BIT")
    signature_data << "CRX3 SignedData\00"
    signature_data << [ signed_data.size ].pack("V")
    signature_data << signed_data
    signature_data << data

    signature = key.sign(digest, signature_data)

    proof =
    proof.public_key = key.public_key.to_der
    proof.signature = signature

    header_struct =
    header_struct.sha256_with_rsa = [proof]
    header_struct.signed_header_data = signed_data
    header_struct = header_struct.encode

    header = "ASCII-8BIT")
    header << "Cr24"
    header << [ 3 ].pack("V") # version
    header << [ header_struct.size ].pack("V")
    header << header_struct

    return header

  # @param file [String] to write result to
  # @param dir [String] to read extension from
  # @param key [OpenSSL::PKey]
  # @param crxv [String] version of CRX file to create
  # @param erb_binding [Binding] optional if you want to process ERB files
  # @return undefined
  def self.pack_extension(file:, dir:, key: nil, crxv: "v3", erb_binding: nil)
    zip = zip(dir: dir, erb_binding: erb_binding), 'wb') do |io|
      io.write self.send(:"header_#{crxv}_extension", zip, key: key)
      io.write zip

  # @param dir [String] to read extension from
  # @param erb_binding [Binding] optional if you want to process ERB files
  # @return [String] the zip file content
  def, erb_binding: nil)
    dir_prefix_len = dir.end_with?("/") ? dir.length : dir.length + 1
    zip =
    zip.set_encoding "ASCII-8BIT"
    Zip::OutputStream::write_buffer(zip) do |zio|
      Find.find(dir) do |file|
        if File.file? file
          if erb_binding && file.end_with?(".erb")
            erb = file)
            erb.location = file
            Kernel.puts erb.result(erb_binding)
    return zip.string

Using the packager

Packing the extension is as simple as:
require 'chrome_extension'

ChromeExtension.pack_extension(file: "/path/of/target/extension.crx", dir: "/path/of/proxy/extension")

Using the extension with Watir

proxy_proto, proxy_user, proxy_pass, proxy_host, proxy_port = <...>
chrome_caps =
chrome_caps.proxy ={http: "#{proxy_proto}://#{proxy_host}:#{proxy_port}", :ssl => "#{proxy_proto}://#{proxy_host}:#{proxy_port}")
# there is a bug in Watir where providing an object here results in an error 
# options =
# options.add_extension proxy_chrome_ext_file if proxy_chrome_ext_file
options = {}
options[:extensions] = [proxy_chrome_ext_file] if proxy_chrome_ext_file
browser = :chrome, desired_capabilities: chrome_caps, switches: chrome_switches, options: options

Bonus content - CRX2 method

  #  @note original crx2 format description
  def self.header_v2_extension(data, key: nil)
    key ||= gen_rsa_key()
    digest ='sha1')
    header = "ASCII-8BIT")

    # it is exactly same signature as `ssh_do_sign(data)` from net/ssh does
    signature = key.sign(digest, data)
    signature_length = signature.length
    pubkey_length = key.public_key.to_der.length

    header << "Cr24"
    header << [ 2 ].pack("V") # version
    header << [ pubkey_length ].pack("V")
    header << [ signature_length ].pack("V")
    header << key.public_key.to_der
    header << signature

    return header


Monday, April 15, 2019

accessing namespaces of a docker/podman container (nsenter)

There is a nice utility `nsenter` that allows you to switch to the namespace of another process. It took me considerable time to search it out today so thought to write a short blog about it.

Now I have a Podman container (for docker just use `docker` command instead of `podman` below). I started that container by:

$ sudo podman run -t -a STDIN -a STDOUT -a STDERR --rm=true --entrypoint /bin/bash

And I've been running some testing on it but it turned out I want to increase limits without destroying my preparations if I exit the process. So first thing is to figure out pid namespace of my container:

$ sudo podman ps --ns
CONTAINER ID  NAMES                PID   CGROUPNS    IPC         MNT         NET         PIDNS       USERNS      UTS
a147a3a5b35f  fervent_stonebraker  1408  4026531835  4026532431  4026532429  4026532360  4026532432  4026531837  4026532430

I see different namespaces but `nsenter` requires a file name to switch to a PID namespace. SO I will use the PID information in above output.

$ sudo nsenter --pid=/proc/1408/ns/pid

The above starts a shell for me in the PID namespace of my container. Now I want to change limits. Interesting to note here is that I change pid 1 as it is the PID of my bash shell in the container:

$ sudo prlimit --rss=-1 --memlock=33554432 --pid 1

Finally verify limits in my container shell:

bash-4.2$ ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 23534
max locked memory       (kbytes, -l) 32768
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1048576
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 16384
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1048576
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

One interesting thing is `ps` inside namespace. If I run these two

$ ps -ef
$ sudo nsenter --pid=/proc/1408/ns/pid ps -ef

They will show exactly the same output. It is because I still have same `/proc` mounted even though my PID namespace is changed. And it is what `ps` looks at.

With `nsenter` you can switch any namespace, not only PID. I hope this is a useful short demonstration how to do fun things with linux namespaces.

Some links:
  • - namespaces overview series

Saturday, January 19, 2019

Install OKD 3.11 with source version of openshift-ansible installer

To install OpenShift by openshift-ansible from sources, one needs to build the openshift-ansible RPMs and install them as a repo on the machine performing the installation. For 3.11 in CI this is done by the following YAML.

First clone openshift-ansible repo.

$ git clone --depth=1 --branch=release-3.11

Then build base image as described in the YAML.

$ cd openshift-ansible
$ BUILDAH_LAYERS=false sudo podman build -f images/installer/Dockerfile -t ocp-ansible --layers=false .

Run the image and prepare for RPM building

$ sudo podman run -t -a STDIN -a STDOUT -a STDERR --rm=true -u root ocp-ansible /bin/bash
# yum install tito createrepo
# git clone --depth=1 --branch=release-3.11
# git config --add
# git config --add myname

Build RPMs as pointed in the rpm building section of the YAML with slight modifications. In bold I write things that differ.

# tito tag --offline --no-auto-changelog
# tito build --output="_output/local/releases" --rpm --test --offline --quiet
# createrepo _output/local/releases/noarch

Now RPM repo is under `_output/local/releases/noarch/`.  Copy it to a web server or locally on the machine where you would run the installation. Then create a file /etc/yum.repos.d/my-ocp-ansible.conf:

baseurl = <file:// or http:// url of RPM repo>
enabled = 1
gpgcheck = 0
name = Custom built OpenShift Ansible repo

Finally perform the installation as described in the official docs.

$ ansible-playbook ....

Make sure that you see your RPMs in the install log under `List all openshift ansible packages`.

Thursday, January 10, 2019

Building debug firefox build from source RPM on Red Hat Enterprise Linux

In short:
  • Create an account on
  • Get Red Hat Enterprise Linux (RHEL)
    • Download and install RHEL server on a local physical or virtual machine (it is free with developer subscription).
    • Or spawn a RHEL machine in some cloud service.
    • Important:  you will need a large machine. For me 4GB failed [*] and I used a 16GB one. I didn't check what is the minimum required.
  • If you installed your own RHEL, then you need to subscribe the machine.
    • subscription-manager register # use your credentials
    • subscription-manager attach
      • if the above does not work automatically try the below
      • subscription-manager list --available
      • subscription-manager attach --pool=<whatever you find useful above>
  • sudo yum install yum-utils rpm-build
  • yumdownloader --source firefox
  • rpm -ivh firefox-*.rpm
  • sudo yum-builddep rpmbuild/SPECS/firefox.spec
    • on a vanilla system you will see missing dependencies
    • if you wanted to figure that out by yourself, you'd go to and search for the packages to see what repos they come from (or maybe use some clever yum command that I don't know atm)
  • yum-config-manager --enable rhel-7-server-devtools-rpms rhel-7-server-optional-rpms
    • or edit /etc/yum.repos.d/redhat.repo
  • sudo yum-builddep rpmbuild/SPECS/firefox.spec # this time it will succeed
  • rpmbuild -ba --with=debug_build rpmbuild/SPECS/firefox.spec
  • find the built rpm at
    • ~/rpmbuild/RPMS/x86_64/firefox-60.4.0-1.el7.x86_64.rpm
    • ~/rpmbuild/RPMS/x86_64/firefox-debuginfo-60.4.0-1.el7.x86_64.rpm
    • ~/rpmbuild/SRPMS/firefox-60.4.0-1.el7.src.rpm

[*] it is really sad, in the past one could learn to be a developer on a budget machine. Nowadays it seems like even compiling your code takes a beefy one :/