Friday, December 4, 2020

Why Linux sucks with drivers?

I just found this blog post as an unpublished drart and only one line

I never liked Microsoft in particular. But no

I honestly can't remember what I intended to write here. My guess is that I was frustrated with ROCm and state of GPU drivers and frameworks in Linux. And situation is still quiet frustrating.

But I recently saw the situation with Windows drivers and I'm now convinced Windows is no better with drivers unless you're buying latest hardware.

Last week I've upgraded home wifi router to latest OpenWRT with more secure settings including WPA3 optional support. And while everything else started to work better, one Windows 10 laptop started to disconnect from network very often and network performance was not enough to play Youtube videos.

The machine a pretty descent one, HP Inspiron 15 3000 series with i7 CPU and descent amount of RAM, still a few years old. So I thought that the old Atheros/Qualcomm wifi card needs a driver update.

What I found in HP website was from 2017 and didn't yield any better results. Then with fear I tried as an unofficial driver source and latest driver for the card from 2019. No luck either.

Now I had the option to configure the old router just for this laptop. But this didn't sound right and still compromises the whole local network. Then I decided to find a second hand mPCIe Wi-Fi card. Choices basically boil down to old Intel, Broadcom and Realtek models. Realtek is the one I did *not* try due to lack of reputation.

I found a guy who had both Intel and Broadcom models. So I could take both home and see which one works better. The Intel model had drivers in windows update from 2013 only (looking at available model, the latest model mPCIe Intel I found is discontinued and has 2019 drivers). The Broadcom had 2016 or something.

I wanted to try the Broadcom first due to Bluetooth 4.0 LE support. Some new mice and other devices only support that version. It performed well but computer crashed a few times for one day.

Finally tried the Intel with 2013 driver. Now that worked rock solid and fast. Downside is Bluetooth 3.0 support but mouse can also be used with a receiver so I guess it should be good enough. I see Bluetooth 4 USB dongles for $5 so not a big deal to add such support if needed in the future.

Unfortunately this card will never get WPA3 support and I have no idea whether the recent WPA2 vulnerabilities have been fixed with it somehow or not.

In conclusion I see that for older hardware, that is not ancient, just little old but perfectly fine, Linux still gets much better support.

I'm sorry if you didn't expect just another Windows rant with a click-bait title. Still I needed to express my frustration with the state of computing. And I don't mention Apple here, it is so closed ecosystem that no amount of polish can fix.

Wednesday, April 1, 2020

Back to third grade with SSH or how to setup ~/.ssh/authorized_keys

Very often I am asked to SSH to a machine just to hit access denied. A few roundtrips then needed until issue is resolved. Here are my commands to get it working from the first time.

# mkdir .ssh
# vi .ssh/authorized_keys # add user's public key here
# chown -R user.user .ssh
# chmod 700 .ssh
# chmod 600 .ssh/authorized_keys
# restorecon -R .ssh

Last one is for SELinux enabled distributions.

Hope you find useful.

Thursday, November 21, 2019

Using authenticated proxy with Selenium / Packaging Chrome extensions with Ruby


Recently I've got the request to implement authenticated proxy support for our product test framework. The problem is that recent browsers do not allow the widely popular syntax and still ask you to manually enter credentials.

The next problem is that Selenium does not let you interact with these basic auth dialogs [1][2]. So how should one go about this?

Chrome allows you to do this with a custom extension that you can insert with selenium/watir.

One additional complication is that we can use a different proxy server each time. Thus extension needs to be packaged on the fly.

Chrome extension

This is the proxy extension as I use it. See it as an example for whatever you'll be trying to do. It consists of only 2 files you can put in an empty directory.


    "version": "0.0.1",
    "manifest_version": 2,
    "name": "Authenticated Proxy",
    "permissions": [
    "background": {
        "scripts": ["background.js"]


var config = {
  mode: "fixed_servers",
  rules: {
    singleProxy: {
      scheme: "<%= proxy_proto %>",
      host: "<%= proxy_host %>",
      port: parseInt(<%= proxy_port %>)
    bypassList: <%= proxy_bypass.split(/[ ,]/).delete_if(&:empty?).to_json %>

chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});

function callbackFn(details) {
  return {
    authCredentials: {
      username: "<%= proxy_user %>",
      password: "<%= proxy_pass %>"

  {urls: ["<all_urls>"]},

Protocol Buffers

As you can see on the web site, Protocol Buffers is a method of serializing structured data. For CRX3 (unlike CRX2) format it is part of the required header for the extension.

I decided to use ruby-protobuf project instead of the google ruby library because it appeared well maintained and pure ruby. I assume google ruby library will work well too.

The Packager

 A CRX v3 file would consist of:
  • Cr24 - ASCII 8bit magic string
  • 3 - protocol version in unsigned 32bit little endian
  • header length in bytes in unsigned 32bit little endian
  • header itself - the protobuf serialized object
    • crx3.proto - the protobuf descriptor
    • as a rule of thumb
      •  all lengths inside are given as unsigned 32bit little-endian integers
      • key files are inserted in PKCS#8 binary encoding (Ruby's key.to_der worked fine)
  • ZIP archive of the extension files

Generating protobuf stub

We need to install Google protobuf compiler protoc. You can save the protocol file in a directory where you want stub to live in. Then generate by

protoc --plugin=protoc-gen-ruby-protobuf=`ls ~/bin/protoc-gen-ruby` --ruby-protobuf_out=./ path/chrome_crx3/crx3.proto
This will create a file crx3.pb.rb in the same directory as the protocol file. All you need is to require 'path/crx3.pb.rb' wherever you want to use that format.

Actual packager

At this point the packager is straightforward to implement. Pasting the whole logic here.

We have one ::zip method to generate a ZIP archive in memory. If an ERB binding is provided by caller, any .erb files are processed. That's how the above background.js.erb works.

The method ::header_v3_extension generates the signature and constructs the whole file header.

Finally ::pack_extension just glues the two methods above to generate the final extension.


require 'erb'
require 'find'
require 'openssl'
require 'zip'

require_relative 'resource/chrome_crx3/crx3.pb.rb'

class ChromeExtension
  def self.gen_rsa_key(len=2048)

  #  @note file format spec pointers:
  def self.header_v3_extension(data, key: nil)
    key ||= gen_rsa_key()

    digest ='sha256')
    signed_data =
    signed_data.crx_id = digest.digest(key.public_key.to_der)[0...16]
    signed_data = signed_data.encode

    signature_data = "ASCII-8BIT")
    signature_data << "CRX3 SignedData\00"
    signature_data << [ signed_data.size ].pack("V")
    signature_data << signed_data
    signature_data << data

    signature = key.sign(digest, signature_data)

    proof =
    proof.public_key = key.public_key.to_der
    proof.signature = signature

    header_struct =
    header_struct.sha256_with_rsa = [proof]
    header_struct.signed_header_data = signed_data
    header_struct = header_struct.encode

    header = "ASCII-8BIT")
    header << "Cr24"
    header << [ 3 ].pack("V") # version
    header << [ header_struct.size ].pack("V")
    header << header_struct

    return header

  # @param file [String] to write result to
  # @param dir [String] to read extension from
  # @param key [OpenSSL::PKey]
  # @param crxv [String] version of CRX file to create
  # @param erb_binding [Binding] optional if you want to process ERB files
  # @return undefined
  def self.pack_extension(file:, dir:, key: nil, crxv: "v3", erb_binding: nil)
    zip = zip(dir: dir, erb_binding: erb_binding), 'wb') do |io|
      io.write self.send(:"header_#{crxv}_extension", zip, key: key)
      io.write zip

  # @param dir [String] to read extension from
  # @param erb_binding [Binding] optional if you want to process ERB files
  # @return [String] the zip file content
  def, erb_binding: nil)
    dir_prefix_len = dir.end_with?("/") ? dir.length : dir.length + 1
    zip =
    zip.set_encoding "ASCII-8BIT"
    Zip::OutputStream::write_buffer(zip) do |zio|
      Find.find(dir) do |file|
        if File.file? file
          if erb_binding && file.end_with?(".erb")
            erb = file)
            erb.location = file
            Kernel.puts erb.result(erb_binding)
    return zip.string

Using the packager

Packing the extension is as simple as:
require 'chrome_extension'

ChromeExtension.pack_extension(file: "/path/of/target/extension.crx", dir: "/path/of/proxy/extension")

Using the extension with Watir

proxy_proto, proxy_user, proxy_pass, proxy_host, proxy_port = <...>
chrome_caps =
chrome_caps.proxy ={http: "#{proxy_proto}://#{proxy_host}:#{proxy_port}", :ssl => "#{proxy_proto}://#{proxy_host}:#{proxy_port}")
# there is a bug in Watir where providing an object here results in an error 
# options =
# options.add_extension proxy_chrome_ext_file if proxy_chrome_ext_file
options = {}
options[:extensions] = [proxy_chrome_ext_file] if proxy_chrome_ext_file
browser = :chrome, desired_capabilities: chrome_caps, switches: chrome_switches, options: options

Bonus content - CRX2 method

  #  @note original crx2 format description
  def self.header_v2_extension(data, key: nil)
    key ||= gen_rsa_key()
    digest ='sha1')
    header = "ASCII-8BIT")

    # it is exactly same signature as `ssh_do_sign(data)` from net/ssh does
    signature = key.sign(digest, data)
    signature_length = signature.length
    pubkey_length = key.public_key.to_der.length

    header << "Cr24"
    header << [ 2 ].pack("V") # version
    header << [ pubkey_length ].pack("V")
    header << [ signature_length ].pack("V")
    header << key.public_key.to_der
    header << signature

    return header


Monday, April 15, 2019

accessing namespaces of a docker/podman container (nsenter)

There is a nice utility `nsenter` that allows you to switch to the namespace of another process. It took me considerable time to search it out today so thought to write a short blog about it.

Now I have a Podman container (for docker just use `docker` command instead of `podman` below). I started that container by:

$ sudo podman run -t -a STDIN -a STDOUT -a STDERR --rm=true --entrypoint /bin/bash

And I've been running some testing on it but it turned out I want to increase limits without destroying my preparations if I exit the process. So first thing is to figure out pid namespace of my container:

$ sudo podman ps --ns
CONTAINER ID  NAMES                PID   CGROUPNS    IPC         MNT         NET         PIDNS       USERNS      UTS
a147a3a5b35f  fervent_stonebraker  1408  4026531835  4026532431  4026532429  4026532360  4026532432  4026531837  4026532430

I see different namespaces but `nsenter` requires a file name to switch to a PID namespace. SO I will use the PID information in above output.

$ sudo nsenter --pid=/proc/1408/ns/pid

The above starts a shell for me in the PID namespace of my container. Now I want to change limits. Interesting to note here is that I change pid 1 as it is the PID of my bash shell in the container:

$ sudo prlimit --rss=-1 --memlock=33554432 --pid 1

Finally verify limits in my container shell:

bash-4.2$ ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 23534
max locked memory       (kbytes, -l) 32768
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1048576
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 16384
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1048576
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

One interesting thing is `ps` inside namespace. If I run these two

$ ps -ef
$ sudo nsenter --pid=/proc/1408/ns/pid ps -ef

They will show exactly the same output. It is because I still have same `/proc` mounted even though my PID namespace is changed. And it is what `ps` looks at.

With `nsenter` you can switch any namespace, not only PID. I hope this is a useful short demonstration how to do fun things with linux namespaces.

Some links:
  • - namespaces overview series

Saturday, January 19, 2019

Install OKD 3.11 with source version of openshift-ansible installer

To install OpenShift by openshift-ansible from sources, one needs to build the openshift-ansible RPMs and install them as a repo on the machine performing the installation. For 3.11 in CI this is done by the following YAML.

First clone openshift-ansible repo.

$ git clone --depth=1 --branch=release-3.11

Then build base image as described in the YAML.

$ cd openshift-ansible
$ BUILDAH_LAYERS=false sudo podman build -f images/installer/Dockerfile -t ocp-ansible --layers=false .

Run the image and prepare for RPM building

$ sudo podman run -t -a STDIN -a STDOUT -a STDERR --rm=true -u root ocp-ansible /bin/bash
# yum install tito createrepo
# git clone --depth=1 --branch=release-3.11
# git config --add
# git config --add myname

Build RPMs as pointed in the rpm building section of the YAML with slight modifications. In bold I write things that differ.

# tito tag --offline --no-auto-changelog
# tito build --output="_output/local/releases" --rpm --test --offline --quiet
# createrepo _output/local/releases/noarch

Now RPM repo is under `_output/local/releases/noarch/`.  Copy it to a web server or locally on the machine where you would run the installation. Then create a file /etc/yum.repos.d/my-ocp-ansible.conf:

baseurl = <file:// or http:// url of RPM repo>
enabled = 1
gpgcheck = 0
name = Custom built OpenShift Ansible repo

Finally perform the installation as described in the official docs.

$ ansible-playbook ....

Make sure that you see your RPMs in the install log under `List all openshift ansible packages`.

Thursday, January 10, 2019

Building debug firefox build from source RPM on Red Hat Enterprise Linux

In short:
  • Create an account on
  • Get Red Hat Enterprise Linux (RHEL)
    • Download and install RHEL server on a local physical or virtual machine (it is free with developer subscription).
    • Or spawn a RHEL machine in some cloud service.
    • Important:  you will need a large machine. For me 4GB failed [*] and I used a 16GB one. I didn't check what is the minimum required.
  • If you installed your own RHEL, then you need to subscribe the machine.
    • subscription-manager register # use your credentials
    • subscription-manager attach
      • if the above does not work automatically try the below
      • subscription-manager list --available
      • subscription-manager attach --pool=<whatever you find useful above>
  • sudo yum install yum-utils rpm-build
  • yumdownloader --source firefox
  • rpm -ivh firefox-*.rpm
  • sudo yum-builddep rpmbuild/SPECS/firefox.spec
    • on a vanilla system you will see missing dependencies
    • if you wanted to figure that out by yourself, you'd go to and search for the packages to see what repos they come from (or maybe use some clever yum command that I don't know atm)
  • yum-config-manager --enable rhel-7-server-devtools-rpms rhel-7-server-optional-rpms
    • or edit /etc/yum.repos.d/redhat.repo
  • sudo yum-builddep rpmbuild/SPECS/firefox.spec # this time it will succeed
  • rpmbuild -ba --with=debug_build rpmbuild/SPECS/firefox.spec
  • find the built rpm at
    • ~/rpmbuild/RPMS/x86_64/firefox-60.4.0-1.el7.x86_64.rpm
    • ~/rpmbuild/RPMS/x86_64/firefox-debuginfo-60.4.0-1.el7.x86_64.rpm
    • ~/rpmbuild/SRPMS/firefox-60.4.0-1.el7.src.rpm

[*] it is really sad, in the past one could learn to be a developer on a budget machine. Nowadays it seems like even compiling your code takes a beefy one :/

Friday, November 23, 2018

Running Logstash container under OpenShift

What is the issue?

Main problem for running random images under OpenShift is that OpenShift starts containers as a random user. This is done for security reasons (isolation of workloads). A user can be given permissions to run `privileged` containers but this is not recommended if it can be avoided.

You can check my earlier blog Creating docker images suitable for OpenShift (ssh-git image HowTo) for openshift for more information and a more complicated example.

Logstash official container image

Official logstash image can be found on dockerhub and is built off logstash-docker github project. It is not specifically built to run in OpenShift but it is still straightforward to run it unmodified. There are only 2 issues:
  • it tries to run as user 1000 and expects to find logstash code in user's home directory
  • some configuration files lack needed permissions to be modified by a randim user id

Get running it

Depending on what you're trying to do, you can approach in a somehow different way. I will give a specific example by mostly retaining original configuration (beats input and stdout output) but adding `config` file with Kubernetes audit setup and disabling elasticsearch monitoring as don't have an elasticsearch backend. I hope this will provide enough of an example so you can setup your instance the way you desire.

Creating configuration

To store our custom configuration files, we will create a config map with the file content.
$ cat logstash-cfgmap.yml
apiVersion: v1
data: |-
      set -x -e
      rm -vf "/usr/share/logstash/config/logstash.yml"
      echo "xpack.monitoring.enabled: false" > "/usr/share/logstash/config/logstash.yml"
      exec /usr/local/bin/docker-entrypoint "$@"
  config: |-
            #TODO, figure out a way to use kubeconfig file to authenticate to logstash
            # Webhook audit backend sends several events together with EventList
            # split each event here.
            # We only need event subelement, remove others.
            remove_field=>[headers, metadata, apiVersion, "@timestamp", kind, "@version", host]
            rename => {items=>event}
            # Audit events from different users will be saved into different files.
kind: ConfigMap
  name: logstash
$ oc create -f logstash-cfgmap.yml
configmap/logstash created

With the above config map we have two files.
  • - this we need to run some custom commands before we delegate back to image original entry point. Namely to remove original `logstash.yml` that lacks group write permissions. As well disable elasticsearch monitoring that is enabled by default. The write permissions are needed in case logstash image startup script notice env variables that need to be converted to configuration entries and put into it. See env2yaml.go and docker-config docs.
  • config - this file contains logstash configuration file and is a copy of what I presently see in kubernetes auditing docs.
Note that at this step you can create full Logstash configuration inside the config map together with `logstash.yml`,``, `pipelines.yml`, etc. Then we can ignore default config from image.

Creating deployment config

$ oc run logstash  --image=logstash:6.5.0 --env=LOGSTASH_HOME\=/usr/share/logstash --command=true bash -- /etc/logstash/ -f /etc/logstash/config created

A few things to explain:
  • we are setting LOGSTASH_HOME environment variable to `/usr/share/logstash` because we are running as a random user thus user home directory will not work
  • we override container start command to our wrapper script
    • we add `-f  /etc/logstash/config` to point at our custom config
    • in case we wanted to put all our configuration in the config map, then we can set instead `--path.settings /etc/logstash/`
    • once pull/113 is merged, the custom startup script wrapper will not be needed, but we may still want to provide additional arguments like `-f` and `--path.settings`
 Further we need to make sure our custom configuration is mounted under  `/usr/share/logstash`
$ oc set volume --add=true --configmap-name=logstash --mount-path=/etc/logstash dc/logstash volume updated

Finally, because our custom config wants to write under /var/log, we need to mount a volume on that path.
oc set volume --add=true --mount-path=/var/log dc/logstash

What we did is create an emptyDir volume that will go away when pod dies. If you want to persist these logs, then a Persistent Volume needs to be used instead.

Exposing logstash service to the world

First we need to create a service that will allow other project pods and Kubernetes to reach Logstash.
$ oc expose dc logstash --port=8888
service/logstash exposed
Port 8888 is what we have set as an HTTP endpoint in `config`. If you expose other ports, then you'd have to create one service per each port that you care about.

We can easily expose HTTP endpoints to the great Internet so that we can collect logs from services external of the OpenShift environments. We can also expose non-HTTP endpoints to the internet with the node port service type but there are more limitations. Or for 4.x Ingress Controller. Below see how to do with the HTTP traffic.
$ oc expose service logstash --name=logstash-http-input exposed

Important: Only expose secured endpoints to the Internet! In the above example the endpoint is insecure and no authentication is required. Thus somebody can DoS your Logstash service easily.

That's all.