TTimo's blog

Certificate verification of sites backed by letsencrypt in python

A follow up to my post a few months back about setting up letsencrypt certificates for appengine sites .. I found that when accessing the site with python requests or urllib.request modules, I was still getting a SSL certificate verification failure "unable to get local issuer certificate". Browsers however have no problem with the site so it didn't seem like a general problem with my setup.

I found a somewhat old SO issue about this but none of the solutions really worked out. That did put me on the right track to produce this fix though:

# appengine backend by a letsencrypt certificate
TEST_URL='https://core-drones.corecomplex.cc/testSSL'

import os
import ssl
R10_PEM=os.path.abspath('letsencrypt-r10.pem')
context = ssl.create_default_context()
context.load_verify_locations(cafile=R10_PEM)

import urllib.request
response = urllib.request.urlopen(TEST_URL, context=context)
print('urllib.request success')

# verify takes the path to a CA bundle, we want the certifi bundle + our extra R10
import shutil
import certifi
ca_bundle_path = os.path.abspath('ca_bundle.pem')
shutil.copyfile(certifi.where(), ca_bundle_path)
ca_bundle = open(ca_bundle_path, 'at')
ca_bundle.write(open(R10_PEM, 'rt').read())
ca_bundle.close()
print(f'prepared {ca_bundle_path} from R10 cert and {certifi.where()}')

import requests
response = requests.get(TEST_URL, verify=ca_bundle_path)
print('requests.get success')

And the accompanying letsencrypt-r10.pem issuer certificate. I extracted this by following the certificate information in my browser and downloading the R10 PEM file.

This url on my test site gives a good overview of what is going on and reports "certificate chain is incomplete". My understanding is that browsers don't carry the R10 certificate, but they are smart enough to download it on the fly to verify the chain. Python needs a little help, requests being the more annoying module as it doesn't support adding certificates, so you need to pull the current set from certifi and append to it yourself. Pfew!

Update 1/20 - I had a very instructive follow up conversation with another Mastodon user who obviously understands the intricacies of certificate verification better than I do, concluding that this is likely a bug in google's appengine setup .. click below:

Post by @[email protected]

View on Mastodon

Improving Steam Client stability on Linux: setenv and multithreaded environments

The Steam client update on November 5th mentions "Fixed some miscellaneous common crashes." in the Linux notes, which I wanted to give a bit of background on. There's more than one fix that made it in under the somewhat generic header, but the one change that made the most significant impact to Steam client stability on Linux has been a revamping of how we are approaching the setenv and getenv functions.

One of my colleagues rightly dubbed setenv "the worst Linux API". It's such a simple, common API, available on all platforms that it was a little difficult to convince ourselves just how bad it is. I highly encourage anyone who writes software that will run on Linux at some point to read through "RachelByTheBay"'s very engaging post on the subject.

Being the consummate Linux developers that we are, of course we already knew that setenv and getenv aren't safe to use in multithreaded environments. Our policy up to now had been to minimize usage of these functions, and hope for the best.

The Steam client collects basic crash information. The reports have a backtrace of where the crash happened, and what the other threads were doing. On Linux this data is very noisy, there is so much variation across distributions, driver versions, window manager choices, extensive user customization etc., that the reports do not bucket as nicely as they do on Windows.

After a concerted effort to improve our grouping, a pattern emerged. It turns out that if you call setenv in a multithreaded program, sometimes you will crash outright, but that's pretty rare. That happens, but the volumes we saw were always low.

We found that other threads would blow up though, usually with a SIGABRT, shortly after calling getenv themselves. The backtraces for such crashes were all over the place and could not be easily tied to a single cause, but there were several orders of magniture more of those that we had direct setenv crashes.

There is no silver bullet to address this. These APIs are thread safe on Windows and Mac, so developers use them. Mac opted to leak the strings rather than crash (see BUGS in the Mac OS X manual page for getenv).

In the latest release of the Steam client we changed several things:

• We removed the majority of setenv calls. It was mostly used when spawning processes, and refactoring to use exevpe to pass down a prepared environment is an all around improvement.
• We reduced how much we rely on getenv, mostly by caching the calls. There is still an uncomfortable amount of it, but it's in OS libraries at this point (x11, xcb, dbus etc.) and we continue reducing it's usage.
• For the few remaining setenv use cases that could not be easily refactored, we introduced an 'environment manager' that pre-allocates large enough value buffers at startup for fixed environment variable names, before any threading has started.

This last change is what really made a difference for us. Large enough buffers are preallocated at startup, the targeted environment variables exist for the whole process lifetime but start as an empty string. Wherever a setenv would previously happen, we call getenv first to make sure the buffer hasn't moved, and use a direct string copy to update the value. That's not a completely reliable fix, a third party library could still call setenv and trigger crashes, there's still a risk of data races, but we've observed a significant reduction in SIGABRT volumes.

If this can be addressed in glibc, it may involve a tradeoff on features, maybe an opt-in mechanism with a slight departure from the "impossible" POSIX spec. That's something we may pursue in the long term if we can propose something sensible.

Using a SSL certificate from letsencrypt with google app engine

I'm reviving some antique code of mine that uses google app engine. The SSL certificate was self-signed and long expired so this time around I tried to use letsencrypt.

If you search you'll likely come across this post from 2015 which is mostly correct but has a few gotchas. I'm taking a few notes for next time I need to renew. I was doing this on Windows for a change.

• Install certbox.exe via pip install certbot


• Request a certificate from letsencrypt:
  
  certbot.exe certonly --manual --preferred-challenges=dns --email <your email> --agree-tos --no-eff-email --key-type rsa -d <your hostname>
  
  I prefer the DNS challenge, I find adding a TXT record is easier than uploading some custom request handler.
  
  Make sure to set --key-type rsa ! This is the important bit, certbot has switched to ECDSA keys by default and app engine only supports RSA.
  
  
• Convert the private key from "OpenSSH key" to "PEM encoded RSA key":
  
  Make a copy of the private key that certbot downloaded (privkey2-rsa.pem below) and convert it
  ssh-keygen.exe -p -N "" -m pem -t rsa -f privkey2-rsa.pem

See this SO post for more details.
  
  
• You can now upload the fullchain file and the key file to app engine. If you are getting errors such as "The private key you've selected does not appear to be valid." or "the certificate data is invalid", you didn't correctly configure to use RSA or didn't convert the key correctly.

gdb the hard way: using add-symbol-file

gdb's add-symbol-file command is versatile and powerful, but it has some gotchas that will catch the unaware. I haven't found good examples showing how to use it, so I thought I'd write down a few notes:

First, you can read the official documentation, to get a general idea: Commands to specify files.

There is a variety of situations where you'll have gdb attached to a process and it won't find the dynamic modules and debug symbols. In the example below we have a game running under proton that spits out an error message and crashes: *** bit out of range 0 - FD_SETSIZE on fd_set ***: terminated.

A bit of googling and you'll know this crashes in glibc. We would like to obtain a backtrace and investigate, but here is what we have as a starting point:

Thread 248 "IPC:CSteamEngin" received signal SIGABRT, Aborted.
[Switching to LWP 1059407]
0x00007b0c1edd683c in ?? ()
(gdb) bt
#0  0x00007b0c1edd683c in ?? ()
#1  0x0000000000000000 in ?? ()

We know we should be in glibc, we can verify this by looking at the maps:

root@vanguard ~# cat /proc/1059034/maps | grep libc
7b0c1ed48000-7b0c1ed6e000 r--p 00000000 00:19 43674755                   /run/host/usr/lib/libc.so.6
7b0c1ed6e000-7b0c1eec8000 r-xp 00026000 00:19 43674755                   /run/host/usr/lib/libc.so.6
7b0c1eec8000-7b0c1ef1c000 r--p 00180000 00:19 43674755                   /run/host/usr/lib/libc.so.6

We are interested in the executable section (r-xp), also known as the .text section. 7b0c1edd683c is within 7b0c1ed6e000-7b0c1eec8000, that checks out.

But gdb didn't load the symbols for glibc, can we fix that?

This is where the fun begins! Based on the documentation, one might be tempted to type:

add-symbol-file /usr/lib/libc.so.6 0x7b0c1ed6e000

gdb happily eats that up, finds the glibc debug symbols on Arch's debuginfod servers, and shows you this:

(gdb) bt
#0  0x00007b0c1edd683c in __pthread_mutex_cond_lock_full (mutex=0x102a4f) at ../nptl/pthread_mutex_lock.c:514
#1  0x0000000000000000 in ?? ()

Still no backtrace. Mutexes when we should be in FD_SET and friends? What gives?

It turns out we gave gdb the wrong address, and since it doesn't know any better, it did what we asked for, loaded the symbols in the wrong location and now they are off. We need to figure the correct address for the .text section.

We can use readelf to find the offset of the .text section:

readelf -S /usr/lib/libc.so.6

The offset is 263c0

And the /proc/1059034/maps above reports that offset 26000 in libc.so.6 was loaded at address 7b0c1ed6e000

So that's the correction we need. The .text section is actually at 7b0c1ed6e000 + 3c0:

add-symbol-file /usr/lib/libc.so.6 -s .text 0x7b0c1ed6e3c0

Now we get the beginnings of a valid backtrace:

(gdb) bt
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007b0c1edd68a3 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2  0x00007b0c1ed86668 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007b0c1ed6e4b8 in __GI_abort () at abort.c:79
#4  0x00007b0c1ed6f390 in __libc_message (fmt=fmt@entry=0x7b0c1eee62fc "*** %s ***: terminated\n") at ../sysdeps/posix/libc_fatal.c:150
#5  0x00007b0c1ee66b4b in __GI___fortify_fail (msg=msg@entry=0x7b0c1eeeb418 "bit out of range 0 - FD_SETSIZE on fd_set") at fortify_fail.c:24
#6  0x00007b0c1ee66642 in __GI___fdelt_chk (d=<optimized out>) at fdelt_chk.c:26
#7  0x00007b0c1a067f26 in ?? ()

Frame #7 is in steamclient.so, which I can't show you here, but the same process applies to load the symbols in and get the complete trace (if you have them that is).

I've been bit by this on several occasions. Now I have some notes I can go back to! I hope it'll be useful to someone else.

GtkRadiant is now available on Flathub

For roughly 25 years I've told folks looking to run GtkRadiant on Linux to download the source and build it themselves. Last week this finally changed, and users now have the option to install an official GtkRadiant release from Flathub.

The source code for GtkRadiant is very old and hardly evolves anymore. It depends on gtk2 and gtkglext, libraries that are no longer in mainline distributions and require patching in order to build on modern systems. Flatpak offers a stable SDK and runtime, I'm hoping this shields us from distribution and compiler issues in the future.

Doing feature development iteration for an app via Flatpak introduces some annoying friction. It's possible to work with a local checkout of the source at least,  but I have not found a way to do incremental rebuilds .. please comment on this post if you have a solution!

Looking at development workflows based on distrobox for instance could be a great direction to follow in order to improve this. But for now, the insulation we get from distro variations is well worth it.

 

 

CloudGFX - Cloud Linux VMs with GPU acceleration

(should I move to substack .. hurrrgh)

I'm releasing a bit of terraform and saltstack code that I've been sitting on for months: https://github.com/TTimo/CloudGFX-Linux

I wanted to expand my test lab without dealing with local stuff, and generally be able to spin up a distro (Fedora, Mint, Endeavour etc.) on AMD or NVidia hardware easily. I wanted to use this to test the Steam client as we were seeing a few distro-specific odd behaviors that maybe I could just remote in and reproduce over VNC (it turned out this wasn't all that useful for that goal).

But! AWS has been offering systems with AMD GPUs and that's kind of neat and new. For many years the only option in cloud was NVidia. So I cleaned my stuff up and made it easy to use. It'll pop a Fedora 38 VM that you can VNC into and has ready to go mesa drivers.

So of course it's already outdated. Updating to Fedora 39 would be nice, remoting into a Wayland setup like this would be interesting. And of course adding support for NVidia, and a bunch of other distros. (I might look into the Wayland part).

 

GtkRadiant 1.6.7 package for Windows, with Quake II: Remaster support

You can now download an official package of GtkRadiant 1.6.7 binaries for Windows, with the new support for Quake II: Remastered.

Please see my previous post for details. This build comes with a few additional fixes, and a new github repository for the Q2RePack.

The entities.def in the pack is probably a little outdated, a conversion effort from the latest fgd may be needed (or directly adding fgd support to radiant).

 

Quake 2 remastered support in GtkRadiant

Quake 2 remastered was released at QuakeCon over the weekend. I'm very happy that I got to help a bit with id and nightdive getting it shipped.

As I mentioned on Mastodon I had a few bits of GtkRadiant support stashed away until the release. I'd like to get an updated release out, but I don't have much time right now so I've pushed the patches I have to github and zipped up the gamepack. If you have some GtkRadiant smarts you should be able to compile a working build.

The gamepack is a bit outdated, I recommend replacing the ericw-tools folder with latest. Also you may want to use TrenchBroom altogether to map instead of GtkR. TrenchBroom is easier to pickup and more active than GtkR so a better choice if you are new.

 I'm being told there's a Quake Mapping discord where you can get more info about mapping for the new release. HF!

Recent interview with Linux Gaming Central

An update on what I've been up to!

Edited 11/29/2024: LGC has shut down and the original interview link does not work anymore.

Here is a local copy of the interview, retrieved from the LGC archive:

Download Interview-with-tim-besset-valve-contractor

 

Cheap house monitoring with no vendor lock in

Radiant floor heating is awesome. But if you are going to be away for extended periods of time, you'll want to make sure the pipes don't freeze on you and incur costly damage.

When I googled off-the-shelf solutions, I kept finding somewhat expensive products which often came with proprietary apps and assorted vendor lock-ins.

Until I checked my trusty CloudFree website and saw these Aqara Temperature Sensors ($15!). After chatting with CloudFree support for a bit, I found that those sensors work over Zigbee, and don't need an Aqara hub .. any Zigbee hub will do, such as the Sonoff Zigbee 3.0 Dongle Plus, which comes flashed with the open source Z-Flash firmware from TI.

Plugging the dongle into a Raspberry Pi wouldn't be the end of the world, but I wanted to avoid that if possible, and use my EA8300 Linksys router flashed with OpenWrt instead.

Typically, one would run zigbee2mqtt or Home Assistant on the device to which the dongle is plugged in to get the readings. I didn't want to run a node app on the router, and after stumbling on a very useful stackexchange post, this is the setup I landed on:

• ser2net on the router: /usr/sbin/ser2net -d -C "8888:raw:0:/dev/ttyUSB0:115200 remctl"
• Proxy from a VPS system: /usr/bin/socat pty,raw,link=/dev/ttyVUSB0,waitslave tcp:<home dyndns>:8888,forever,reuseaddr,keepalive
  
• Run zigbee2mqtt on the VPS (+ Mosquitto MQTT broker)

From there it gets piped into my Home Assistant instance, with adequate monitoring and alerts.

The Zigbee Home Automation integration in Home Assistant can be used to add the dongle directly in HA rather than use zigbee2mqtt. This is nicely documented by Sonoff. I couldn't make this work with the proxy configuration though.