TTimo's blog

gdb the hard way: using add-symbol-file

gdb's add-symbol-file command is versatile and powerful, but it has some gotchas that will catch the unaware. I haven't found good examples showing how to use it, so I thought I'd write down a few notes:

First, you can read the official documentation, to get a general idea: Commands to specify files.

There is a variety of situations where you'll have gdb attached to a process and it won't find the dynamic modules and debug symbols. In the example below we have a game running under proton that spits out an error message and crashes: *** bit out of range 0 - FD_SETSIZE on fd_set ***: terminated.

A bit of googling and you'll know this crashes in glibc. We would like to obtain a backtrace and investigate, but here is what we have as a starting point:

Thread 248 "IPC:CSteamEngin" received signal SIGABRT, Aborted.
[Switching to LWP 1059407]
0x00007b0c1edd683c in ?? ()
(gdb) bt
#0  0x00007b0c1edd683c in ?? ()
#1  0x0000000000000000 in ?? ()

We know we should be in glibc, we can verify this by looking at the maps:

root@vanguard ~# cat /proc/1059034/maps | grep libc
7b0c1ed48000-7b0c1ed6e000 r--p 00000000 00:19 43674755                   /run/host/usr/lib/libc.so.6
7b0c1ed6e000-7b0c1eec8000 r-xp 00026000 00:19 43674755                   /run/host/usr/lib/libc.so.6
7b0c1eec8000-7b0c1ef1c000 r--p 00180000 00:19 43674755                   /run/host/usr/lib/libc.so.6

We are interested in the executable section (r-xp), also known as the .text section. 7b0c1edd683c is within 7b0c1ed6e000-7b0c1eec8000, that checks out.

But gdb didn't load the symbols for glibc, can we fix that?

This is where the fun begins! Based on the documentation, one might be tempted to type:

add-symbol-file /usr/lib/libc.so.6 0x7b0c1ed6e000

gdb happily eats that up, finds the glibc debug symbols on Arch's debuginfod servers, and shows you this:

(gdb) bt
#0  0x00007b0c1edd683c in __pthread_mutex_cond_lock_full (mutex=0x102a4f) at ../nptl/pthread_mutex_lock.c:514
#1  0x0000000000000000 in ?? ()

Still no backtrace. Mutexes when we should be in FD_SET and friends? What gives?

It turns out we gave gdb the wrong address, and since it doesn't know any better, it did what we asked for, loaded the symbols in the wrong location and now they are off. We need to figure the correct address for the .text section.

We can use readelf to find the offset of the .text section:

readelf -S /usr/lib/libc.so.6

The offset is 263c0

And the /proc/1059034/maps above reports that offset 26000 in libc.so.6 was loaded at address 7b0c1ed6e000

So that's the correction we need. The .text section is actually at 7b0c1ed6e000 + 3c0:

add-symbol-file /usr/lib/libc.so.6 -s .text 0x7b0c1ed6e3c0

Now we get the beginnings of a valid backtrace:

(gdb) bt
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007b0c1edd68a3 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2  0x00007b0c1ed86668 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007b0c1ed6e4b8 in __GI_abort () at abort.c:79
#4  0x00007b0c1ed6f390 in __libc_message (fmt=fmt@entry=0x7b0c1eee62fc "*** %s ***: terminated\n") at ../sysdeps/posix/libc_fatal.c:150
#5  0x00007b0c1ee66b4b in __GI___fortify_fail (msg=msg@entry=0x7b0c1eeeb418 "bit out of range 0 - FD_SETSIZE on fd_set") at fortify_fail.c:24
#6  0x00007b0c1ee66642 in __GI___fdelt_chk (d=<optimized out>) at fdelt_chk.c:26
#7  0x00007b0c1a067f26 in ?? ()

Frame #7 is in steamclient.so, which I can't show you here, but the same process applies to load the symbols in and get the complete trace (if you have them that is).

I've been bit by this on several occasions. Now I have some notes I can go back to! I hope it'll be useful to someone else.

GtkRadiant is now available on Flathub

For roughly 25 years I've told folks looking to run GtkRadiant on Linux to download the source and build it themselves. Last week this finally changed, and users now have the option to install an official GtkRadiant release from Flathub.

The source code for GtkRadiant is very old and hardly evolves anymore. It depends on gtk2 and gtkglext, libraries that are no longer in mainline distributions and require patching in order to build on modern systems. Flatpak offers a stable SDK and runtime, I'm hoping this shields us from distribution and compiler issues in the future.

Doing feature development iteration for an app via Flatpak introduces some annoying friction. It's possible to work with a local checkout of the source at least,  but I have not found a way to do incremental rebuilds .. please comment on this post if you have a solution!

Looking at development workflows based on distrobox for instance could be a great direction to follow in order to improve this. But for now, the insulation we get from distro variations is well worth it.

 

 

CloudGFX - Cloud Linux VMs with GPU acceleration

(should I move to substack .. hurrrgh)

I'm releasing a bit of terraform and saltstack code that I've been sitting on for months: https://github.com/TTimo/CloudGFX-Linux

I wanted to expand my test lab without dealing with local stuff, and generally be able to spin up a distro (Fedora, Mint, Endeavour etc.) on AMD or NVidia hardware easily. I wanted to use this to test the Steam client as we were seeing a few distro-specific odd behaviors that maybe I could just remote in and reproduce over VNC (it turned out this wasn't all that useful for that goal).

But! AWS has been offering systems with AMD GPUs and that's kind of neat and new. For many years the only option in cloud was NVidia. So I cleaned my stuff up and made it easy to use. It'll pop a Fedora 38 VM that you can VNC into and has ready to go mesa drivers.

So of course it's already outdated. Updating to Fedora 39 would be nice, remoting into a Wayland setup like this would be interesting. And of course adding support for NVidia, and a bunch of other distros. (I might look into the Wayland part).

 

GtkRadiant 1.6.7 package for Windows, with Quake II: Remaster support

You can now download an official package of GtkRadiant 1.6.7 binaries for Windows, with the new support for Quake II: Remastered.

Please see my previous post for details. This build comes with a few additional fixes, and a new github repository for the Q2RePack.

The entities.def in the pack is probably a little outdated, a conversion effort from the latest fgd may be needed (or directly adding fgd support to radiant).

 

Quake 2 remastered support in GtkRadiant

Quake 2 remastered was released at QuakeCon over the weekend. I'm very happy that I got to help a bit with id and nightdive getting it shipped.

As I mentioned on Mastodon I had a few bits of GtkRadiant support stashed away until the release. I'd like to get an updated release out, but I don't have much time right now so I've pushed the patches I have to github and zipped up the gamepack. If you have some GtkRadiant smarts you should be able to compile a working build.

The gamepack is a bit outdated, I recommend replacing the ericw-tools folder with latest. Also you may want to use TrenchBroom altogether to map instead of GtkR. TrenchBroom is easier to pickup and more active than GtkR so a better choice if you are new.

 I'm being told there's a Quake Mapping discord where you can get more info about mapping for the new release. HF!

Recent interview with Linux Gaming Central

An update on what I've been up to!

https://linuxgamingcentral.com/posts/interview-with-tim-besset-valve-contractor/

Cheap house monitoring with no vendor lock in

Radiant floor heating is awesome. But if you are going to be away for extended periods of time, you'll want to make sure the pipes don't freeze on you and incur costly damage.

When I googled off-the-shelf solutions, I kept finding somewhat expensive products which often came with proprietary apps and assorted vendor lock-ins.

Until I checked my trusty CloudFree website and saw these Aqara Temperature Sensors ($15!). After chatting with CloudFree support for a bit, I found that those sensors work over Zigbee, and don't need an Aqara hub .. any Zigbee hub will do, such as the Sonoff Zigbee 3.0 Dongle Plus, which comes flashed with the open source Z-Flash firmware from TI.

Plugging the dongle into a Raspberry Pi wouldn't be the end of the world, but I wanted to avoid that if possible, and use my EA8300 Linksys router flashed with OpenWrt instead.

Typically, one would run zigbee2mqtt or Home Assistant on the device to which the dongle is plugged in to get the readings. I didn't want to run a node app on the router, and after stumbling on a very useful stackexchange post, this is the setup I landed on:

• ser2net on the router: /usr/sbin/ser2net -d -C "8888:raw:0:/dev/ttyUSB0:115200 remctl"
• Proxy from a VPS system: /usr/bin/socat pty,raw,link=/dev/ttyVUSB0,waitslave tcp:<home dyndns>:8888,forever,reuseaddr,keepalive
  
• Run zigbee2mqtt on the VPS (+ Mosquitto MQTT broker)

From there it gets piped into my Home Assistant instance, with adequate monitoring and alerts.

The Zigbee Home Automation integration in Home Assistant can be used to add the dongle directly in HA rather than use zigbee2mqtt. This is nicely documented by Sonoff. I couldn't make this work with the proxy configuration though.

 

Sourcetrail with Unreal Engine 4 - a quick primer

TL;DR - It sure would be nice if someone contributed support for JSON Compilation Database to Unreal Build Tool. Until then, here's what you can do:

• Install a patched version of Bear from https://github.com/TTimo/Bear/tree/ttimo (note: ttimo branch)
• From a clean UE4 tree, capture a compilation database:
  bear -o ./UE4Game.json ./Engine/Build/BatchFiles/Linux/Build.sh UE4Game Linux Development -disableunity -nopch
• Create a Sourcetrail project, C/C++ from Compilation Database. Add Engine/ to the Header Files & Directories to Index, no other changes needed (my project).
• Index, wait for a while .. and voila!

Software:

• Unreal Engine 4.20 source from github
• Sourcetrail 2018.3.55

Stats:

• Indexing took 1h40 on an AMD threadripper 1950X, configured with 16 indexer threads, peaked to around 50GB RAM usage.
• Database is 7.6GB on disk, 12k files indexed.

Discussion:

The current release of bear has two issues that will prevent it from capturing an UE4 compile, hence my fork: https://github.com/rizsotto/Bear/issues/194 and https://github.com/rizsotto/Bear/pull/224. Other than that the tool is pretty much perfect.

The database is large and queries against it are noticeably slower than with smaller projects. The indexing used to be in the 6+ hour range prior to Sourcetrail 2018.3.55, and new versions of ST allow you to index in the background, so there is great progress there.

I have not tried, but the same steps would likely work with UE4Editor, and result in an even larger database.

I don't use Sourcetrail with UE4 on Windows. Bear will not be applicable, I don't know if the Sourcetrail Visual Studio plugin will capture anything.

GtkRadiant 1.6.5 builds, new website and gratipay

We have had a steady stream of fixes coming in from contributors over the last few months, to fix a number of memory errors and bugs across the map compiler (q3map2) and the bot routes compiler (bspc). Several of those fixes also improve the stability and correctness of the tools on 64 bit Linux.

To get an overview of the changes in 1.6.5, please consult the Milestone issues list on github. Latest changes on GtkRadiant and BSPC for more details.

To celebrate, we are rolling out an updated website that Obsidian has put together on the gtkradiant.com domain, where you can download the latest build.

Last but not least, we are now accepting donations via gratipay. Please show your love and support to the project!

Some thoughts about Yubikeys

Hello there!

I have several things in the pipeline that will benefit from some public exposure, so I will try to post here more regularly.

We'll start easy and talk about Yubico hardware keys today. I bought a Yubikey Edge last year, thinking that it was finally the end of having to manage endless lists of passwords.

That was very naive of course. The main benefit I got out of the purchase is that I now have two factor auth for all my critical apps and services (Gmail, Amazon AWS, Dropbox, GitHub and Xero). This is all done through the Google Authenticator app on iOS, and does not make use the Yubikey at all.

I found Yubikey's documentation to be a bit lacking about where and how it could be used and how to set it up. I assumed it would replace the two factor auth apps but that wasn't the case.

I also found that U2F only works with Google Chrome on Windows. I'm running Chrome on Linux and web-based U2F doesn't work there.

Adding the key to your GitHub account will have interesting effects. All your authenticated https access will be disabled, and you will have to rely on ssh keys only. I understand that private ssh keys are more secure than a login/password over https, but I feel that disabling https shouldn't be an implicit requirement.

Essentially, the Yubikey on github secures your website login only. In order to lock things some more you will have to do some extra work to secure your private keys.

Things will continue to improve with broader adoption. Better, easier OATH–TOTP would have the most impact for me. In the meantime the cheap and easy improvement to your online security is to simply get the Google Authenticator app and set it up on a handful of essential services.