@jnoxon's thoughts

Using custom DNS-over-HTTPS with CoreDNS and Google Chrome

jeff@noxon.cc (Jeff Noxon) — Thu, 17 Sep 2020 00:00:00 +0000

I’ve been using CoreDNS on my home network infrastructure for a few years, and it’s worked really well. One of the things I use it for is translating local DNS queries into DNS-over-TLS queries, which keeps my ISP from inspecting and manipulating DNS lookups. That use case is well documented and works well.

Today I found myself wanting to override the DNS provider used by just one browser on my computer. I’m often connected to a corporate VPN that hijacks DNS and makes it difficult to connect to sites on my home network. Having a dedicated browser for these sites means I don’t need to disconnect from the VPN or mess with /etc/hosts to reach them.

I don’t think any browsers allow overriding normal DNS resolvers anymore, but both Chrome and Firefox support DNS-over-HTTPS with custom resolvers.

Here’s how to make that work with Chrome and CoreDNS:

First, you’ll need a valid TLS certificate for your resolver. I used the awesome LEGO client for Let’s Encrypt to generate one, along with some simple automation to keep it fresh. It might be possible to create your own CA and add it to your computer’s trust store; I haven’t tried that. The CN for the cert must be a valid DNS name.

The CoreDNS config should look something like this:

https://.:853 {
tls /etc/coredns/ssl/coredns.crt /etc/coredns/ssl/coredns.key {
client_auth nocert
}
forward . 127.0.0.1:53
# log
}

This tells CoreDNS to listen on port 853 using the DNS-over-HTTPS protocol, and to resolve anything using the . wildcard. It then forwards those queries to itself using standard DNS. That’s it! I suggest enabling logging at first but turn it off once everything works.

Note: CoreDNS supports RFC 8484 queries only, not Google’s JSON query language.

You can test that it works with cURL:

curl -i 'https://you-resolver-fqdn:853/dns-query?dns=AAABAAABAAAAAAAAA3d3dwdleGFtcGxlA2NvbQAAAQAB'
HTTP/1.1 200 OK
Cache-Control: max-age=3455.000000
Content-Length: 64
Content-Type: application/dns-message
Date: Thu, 17 Sep 2020 16:56:23 GMT
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.

The 200 OK response is what you’re looking for.

If you get a certificate validation problem, don’t ignore it with -k. Fix that before you continue.

The query above will resolve example.com. (The base64 is an encoded DNS query.) If you have logging enabled in CoreDNS, you should see that query in your logs.

Now you can configure it in Chrome:

In Chrome 85, the setting lives in Preferences under Privacy and security, Security, Advanced, Use secure DNS. Turn it on, select the custom provider, and enter your resolver like so:

https://your-resolver-fqdn:853/dns-query

It’s the last bit that took me forever to figure out. RFC 8484 specifies the /dns-query path, but Chrome doesn’t append it implicitly.

If Chrome is satisfied, that’s it. If it isn’t, you’ll see Please verify that this is a valid provider or try again later. If you see that after following the advice here, I’m not sure what to suggest, but it’s most likely a problem with the cert, or Chrome can’t resolve the resolver itself (try using /etc/hosts), or CoreDNS isn’t resolving correctly (check the logs, try cURL).

Since DoH uses standard HTTPS, you can do further troubleshooting with mitmproxy in reverse proxy mode, using the same cert/key as your resolver.

I tried this with Firefox and I never got it working. Firefox 80.0 accepts a custom DoH config and then silently refuses to use it. I didn’t put Firefox under the mitmproxy microscope, however, since getting it to work with Chrome was enough for me.

Expanding btrfs filesystems

jeff@noxon.cc (Jeff Noxon) — Sun, 17 Mar 2019 19:29:06 -0500

A few months ago, I migrated a bunch of media from a ZFS mirror on my Mac to my home server VM. I used mismatched drives that I happened to have on hand, and while it works fine, I might as well replace the disks with the ones from the ZFS pool that I’m not using any more.

While ZFS works reasonably well, it carries a ton of technical and political debt, particularly on Linux. Since btrfs has matured significantly in the last few years, I’m using it on my Linux servers now. I like btrfs a lot. It has good performance, and offers advanced features like data and metadata checksumming, snapshots, RAID, and compression.

What follows is mostly for my own notes, but maybe it’ll help someone.

Note that /var/lib/plex/data is just one of the subvolumes that belongs to the btrfs filesystem I’m working on.

First, let’s identify the existing devices in the filesystem:

root@proteus:~# btrfs filesystem show /var/lib/plex/data
Label: 'Media' uuid: 290c5b77-9bff-47d5-9f82-3d7576f134f5
Total devices 2 FS bytes used 1.62TiB
devid 1 size 5.46TiB used 1.62TiB path /dev/sdd2
devid 2 size 3.64TiB used 1.62TiB path /dev/sdc2

/dev/sdd2, which btrfs regards as devid 2, is the disk I’ll replace first. I’ve already partitioned a replacement disk, /dev/sde2

Let’s start the replacement:

root@proteus:~# btrfs replace start -B 2 /dev/sde2 /var/lib/plex/data

The -B option makes the command block until it’s completed, so it’s completely optional. 2 is the devid we’re replacing. The filesystem will remain online and read/write during the migration. Btrfs doesn’t appear to prioritize regular filesystem I/O during the migration, so expect performance to suffer until the migration is done.

It’s also possible to monitor the status in a blocking way:

root@proteus:~# btrfs replace status /var/lib/plex/data
7.4% done, 0 write errs, 0 uncorr. read errs

With the migration in progress, you can also see that the mirror has three devices attached:

root@proteus:~# btrfs filesystem show /var/lib/plex/data/
Label: 'Media' uuid: 290c5b77-9bff-47d5-9f82-3d7576f134f5
Total devices 3 FS bytes used 1.63TiB
devid 0 size 3.64TiB used 1.62TiB path /dev/sde2
devid 1 size 5.46TiB used 1.64TiB path /dev/sdd2
devid 2 size 3.64TiB used 1.64TiB path /dev/sdc2

The new disk, devid 0, is only reporting 3.64TiB, even though it’s an 8TB disk. We’ll fix that once the migration is done.

Hours later, the process is done:

root@proteus:~# btrfs filesystem show /var/lib/plex/data/
Label: 'Media' uuid: 290c5b77-9bff-47d5-9f82-3d7576f134f5
Total devices 2 FS bytes used 1.62TiB
devid 1 size 5.46TiB used 1.63TiB path /dev/sdd2
devid 2 size 3.64TiB used 1.63TiB path /dev/sde2

Note how devid 2 is the new disk on /dev/sde2, but it still shows the wrong capacity. It’s unclear why resizing is not automatic, but it can be done manually:

root@proteus:~# btrfs filesystem resize 2:max /var/lib/plex/data
Resize '/var/lib/plex/data' of '2:max'

The resize operation completes almost instantly, and now we have:

root@proteus:~# btrfs filesystem show /var/lib/plex/data/
Label: 'Media' uuid: 290c5b77-9bff-47d5-9f82-3d7576f134f5
Total devices 2 FS bytes used 1.62TiB
devid 1 size 5.46TiB used 1.63TiB path /dev/sdd2
devid 2 size 7.28TiB used 1.63TiB path /dev/sde2

Finally, we can see that full redundancy is maintained:

root@proteus:~# btrfs device usage /var/lib/plex/data
/dev/sdd2, ID: 1
Device size: 5.46TiB
Device slack: 0.00B
Data,RAID1: 1.63TiB
Metadata,RAID1: 3.00GiB
System,RAID1: 64.00MiB
Unallocated: 3.83TiB
/dev/sde2, ID: 2
Device size: 7.28TiB
Device slack: 3.50KiB
Data,RAID1: 1.63TiB
Metadata,RAID1: 3.00GiB
System,RAID1: 64.00MiB
Unallocated: 5.65TiB

There are some relevant messages in the kernel log:

[1987714.818696] BTRFS info (device sdd2): dev_replace from /dev/sdc2 (devid 2) to /dev/sde2 started
[1998011.747252] BTRFS info (device sdd2): dev_replace from /dev/sdc2 (devid 2) to /dev/sde2 finished
[1999100.104284] BTRFS info (device sdd2): resizing devid 2
[1999100.104289] BTRFS info (device sdd2): new size for /dev/sde2 is 8001352437760

Now /dev/sdc can be removed from the system.

root@proteus:~# lsblk -S /dev/sdc
NAME HCTL TYPE VENDOR MODEL REV TRAN
sdc 4:0:0:0 disk Mercury Elite Pro Quad C 0 usb

The above command shows that the disk occupies bay C in the disk enclosure.

After removing the disk and replacing it with another 8TB disk, the entire process was repeated to replace /dev/sdd2.

Using Commento with Hugo

jeff@noxon.cc (Jeff Noxon) — Sun, 17 Mar 2019 19:29:06 -0500

Although this site has never received much comment volume, there have been occasions where comments have been quite valuable. One popular solution for static sites is to use Disqus, which is a fully hosted comment system that many Hugo themes support right out of the box. Disqus is free, but you pay for it with lost privacy, third-party trackers, and bloat.

I’ve decided instead to use Commento. There’s a cheap hosted version of Commento, but why not try out the fully self-hosted version?

What follows are some notes on how I got Commento working and integrated with Hugo.

Setting up a dedicated database

Even though Commento is small, it requires a PostgreSQL database. That’s a bit of overkill for a small blog like this one, but considering a single Commento server could potentially handle dozens if not thousands of domains, it’s not an unreasonable choice.

I decided to use Docker to run a PostgreSQL database dedicated to this purpose. First, I installed postgresql-client and postgresql-common on my host, and took advantage of that creating a postgres user and group for me.

The initial startup looked like this:

docker run -e POSTGRES_PASSWORD=censored -it --rm -v commento-postgres:/var/lib/postgresql/data postgres
.. wait for the container to settle down ..
^C

This created a docker volume called commento-postgres to hold the database and initialized a postgres user in the database engine.

I then changed the permissions on the data, where 113:120 is the user:group of my host’s postgres user:

docker run -it --rm -v commento-postgres:/var/lib/postgresql/data bash chown -R 113:120 /var/lib/postgresql/data

Commento wants TLS

I’ll save you a bit of trial-and-error here. It turns out Commento expects to connect to the database using TLS. It doesn’t care if the cert is self-signed or not, so I made one and stuffed it into my commento-postgres volume:

openssl req -x509 -newkey rsa:1024 -nodes -keyout key.pem -out cert.pem -days 3650

(Use docker volume inspect commento-postgres if you’re not sure where the volume is mounted.)

While I was there, I edited postgresql.conf to enable TLS:

ssl = on
ssl_cert_file = 'cert.pem'
ssl_key_file = 'key.pem'

Note that this database isn’t accessible over the Internet or even the LAN, thanks to iptables rules that aren’t shown here.

Starting the database

docker run -d \
--restart=always \
--name=commento-postgres \
--user 113:120 \
-p5432:5432/tcp \
-v commento-postgres:/var/lib/postgresql/data \
postgres

Testing the connection and creating a commento database

# psql -h localhost -p 5432 -U postgres
Password for user postgres:
psql (10.6 (Ubuntu 10.6-0ubuntu0.18.04.1), server 11.2 (Debian 11.2-1.pgdg90+1))
WARNING: psql major version 10, server major version 11.
Some psql features might not work.
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.
postgres=# create database commento;
CREATE DATABASE
^D

You can see above that the connection used TLS, which is good. I also created the database.

Starting commento

First, I created a system user and group for commento.

I went through a ton of trial-and-error here as well, and the details aren’t that interesting, so here is what I ended up with:

docker run -d \
-p 8080:8080 \
-e COMMENTO_ORIGIN=https://commento.noxon.cc \
-e COMMENTO_POSTGRES=postgres://postgres:censored@docker_host_ip:5432/commento \
-e COMMENTO_SMTP_USERNAME=nobody \
-e COMMENTO_SMTP_PASSWORD=nopass \
-e COMMENTO_SMTP_HOST=my_mail_host.example.com \
-e COMMENTO_SMTP_PORT=25 \
-e COMMENTO_FORBID_NEW_OWNERS=true \
-e COMMENTO_SMTP_FROM_ADDRESS=no-reply@noxon.cc \
-e COMMENTO_GOOGLE_KEY=censored \
-e COMMENTO_GOOGLE_SECRET=censored \
-e COMMENTO_TWITTER_KEY=censored \
-e COMMENTO_TWITTER_SECRET=censored \
--dns=my_dns_server_ip \
--user=122:122 \
--name commento \
--restart=always \
registry.gitlab.com/commento/commento

Some important notes:

I published a DNS record for COMMENTO_ORIGIN
I configured an NGINX reverse proxy to add TLS and expose Commento on port 443.
The first time running this, don’t set COMMENTO_FORBID_NEW_OWNERS. Register a user through the web portal, then restart it with new owners disabled.
To test SMTP, use the “forgot password” workflow, and docker logs commento to check for errors (and, obviously, look for the email.)
All of the SMTP parameters are necessary. If your SMTP relay doesn’t use SASL authentication, make sure it doesn’t offer it, and just use a bogus SMTP username and password. If your relay is Postfix, take a look at smtpd_sasl_exceptions_networks.
I highly recommend using a local SMTP relay to store and forward emails.
You can get OAuth keys from Google and Twitter pretty easily. The callbacks are /api/oauth/google/callback and /api/oauth/twitter/callback. Github and Gitlab are also supported using the same pattern.

Putting the pieces together

Create layouts/partials/commento.html:

<p/>
<div id="commento"></div>
<script defer
src="https://commento.noxon.cc/js/commento.js"
data-div="#commento">
</script>

Copy themes/<your-theme>/layouts/_default/single.html to layouts/_default/single.html and add {{ partial "commento.html" . }} right around where the Disqus comment code is.

What’s left to figure out?

How to automate database dumps for backups
Investigate ways to import old WordPress comments

I’ll come back and update this post as I make progress.

Enjoy some comment spam.

That’s roughly it. I’d like to thank oct8l for his helpful blog post on the same subject!

Feedback is welcome. Add a comment below!

Migrated From WordPress to Hugo

jeff@noxon.cc (Jeff Noxon) — Thu, 14 Mar 2019 13:17:12 -0500

I’ve migrated this blog from WordPress to Hugo. It’s something I’ve wanted to do for a long time. WordPress is a bit of a security nightmare, and although I’ve somehow never had my server owned, I did notice a couple of successful exfiltration attacks against WordPress while reviewing logs. Thankfully, the attacks were harmless.

If you subscribe to my RSS feed, you’ll notice some repeated content. It wasn’t worth the effort to match the content GUIDs.

I’m still not sure about this theme, but it will do for now.

As part of this migration, I’ve finally made the site HTTPS-only. HTTPS has been enabled for years with HSTS, but now HTTP is permanently redirected to HTTPS.

There are still a bunch of tweaks I’d like to make, particularly with responsive image sizes and re-enabling the CDN.

Last but not least, comments are gone, at least temporarily. I’d like to add Commento or another open-source comment engine at some point, but for now, I’m exhaused.

If you encounter any issues or have any feedback, send me an email!