Know It Not

Anthropic ships fast: from feature request to merged in days

2026-05-22T00:00:00+00:00

This is a story about what it looks like when the people who build a product are directly reachable, and why it makes me believe Anthropic isn’t just going to be around in twelve months, but is the kind of company that compounds over time.

The problem: tmux-resurrect and Claude sessions

I use tmux-resurrect to save and restore my terminal sessions across reboots. It works beautifully for most things, but Claude Code sessions were a persistent headache.

The dream is simple: close the laptop, open it the next morning, run a restore, and have your Claude windows reopen with claude --resume <session-id> exactly where they left off. The reality was that I had accumulated about 140 lines of increasingly baroque shell heuristics to make this work:

Encode the working directory into the window name so I had something to decode on restore
Correlate JSONL files by birth time to guess which session belonged to which process
Exclude any UUID-looking strings that looked “claimed” by another process
Cap the time window to avoid picking up stale sessions

It was fragile. Four scenarios I cared about still failed regularly, including the case where two live Claude processes legitimately shared one session (which is valid: --resume lets you attach multiple terminals to the same session).

The accidental discovery

I was about to file a feature request for “some way to introspect running Claude sessions” when I decided to poke around the filesystem first. I found ~/.claude/sessions/<pid>.json:

{
  "pid": 70927,
  "sessionId": "956af9b5-808e-47a1-8278-bea9dc679d5e",
  "cwd": "/home/coder",
  "startedAt": 1778933953596,
  "procStart": "8715888",
  "version": "2.1.143",
  "peerProtocol": 1,
  "kind": "interactive",
  "entrypoint": "cli",
  "status": "idle",
  "updatedAt": 1778933953487,
  "bridgeSessionId": "session_01JAoSBnhk14tdaFsa49cxQ1"
}

It’s written at startup; I verified it exists even for a bare-launched Claude that hasn’t received its first prompt. Every live Claude on my machine has one. And sessionId matches the --resume argument exactly.

My tmux-resurrect strategy went from ~140 lines down to ~70:

read /proc/<pid>/stat tpgid
→ read ~/.claude/sessions/<pid>.json
→ splice --resume <sessionId> into the restore command

All four failing scenarios resolved. The shared-session case worked correctly because the file is per-pid, not per-session.

What I actually asked

Even though the discovery solved the immediate problem, I had real questions before I could depend on this in tooling:

Documentation: is this a supported surface, or did I just stumble onto an internal file?
Schema stability: are pid and sessionId committed field names, or could they be renamed?
Cleanup semantics: if a Claude process segfaults, when does its file disappear?

I also mentioned a couple of nice-to-haves: a claude session list --running --json subcommand that would save every consumer from reimplementing the directory-scan + pid-liveness dance, and surfacing the --name flag in the pidfile for status bars.

The response, and what happened next

The reply from the Claude Code team came with exactly the detail I needed:

Those files power Claude’s internal agent and cross-session messaging. Not a documented API today, but pid and sessionId have been present since the files were introduced in March and every internal consumer depends on them; stable in practice.
Cleanup is both: on clean exit the file is removed; on crash it survives until the next Claude launch, which sweeps any file whose pid is dead. (procStart is the pid-reuse guard, which is why it’s in the schema; pid recycling is a real concern on long-running systems.)
They’d raise the documentation + additive-only ask with the team.

Less than 24 hours later, a follow-up: “we just merged claude agents --json as the supported scripting surface for this, should ship on Tuesday.”

The --json output gives you pid and sessionId with an additive-only commitment and already filters to live processes, so consumers don’t need to think about cleanup at all.

I posted on a Saturday morning. By Tuesday, three days later, a new subcommand was merged and on its way to shipping. That’s a turnaround I’d expect from a three-person startup, not a company building frontier AI models at scale.

Why this matters beyond the feature

The speed is impressive. But what actually signals something deeper is how it happened.

After I shared this post, the person I talked to pointed out something I hadn’t fully appreciated: there is no support team on the other side. The reason they could move so fast is that job functions are collapsing at Anthropic. The person who answered my questions was the same person who could push to main. They just needed a quick review from the Claude Code team and it was done.

That distinction matters. It’s not “support escalated to engineering.” It’s that the people who build the product are directly reachable, understand it deeply, and have the latitude to act on what they hear. When I described the problem, the person I was talking to already knew exactly what the files were for, how cleanup worked, and what the right long-term surface should be, because they built it.

I’ve worked with a lot of developer tools over the years. The ones that compound, the ones that are still the dominant choice five years after they launched, almost always have this quality early: they treat the gap between “user found a workaround” and “user has a real API” as a bug, not a feature request backlog item.

Anthropic does that. And that’s why I think they’re not just going to succeed this year. They’re building the kind of institutional behavior that wins for a long time.

If you want to see the tmux-resurrect strategy, I’ll write it up in a follow-on post. The happy path is now a single claude agents --json call; the old heuristic pile is still there as a fallback for older Claude versions, but the common case is clean.

How a missing TERM entry broke my UDM-Pro shell for years

2026-04-26T00:00:00+00:00

I have been on Linux for 16+ years, and yesterday I learned something about TERM that I really should have learned a long time ago. This is the story of how an unrelated AdGuard Home deploy on a couple of new Raspberry Pi 5’s solved a problem on my UDM-Pro that had been quietly annoying me for years.

The AdGuard Home symptom

I was bringing up two new Pi 5’s as DNS hosts and went to check the service:

root@dns03:~# systemctl status AdGuardHome.service
WARNING: terminal is not fully functional
Press RETURN to continue
 ESC[A● AdGuardHome.service - AdGuard Home: Network-level blocker
     Loaded: loaded (/etc/systemd/system/AdGuardHome.service; enabled; preset: enabled)
     Active: active (running) since Sat 2026-04-25 13:32:20 MDT; 19s ago
   Main PID: 1477 (AdGuardHome)

The WARNING: terminal is not fully functional line stopped me. In 16+ years on Linux I had somehow never seen it before, so I asked Claude what it meant. The answer was the kind of answer where you immediately feel a little dumb:

That warning isn’t from AdGuard Home, it’s from less, the pager that systemctl status pipes its output through. less is checking your TERM environment variable to decide what terminal capabilities (cursor movement, color, etc.) it can use, and it’s getting back something it considers unusable. It falls back to a degraded mode and prints that warning to let you know fancy navigation is off. The literal ESC[A you see in the output is the same thing. less couldn’t translate the up-arrow key into anything useful and just printed the raw escape bytes.

I’ll be honest, I hadn’t even noticed the ESC[A in the output until Claude pointed it out. My up-arrow had eaten the keypress in the moment, but the broken echo was sitting right there in plain text and I had skimmed past it. That was its own little lesson in what I’m not noticing in terminal output.

I checked echo $TERM on the Pi:

TERM=xterm-ghostty

There it was. I run Ghostty as my terminal emulator, and Ghostty advertises itself with TERM=xterm-ghostty. SSH faithfully forwards that value to the remote host. The remote host then asks the system “do you have a terminfo entry named xterm-ghostty?” and Raspberry Pi OS Bookworm, like most distros, does not. So less, vim, htop, even bash’s readline all fall back to the most pessimistic assumption about what the terminal can do. Hence the warning. Hence the raw escape bytes.

The fix

Ghostty’s docs cover this. The canonical fix is to copy your local terminfo entry to the remote host:

infocmp -x xterm-ghostty | ssh root@dns03 -- tic -x -

infocmp -x dumps the compiled terminfo source for the named entry, the pipe sends it to the remote, and tic -x compiles and installs it into the remote’s terminfo database. Run it once per host.

If you can’t SSH directly as root (I can’t), it’s a two-step:

# step 1: drop the source on the remote
infocmp -x xterm-ghostty | ssh user@dns03 'cat > /tmp/xterm-ghostty.ti'

# step 2: install it system-wide (needs a real TTY for sudo)
ssh -t user@dns03 'sudo tic -x /tmp/xterm-ghostty.ti && rm /tmp/xterm-ghostty.ti'

sudo tic -x writes to /etc/terminfo/x/xterm-ghostty, which means every user on the host benefits, including whatever you do under sudo later.

Modern tic will print a warning that looks alarming but isn’t:

"/tmp/xterm-ghostty.ti", line 2, col 31, terminal 'xterm-ghostty':
older tic versions may treat the description field as an alias

That just means: the terminfo source uses a long description in the second pipe-delimited field (xterm-ghostty|Ghostty|...), which an ancient tic on some other host might misread. Your install on Bookworm is fine.

After that, systemctl status is clean, less is happy, vim has color, and the up-arrow does what an up-arrow is supposed to do.

The plot twist: my UDM-Pro

A day later I SSH’d into my UDM-Pro for an unrelated reason. As soon as I started typing, the shell felt off. Same old “off.” For years I have had a bug on the UDM-Pro where I cannot use the left-arrow key to move the cursor backwards over text I had already typed. If I made a typo near the beginning of a long command, my only options were to retype it or backspace the whole thing. Annoying, but I had a workaround: if I started a tmux session locally first and then SSH’d from inside tmux, everything worked. I never knew why that worked. I just did it.

This time, on muscle memory, I opened a tmux window to SSH from. And then it hit me.

tmux re-exports TERM as tmux-256color (or screen-256color depending on config), and every Linux distro on Earth ships terminfo for those. So when I went through tmux, the UDM-Pro was getting a TERM it recognized. When I SSH’d directly from Ghostty, it was getting xterm-ghostty, which the UDM had never heard of, so bash’s readline fell back to the most degraded line-editor mode it has, the one where the left-arrow doesn’t reliably walk back over already-printed text. Same root cause as the Pi. I had been working around it for years without ever asking why the workaround worked.

I checked, and tic exists on UniFi OS (it’s Debian Bullseye underneath), so the fix was the same one-liner:

infocmp -x xterm-ghostty | ssh root@10.0.15.1 -- tic -x -

Reconnected. Arrow keys work. less is clean. vim has color and syntax highlighting. A multi-year papercut, gone in one command.

One footnote for UniFi people: the UDM-Pro’s root filesystem is an overlay that may get reset by firmware upgrades. /etc/terminfo has persisted across reboots for me, but I’d expect to re-run this after a UniFi OS upgrade.

What I actually learned

The thing I am embarrassed about is not that I didn’t know the fix. The thing I am embarrassed about is that I never really understood, at a gut level, how much of “does my shell feel right” is downstream of one environment variable and a lookup in a database I never thought about.

TERM is not a label. It is a key into terminfo, and terminfo is the contract that tells every program (bash’s readline, less, vim, git, htop, systemctl’s pager, every TUI you have ever used) what bytes to send for “move cursor left,” what bytes mean “the user pressed up-arrow,” whether color is safe, whether undercurl is supported, whether the terminal has a status line. If the remote host can’t resolve TERM to an entry, all of those programs fall back to the most paranoid assumptions they have, and your shell starts behaving in subtly broken ways that you might just attribute to “this host is weird.” For sixteen years I had been attributing it to “this host is weird.”

It was never the host. It was always TERM.

Discovering Azure DevOps Pipeline Performance Issues

2024-07-10T00:00:00+00:00

Recently I ran into some very unusual issues with Azure DevOps pipelines. The pipelines were running very slowly, and the logs were not providing any useful information. I decided to dig deeper and see if I could find the root cause of the issue.

I opened a case with Microsoft support, and it become very challenging to even communicate the issue. The support engineer was not able to understand the issue, and I was not able to provide enough information to help them understand the issue.

I ended up authoring a program in Python to help me understand the issue. The program would run and hit the Azure DevOps API to get the status of the pipeline. I would then log the status of the pipeline to a file. I ran this program for a few days and was able to see that the pipeline was running very slowly.

import os
from datetime import datetime

import requests
import yaml
from requests.auth import HTTPBasicAuth

ADO_ORGANIZATION: str = "<fill-me-in>"
ADO_PROJECT: str = "<fill-me-in>"
ADO_BUILD_DEFINITION: int = "<fill-me-in>"


class Node:
    def __init__(self, id, type, name, startTime, finishTime, parentId=None, state=None, result=None):
        self.id = id
        self.type = type
        self.name = name
        self.startTime = self.parse_time(startTime)
        self.finishTime = self.parse_time(finishTime)
        self.parentId = parentId
        self.state = state
        self.result = result
        self.children = []

    def parse_time(self, time_str):
        if time_str is None:
            return None
        # Ensure microseconds have exactly 6 digits
        adjusted_time_str = self.ensure_six_digit_microseconds(time_str)
        return datetime.strptime(adjusted_time_str, "%Y-%m-%dT%H:%M:%S.%fZ")

    def ensure_six_digit_microseconds(self, time_str):
        if "." in time_str:
            parts = time_str.split(".")
            microseconds = parts[1].rstrip("Z")
            microseconds = microseconds[:6].ljust(6, "0")  # Ensure exactly 6 digits
            return f"{parts[0]}.{microseconds}Z"
        elif "Z" in time_str:
            return time_str.replace("Z", ".000000Z")
        else:
            return f"{time_str}.000000Z"

    def duration(self):
        if self.startTime and self.finishTime:
            return (self.finishTime - self.startTime).total_seconds()
        return 0

    def is_successful(self):
        return self.state == "completed" and self.result == "succeeded"


def build_tree(data):
    nodes = {
        item["id"]: Node(
            item["id"],
            item["type"],
            item["name"],
            item["startTime"],
            item["finishTime"],
            item.get("parentId"),
            item.get("state"),
            item.get("result"),
        )
        for item in data
    }
    roots = []

    for node in nodes.values():
        if not node.is_successful():
            continue
        if node.parentId:
            parent = nodes[node.parentId]
            parent.children.append(node)
        else:
            roots.append(node)

    return roots


def check_discrepancies(node, discrepancies, stage_name=None):
    if not node.startTime or not node.finishTime:
        return 0

    if not node.children:
        return node.duration()

    total_child_duration = sum(
        check_discrepancies(child, discrepancies, node.name if node.type == "Stage" else stage_name) for child in node.children
    )
    node_duration = node.duration()

    if node_duration > 0 and total_child_duration > 0 and total_child_duration < node_duration * 0.2:
        discrepancies.append((stage_name, node.name))

    return node_duration


def analyze_timeline(timeline_data):
    roots = build_tree(timeline_data)
    discrepancies = []

    for root in roots:
        check_discrepancies(root, discrepancies)

    return discrepancies


def summarize_discrepancies(discrepancies, build):
    build_summary = {
        "buildId": build["id"],
        "queueTime": build["queueTime"],
        "startTime": build["startTime"],
        "finishTime": build["finishTime"],
        "discrepanciesCount": len(discrepancies),
        "discrepancies": [{"stage": stage, "job": job} for stage, job in discrepancies],
        "results": f"https://dev.azure.com/{ADO_ORGANIZATION}/{ADO_PROJECT}/_build/results?buildId={build['id']}&view=results",
    }
    return build_summary


def main():
    ado_token = os.getenv("ADO_ACCESS_TOKEN")
    if not ado_token:
        print("ADO_ACCESS_TOKEN environment variable not set.")
        return

    session = requests.Session()
    session.auth = HTTPBasicAuth("", ado_token)

    builds_url = f"https://dev.azure.com/{ADO_ORGANIZATION}/{ADO_PROJECT}/_apis/build/builds?definitions={ADO_BUILD_DEFINITION}&statusFilter=completed&api-version=6.0"
    builds_response = session.get(builds_url)
    builds_data = builds_response.json()["value"]

    builds_with_discrepancies = []

    for build in builds_data:
        if build["status"] == "completed" and build["result"] == "succeeded":
            timeline_url = build["_links"]["timeline"]["href"]
            build_id = build["id"]

            timeline_response = session.get(timeline_url)
            timeline_data = timeline_response.json()["records"]

            discrepancies = analyze_timeline(timeline_data)
            if discrepancies:
                build_summary = summarize_discrepancies(discrepancies, build)
                builds_with_discrepancies.append(build_summary)

    output = {"builds_with_discrepancies": builds_with_discrepancies}
    print(yaml.dump(output, default_flow_style=False))


if __name__ == "__main__":
    main()

Viewing processes in a running container without ps

2024-01-03T00:00:00+00:00

First we find all of the process ids in

shopt -s extglob # assuming bash
(cd /proc && echo +([0-9]))

Great, we have all of the processes, but how can we view the running command?

cat /proc/48/cmdline | tr '\000' ' '

Add NeoVim to Running Pod/Container

2024-01-03T00:00:00+00:00

NeoVim

Recently, I had an issue where I was trying to investigate an issue on a running Kubernetes (K8s) pod. It was a pain because I didn’t have root to install additional tools, but I did have curl.

I came up with a quick way to add NeoVim with curl.

curl -LO https://github.com/neovim/neovim/releases/download/v0.9.5/nvim.appimage
chmod +x nvim.appimage
./nvim.appimage --appimage-extract
export PATH=$(pwd)/squashfs-root/usr/bin:$PATH
alias vi=nvim
alias vim=nvim

After that you can use NoeVim anywhere you’d like.

nvim

fd

Bonus points, this container also didn’t have find. I turned to fd instead.

curl -LO https://github.com/sharkdp/fd/releases/download/v9.0.0/fd-v9.0.0-x86_64-unknown-linux-gnu.tar.gz
mkdir fd
tar zxvf fd-v9.0.0-x86_64-unknown-linux-gnu.tar.gz --strip-components 1 -C fd
export PATH=$(pwd)/fd:$PATH

ripgrep

Bonus points, this container had grep, but grep is slow. I turned to ripgrep.

curl -LO https://github.com/BurntSushi/ripgrep/releases/download/14.0.3/ripgrep-14.0.3-x86_64-unknown-linux-musl.tar.gz
mkdir rg
tar zxvf ripgrep-14.0.3-x86_64-unknown-linux-musl.tar.gz --strip-components 1 -C rg
export PATH=$(pwd)/rg:$PATH

doggo

curl -LO https://github.com/mr-karan/doggo/releases/download/v0.5.7/doggo_0.5.7_linux_amd64.tar.gz
mkdir doggo
tar zxvf doggo_0.5.7_linux_amd64.tar.gz -C doggo
export PATH=$(pwd)/doggo:$PATH

If you need other runtimes for these programs, check out their project release pages:

Pi-Hole, Tailscale, and Unifi UDM-Pro

2023-12-27T00:00:00+00:00

Recently, I delved into the challenge of setting up my own DNS server, aiming for a solution that didn’t involve local hardware support. My motivation? Filtering ads and trackers, along with blocking certain Apple call-home services within my home network. Initially, I experimented with several Raspberry Pi units but soon realized managing hardware wasn’t my cup of tea. Consequently, I shifted focus to hosted solutions and opted for DigitalOcean. My choice was driven by curiosity, as I hadn’t previously used their Virtual Machines, and their pricing appeared competitive. In truth, any reputable provider in this sector could have worked.

However, I faced a key concern: I wanted to reduce who can see my home network’s DNS traffic. This added complexity to my task, which I’ll outline in this journey. Join me as I walk you through the steps and processes I employed, not out of an initial grand vision, but as incremental solutions to emerging challenges.

This is a high level overview of all of the components used in the final design.

(2) DigitalOcean Droplets
Pi-Hole
Tailscale
UDM-Pro
DHCP Option 119 for Tailnet name search domain
DHCP Option 121 for Tailnet static route
SierraSoftworks/tailscale-udm
StevenBlack/hosts
Gravity Sync

My initial setup comprised simply of two Pi-Hole servers operating on a pair of DigitalOcean Droplets, each in a different datacenter. I modified the DHCP options in my UDM-Pro to direct all DNS traffic to the public IP addresses of these DigitalOcean Droplets. To ensure minimal security, I implemented a basic firewall ruleset that restricted access to my home’s public IP address. An important detail to remember: for this setup to function correctly, you need to enable “Permit all origins.” This option can be found under Settings > DNS in the Pi-Hole interface.

After setting up the infrastructure, I dedicated time to fine-tuning the Pi-Hole configuration, a tool I was using for the first time. My focus was on understanding the ad lists and exploring available options. During this exploration, I discovered the popular StevenBlack/hosts GitHub repository and decided to use the alternates/porn/hosts from it. Additionally, I created a custom hosts file tailored to block specific Apple services that I preferred not to have communicating back to their servers.

While configuring my Pi-Hole servers, I realized the inefficiency of duplicating each setting on both servers. This redundancy prompted me to find a solution for syncing the configurations. The goal was to ensure that any changes made to the primary Pi-Hole would automatically replicate to the secondary one. My solution was Gravity Sync, which, while not perfect due to some limitations in syncing upstream servers (documented here), proved to be significantly beneficial.

Satisfied with the Minimum Viable Product (MVP) I had created, my attention shifted to concerns about unencrypted DNS traffic traversing the internet. Additionally, the lack of an SSL-secured admin interface for Pi-Hole was a downside, and I preferred not to get involved in SSL certificate management, despite the relative ease of using Let’s Encrypt and certbot. To address these issues, I turned to Tailscale for end-to-end encryption. If you’re unfamiliar with Tailscale, you can learn more about it here.

I began by installing Tailscale on the DigitalOcean Droplets and the devices within my home network. Subsequently, I updated the DHCP DNS settings to use the Tailscale IPv4 addresses, routing DNS traffic via Tailscale’s end-to-end encrypted connection. However, this method had a significant limitation: devices without Tailscale couldn’t access the DNS servers, resulting in DNS resolution failure. This was particularly problematic for devices on my network that couldn’t install Tailscale.

To overcome this, I explored integrating Tailscale with my UDM-Pro. My research led me to SierraSoftworks/tailscale-udm on GitHub. After sifting through various GitHub issues, I found a crucial discussion that detailed the process of setting up a UDM-Pro as a Tailscale subnet router. This setup eliminated the need for individual Tailscale installations on each device.

Despite the GitHub discussion suggesting the use of a persistent bash script, I felt a more efficient approach would be to integrate support for UDM-Pro routing tables directly into Tailscale. This led me to fork Tailscale and start working on adding this functionality in my jwb/add-support-for-udm-pro branch, rebased off of v1.56.1, which is the latest release as of this writing.

My next task was to ensure all devices on my network properly routed 100.64.0.0/10 (What are these 100.x.y.z addresses?) through the UDM-Pro. To accomplish this, I utilized DHCP Option 121. By employing a calculator, I generated the necessary hexadecimal array for Unifi’s configuration. With this implementation, I effectively enabled DNS to function over an end-to-end encrypted network path. OK, so it’s not really end-to-end, you got me. It’s encrypted from the edge of my private network to the the other end. So, edge-to-end encryption.

Before proceeding further, it’s worth mentioning that I disabled Tailscale’s key expiry for both my UDM-Pro and the two DigitalOcean Droplets. This was a deliberate choice to ensure connectivity between my home network and the DNS servers is not disrupted.

Focusing next on DNS resolution within my Tailnet, I aimed to move beyond just IP address-based connectivity between devices. Tailscale provides a handy feature by assigning a Tailnet name to each network, making access more user-friendly. My objective was to establish a conditional forwarder from Pi-Hole to Tailscale’s private DNS server at 100.100.100.100. Without a native Pi-Hole solution available, I opted for a manual configuration. This involved adding settings to /etc/dnsmasq.d/02-tailscale.conf on my setup, followed by restarting the pihole-FTL service (systemctl restart pihole-FTL.service). For those interested, you can find your own Tailnet name here.

# /etc/dnsmasq.d/02-tailscale.conf
server=/hot-mess.ts.net/100.100.100.100

With these configurations in place, my setup really started to come together. I now had a fully functional Pi-Hole DNS server. Traffic from my home network was being securely routed to my DNS Pi-Hole servers, and I could resolve Tailnet names. To further refine the system, I configured DHCP Option 119 on my UDM-Pro, enabling short hostnames to resolve correctly to my Tailnet name.

My next objective was to minimize what DigitalOcean could observe in terms of DNS traffic. At this point, all DNS queries from my home network were forwarded to Pi-Hole, then to an upstream DNS server (like 1.1.1.1). However, this meant that traffic from the Droplets to the upstream servers was unencrypted. In my quest for DNS over HTTPS, I discovered Cloudflare’s cloudflared. Following their Connect to 1.1.1.1 using DoH clients guide, I set up a running DNS proxy service. Subsequently, I configured Pi-Hole to route DNS queries to this proxy by entering 127.0.0.1#5553 in the Custom Upstream DNS Servers section under Settings > DNS. It’s important to note that the Cloudflare documentation omits the port in the systemd.unit file, which is crucial here since Pi-Hole binds to UDP/53 and TCP/53.

# /etc/systemd/system/cloudflared-proxy-dns.service
[Unit]
Description=DNS over HTTPS (DoH) proxy client
Wants=network-online.target nss-lookup.target
Before=nss-lookup.target

[Service]
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
DynamicUser=yes
ExecStart=/usr/local/bin/cloudflared proxy-dns --port 5553

[Install]
WantedBy=multi-user.target

In the end, I made a trade-off, opting to trust Cloudflare over managing a recursive DNS server myself. If you’re uncomfortable with this choice, there’s an alternative: you can set up a recursive DNS server using unbound. For guidance on this, refer to the unbound setup guide, which provides detailed instructions on configuring Pi-Hole with unbound as a recursive DNS server.

Another trade-off I made involves network access: any device connected to my home network has unrestricted access to my entire Tailnet. This compromise seemed reasonable, as I don’t typically have untrusted individuals connecting to my home network. If needed, securing this aspect would be relatively straightforward using Tailscale ACLs. The strategy would involve permitting only the UDM-Pro Tailscale client to access the Pi-Hole servers via UDP/53 and TCP/53. While I might consider implementing this security measure in the future, I currently appreciate the simplicity and convenience of full Tailnet access from my home network.

In conclusion, I’ve successfully set up a DNS server that fulfills the functionality I originally sought. Along the way, I achieved a modest improvement in privacy, though it’s important to acknowledge that the solution isn’t fully privacy-proof. Some remaining privacy concerns include:

Subject Name Indication (SNI) remains unencrypted in most scenarios.
Assuming Cloudflare adheres to their privacy policy, up to 25 hours of DNS queries could still be logged.
My ISP can still detect the destination IP addresses of my traffic.

Thank you for reading. If you have any feedback, comments, or need assistance, please feel free to leave a comment.

Tip of the day: Removing files in linux when the filename starts with a hyphen (-)

2015-07-13T00:00:00+00:00

I kept running into the following problem:

$ \rm -i *.zip
rm: invalid option -- 't'
Try 'rm ./-thevoice-amway-com.zip' to remove the file ‘-thevoice-amway-com.zip’.
Try 'rm --help' for more information.

I’ve been using the rm command for over 5 years and I had never run into this issue. At first glance I had no idea what was going on. I initially started looking to see if maybe rm was actually an alias or function… both type rm and declare -f rm revealed nothing. I decided to try looking at rm --help and boom, I found it.

To remove a file whose name starts with a '-', for example '-foo',
use one of these commands:
  rm -- -foo

  rm ./-foo

I was able to finally get things rolling by using the following:

\rm -i -- *.zip

P.S. I did mean to use \rm instead of rm. For those of you who don’t know, prefixing a command with a backslash temporarily disables the alias if one exists. This is extremely helpful if you have rm aliased to rm -f or something like that.

How to easily get your public / WAN IP address through a public API

2014-08-21T00:00:00+00:00

I’ve had this desire in the back of my mind to create a simple way for programmers / devops / sysadmins or whoever, to get their public IP through an API.

What ended up pushing me over the edge was a requirement at work to automatically add newly provisioned servers to a Google Cloud SQL instance, allowing access to one of our MySQL databases.

I knew there were tons of sites out there that could already provide you with your WAN IP, but they were all HTML based, bulky and slow. I needed a fast, reliable and lightweight way to get this information.

I started hunting for domain names that would fit what I was going for. The only thing available at the .com level and made any sense at all was myjsonip.com.

The goal initially was to create a site that would return my public IP in a JSON object. As I started working on it and moving forward, I decided to add additional formats. The site now support getting your IP address in JSON, YAML and XML. Furthermore, I decided to open source the project. You can find the source code on github. Enjoy! It’s small, simple, versatile and fast.

How to override Google Compute Engine's DHCP MTU Option

2014-04-30T00:00:00+00:00

Recently I ran into a few issues with a linux IPSec Tunnel (Strongswan) that was hosted at GCE. With the suggestion of one of Google’s engineers, I thought that it might have something to do with the MTU. Google’s compute engine network MTU is 1460.

“For best performance, set MTU to 1460. Google Compute Engine’s DHCP server will serve this parameter as the interface-mtu option, which most clients respect.”

source: https://developers.google.com/compute/docs/images

I decided to drop it to 1400 in order to give the Strongswan host some additional headroom on the GCE network side. It ended up being difficult to override the MTU. I attempted to set the MTU option in the /etc/sysconfig/network-scripts/ifcfg-eth0 script like so:

DEVICE="eth0"
BOOTPROTO="dhcp"
DHCP_HOSTNAME="localhost"
MTU="1400"
NM_CONTROLLED="yes"
ONBOOT="yes"
TYPE="Ethernet"

That wasn’t working though, so I needed to somehow override the dhcp options being offered. I had never messed with the linux dhcp client (dhclient) so into the man pages we go! I read through them, but wasn’t able to find anything that worked. I tried multiple things until I stumbled upon a random config on the internet which showed (by observing) that options with numbers as values weren’t surrounded in quotes. I removed the quotes from the option and BAM, it worked! Here is the final config that worked for me:

# /etc/dhcp/dhclent-eth0.conf
send vendor-class-identifier "anaconda-Linux 2.6.32-431.el6.x86_64 x86_64";
timeout 45;
supersede interface-mtu 1400;

I highly suggest reading through the dhclient.conf man page.

man dhclient.conf

Here is a snippet from the man page about the supersede statement.

The supersede statement

supersede [ option declaration ] ;

If for some option the client should always use a locally-configured value or values rather than whatever is supplied by the server, these values can be defined in the supersede statement.

Export a list of all mail enabled public folders with their email address and path

2014-03-05T00:00:00+00:00

First let me start off by saying, this is not a post to show off my powershell skills because let me tell you, they suck and I have no desire to improve them.

I ran into a slight problem at work this past month. We needed to provide a list of all of our Mail-Enabled Public Folder’s email address and location in order that we could migrate them to Google Groups/Users. As I searched the internet, I quickly found that this wasn’t an “easy” task in Exchange. I was in complete and utter shock that Exchange didn’t have some quick way of doing this. The only built in options I could find where to either get all Public Folders (Get-PublicFolder) with their locations or to get all Mail-Enabled Folders (Get-MailPublicFolder) with their email address and no location. This of course is problematic when you need all Public Folder’s email address + location.

I ended up combining a few powershell techniques I found across the internet to get the desired result. This little and simple powershell script will save a CSV with all of the Mail-Enabled Public Folders email address and location.

## export-mail-enabled-public-folders.ps1 ##
############################################

$resultsarray = @()

$mailpub = Get-MailPublicFolder -ResultSize unlimited
foreach ($folder in $mailpub) {
  $email      = $folder.primarysmtpaddress.local + "@" + $folder.primarysmtpaddress.domain
  $pubfolder  = Get-PublicFolder -Identity $folder.identity
  $folderpath = $pubfolder.parentpath + "\" + $pubfolder.name

  # Create a new object for the purpose of exporting as a CSV
  $pubObject = new-object PSObject
  $pubObject | add-member -membertype NoteProperty -name "Email" -Value $email
  $pubObject | add-member -membertype NoteProperty -name "FolderPath" -Value $folderpath


  # Append this iteration of our for loop to our results array.
  $resultsarray += $pubObject
}

# Finally we want export our results to a CSV:
$resultsarray | export-csv -Path $env:userprofile\Desktop\mail-enabled-public-folders.csv

Below is some example output:

#TYPE System.Management.Automation.PSCustomObject
"Email","FolderPath"
"GS_request@company.com","\Company US\Global Systems\GS_request"
"HR@company.com","\Company US\HR"
"IT2@company.com","\Company US\IT"
"Purchases@company.com","\Company US\IT\Purchases"
"Japan@company.com","\Company US\Japan"
"JPN-Sales@company.com","\Company US\Japan\JPN-Sales"
"Marketing@company.com","\Company US\Marketing"
"Webmaster@company.com","\Company US\Marketing\Webmaster"
"BlogFeedback@company.com","\Company US\Marketing\Webmaster\Blog Feedback"
"DomainRegistration@company.com","\Company US\Marketing\Webmaster\Domain Registration"
"Feedblitzsignups@company.com","\Company US\Marketing\Webmaster\Feedblitz signups"
"SampleMaps@company.com","\Company US\Marketing\Webmaster\Sample Maps"
"MarketingCalendar3@company.com","\Company US\Marketing Calendar"
"Product@company.com","\Company US\Product"
"Archive@company.com","\Company US\Product\Archive"
"R&D@company.com","\Company US\R&D"
"Muse@company.com","\Company US\R&D\Muse"
"Bugs@company.com","\Company US\R&D\Muse\Bugs"
"Handleditems@company.com","\Company US\R&D\Muse\Handled items"
"resellerorder@company.com","\Company US\Sales Ops\resellerorder"
"ResellerOrdersCompleted@company.com","\Company US\Sales Ops\resellerorder\Reseller Orders Completed"
"TimeOffSchedule@company.com","\Company US\Sales Ops\resellerorder\Time Off Schedule"
"SalesCalendar@company.com","\Company US\Sales\Sales Calendar"

ifconfig vs ip

2014-01-03T00:00:00+00:00

!! ALL of this was stolen from tty1.net. I liked it so much, I decided to copy it for my own reference. !!

The command /bin/ip has been around for some time now. But people continue using the older command /sbin/ifconfig. Let’s be clear: ifconfig will not quickly go away, but its newer version, ip, is more powerful and will eventually replace it.

The man page of ip may look intimidating at first, but once you get familiar with the command syntax, it is an easy read. This page will not introduce the new features of ip. It rather features a side-by-side comparison if ifconfig and ip to get a quick overview of the command syntax.

Show network devices and configuration

ifconfig

ip addr show

ip link show

Enable a network interface

ifconfig eth0 up

ip link set eth0 up

A network interface is disabled in a similar way:

ifconfig eth0 down

ip link set eth0 down

Set IP address

ifconfig eth0 192.168.0.77

ip address add 192.168.0.77 dev eth0

This was the simple version of the command. Often, also the network mask or the broadcast address need to be specified. The following examples show the ifconfig and ip variants.

Needless to say that the netmask can also be given in CIDR notation, e.g. as 192.168.0.77/24.

ifconfig eth0 192.168.0.77 netmask 255.255.255.0 broadcast 192.168.0.255

ip addr add 192.168.0.77/24 broadcast 192.168.0.255 dev eth0

Delete an IP address

With ip it is also possible to delete an address:

ip addr del 192.168.0.77/24 dev eth0

Add alias interface

ifconfig eth0:1 10.0.0.1/8

ip addr add 10.0.0.1/8 dev eth0 label eth0:1

ARP protocol

Add an entry in your ARP table.

arp -i eth0 -s 192.168.0.1 00:11:22:33:44:55

ip neigh add 192.168.0.1 lladdr 00:11:22:33:44:55 nud permanent dev eth0

Switch ARP resolution off on one device

ifconfig -arp eth0

ip link set dev eth0 arp off

How to get user input without echoing to console with Ruby

2013-11-14T00:00:00+00:00

I am always writing little programs that require the end user to provide a password. One of the biggest problems is that the common methods for receiving user input ends up echoing everything typed in clear text for all to see. I was able to solve this little problem with a quick google search and I wanted to share for all to see.

require 'io/console'

print "Please enter your password: "
password = STDIN.noecho(&:gets).chomp
puts ## This is here to add a line break after grabbing the user input

How to reset Google Drive on Mac OS X

2013-11-02T00:00:00+00:00

I just recently was needing to change my Google Drive from using one Google account to another. When I looked at the Google Drive menu and the “preferences” menu item was greyed out, leaving me without a way to change accounts. This led me down a path of searching out a solution. In combination with my own intuition and the web, I came up with the following solution that worked.

Open Terminal
Copy and paste the following commands:

# The MIT License (MIT)
#
# Copyright (c) 2014 Jason Barnett
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.

# This command prints which directories would be deleted:
find ~ -maxdepth 1 -type d -iname '*google*drive*' -print

# The commands below will actually remove the Google Drive folders
# found in order to reset Google Drive
find ~ -maxdepth 1 -type d -iname '*google*drive*' 2> /dev/null -exec rm -Rf {} \;
rm -Rf ~/Library/Application\ Support/Google/Drive

How to remove all files from a Rackspace Cloud Files container

2013-08-03T00:00:00+00:00

I recently switched from Rackspace Cloud Files to Amazon’s S3 due to the high number of read errors I was getting while using Cloud Files. This presented an interested challenge: How am I going to remove all of these containers that are filled with thousands of files? I searched high and low and couldn’t find any code that could automate the entire process. This led me to hunt for a Ruby Library and get to coding! I quickly stumbled upon fog and began to code. Below is a very quick and dirty script to get the job done. I’m hoping that you’ll be able to modify the script below in a few minutes and you’ll be able to easily remove all files from several hundred containers, if not more. The script even has a neat little counter so you know where you are in the whole process.

#!/usr/bin/env ruby
# Author: Jason Barnett <J@sonBarnett.com>

require 'fog'

def delete_file(container, file_num, max_tries)
  max_retries ||= max_tries
  try = 0
  puts "(#{file_num} of #{container.files.count}) Removing #{container.files[file_num].key}"
  begin
    container.files[file_num].destroy
  rescue Excon::Errors::NotFound, Excon::Errors::Timeout, Fog::Storage::Rackspace::NotFound => e
    if try == max_retries
      $stderr.puts e.message
    else
      try += 1
      puts "Retry \##{try}"
      retry
    end
  end
end

def equal_div(first, last, num_of_groups)
  total      = last - first
  group_size = total / num_of_groups + 1

  top    = first
  bottom = top + group_size
  blocks = 1.upto(num_of_groups).inject([]) do |result, x|
    bottom = last if bottom > last
    result << [ top, bottom ]

    top    += group_size + 1
    bottom =  top + group_size

    result
  end

  blocks
end

service = Fog::Storage.new({
    :provider             => 'Rackspace',               # Rackspace Fog provider
    :rackspace_username   => 'your_rackspace_username', # Your Rackspace Username
    :rackspace_api_key    => 'your_api_key',            # Your Rackspace API key
    :rackspace_region     => :ord,                      # Defaults to :dfw
    :connection_options   => {},                        # Optional
    :rackspace_servicenet => false                      # Optional, only use if you're the Rackspace Region Data Center
})

containers = service.directories.select do |s|
  s.key =~ /^some_regex/  # Only delete containers that match the regexp
end

TOT_THREADS = 4
threads     = []

containers.each do |container|
  puts
  puts "-----------------------------------------"
  puts "-- Removing _ALL_ objects from #{container.key}"
  puts "-----------------------------------------"
  puts

  #puts "container.files.count: #{container.files.count}"

  ## separates the number of files into equal groups to distribute to each thread
  mygroups = equal_div(0, container.files.count - 1, TOT_THREADS)

  0.upto(TOT_THREADS - 1) do |thread|
    threads << Thread.new([ container, mygroups[thread] ]) { |tObject|
      tObject[1][0].upto(tObject[1][1]) do |x|
        delete_file(tObject[0], x, 5)
      end
    }

  end
  threads.each { |aThread|  aThread.join }
  puts "Deleting #{container.key}"
  container.destroy
end

How to compare all yum packages between two hosts

2013-08-03T00:00:00+00:00

Just a quick note! I often find myself needing to migrate a RHEL server from and old VM to a new one and I like to run a few checks and balances to know I have all of the same packages installed. This is an easy way to generate a list of packages that need to be installed on the new host. You can simply copy and paste the list into yum install .

comm -23 <(rpm -qa --qf "%{NAME}\n" | sort | uniq) <(ssh newhost.domain.com 'rpm -qa --qf "%{NAME}\n" | sort | uniq') | xargs

VMware – Linux VLAN tagging with Cisco Trunk Port

2013-07-02T00:00:00+00:00

I wanted to quickly document how you’re able to allow your Linux VMs to use VLAN tagging when using VMware ESXi. I originally ran into this problem when I was converting an avahi physical machine into a VM. After I had migrating the machine over, it was unable to bring up the VLAN interfaces. It ended up being really easy to fix, but it was a matter of knowing where to look. It is as simple as setting “VLAN ID (Optional)” to “ALL (4096)”

Click the ESXi/ESX host.
Click the Configuration tab.
Click the Networking link.
Click Properties.
Click the virtual switch / portgroups (often named VM Network) in the Ports tab and click Edit.
Click the General tab.
Select “All (4096)” in “VLAN ID (optional)”.

Send a Simple HTML Email Using Ruby

2013-05-10T00:00:00+00:00

Most recently in my Ruby endeavors, I needed an easy way to send simple HTML emails. I came up with what I think to be the most concise way of doing so. From the example I’ve given below you should be able to copy + paste and have something running in minutes, if not seconds. It simply leverages the lovely built-in Ruby class Net::SMTP. No need to install any additional RubyGems or anything else. Enjoy and have fun sending emails from your scripts, programs or anything else.

require 'net/smtp'

from    = "First Last <person@example.com>"
to      = "First Last <person@example.com>"
subject = "this is my subject"

message = <<MESSAGE_END
From: #{from}
To: #{to}
Subject: #{subject}
Mime-Version: 1.0
Content-Type: text/html
Content-Disposition: inline

<b>This is my simple HTML message</b><br /><br />

It goes on to tell of wonderful things you can do in ruby.
MESSAGE_END

Net::SMTP.start('localhost', 25) do |smtp|
  smtp.send_message message, from, to
end

iPhone 5 Exchange Server and Gmail – No Longer Supported – “The connection to the server failed.”

2013-05-03T00:00:00+00:00

So, probably like me, you upgraded your iPhone to the latest and greatest iPhone 5, only to realize that your gmail account that you setup as exchange is no longer working. You get that dreadful error, “The connection to the server failed.”

Well, as it turns out, Google is no longer supporting Google Sync. I was so baffled by this problem because my google apps account (Google Apps, NOT personal gmail) was working just fine, but none my other personal gmail accounts were working. After searching for hours, I finally stumbled upon this lovely little post by Google, “Google Sync End of Life“. If you go to that article it well provide alternative ways to sync your email, contacts, and calendar.

The Ultimate MySQL Backup Script in Bash

2013-01-21T00:00:00+00:00

So, just as many smart people in the world do, when I needed a MySQL backup script, I first googled it. To my surprise, I couldn’t find a single script that fit my needs. They were either far too basic (e.g. simple mysqldump without any kind of error reporting) or an entire program that needed to be deployed, configured and setup (e.g. Zmanda). I set out to write a bash script that was easy enough for someone to use who didn’t have the depth and knowledge of a Linux/Bash, but also could be used by a sysadmin who wanted something simple and quick to deploy. I think that I’ve accomplished just that, but I’ll let you be the judge. The source code (at the time of writing) can be found below or the latest at github.

#!/bin/bash
# Author: Jason Barnett <J@sonBarnett.com>
MYSQL_BACKUP_VERSION=1.0

############
## README ##
############
##
##  Assumptions this script makes:
##  ------------------------------
##    -The script itself is run by root
##    -root's home is /root
##    -MySQL Server version is >= 5.1.2 (That's when the whole --routines switch was added to mysqldump)
##    -crontab is configured to email you, help can be found here:
##      https://www.cyberciti.biz/faq/linux-unix-crontab-change-mailto-settings/
##
##  How to use this script:
##  -----------------------
##    Simply run this script as is once and it should ask all the right questions and place them in an easy
##    to reference config file so it never needs to ask again.
##
##    I suggest setting up a cronjob with this script once you've confirmed it's running as expected
##    and then putting it in your crontab, e.g.:
##
##      0 2 * * * /path/to/mysql-backup.sh > /dev/null
##
##    The reason behind my suggestion is that most people have (or sould have) cron setup so that it only
##    emails them when there is any output (both stdout or stderr) from the script/command/etc... I've
##    written this script to output to stderr when you need to be notified by email about a problem, e.g.
##    it's unable to grab a full list of databases, or it can't authenticate, etc... All of the stdout output
##    can be safely ignored. In order for this to work properly though, it requires cron daemon to be configured
##    properly so you can receive email from it when things go awry.
##
###########

## Functions ##
###############

function ask_question {
    question=$1
    read -p "$question: "
    echo $REPLY
}

function ask_yes_no {
    question=$1
    read -p "$question: [y/n] "
    local answer=$(echo $REPLY | tr '[:upper:]' '[:lower:]')

    while [[ "${answer}" != "yes" && "${answer}" != "no" && "${answer}" != "y" && "${answer}" != "n" ]];do
        read -p "y/n only please... $question: [y/n] "
        answer=$(echo $REPLY | tr '[:upper:]' '[:lower:]')
    done

    [[ "${answer}" == "yes" || "${answer}" == "y" ]] && echo true || echo false
}

function msg {
    echo "$1"
}

function err_msg {
    echo "$1" 1>&2
}

function fail_msg {
    echo "$1" 1>&2
    exit 1
}

function update_config {
    local config_property config_value
    config_property=$(echo $1 | awk -F= '{print $1}')
    config_value=$(echo $1 | awk -F= '{print $2}')

    [[ ! -d ${HOME}/.config/mysql-backup ]] && mkdir -p ${HOME}/.config/mysql-backup
    chmod 0700 ${HOME}/.config/mysql-backup

    [[ ! -e ${HOME}/.config/mysql-backup/config ]] && touch ${HOME}/.config/mysql-backup/config
    chmod 0600 ${HOME}/.config/mysql-backup/config

    sed -i "/^${config_property}=/d" ${HOME}/.config/mysql-backup/config
    echo "${config_property}=${config_value}" >> ${HOME}/.config/mysql-backup/config
}

function get_mysql_credentials {
    MYSQL_USER=$(ask_question "MySQL Username")
    [[ -z $MYSQL_USER ]] && exit 1
    MYSQL_HOST=$(ask_question "MySQL Host")
    MYSQL_PASS=$(ask_question "MySQL Password")
    [[ -n ${MYSQL_USER} && -n ${MYSQL_HOST} && -n ${MYSQL_PASS} ]] || {
        fail_msg "You have not specified a MySQL Username, Host and/or Password"
    }

    check_mysql_credentials
    update_config "MYSQL_USER=${MYSQL_USER}"
    update_config "MYSQL_HOST=${MYSQL_HOST}"
    update_config "MYSQL_PASS=${MYSQL_PASS}"
}

function check_mysql_credentials {
    temp_file=$(mktemp /tmp/.mysql-backup.XXXXXX)
    local good_credentials=

    mysql -u${MYSQL_USER} -h${MYSQL_HOST} -p${MYSQL_PASS} \
        -BNe 'show databases;' &> ${temp_file} && good_credentials=true

    if [[ -z $good_credentials ]];then
        fail_msg "You have a bad MySQL Username, Host and/or Password"
    fi

    rm -f ${temp_file}
}

function ask_about_routines {
    ROUTINES=$(ask_yes_no "Dump stored routines?")
    update_config "ROUTINES=${ROUTINES}"
}

function get_backup_destination {
    BACKUP_DEST=$(ask_question "Choose a backup destination, absolute path only")

    while [[ $(echo ${BACKUP_DEST} | egrep '^((\/[a-zA-Z0-9]+(_[a-zA-Z0-9]+)*(\-[a-zA-Z0-9]+)*)+)$') == "" ]];do
        BACKUP_DEST=$(ask_question "I said absolute path only...")
    done

    update_config "BACKUP_DEST=${BACKUP_DEST}"
}

## MAIN SCRIPT ##
#################

# Check if mysql server even exists on the machine and exit if it's not.
[[ -x /usr/bin/mysqld_safe ]] || { echo "MySQL-Server is not installed on this machine."; exit 0; }

# Set HOME and load config (We set HOME here because most likely we're running from the crontab and HOME isn't set)
HOME=/root
[[ -e ${HOME}/.config/mysql-backup/config ]] && . ${HOME}/.config/mysql-backup/config

# Check for MySQL credentials
[[ -n ${MYSQL_USER} && -n ${MYSQL_HOST} && -n ${MYSQL_PASS} ]] && check_mysql_credentials || get_mysql_credentials

# Should we dump routines?
[[ -n ${ROUTINES} && (${ROUTINES} == "true" || ${ROUTINES} == "false") ]] || ask_about_routines
[[ ${ROUTINES} == "true" ]] && ROUTINES='--routines' || ROUTINES=

# Check for backup destination, create it if it doesn't exist, and set the correct permissions
[[ -n ${BACKUP_DEST} ]] || get_backup_destination
[[ ! -d ${BACKUP_DEST} ]] && mkdir -p ${BACKUP_DEST}
chown root:root -R ${BACKUP_DEST}
chmod 0700 ${BACKUP_DEST}

# Locate gzip binary and try to use pigz (multi-threaded gzip), or fallback and use gzip
GZIP=`which pigz 2> /dev/null`
[[ -z ${GZIP} ]] && { GZIP=`which gzip 2> /dev/null`; msg "INFO: You don't have pigz installed, using gzip instead."; msg "      This is not a big deal, pigz simply speeds up the backup process."; }

# Locate mysqldump binary
MYSQLDUMP=`which mysqldump 2> /dev/null`
[[ -z ${MYSQLDUMP} ]] && fail_msg "Unable to locate \"mysqldump\". Make sure it's in your \$PATH."

# Get a list of all databases
DBs="$(mysql -u${MYSQL_USER} -h${MYSQL_HOST} -p${MYSQL_PASS} -BNe 'show databases;' | egrep -v '^(information_schema)$')"
mysql_status=$?

[[ $mysql_status != "0" ]] && fail_msg "There was an issue grabbing a complete list of databases to backup."

backup_failed=
for db in ${DBs};do
    FILE="${BACKUP_DEST}/${db}.sql.gz"
    [ -f ${FILE}.2 ] && rm -f ${FILE}.2
    [ -f ${FILE}.1 ] && mv ${FILE}.1 ${FILE}.2
    [ -f ${FILE}.0 ] && mv ${FILE}.0 ${FILE}.1
    [ -f ${FILE} ] && mv ${FILE} ${FILE}.0
    echo -n "Backing up $db... "
    ${MYSQLDUMP} -u${MYSQL_USER} -h${MYSQL_HOST} -p${MYSQL_PASS} $ROUTINES -B ${db} | $GZIP -9 > ${FILE}
    if [[ $? == "0" ]];
        then
            msg Success!
        else
            err_msg Failed!
            backup_failed=true
            failed_dbs="$db $failed_dbs"
    fi
done

[[ $backup_failed == "true" ]] && fail_msg "There was an issue backing up the following databases: ${failed_dbs}"

Parsing Tomcat's server.xml using Ruby

2013-01-19T00:00:00+00:00

I needed a tool to create a flat file database from Tomcat’s server.xml that I could then later reference when migrating sites, when creating a snapshot, etc. This is what I came up with:

#!/usr/bin/ruby

require 'rubygems'
require 'optparse'
require 'nokogiri'

options = {}
optparse = OptionParser.new do |opts|
  opts.banner = "Usage: #{File.basename($0)} [options]"

  opts.on("-d", "--debug",
    "Enables some helpful debugging output."
  ) { |value| options[:debug] = true }

  opts.on("-f", "--file FILE",
    "The server.xml file you would like to load."
  ) { |value| options[:file] = value }

  opts.on("-o", "--old",
    "Generate using the old formatting."
  ) { |value| options[:old] = value }

  opts.on("-h", "--help", "Display this help message.") do
    puts opts
    exit
  end
end

begin
  optparse.parse!
rescue OptionParser::MissingArgument
  puts "You're missing an argument...\n\n"
  puts optparse
  exit
end

p options if options[:debug] == true
p ARGV    if options[:debug] == true

## MAIN PROGRAM: ##
###################
file_to_parse = options[:file] ? options[:file] : "/opt/tomcat/conf/server.xml"

if ! FileTest.exist? file_to_parse
  $stderr.puts "#{file_to_parse} doesn't exist."
  $stderr.puts "   maybe you should use the -f option?\n\n"
  $stderr.puts optparse
  exit
elsif ! FileTest.readable? file_to_parse
  $stderr.puts "#{file_to_parse} isn't readable, exiting..."
  exit
end
puts "Parsing: " + file_to_parse if options[:debug] == true

## Create a table so we can lookup the line number for the </Service> for the particular service we want
lines = File.open(file_to_parse).readlines.collect { |x| x.strip }
lines.unshift('placing item in array so the index reflects the actual line number')

services_start = (0 .. lines.count - 1).find_all { |x| lines[x,1].to_s.match(/<Service/) }
services_end   = (0 .. lines.count - 1).find_all { |x| lines[x,1].to_s.match(/<\/Service/) }
service_line_numbers = services_start.zip(services_end)
p service_line_numbers if options[:debug] == true

end_line_lookup = service_line_numbers.inject({}) do |result, element|
  key   = element.first
  value = element.last
  result[key] = value
  result
end

xml_doc = Nokogiri::XML(File.open(file_to_parse))

xml_doc.xpath('//Service').each do |service|
  name = service[:name]
  start_line = service.line
  end_line   = end_line_lookup[start_line]

  # Create an array with all connectors and the needed attributes in a hash
  connectors = []
  service.children.css('Connector').each do |connector|
    connectors << {:ip => connector[:address], :port => connector[:port]}
  end

  # Create an array with all docBases
  docBase = []
  service.children.css('Context').each do |context|
    docBase << context[:docBase]
  end

  resources = []
  service.children.css('Resource').each do |resource|
    db_host     = resource[:url].scan(%r{mysql://(.*)/}).join.sub(/^127\.0\.0\.1$/, "localhost")
    schema_name = resource[:url].scan(%r{/([^/]+)\?}).join

    if resources.empty?
      resources << {:db_host => db_host, :schema_name => schema_name}
    else
      duplicate=""
      resources.each do |i|
        if i[:db_host] == db_host and i[:schema_name] == schema_name
          duplicate=true
        end
      end
      if duplicate != true
        resources << {:db_host => db_host, :schema_name => schema_name}
      end
    end
  end

  if options[:old] == true
    # Create schemas array for all schema names, then add NAMEuser schema.
    schemas = resources.inject([]) do |result, resource|
      result << resource[:schema_name]
      result
    end

    # Create dbhost variable and check some stuff out
    db_hosts = resources.inject([]) do |result, resource|
      result << resource[:db_host]
      result
    end.uniq
    if db_hosts.count > 1
      puts "WARNING: We found miltiple hosts for the schema's, exiting..."
      p db_hosts if options[:debug] == true
      exit 1
    end

    schemas << "#{schemas[0]}user"
    # Create ips array for all ip addresses
    ips = connectors.inject([]) do |result, connector|
      result << connector[:ip]
      result
    end
    puts "#{start_line},#{end_line}:#{ips.uniq.join(',')}:#{docBase.uniq.join(',')}:#{db_hosts.join(',')}:#{schemas.join(',')}:#{name}"
  else
    serviceDefintion = {:start_line => start_line, :end_line => end_line, :name => name, :connectors => connectors.uniq, :docBases => docBase.uniq, :resources => resources}
    p serviceDefintion
  end
end