Wednesday, November 27, 2013

sshd: why you have to type your password
Or: Why can't nxclient authenticate?

I set up passwordless login on my home computers literally a decade ago, but after a recent upgrade I found myself confronted by this nonsense:
collin@v2:~ 
% ssh p3
Password: 
My initial reaction—viz., "wtf?"—gave way to my usual "let's-get-on-with-it" attitude, and so for a while I've just been typing in my password.

Perhaps you've heard the quip that a reasonable man adapts himself to circumstances, but the unreasonable man adapts the circumstances to himself? And therefore, all progress depends upon the unreasonable man? Well, today I got unreasonable.

The catalyst was my totally unreasonable desire to use FreeNX at work. I ran nxclient and typed my password in (that's expected btw). But I got some message about can't authenticate. Wha...? Why would this start failing? It worked before I upgraded to OpenSUSE 12.3 (64-bit).

Various websites told me to look in /var/log/auth.log, which my computer doesn't have. So I went looking in the config file for logfile information. No joy. But I happened to see this:

$ grep -A2 authorized_keys2 /etc/ssh/sshd_config
# The default is to check both .ssh/authorized_keys and .ssh/authorized_keys2
# but this is overridden so installations will only check .ssh/authorized_keys
AuthorizedKeysFile      .ssh/authorized_keys
$
A-HA! So in /etc/NX/home/nx/.ssh and also in my own $HOME/.ssh, I said:
collin@p3:~/.ssh> ll auth*
-rw-r--r-- 1 collin users 1341 Jul 26  2003 authorized_keys
-rw-r--r-- 1 collin users 2031 May 20  2006 authorized_keys2
collin@p3:~/.ssh> mv authorized_keys authorized_keys-2003-07-26
collin@p3:~/.ssh> mv authorized_keys2 authorized_keys
collin@p3:~/.ssh>
My next nxclient connection attempt worked. As did my next login at home.

Life's little victories...

Saturday, November 16, 2013

Sending SMTP mail through my ISP’s server while VPNed to the office

Until today, my home computer's relay host settings (both t-bird’s outgoing mail host setting and Postfix’s relayhost) were set to my ISP’s mail server, mail.myISP.com. This works well, since most of the time that's how it gets to the internet. You can see my routing table here:
collin@p3:~> netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         192.168.1.254   0.0.0.0         UG        0 0          0 eth0
127.0.0.0       0.0.0.0         255.0.0.0       U         0 0          0 lo
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 eth0
collin@p3:~> 
My ISP of course doesn't have an rfc1918 address, so when I want to send SMTP email, my computer connects to mail.myisp.com port 25 by going through my router/firewall (see the default gateway line in the table, shown in this color) then out through the modem.

Accordingly, my ISP's MTA (also Postfix I think) sees an incoming connection from a DHCP address assigned by my ISP (i.e., one of its own) and allows relaying to gmail.com or earthlink.net or wherever I want to send email.

But if I'm connected to my office via vpnc(8), my routing table will be altered to look something like this:

collin@p3:~> netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         0.0.0.0         0.0.0.0         U         0 0          0 tun0
10.55.72.0      0.0.0.0         255.255.248.0   U         0 0          0 tun0
127.0.0.0       0.0.0.0         255.0.0.0       U         0 0          0 lo
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 eth0
216.240.19.24   192.168.1.254   255.255.255.255 UGH       0 0          0 eth0
collin@p3:~>
In this condition, when I connect to my ISP, it won't see one of its own IP addresses; the routing table's default gateway line directs packets through the tunnel, thence over the VPN to my office. Then my ISP's MTA sees a connection from an alien address, rather than one of its own. Accordingly, it won't relay any email; I can only send to SOMEONE@myisp.com. (And yes, I have tried it and seen the bounce message -- "You, Mr. Foreign IP address, aren't allowed to relay email through this server.")

Today I figured out what to do about it, and wonder of wonders, it actually worked. So I thought I'd share it with you. Here's the secret: I have another computer. Actually it belongs to the lovely Carol. It's a mac mini, running the Postfix MTA. I could use it as my relay host, rather than my ISP. The mac mini would relay email to my ISP, and the ISP would see email coming from one of "its own" IP addresses, since the mac doesn't VPN anywhere in this reality. So on my Linux box (OpenSuSE 12.3) I modified main.cf to read:

collin@p3:/etc/postfix> grep relayhost main.cf
# The relayhost parameter specifies the default host to send mail to
# no relayhost is given, mail is routed directly to the destination.
#relayhost = [mail.myisp.com]
relayhost = 192.168.1.99
collin@p3:/etc/postfix> 
(where of course 192.168.1.99 is the IP address of the mac).

Now this didn't quite do it, as the mac won't relay email for any other host. Web searches told me where to find the config files, and the config files told me what to tweak. Basically I did this:

bash-3.2# cd /etc/postfix/
bash-3.2# diff main.cf.install main.cf
…
663c668,669
< mynetworks = 127.0.0.0/8
---
> # collin 2013-11-16.  This should be safe as DHCP assigns above 200 or so
> mynetworks = 127.0.0.0/8, 192.168.1.0/28
bash-3.2# 
You see, the mac formerly would only relay email for, ah, itself. That 127.0.0.0/8 thing—so elitist! So I switched it to let in the first, oh, 16 or so addresses from my local net. As the comment says, DHCP is configured to assign only addresses near the upper end of the 192.168.1.0/24 range.

That was it. I gave it a quick test, emailing to a couple different domains (gmail, yahoo) and reading the headers.

Friday, November 08, 2013

Where Am I Going?

This year has been disorienting: three deaths (one untimely) in the family, and two reunions of a sort.

My father's two surviving brothers died, just a few weeks apart; they were in their 90s. My brother-in-law, though, was under 64: too young to die, really, yet with multiple health challenges he beat the odds. I don't know who said that each of us should (daily?) consider their own death, but I'll grant the point. We westerners are great death-deniers, and considering with some regularity the question of my own death could be a healthy and actually life-affirming thing.

Without that I could sleepwalk through my days, arriving at my last one with astonishment and needless regret.

I mentioned reunions: I saw several people from my college class last month. One said, "I still tell that story, about how you finished the final project…" I really didn't remember that achievement, but it started to make me wonder if I've lived up to my potential. I don't mean becoming rich or famous; rather "How well have I used what I've received?" Similarly, does my life preach the doctrine of ignoble ease? Have I taken the easy path too often?

At my brother-in-law's memorial, someone from my high school class asked me if my job used all my mental capabilities, and that reminded me of a sermon I heard a few months back. The preacher, Jordan Seng, used to be an academic; he published numerous papers during his post-doc fellowships. His recent book is about miracles, and some past colleagues seem surprised at his change in focus and apparent level of intellectual activity.

At some point during his academic career, Jordan visited a developing country, where he taught children about God. It was on that trip, he said, that he found his teaching abilities stretched. "Teaching smart academics about political theory—that's easy. Teaching an illiterate child with uneducated parents about the Kingdom of God—that fully used all my teaching skills."

I think it was Rilke who said that if something is hard, that's what makes it worth doing. That's an oversimplification of course, but doing only easy things makes for a boring and probably useless life.

As Buechner (or should that be Büchner?) wrote, we are great avoiders. We have some spare minutes with nothing really going on, a time that would be perfect for considering where we have come from, where we are going, and what kind of person, for better or worse, we are becoming. But we don't think about that; we pick up the paper or turn on the television or start on some task that really needn't be done today or even this week.

So where am I going? What kind of person am I becoming? If I were to die tonight, what would be my regrets?

And what hard things am I doing, or should I be doing?

Coincidentally (hah!) I had two reassuring incidents this week, which showed me that although I sometimes take the easy path, I don't do it all the time. At the farewell dinner for our church board (I rotated off this week), kind words were said about the outgoing members. Someone described me as adventurous, citing the years we'd spent in Japan. And at lunch today (are you detecting a theme here?) where we were all supposed to speak each member's spiritual gifts, someone said I had the gift of apostleship! So if I am indeed guilty of preaching the doctrine of ignoble ease, I'm apparently not doing it very well.

But when I think about hard problems, I tend to think rather of some easier problem to work on—one I have a chance of succeeding at. What's a hard problem? There are lots of them of course, even if I restrict myself to ones where I think I have any chance at all of figuring them out. So, writing and selling a book would be hard. Explaining the gospel to a Silicon Valley denizen would be hard.

I mean, I think about what I used to say to students in 1980: "Because of sin, there's a chasm between us and God..." By this time, the Si-Valley guy would be checking his phone, if he hadn't already walked off.

Well, no answers, but I thought I'd share these thoughts with you anyway.

Monday, November 04, 2013

Combining lots of ".aiff" files using sox

There was a memorial service yesterday, and the sound guy at the church made an audio CD. I'm surprised everything fit, really; it was about 81 minutes.

My dad noted that the CD had 81 tracks, each about a minute long. When trying to listen to the CD on my mom's computer, he could only hear a minute at a time, and a track would typically end in the middle of a song or sentence. Then he'd have to hit the "Play" button and wait for the CD to spin up again.

I supposed that there was no actual pause on the CD, but that the CD player software just stopped when it saw a track marker. To make this friendlier for playback on computer, we decided to concatenate the "tracks" into a single long file. We could divide it up into smaller pieces (inserting a track marker when a particular song or speech starts, say) later.

My guess is that the sound guy had a single very long audio track, and he told his CD burning software to insert a track marker every 60 seconds. Why was this a good idea? Well, suppose you're playing the CD on a CD player (these things still exist). Some CD players don't tell you how far into a given track they've gotten. So if you wanted to find a spot 23 minutes into the thing, you'd be pretty much stuck with either waiting 23 minutes after you hit "PLAY," or using the skip/review function (if your CD player has one) while listening... assuming of course that you knew what came before and after the point of interest.

With the every-60-second track markers, you can skip 23 tracks. It still might start in the middle of a song or sentence, but that's better than waiting 23 minutes, right?

My newphew came into the room, and upon hearing the problem, proceeded to copy 81 "AIFF" files into a directory on the computer's hard drive. (Do audio CDs actually have AIFF files? No, but on Mac OS X that's what it looks like in the Finder.) This took a long time. After looking at the Activity Monitor app and seeing no CPU or RAM hogs, we decided that having a 98% full hard drive might have something to do with the issue. That took care of some of our waiting time, anyway.

Anyway, he seemed to remember hearing that Quick Time 7 could concatenate sound files, and he tried it, taking the first file (I think the filename was "1 Track 1.aiff" or something like this) and doing the click-and-drag thing with "2 Track 2.aiff". After some seconds of computing, the computer showed us that we indeed had a two-minute file. We played it and there was no audible gap or pause as we crossed the one-minute mark.

But nobody had the patience to drag the other 79 files, and I certainly didn't feel confident that we could do it without error. Attention wanders when doing tasks like that. The worst thing was, if we did make a mistake (leaving out track #24 for example, or appending track #72 twice), we might not notice it for some hours.

So we did a web search on concatenating .AIFF files. One result mentioned using cat(1), so I gamely opened a terminal window. My thought was to get all the files listed in order, then pass them to cat, which would concatenate them end-to-end. We would route the output to an output file, say x.aiff.

How to get a list of all the files? Well, one could type something like "echo *.aiff" and get a result that looks something like this:

% echo *.aiff
1 Track 1.aiff 10 Track 10.aiff 11 Track 11.aiff 12 Track 12.aiff
13 Track 13.aiff 14 Track 14.aiff 15 Track 15.aiff 16 Track 16.aiff
17 Track 17.aiff 18 Track 18.aiff 19 Track 19.aiff 2 Track 2.aiff
20 Track 20.aiff 21 Track 21.aiff 22 Track 22.aiff…
Actually the result is one very long line, which ends as follows:
79 Track 79.aiff 8 Track 8.aiff 80 Track 80.aiff 81 Track 81.aiff 9 Track 9.aiff
Is the first problem obvious? When we say "echo *.aiff" or whatever, the output is sorted character-by-character, so for example "19" comes before "2" because the first character of "19" (i.e., "1") sorts lower than the first character of "2". The ls command does that too. If you knew the files were written in a certain order (e.g., track 2 has an earlier modification time than track 19 for example) then ls might help, but since the source was an audio CD with no mtime data, this didn't occur to me.

What did occur to me, though, was to say "sort -n"; as the manpage tells us (type "man sort")

   -n     Compare according to arithmetic value an initial numeric  string
          consisting of optional white space, an optional - sign, and zero
          or more digits, optionally followed by a decimal point and  zero
          or more digits.
So if we said "ls|sort -n"...
collin@v2:~/tmp/CD-tracks 
% ls *.aiff|sort -n
1 Track 1.aiff
2 Track 2.aiff
3 Track 3.aiff
4 Track 4.aiff
5 Track 5.aiff
6 Track 6.aiff
7 Track 7.aiff
8 Track 8.aiff
9 Track 9.aiff
10 Track 10.aiff
…
78 Track 78.aiff
79 Track 79.aiff
80 Track 80.aiff
81 Track 81.aiff
collin@v2:~/tmp/CD-tracks 
% 
OK, that's more like it. Suppose then we piped all these to cat(1). Umm, let's try with just the first three files.
collin@v2:~/tmp/CD-tracks 
% ls *.aiff | sort -n | head -n3
1 Track 1.aiff
2 Track 2.aiff
3 Track 3.aiff
collin@v2:~/tmp/CD-tracks 
% 
The head command gives the first <howevermany> lines of its input. To say how many lines (the default I think is 10), we use -n# where # is the number of lines we want -- in this case, three.

Given the three lines, how do we pass all of them to cat? Easiest thing is the xargs command. If it had been invented last week, there would probably be a patent application in the works, but fortunately the idea came in an earlier era so all can use it for free. Here's the thing: If you give xargs five lines, then it will append them to the end of a single command line. So if we said for example

collin@v2:~/tmp/CD-tracks 
% ls *.aiff | head -n3 | xargs echo
1 Track 1.aiff 10 Track 10.aiff 11 Track 11.aiff
collin@v2:~/tmp/CD-tracks 
% 
Right? We can do that with cat and send the output to a file we'll call "x", maybe like this:
collin@v2:~/tmp/CD-tracks 
% ls *.aiff | sort -n | head -n3 | xargs cat > x.aiff
cat: 1: No such file or directory
cat: Track: No such file or directory
cat: 1.aiff: No such file or directory
cat: 2: No such file or directory
cat: Track: No such file or directory
cat: 2.aiff: No such file or directory
cat: 3: No such file or directory
cat: Track: No such file or directory
cat: 3.aiff: No such file or directory
collin@v2:~/tmp/CD-tracks 
% 
Were you surprised, or did you wonder how cat would be able to tell where one filename ended and the next began? Let's get rid of the spaces in those filenames. There's any number of ways to do that, but I typed something like this to see if it would work:
collin@v2:~/tmp/CD-tracks 
% for X in *.aiff; do echo mv "$X" ${X// /.}; done | head -n3
mv 1 Track 1.aiff 1.Track.1.aiff
mv 10 Track 10.aiff 10.Track.10.aiff
mv 11 Track 11.aiff 11.Track.11.aiff
collin@v2:~/tmp/CD-tracks 
% 
What's "${X// /.}"? Well, let me go back to the beginning of the line.
  1. for X in *.aiff; do
    says to assign the variable X to successive values of whatever filenames match *.aiff, and execute everything until the done keyword.
  2. echo &hellip
    I want to see what's about to happen, rather than just executing it.
  3. mv "$X" …
    The command mv is the Unix™ command "move", which is how we rename things. What do we rename? The $X says "whatever's in shell variable X".

    Why do I put the $X in "double quotes"? Because that tells the shell to treat the entire name (1 Track 1.aiff for example) as a single "word".

    What new name will we give to $X? That's coming next.

  4. ${X// /.}
    This tells the shell to start with the variable X (which will be "1 Track 1.aiff" the first time through, etc.) and replace all instances of " " by ".". How did I know this? From reading the manpage for the shell. Type "man sh" or "man bash" some evening when you're having trouble getting to sleep:
    BASH(1)                                                          BASH(1)
    
    NAME
           bash - GNU Bourne-Again SHell
    
    SYNOPSIS
           bash [options] [file]
    …
    Parameter Expansion
       The `$' character introduces parameter expansion, command substitution,
       or arithmetic expansion.  The parameter name or symbol to  be  expanded
       may  be enclosed in braces, which are optional but serve to protect the
       variable to be expanded from characters immediately following it  which
       could be interpreted as part of the name.
    …
       ${parameter/pattern/string}
       ${parameter//pattern/string}
          The pattern is expanded to produce a pattern just as in pathname
          expansion.  Parameter is expanded and the longest match of  pat-
          tern  against  its  value is replaced with string.  In the first
          form, only the first match is replaced.  The second form  causes
          all  matches  of pattern to be replaced with string.  If pattern
          begins with #, it must match at the beginning  of  the  expanded
          value  of parameter.  If pattern begins with %, it must match at
          the end of the expanded value of parameter.  If string is  null,
          matches  of  pattern are deleted and the / following pattern may
          be omitted.  If parameter is @ or *, the substitution  operation
          is  applied to each positional parameter in turn, and the expan-
          sion is the resultant list.  If parameter is an  array  variable
          subscripted  with  @ or *, the substitution operation is applied
          to each member of the array in turn, and the  expansion  is  the
          resultant list.  
  5. done;
    see item 1 above.
Convinced that that ought to work now, I do this:
collin@v2:~/tmp/CD-tracks 
% for X in *.aiff; do mv "$X" ${X// /.}; done
collin@v2:~/tmp/CD-tracks 
% ls *.aiff | sort -n | head -n3 | xargs echo cat
cat 1.Track.1.aiff 2.Track.2.aiff 3.Track.3.aiff
collin@v2:~/tmp/CD-tracks 
% 
That's more like it. Now let's do the real thing:
collin@v2:~/tmp/CD-tracks 
% ls *.aiff | sort -n | xargs cat > x.aiff
collin@v2:~/tmp/CD-tracks 
% 
Great! We looked at the size of each file (about 10 Mbytes per track, except the last), and the output file, x.aiff, was about 800 Mbytes, so we were OK. Opening the result in Quick Time Player 7, we found it was only as long as the last track, i.e., less than 40 seconds.

Disillusionment

Well, we went back to the web search, to the post that mentioned using cat, and found that basically, you can't do that.

I did another web search and found that sox can in fact concatenate sound files, including aiff files. Yes! We went to the sourceforge site and downloaded the Mac OS X zip file. I think it was a zip file anyway (actually happened about 12 hours ago). Unlike a lot of Mac OS apps where you click here to install, etc., this one you just unpack and run it. I ended up typing something like

% ~/Downloads/sox-14.2/sox… 
but I'm getting ahead of myself.

First we looked at the documentation for sox, which told us that if you want to concatenate sound files, you put them on the command line, with the output file last. If I recall correctly the example was

% sox short.aiff long.aiff longer.aiff
but of course we had 81 input files. What would the output file be? I would want it to sort last, so I did something like this:
% touch 999.aiff; ls *.aiff | sort -n | xargs ~/Downloads/sox-14.2/sox
Then, opening 999.aiff in Quick Time Player 7, we found that it indeed appeared to be about 81 minutes long.

Technology is great when it works. I copied the result to my USB flash drive, so the lovely Carol can listen to it after I return home.

Remembering Danny

My brother-in-law Danny died a week ago. He was in great pain and ready to go to his reward. I called my sister the day before, and she filled me in a little on his condition. He'd been hallucinating off and on. Did I want to talk with him? She handed him the phone.

"Hi Collin!" he said. He sounded cheerful, or was that just me wanting him not to be in pain? I told him I was sorry he was going through all this. He seemed not to hear me, but said, "I'll call you back, OK?"

No you won't, I thought, but what I said was, "OK." What a miserable situation! He wanted to go, he was deeply uncomfortable, and if he recovered, my sister would have had a horrific time trying to care for him: a leg was due to be amputated, and his heart was going to need a valve job within a year.

And yet how terrible for him not to be there any more!

Well, I wanted to write a few things about him while I can still see.

The lovely Carol mentioned a time when our kids were very young. There were frustrations, but she never heard Danny yell at his boys. She asked him about it, and he explained that he'd put a hand on the kid's shoulder and lower his face to be directly in front of the kid's. (Maybe that was one hand on each shoulder—I don't recall exactly.) His nose no more than 3 inches from his son's, he would say firmly, "You need to put away your toys now." It was not loud, but it was quite effective.

Danny's musical gifts were considerable, and he used them at various churches. At yesterday's memorial service, I kept reminding myself that he wasn't about to appear on stage with his cane and sit down at the piano. Several people mentioned yesterday that he started many musical groups and worked patiently with almost anyone who wanted to sing or play.

What I remember most about him most was his gentle manner and the way he talked about his health challenges. "Everybody's got something," he would say, and he was certainly right about that. But Danny had way more than his fair share.

"I'll call you back, OK?" he said. When I get there, some decades hence I suppose, I'll hold him to that promise.