Wednesday, November 27, 2013

sshd: why you have to type your password
Or: Why can't nxclient authenticate?

I set up passwordless login on my home computers literally a decade ago, but after a recent upgrade I found myself confronted by this nonsense:
collin@v2:~ 
% ssh p3
Password: 
My initial reaction—viz., "wtf?"—gave way to my usual "let's-get-on-with-it" attitude, and so for a while I've just been typing in my password.

Perhaps you've heard the quip that a reasonable man adapts himself to circumstances, but the unreasonable man adapts the circumstances to himself? And therefore, all progress depends upon the unreasonable man? Well, today I got unreasonable.

The catalyst was my totally unreasonable desire to use FreeNX at work. I ran nxclient and typed my password in (that's expected btw). But I got some message about can't authenticate. Wha...? Why would this start failing? It worked before I upgraded to OpenSUSE 12.3 (64-bit).

Various websites told me to look in /var/log/auth.log, which my computer doesn't have. So I went looking in the config file for logfile information. No joy. But I happened to see this:

$ grep -A2 authorized_keys2 /etc/ssh/sshd_config
# The default is to check both .ssh/authorized_keys and .ssh/authorized_keys2
# but this is overridden so installations will only check .ssh/authorized_keys
AuthorizedKeysFile      .ssh/authorized_keys
$
A-HA! So in /etc/NX/home/nx/.ssh and also in my own $HOME/.ssh, I said:
collin@p3:~/.ssh> ll auth*
-rw-r--r-- 1 collin users 1341 Jul 26  2003 authorized_keys
-rw-r--r-- 1 collin users 2031 May 20  2006 authorized_keys2
collin@p3:~/.ssh> mv authorized_keys authorized_keys-2003-07-26
collin@p3:~/.ssh> mv authorized_keys2 authorized_keys
collin@p3:~/.ssh>
My next nxclient connection attempt worked. As did my next login at home.

Life's little victories...

Saturday, November 16, 2013

Sending SMTP mail through my ISP’s server while VPNed to the office

Until today, my home computer's relay host settings (both t-bird’s outgoing mail host setting and Postfix’s relayhost) were set to my ISP’s mail server, mail.myISP.com. This works well, since most of the time that's how it gets to the internet. You can see my routing table here:
collin@p3:~> netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         192.168.1.254   0.0.0.0         UG        0 0          0 eth0
127.0.0.0       0.0.0.0         255.0.0.0       U         0 0          0 lo
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 eth0
collin@p3:~> 
My ISP of course doesn't have an rfc1918 address, so when I want to send SMTP email, my computer connects to mail.myisp.com port 25 by going through my router/firewall (see the default gateway line in the table, shown in this color) then out through the modem.

Accordingly, my ISP's MTA (also Postfix I think) sees an incoming connection from a DHCP address assigned by my ISP (i.e., one of its own) and allows relaying to gmail.com or earthlink.net or wherever I want to send email.

But if I'm connected to my office via vpnc(8), my routing table will be altered to look something like this:

collin@p3:~> netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         0.0.0.0         0.0.0.0         U         0 0          0 tun0
10.55.72.0      0.0.0.0         255.255.248.0   U         0 0          0 tun0
127.0.0.0       0.0.0.0         255.0.0.0       U         0 0          0 lo
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 eth0
216.240.19.24   192.168.1.254   255.255.255.255 UGH       0 0          0 eth0
collin@p3:~>
In this condition, when I connect to my ISP, it won't see one of its own IP addresses; the routing table's default gateway line directs packets through the tunnel, thence over the VPN to my office. Then my ISP's MTA sees a connection from an alien address, rather than one of its own. Accordingly, it won't relay any email; I can only send to SOMEONE@myisp.com. (And yes, I have tried it and seen the bounce message -- "You, Mr. Foreign IP address, aren't allowed to relay email through this server.")

Today I figured out what to do about it, and wonder of wonders, it actually worked. So I thought I'd share it with you. Here's the secret: I have another computer. Actually it belongs to the lovely Carol. It's a mac mini, running the Postfix MTA. I could use it as my relay host, rather than my ISP. The mac mini would relay email to my ISP, and the ISP would see email coming from one of "its own" IP addresses, since the mac doesn't VPN anywhere in this reality. So on my Linux box (OpenSuSE 12.3) I modified main.cf to read:

collin@p3:/etc/postfix> grep relayhost main.cf
# The relayhost parameter specifies the default host to send mail to
# no relayhost is given, mail is routed directly to the destination.
#relayhost = [mail.myisp.com]
relayhost = 192.168.1.99
collin@p3:/etc/postfix> 
(where of course 192.168.1.99 is the IP address of the mac).

Now this didn't quite do it, as the mac won't relay email for any other host. Web searches told me where to find the config files, and the config files told me what to tweak. Basically I did this:

bash-3.2# cd /etc/postfix/
bash-3.2# diff main.cf.install main.cf
…
663c668,669
< mynetworks = 127.0.0.0/8
---
> # collin 2013-11-16.  This should be safe as DHCP assigns above 200 or so
> mynetworks = 127.0.0.0/8, 192.168.1.0/28
bash-3.2# 
You see, the mac formerly would only relay email for, ah, itself. That 127.0.0.0/8 thing—so elitist! So I switched it to let in the first, oh, 16 or so addresses from my local net. As the comment says, DHCP is configured to assign only addresses near the upper end of the 192.168.1.0/24 range.

That was it. I gave it a quick test, emailing to a couple different domains (gmail, yahoo) and reading the headers.

Friday, November 08, 2013

Where Am I Going?

This year has been disorienting: three deaths (one untimely) in the family, and two reunions of a sort.

My father's two surviving brothers died, just a few weeks apart; they were in their 90s. My brother-in-law, though, was under 64: too young to die, really, yet with multiple health challenges he beat the odds. I don't know who said that each of us should (daily?) consider their own death, but I'll grant the point. We westerners are great death-deniers, and considering with some regularity the question of my own death could be a healthy and actually life-affirming thing.

Without that I could sleepwalk through my days, arriving at my last one with astonishment and needless regret.

I mentioned reunions: I saw several people from my college class last month. One said, "I still tell that story, about how you finished the final project…" I really didn't remember that achievement, but it started to make me wonder if I've lived up to my potential. I don't mean becoming rich or famous; rather "How well have I used what I've received?" Similarly, does my life preach the doctrine of ignoble ease? Have I taken the easy path too often?

At my brother-in-law's memorial, someone from my high school class asked me if my job used all my mental capabilities, and that reminded me of a sermon I heard a few months back. The preacher, Jordan Seng, used to be an academic; he published numerous papers during his post-doc fellowships. His recent book is about miracles, and some past colleagues seem surprised at his change in focus and apparent level of intellectual activity.

At some point during his academic career, Jordan visited a developing country, where he taught children about God. It was on that trip, he said, that he found his teaching abilities stretched. "Teaching smart academics about political theory—that's easy. Teaching an illiterate child with uneducated parents about the Kingdom of God—that fully used all my teaching skills."

I think it was Rilke who said that if something is hard, that's what makes it worth doing. That's an oversimplification of course, but doing only easy things makes for a boring and probably useless life.

As Buechner (or should that be Büchner?) wrote, we are great avoiders. We have some spare minutes with nothing really going on, a time that would be perfect for considering where we have come from, where we are going, and what kind of person, for better or worse, we are becoming. But we don't think about that; we pick up the paper or turn on the television or start on some task that really needn't be done today or even this week.

So where am I going? What kind of person am I becoming? If I were to die tonight, what would be my regrets?

And what hard things am I doing, or should I be doing?

Coincidentally (hah!) I had two reassuring incidents this week, which showed me that although I sometimes take the easy path, I don't do it all the time. At the farewell dinner for our church board (I rotated off this week), kind words were said about the outgoing members. Someone described me as adventurous, citing the years we'd spent in Japan. And at lunch today (are you detecting a theme here?) where we were all supposed to speak each member's spiritual gifts, someone said I had the gift of apostleship! So if I am indeed guilty of preaching the doctrine of ignoble ease, I'm apparently not doing it very well.

But when I think about hard problems, I tend to think rather of some easier problem to work on—one I have a chance of succeeding at. What's a hard problem? There are lots of them of course, even if I restrict myself to ones where I think I have any chance at all of figuring them out. So, writing and selling a book would be hard. Explaining the gospel to a Silicon Valley denizen would be hard.

I mean, I think about what I used to say to students in 1980: "Because of sin, there's a chasm between us and God..." By this time, the Si-Valley guy would be checking his phone, if he hadn't already walked off.

Well, no answers, but I thought I'd share these thoughts with you anyway.

Monday, November 04, 2013

Combining lots of ".aiff" files using sox

There was a memorial service yesterday, and the sound guy at the church made an audio CD. I'm surprised everything fit, really; it was about 81 minutes.

My dad noted that the CD had 81 tracks, each about a minute long. When trying to listen to the CD on my mom's computer, he could only hear a minute at a time, and a track would typically end in the middle of a song or sentence. Then he'd have to hit the "Play" button and wait for the CD to spin up again.

I supposed that there was no actual pause on the CD, but that the CD player software just stopped when it saw a track marker. To make this friendlier for playback on computer, we decided to concatenate the "tracks" into a single long file. We could divide it up into smaller pieces (inserting a track marker when a particular song or speech starts, say) later.

My guess is that the sound guy had a single very long audio track, and he told his CD burning software to insert a track marker every 60 seconds. Why was this a good idea? Well, suppose you're playing the CD on a CD player (these things still exist). Some CD players don't tell you how far into a given track they've gotten. So if you wanted to find a spot 23 minutes into the thing, you'd be pretty much stuck with either waiting 23 minutes after you hit "PLAY," or using the skip/review function (if your CD player has one) while listening... assuming of course that you knew what came before and after the point of interest.

With the every-60-second track markers, you can skip 23 tracks. It still might start in the middle of a song or sentence, but that's better than waiting 23 minutes, right?

My newphew came into the room, and upon hearing the problem, proceeded to copy 81 "AIFF" files into a directory on the computer's hard drive. (Do audio CDs actually have AIFF files? No, but on Mac OS X that's what it looks like in the Finder.) This took a long time. After looking at the Activity Monitor app and seeing no CPU or RAM hogs, we decided that having a 98% full hard drive might have something to do with the issue. That took care of some of our waiting time, anyway.

Anyway, he seemed to remember hearing that Quick Time 7 could concatenate sound files, and he tried it, taking the first file (I think the filename was "1 Track 1.aiff" or something like this) and doing the click-and-drag thing with "2 Track 2.aiff". After some seconds of computing, the computer showed us that we indeed had a two-minute file. We played it and there was no audible gap or pause as we crossed the one-minute mark.

But nobody had the patience to drag the other 79 files, and I certainly didn't feel confident that we could do it without error. Attention wanders when doing tasks like that. The worst thing was, if we did make a mistake (leaving out track #24 for example, or appending track #72 twice), we might not notice it for some hours.

So we did a web search on concatenating .AIFF files. One result mentioned using cat(1), so I gamely opened a terminal window. My thought was to get all the files listed in order, then pass them to cat, which would concatenate them end-to-end. We would route the output to an output file, say x.aiff.

How to get a list of all the files? Well, one could type something like "echo *.aiff" and get a result that looks something like this:

% echo *.aiff
1 Track 1.aiff 10 Track 10.aiff 11 Track 11.aiff 12 Track 12.aiff
13 Track 13.aiff 14 Track 14.aiff 15 Track 15.aiff 16 Track 16.aiff
17 Track 17.aiff 18 Track 18.aiff 19 Track 19.aiff 2 Track 2.aiff
20 Track 20.aiff 21 Track 21.aiff 22 Track 22.aiff…
Actually the result is one very long line, which ends as follows:
79 Track 79.aiff 8 Track 8.aiff 80 Track 80.aiff 81 Track 81.aiff 9 Track 9.aiff
Is the first problem obvious? When we say "echo *.aiff" or whatever, the output is sorted character-by-character, so for example "19" comes before "2" because the first character of "19" (i.e., "1") sorts lower than the first character of "2". The ls command does that too. If you knew the files were written in a certain order (e.g., track 2 has an earlier modification time than track 19 for example) then ls might help, but since the source was an audio CD with no mtime data, this didn't occur to me.

What did occur to me, though, was to say "sort -n"; as the manpage tells us (type "man sort")

   -n     Compare according to arithmetic value an initial numeric  string
          consisting of optional white space, an optional - sign, and zero
          or more digits, optionally followed by a decimal point and  zero
          or more digits.
So if we said "ls|sort -n"...
collin@v2:~/tmp/CD-tracks 
% ls *.aiff|sort -n
1 Track 1.aiff
2 Track 2.aiff
3 Track 3.aiff
4 Track 4.aiff
5 Track 5.aiff
6 Track 6.aiff
7 Track 7.aiff
8 Track 8.aiff
9 Track 9.aiff
10 Track 10.aiff
…
78 Track 78.aiff
79 Track 79.aiff
80 Track 80.aiff
81 Track 81.aiff
collin@v2:~/tmp/CD-tracks 
% 
OK, that's more like it. Suppose then we piped all these to cat(1). Umm, let's try with just the first three files.
collin@v2:~/tmp/CD-tracks 
% ls *.aiff | sort -n | head -n3
1 Track 1.aiff
2 Track 2.aiff
3 Track 3.aiff
collin@v2:~/tmp/CD-tracks 
% 
The head command gives the first <howevermany> lines of its input. To say how many lines (the default I think is 10), we use -n# where # is the number of lines we want -- in this case, three.

Given the three lines, how do we pass all of them to cat? Easiest thing is the xargs command. If it had been invented last week, there would probably be a patent application in the works, but fortunately the idea came in an earlier era so all can use it for free. Here's the thing: If you give xargs five lines, then it will append them to the end of a single command line. So if we said for example

collin@v2:~/tmp/CD-tracks 
% ls *.aiff | head -n3 | xargs echo
1 Track 1.aiff 10 Track 10.aiff 11 Track 11.aiff
collin@v2:~/tmp/CD-tracks 
% 
Right? We can do that with cat and send the output to a file we'll call "x", maybe like this:
collin@v2:~/tmp/CD-tracks 
% ls *.aiff | sort -n | head -n3 | xargs cat > x.aiff
cat: 1: No such file or directory
cat: Track: No such file or directory
cat: 1.aiff: No such file or directory
cat: 2: No such file or directory
cat: Track: No such file or directory
cat: 2.aiff: No such file or directory
cat: 3: No such file or directory
cat: Track: No such file or directory
cat: 3.aiff: No such file or directory
collin@v2:~/tmp/CD-tracks 
% 
Were you surprised, or did you wonder how cat would be able to tell where one filename ended and the next began? Let's get rid of the spaces in those filenames. There's any number of ways to do that, but I typed something like this to see if it would work:
collin@v2:~/tmp/CD-tracks 
% for X in *.aiff; do echo mv "$X" ${X// /.}; done | head -n3
mv 1 Track 1.aiff 1.Track.1.aiff
mv 10 Track 10.aiff 10.Track.10.aiff
mv 11 Track 11.aiff 11.Track.11.aiff
collin@v2:~/tmp/CD-tracks 
% 
What's "${X// /.}"? Well, let me go back to the beginning of the line.
  1. for X in *.aiff; do
    says to assign the variable X to successive values of whatever filenames match *.aiff, and execute everything until the done keyword.
  2. echo &hellip
    I want to see what's about to happen, rather than just executing it.
  3. mv "$X" …
    The command mv is the Unix™ command "move", which is how we rename things. What do we rename? The $X says "whatever's in shell variable X".

    Why do I put the $X in "double quotes"? Because that tells the shell to treat the entire name (1 Track 1.aiff for example) as a single "word".

    What new name will we give to $X? That's coming next.

  4. ${X// /.}
    This tells the shell to start with the variable X (which will be "1 Track 1.aiff" the first time through, etc.) and replace all instances of " " by ".". How did I know this? From reading the manpage for the shell. Type "man sh" or "man bash" some evening when you're having trouble getting to sleep:
    BASH(1)                                                          BASH(1)
    
    NAME
           bash - GNU Bourne-Again SHell
    
    SYNOPSIS
           bash [options] [file]
    …
    Parameter Expansion
       The `$' character introduces parameter expansion, command substitution,
       or arithmetic expansion.  The parameter name or symbol to  be  expanded
       may  be enclosed in braces, which are optional but serve to protect the
       variable to be expanded from characters immediately following it  which
       could be interpreted as part of the name.
    …
       ${parameter/pattern/string}
       ${parameter//pattern/string}
          The pattern is expanded to produce a pattern just as in pathname
          expansion.  Parameter is expanded and the longest match of  pat-
          tern  against  its  value is replaced with string.  In the first
          form, only the first match is replaced.  The second form  causes
          all  matches  of pattern to be replaced with string.  If pattern
          begins with #, it must match at the beginning  of  the  expanded
          value  of parameter.  If pattern begins with %, it must match at
          the end of the expanded value of parameter.  If string is  null,
          matches  of  pattern are deleted and the / following pattern may
          be omitted.  If parameter is @ or *, the substitution  operation
          is  applied to each positional parameter in turn, and the expan-
          sion is the resultant list.  If parameter is an  array  variable
          subscripted  with  @ or *, the substitution operation is applied
          to each member of the array in turn, and the  expansion  is  the
          resultant list.  
  5. done;
    see item 1 above.
Convinced that that ought to work now, I do this:
collin@v2:~/tmp/CD-tracks 
% for X in *.aiff; do mv "$X" ${X// /.}; done
collin@v2:~/tmp/CD-tracks 
% ls *.aiff | sort -n | head -n3 | xargs echo cat
cat 1.Track.1.aiff 2.Track.2.aiff 3.Track.3.aiff
collin@v2:~/tmp/CD-tracks 
% 
That's more like it. Now let's do the real thing:
collin@v2:~/tmp/CD-tracks 
% ls *.aiff | sort -n | xargs cat > x.aiff
collin@v2:~/tmp/CD-tracks 
% 
Great! We looked at the size of each file (about 10 Mbytes per track, except the last), and the output file, x.aiff, was about 800 Mbytes, so we were OK. Opening the result in Quick Time Player 7, we found it was only as long as the last track, i.e., less than 40 seconds.

Disillusionment

Well, we went back to the web search, to the post that mentioned using cat, and found that basically, you can't do that.

I did another web search and found that sox can in fact concatenate sound files, including aiff files. Yes! We went to the sourceforge site and downloaded the Mac OS X zip file. I think it was a zip file anyway (actually happened about 12 hours ago). Unlike a lot of Mac OS apps where you click here to install, etc., this one you just unpack and run it. I ended up typing something like

% ~/Downloads/sox-14.2/sox… 
but I'm getting ahead of myself.

First we looked at the documentation for sox, which told us that if you want to concatenate sound files, you put them on the command line, with the output file last. If I recall correctly the example was

% sox short.aiff long.aiff longer.aiff
but of course we had 81 input files. What would the output file be? I would want it to sort last, so I did something like this:
% touch 999.aiff; ls *.aiff | sort -n | xargs ~/Downloads/sox-14.2/sox
Then, opening 999.aiff in Quick Time Player 7, we found that it indeed appeared to be about 81 minutes long.

Technology is great when it works. I copied the result to my USB flash drive, so the lovely Carol can listen to it after I return home.

Remembering Danny

My brother-in-law Danny died a week ago. He was in great pain and ready to go to his reward. I called my sister the day before, and she filled me in a little on his condition. He'd been hallucinating off and on. Did I want to talk with him? She handed him the phone.

"Hi Collin!" he said. He sounded cheerful, or was that just me wanting him not to be in pain? I told him I was sorry he was going through all this. He seemed not to hear me, but said, "I'll call you back, OK?"

No you won't, I thought, but what I said was, "OK." What a miserable situation! He wanted to go, he was deeply uncomfortable, and if he recovered, my sister would have had a horrific time trying to care for him: a leg was due to be amputated, and his heart was going to need a valve job within a year.

And yet how terrible for him not to be there any more!

Well, I wanted to write a few things about him while I can still see.

The lovely Carol mentioned a time when our kids were very young. There were frustrations, but she never heard Danny yell at his boys. She asked him about it, and he explained that he'd put a hand on the kid's shoulder and lower his face to be directly in front of the kid's. (Maybe that was one hand on each shoulder—I don't recall exactly.) His nose no more than 3 inches from his son's, he would say firmly, "You need to put away your toys now." It was not loud, but it was quite effective.

Danny's musical gifts were considerable, and he used them at various churches. At yesterday's memorial service, I kept reminding myself that he wasn't about to appear on stage with his cane and sit down at the piano. Several people mentioned yesterday that he started many musical groups and worked patiently with almost anyone who wanted to sing or play.

What I remember most about him most was his gentle manner and the way he talked about his health challenges. "Everybody's got something," he would say, and he was certainly right about that. But Danny had way more than his fair share.

"I'll call you back, OK?" he said. When I get there, some decades hence I suppose, I'll hold him to that promise.

Sunday, October 20, 2013

An almost-painless OS upgrade: OpenSUSE 12.3

I must be prone to bouts of gratuitous optimism, because this upgrade had its rough spots.

Getting the download image was easy enough. Went to the OpenSUSE website and downloaded it. ("It" here means the 32-bit DVD image.) I used an ancient powerbook to burn a DVD, as my old DVD writer stopped working some time back. I slid the DVD into the drive on this old IBM lease return I got a few years ago for $150 or whatever. It's got 1GB of RAM and a 32-bit P4 processor (bogomips : 6384.75).

It booted fine, with speed what you'd expect from an optical drive. But it took the longest time to probe disk drives and to find Linux partitions. Some time (an hour?) later, there was an error message... I suspected too many retries on the DVD reader.

Fine. Made a bootable USB drive following these instructions and booted it.

Well, it still took too long (IMO) to probe disk drives and find Linux partitions; in fact I gave up once, powered the box down and disconnected the DVD drive cable because I suspected it was goofing things up. It still took an awfully long time (literally hours) but finally I got to the place where it asked me if I wanted to continue.

I clicked the Next button, and then... installation just hung while updating the boot configuration. Huh? The lovely Carol was out, so I slid my chair to her computer and did a web search on "opensuse installation hangs" which landed me here, which included this:

are you sure you are not running out of disk space? (sorry, i know you
are probably have plenty of room, but what partitioning scheme did you
elect to use...that is: is /tmp on its own partition and is it REALLY
REALLY big?)
Whoa! ctl-alt-f2 and "df" -- my root partition was like 93% full. Ouch!

Freed up some space, and things started moving again. Whew! I was working on a shell script for the lovely Carol, and the next time I looked at my Linux box, there was the login screen. Yippee! I typed my login and password and... can't find home directory. Dang!

My homedir is NFS mounted; more than that, it's automounted. So I asked if nfs was running. Yes, but NFS whined at me: "portmap/rpcbind" was not running. OK, chkconfig portmap but apparently OpenSUSE doesn't use "portmap" any more.

$ chkconfig portmap
portmap: unknown service
$ chkconfig rpcbind
rpcbind  off
$ 
O...kay. I started rpcbind, and now nfs was happy. Was automounter running? You guessed it. So I said "chkconfig autofs" and of course it was off.

Enabled that, explicitly started it (going Neanderthal, I typed
$ /etc/init.d/autofs start
then alt-f7 back to the X11 screen, logged out and back in and... voilà! all was good.

I ejected the USB stick (which by now was /dev/sdb) and rebooted, and things pretty much worked. Google-chrome (upon which I'm typing this post) didn't, though; library incompatibility. I downloaded a new RPM, then
$ sudo rpm -Uvh Downloads/google-chrome-stable_current_i386
and after a few minutes (it had to delete the previous version) that was working too.

But wait; there's more!

I decided to try to get mail(1) working. xhost +; sudo yast2 and said I had a permanent connection to the mail server. Then I typed something like
date|mail -s test other-email@yahoo.com
and messages like these appeared:
postdrop: warning: mail_queue_enter: create file maildrop/856011.3977: Permission denied
postdrop: warning: mail_queue_enter: create file maildrop/856204.3977: Permission denied
postdrop: warning: mail_queue_enter: create file maildrop/856417.3977: Permission denied
postdrop: warning: mail_queue_enter: create file maildrop/856738.3977: Permission denied
postdrop: warning: mail_queue_enter: create file maildrop/857467.3977: Permission denied
…and just kept coming. A web search led me to a couple of helpful links: And now I've got working mail too.

Tuesday, October 08, 2013

Matthew Explains Himself

No one knows the Son except the Father, and no one knows the Father except the Son and those to whom the Son chooses to reveal him. from Matthew 11:27

We are going through Matthew in our marvelous Sunday morning group. Last week, our teacher asked, How does Jesus (“the Son” in the passage above) decide who will get to know the Father?

Immediately John 14:23 came to mind, where Jesus says, If anyone loves me, he will obey my teaching. My father will love him, and we will come to him and make our home with him. But another answer, probably a better one, is close at hand. Rather than going to another gospel, we can look around, right where we are. Here's a bit more of the context:

25At that time Jesus said, “I praise you, Father, Lord of heaven and earth, because you have hidden these things from the wise and learned, and revealed them to little children. 26Yes, Father, for this was your good pleasure.

27“All things have been committed to me by my Father. No one knows the Son except the Father, and no one knows the Father except the Son and those to whom the Son chooses to reveal him.

28“Come to me, all you who are weary and burdened, and I will give you rest. 29Take my yoke upon you and learn from me, for I am gentle and humble in heart, and you will find rest for your souls. 30For my yoke is easy and my burden is light.”

Matthew 11:27-30
To whom does the Son reveal the Father? Those who come to him and take his (easy) yoke upon them! Now that's not a mathematical proof, but look at verse 25: things (such as "who is the Father?") have been revealed to little children (i.e., those who follow Jesus—that is, those who came to Jesus) but were hidden from the "wise and learned" who did not.

The knowledge of the secrets of the kingdom

That was last week; this week's passage included a verse about secrets: The knowledge of the secrets of the kingdom of heaven has been given to you, but not to them. (from Matthew 13:11, NIV)

What are the secrets of the kingdom of heaven, and how did Jesus’ listeners get this knowledge? Again my mind went quickly to John (in 8:32 Jesus says, "Then you will know the truth") but why go there when there's plenty right here?

2[H]e got into a boat and sat in it, while all the people stood on the shore. 3Then he told them many things in parables, saying: "A farmer went out to sow his seed. … 9He who has ears, let him hear."

10The disciples came to him and asked, "Why do you speak to the people in parables?"

11He replied, "The knowledge of the secrets of the kingdom of heaven has been given to you, but not to them. 12Whoever has will be given more, and he will have an abundance. Whoever does not have, even what he has will be taken from him.

from Matthew 13:2–12
What am I saying here? As our teacher pointed out this week, Jesus was in a boat, and a crowd stood on the shore. As he spoke, his words landed on various kinds of people in the crowd, just as the farmer's seeds landed on various kinds of soil in the parable.

After Jesus’s closing line, He who has ears, let him hear, many in the crowd said, "Great sermon, Rabbi," and went home with no idea of what Jesus had just said. But some came to him—Matthew calls them "disciples" (i.e., followers)—and said, basically, "Huh?"

It is exactly to these people that Jesus says, "You've got the knowledge of kingdom secrets." The secrets actually come to one big secret, an open one: it's Jesus. To paraphrase Archilochus: the wise and learned know many things; Jesus’s disciples know one big thing.

Thursday, October 03, 2013

Nobody has the right to be this happy

Have you ever had one of those days when you can't believe how fortunate you are? I get together with a small group on Fridays, and we tell each other what God has done, as Psalm 16:23-24 says. At one of these gatherings, I told my friends it's easy for me to love God because he is so good to me.

Tuesday morning started early—earlier than expected, because I had trouble sleeping, worrying uselessly about something or another. Happily, I found this beautiful passage from Merton's No Man Is an Island:

All nature is meant to make us think of paradise. Woods, fields, valleys, hills, the rivers and the sea, the clouds traveling across the sky, light and darkness, sun and stars, remind us that the world was first created as a paradise for the first Adam…. Heaven is even now mirrored in created things. All God's creatures invite us to forget our vain cares and enter into our own hearts, which God Himself has made to be His paradise and our own. If we have God dwelling within us, making our souls His paradise, then the world around us can also become for us what it was meant to be for Adam—his paradise. But if we seek paradise outside ourselves, we cannot have paradise in our hearts.
chapter 6 (Asceticism and Sacrifice) 15 (p. 115)
I went back to sleep with the thought that God is working to make my heart a paradise, for him and for me.

A few hours later I rose for a prayer meeting at church. They even provided coffee! It was a sweet time of fellowship, talking and praying with men and women who so love the church[audio].

After the prayer and fellowship, I drove my '86 Corolla (which still runs great!) to a side street, then ran like heck to catch the just-arriving Caltrain. Little did I know that my younger daughter had seen me running, and was leaving a note for me even as I rode the train—yes, I made it!

At work, I was putting a few finishing touches on a project when the boss cropped by my cubicle. She was happy about my results (we'd had a phone conference with India the night before) and stopped by to congratulate me on well-done job. That's not an everyday occurrence! The other thing was, she asked me how I would feel about working on a certain other project—something I'd mentioned before as an area I'd like to focus. So "I'd love to!" was no surprise.

At lunch, I went for a walk around the block, a little over a mile, and enjoyed the beautiful weather.

I had another chat with the boss after lunch, where she went further and told me to go ahead and spend some time on that new project, managing my time with the other two things I'm working on. She also indicated that there were things I could do that would gain "points" and that would be interesting (if not as much fun as actually writing code) and not all that difficult. I spent the rest of the day working on some code, which as usual put me in a good mood.

Walking out to catch the light rail, I pulled out my phone and called Peet's, because I had coffee duty for the evening meeting at church. The barista/manager said they'd start on those right away; it had been a hectic day, she said. I told her I'd be by just about six (i.e., about 10 minutes later than the "reservation" time) so no need to rush. I congratulated myself for remembering to call ahead.

Returning to my car in Menlo Park, I found the note Sheri had left for me. My heart overflowed with thankfulness. It occurred to me that we can bring joy to God's heart by expressing our love to him.

Dinner was served at our meeting, which is always nice; the meeting itself started with Scripture, a little sharing of things we were thankful for, and prayer. (I didn't say anything because once I started I might have taken the whole time.) There were some significant issues, but I enjoyed the blessings of fellowship and the presence of God's Spirit as we thought and discussed together.

Sing to the Lord, all the earth;
    proclaim his salvation day after day. 
Declare his glory among the nations, 
     his marvelous deeds among all peoples.
Psalm 16:23-24

Wednesday, October 02, 2013

How to Use Your Lottery Winnings

That wasn't the subtitle of Babette’s Feast, but that’s how I took it. Let me explain.

There's a backstory (several actually), but here's a very brief outline: The pastor in a village in Jutland had two daughters, who have spent their lives caring for the poor in their village. Now elderly themselves, they take in an impoverished French cook to help them with their work. One day the news arrives: their cook has won the lottery! She desires to use her bounty to prepare a feast for the village, as the centenary of the pastor's birth approaches. The sisters are reluctant, but are won over by their cook's earnest desire.

Babette prepares her eponymous feast. The sisters assume that after this feast is served, Babette will return to France to live in luxury. They are surprised, however, to learn that Babette has spent all her winnings to host this fabulous meal.

Earlier in life, I would have thought Babette ridiculous, but I begin to appreciate Babette's extravagance.

In reality, is Babette brilliantly generous, or imprudently profligate? What do you think? Suppose Babette died the next morning; would that change your thinking?

And what would/should/could I do with my winnings? Because in a real sense, I have won the lottery: my prize is the days that lie ahead. How should I use them? Who should I bless with them? Because I certainly can't take them with me.

Sunday, September 15, 2013

What made the past week so great

Sorry, this isn't about Assad supposedly agreeing to turn over his stockpile of chemical weapons or about progress containing California fires—great though those things are. And I am grateful that violence and devastation are averted or reduced or at least not increased.

No, what I want to tell you about is just stuff about my life. Starting with about a week ago, when I had a wonderful time interacting with our fellowship group about Jesus in Matthew 8–9. What hit me then, and I am still processing today, is the question: "What is my life about?" Because it's so different from the life of Jesus.

Looking at Jesus in those two particular chapters (which happened to be my teaching assignment) made me see that his life was all about bringing grace and truth into the world.

Grace and truth—an important combination. Earlier in the week, when talking with the lovely Carol about this, I said that Jesus was all about bringing salvation—salvation in the broad sense: freedom from debilitating illness, illusions, alienation &c.—but perhaps “grace and truth” is a better phrase.

My life on the other hand is a lot about staying out of trouble. Yes, I want to do good, and sometimes I actually do. (And now that I think of it, I actually have a lot of joy at times.) But am I all about bringing grace and truth into the world? Healing and mercy? Not so much.

So yes, last Sunday's class was a lot of fun for me. I even wrote a paper (I spent about 5 minutes of class time introducing it) on an engineering/inductive approach to faith and miracles. I'll send you softcopy if you're interested; let me know.

More joy came in the form of writing code. Some of that is for work (I could show it to you, but then my boss would have to kill me) but more of it had to do with a couple of little projects. One was related to "kenken" puzzles. I wrote a solver on last month's vacation, but the input was kinda unfriendly. I mean it looked like this:

    11+ 1A 2A
    2/ 1b 1c
    3- 2b 2c
    20* 1d 2d
    6* 1e 1f 2f 3f
    3/ 2e 3e
    6x 3c 3d
    240* 3a 3b 4a 4b
    …remainder elided
What I did this week was change the front-end to accept input that looks more like this:
11+    2/    <    20*   6*   <
^      3-    <    ^     3/   ^
240*   <     6x   <     ^    ^
>      ^     …remainder elided
So now the kenken solver takes friendlier (or at least not-so-cryptic) input. Am I getting my inner nerd on or what?

My old friend Jan (she's younger than I am; what I mean is we met in 1968, before either of us met Jesus) was in the area, and we had breakfast Friday morning. I heard about some of the joys and disappointments in her family, some close calls and deliverance. Jan exemplifies Colossians 4:6 for me: Let your speech always be with grace, seasoned with salt…, as well as those verses in Psalms that encourage us to tell of [God's] works (see for example Psalm 73:28).

Another cause for celebration: my keys, which had been missing for the past week, were discovered Friday. They once were lost, but now are found. Happy day!

And yesterday, I worked with my daughter Sheri on her new website. Javascript was involved. So was coffee. As of about 9 this morning, it worked. I discovered happily that network solutions have Python 2.6; they do support cgi-bin. The layout is a big quirky, though; if you create a directory "icons" under htdocs, you can't ever get to it, because "/icons" redirects to some magic place where they put their icons. Arrows and dots and such. My icons, which were a few arrows and dots I created with convert(1), were nowhere to be found. So of course I renamed the directory my_icons. Hurmpf.

But it works! The challenge now is to make it easy to update and maintain.

Church was lovely this morning; besides the amusing quote If you have to ask whether you're young or old, you're old, the message was powerful: the importance of community. I heard something recently on NPR about how with economic prosperity comes a breakdown of community. Exactly how does this happen? I'll tell you. I don't know. The basic idea described on the radio was the idea that in poor villages, people have to rely on each other a lot more. But as prosperity comes, people become less reliant on each other; they're more "self-sufficient" (or so they/we tend to think) and therefore become more isolated. Dumb, huh? Yet I resemble that remark. I need to ponder that one too.

The lovely Carol and I went for a walk at Pulgas Ridge this afternoon. Our faithful pup also enjoyed the walk, though it was rather warm. It was great being outdoors and moving our bodies around.

Dinner tonight was fried rice—many leftover veggies make for an interesting dish. The lovely Carol offered to do the dishes since I'd done the cooking.

So I am thankful today. Sometimes I wonder how it was that I got to live in such a time and place as this, to have only minor problems in life, and so on. Not that I'm complaining. Praise God from whom all blessings flow! Amen.

Friday, August 16, 2013

Buying a lei in Kailua (as in Kaneohe, not as in Kona)

Aren't you glad I didn't title this "getting lei'd in Kailua"?

Suppose you're a young newlywed and you want to buy a lei for your honey, and you're staying on the windward side of Oahu. You type "lei" into Google maps and you notice this entry.

Well, it doesn't work. First, 776 Kailua Road isn't where google maps says it is. Second, 776 Kailua Road actually has a surfboard shop and an Arthur Murray studio. So what's a young lei-seeker to do?

Well, it turns out that Walgreen's has leis. At least an orchid lei (a "vanda lei" as I would call it). The folks at the Portuguese Bake Shop told me that Foodland, 7-eleven, and Long's (yes, it's still called Long's here in the islands) all have leis.

But not Lavender Lei, and not at 776 Kailua Road.

Tuesday, August 13, 2013

A Praying Life

The younger ex-teen got this book, A Praying Life by Paul E. Miller (NavPress, 2009). My first comment: Wow!

Chapter 1 is titled “What good does it do?”—something we've asked in the quiet (or noise) of our hearts, but the author recounts a time when he said to his then-teenager, “Let’s pray” (about a missing contact lens). In response she

… burst into tears. “What good does it do? I’ve prayed for Kim to speak, and she isn’t speaking”

Kim struggles with autism and developmental delay. …[S]he is also mute.

page 15
Indeed. What good does prayer do? Doesn't God already know what we need before we ask?

The author does address that question, and many others. I'll tell you about this other part from chapter 4, “Learn to talk to your father”—related to Matthew 7:11 If you, then, though you are evil, know how to give good gifts to your children, how much more will your Father in heaven give good gifts to those who ask him!

Kim got her first speech computer when she was five years old. … We explained the keys to her and waited. She leaned over and pressed the key with little McDonald’s golden arches on it. It was two o’clock in the afternoon, and we’d just eaten lunch. We dropped everything, leapt into the car with Kim raced off to McDonald’s, and got Kim a hamburger and a soda. We were thrilled.
page 58
What a lovely picture of our heavenly father's delight in giving good gifts to his children!

Another thing that's impressed me so far: when Jesus says we need to become like little children, we also need to pray like little children. Quiz: How do little children ask for things?

Well, messy might be a good word. They ask distractedly, repeatedly, annoyingly; they ask with abandon, without pretense or consideration. I used to hang around with a guy who said his default method of praying involved listing a dozen or two things he wanted from God. And he talked about that as though it were a bad thing.

But really, if that's on our minds, that's what we should talk about. I'll chew on that one for a while.

Tuesday, July 02, 2013

Why fast?

Our self-denial is sterile and absurd if we practice it for the wrong reasons or, worse still, without any valid reason at all.
Merton, No Man Is an Island 6.7 (p. 101)
So what's a reason that's not wrong? What's a valid reason for fasting?

For a while I had a habit of fasting on Sundays: I'd skip lunch. But why did I do it? Well, I didn't have a valid reason, and I stopped after a while.

So what is a valid reason? Merton helps us again here:

… The perfection of Christian renunciation is the total offering of ourselves to God in union with the sacrifice of Christ.

To offer this sacrifice perfectly we must practice asceticism, without which we cannot gain enough control over our hearts and their passions to reach such a degree of indifference to life and death.

Merton, op. cit., p. 102
So that's the point: to gain control over our hearts and their passions, so that we can become indifferent to food or whatever we're attracted (or addicted) to, so that we can get closer to the ideal: to offer our whole selves to God, to be indifferent to everything except the will of God.

And that's a valid reason: to be free of our own appetites and passions, not just regarding food, but anything that one might be (or become) addicted to. As Paul writes to married couples in 1 Corinthians 7.5, which I had not understood before: Do not deprive each other except by mutual consent and for a time, so that you may devote yourselves to prayer.

The point, of course, isn't that husbands and wives spend so much time in bed that they have no time to pray; rather, it's that by abstaining voluntarily for a (brief) time they can surrender more of themselves to God.

For a brief time, that is.

Let's see if I can free up 11GB (or: finding and taking care of duplicate files on a mac mini)

The lovely Carol has a mac mini, which currently serves[sic] as an NFS server for my desktop. It also backs up our laptops. This machine, which is backed up off-site, has a huge hard drive that I thought would keep us in disk space for a long long time.

You can guess what happened: photos and music—especially photos—tend to expand to fill the space available. It doesn't help that we have multiple copies of stuff. So I thought to run some Perl or Python script to help me find said copies.

Since I've become a Python partisan I went that route. A web search turned up some helpful hints on stackoverflow and particularly this post on endlesslycurious.com. The mac mini has python2.6, so I made a few modifications; you can see the whole thing at http://cpwriter.net/dup2/.

I ran that on /Users on the lovely Carol's mac mini, putting the results into dups.out.

mini1:~ collin$ wc -l dups.out
   12489 dups.out
mini1:~ collin$ 
Yep, that's a lot of files. A couple of big offenders:
mini1:~ collin$ grep Best.*Wedding dups.out
[621850501, ['/Users/carol/from-macbook/Movies/Best of Wedding.mov', \
   '/Users/collin/from-pbook/Desktop/Best of Wedding.mov']]
[989954048, ['/Users/carol/from-macbook/Desktop/Redeemer Marriage Series/Best of Wedding-DVD.img', \
   '/Users/collin/from-pbook/Desktop/Best of Wedding-DVD.img']]
mini1:~ collin$ 
That's 621Mbytes and 989Mbytes. So about 1.5GB freed up just like that. But I think we have a lot more. I discovered a lot of files under "archives" and "from-pbook" that are the same, like this:
mini1:~ collin$ grep archives dups.out|grep -m5 from-pbook
[1049177, ['/Users/collin/archives/collin-laptop/Pictures/iPhoto Library/2009/01/01/IMG_3001.JPG', \
   '/Users/collin/archives/data1/pix-dec08/img_3001.jpg', \
   '/Users/collin/from-pbook/Pictures/iPhoto Library/2009/01/01/IMG_3001.JPG', 
   '/Users/collin/pix/2008/12/pix-dec08/img_3001.jpg']]
…
Wow, four paths to the same file. Hey, can I get rid of all those pix-dec08 paths? Yes, because:
  1. A "diff -r archives/data1/pix-dec08 pix/2008/12/pix-dec08" showed that these two directories are identical;
  2. every "large" (not a thumbnail or slide) image file under pix/2008/12/pix-dec08/ appeared in dups.out. Except those under 1024×1024 bytes:
    mini1:~ collin$ for F in pix/2008/12/pix-dec08/*jpg; do if grep -qF $F dups.out; then : OK; else ls -l $F; fi; done       
    -rwxr-xr-x  1 collin  _lpoperator  1002328 Dec 31  2008 pix/2008/12/pix-dec08/img_2961.jpg
    -rwxr-xr-x  1 collin  _lpoperator  858104 Jan  1  2009 pix/2008/12/pix-dec08/img_2988.jpg
    -rwxr-xr-x  1 collin  _lpoperator  863361 Jan  1  2009 pix/2008/12/pix-dec08/img_2994.jpg
    -rwxr-xr-x  1 collin  _lpoperator  865777 Jan  1  2009 pix/2008/12/pix-dec08/img_2995.jpg
    -rwxr-xr-x  1 collin  _lpoperator  994298 Jan  1  2009 pix/2008/12/pix-dec08/img_2996.jpg
    -rwxr-xr-x  1 collin  _lpoperator  811491 Jan  1  2009 pix/2008/12/pix-dec08/img_2997.jpg
    mini1:~ collin$
I'm going to take the leap of faith that the remaining files are in fact there in the other paths... well, no I won't:
mini1:~ collin$ for F in pix/2008/12/pix-dec08/*jpg; do if grep -qF $F dups.out; then : OK; else \
     Y=`basename $F|tr [:lower:] [:upper:]`; \
     Z=`/bin/ls from-pbook/Pictures/iPhoto\ Library/200*/*/*/$Y`; \
     echo $Z;cmp "$F" "$Z"; echo; fi; done                                                          
from-pbook/Pictures/iPhoto Library/2008/12/31/IMG_2961.JPG

from-pbook/Pictures/iPhoto Library/2009/01/01/IMG_2988.JPG

from-pbook/Pictures/iPhoto Library/2009/01/01/IMG_2994.JPG

from-pbook/Pictures/iPhoto Library/2009/01/01/IMG_2995.JPG

from-pbook/Pictures/iPhoto Library/2009/01/01/IMG_2996.JPG

from-pbook/Pictures/iPhoto Library/2009/01/01/IMG_2997.JPG

mini1:~ collin$ 
So we can kill off those two paths. That might have saved another Gbyte or so.

Now, can we maybe hardlink the /Users/collin/archives/collin-laptop/Pictures/ stuff to/from the /Users/collin/from-pbook/Pictures/ stuff? And how much space might that save?

mini1:~ collin$ du -sh archives/collin-laptop/Pictures/iPhoto\ Library/ from-pbook/Pictures/iPhoto\ Library/                                    
 12G archives/collin-laptop/Pictures/iPhoto Library/
 11G from-pbook/Pictures/iPhoto Library/
mini1:~ collin$ 
Quite a bit. That plus the 1.5GB already saved earlier would be a significant help here:
collin@p3:/mnt/home/collin> df -h .
Filesystem            Size  Used Avail Use% Mounted on
mini1:/Users          298G  257G   42G  87% /mnt/home
collin@p3:/mnt/home/collin> ssh mini1 df -h .
Filesystem     Size   Used  Avail Capacity  Mounted on
/dev/disk0s2  298Gi  256Gi   41Gi    87%    /
collin@p3:/mnt/home/collin> 
Not sure why the difference, but there it is. Anyway, I wanted to hardlink one set of files to the other. (Why? Because the from-pbook directory may get rsync'd. If I delete the from-pbook directory, then it may come back later. And if I delete the other directory, and subsequently decide to remove the files from the pbook, then we'll lose the photos. So hardlink is the way to go.) Consequently I wrote this silly script:
collin@p3:/mnt/home/collin> cat tmp/photos.sh 
#!/bin/sh
D2="archives/collin-laptop/Pictures/iPhoto Library"
D1="from-pbook/Pictures/iPhoto Library"

find "$D1" -type f | while read AFILE; do
    SUB=${AFILE#$D1/}
    #echo SUB=$SUB
    BFILE=$D2/$SUB 
    if [[ -s $BFILE ]] && [[ ! "$AFILE" -ef "$BFILE" ]] && cmp -s "$AFILE" "$BFILE"; then 
        if [[ $AFILE -ot $BFILE ]] ; then
     echo ln -f "'$AFILE'" "'$BFILE'"
 else
     echo ln -f "'$BFILE'" "'$AFILE'"
 fi
    fi
done
collin@p3:/mnt/home/collin> time tmp/photos.sh > foo.out

real 41m18.878s
user 0m11.009s
sys 0m33.146s
collin@p3:/mnt/home/collin> 
then ran it as you see above. A quick sanity check of "foo.out" looked reasonable. Ah, I probably should have run it on mini1, rather than on the NFS client. And the same here:
collin@p3:/mnt/home/collin> df -h .; ./foo.out; df -h .
Filesystem            Size  Used Avail Use% Mounted on
mini1:/Users          298G  257G   42G  87% /mnt/home
-bash: ./foo.out: Permission denied   # D'oh! I didn't say "chmod +x"; well, let me fix it the easy way...
Filesystem            Size  Used Avail Use% Mounted on
mini1:/Users          298G  257G   42G  87% /mnt/home
collin@p3:/mnt/home/collin> df -h .; sh ./foo.out; df -h .
Filesystem            Size  Used Avail Use% Mounted on
mini1:/Users          298G  257G   42G  87% /mnt/home
Filesystem            Size  Used Avail Use% Mounted on
mini1:/Users          298G  246G   53G  83% /mnt/home
collin@p3:/mnt/home/collin> 
OK, that's enough for now.

Wednesday, June 26, 2013

A gift at the supermarket

The other day I received a gift while shopping at our neighborhood grocery store.

It wasn't a gift from Key Market; it was a gift from God. Here's what it was: a sense of wonder and gratitude. I was heading over to the baking aisle, and I thought of how wonderful it was to be in a land of plenty, where flour and sugar and salt and baking powder and dozens of herbs and spices, are all on the shelf. And not just flour, but bags of the stuff in various sizes, "all-purpose" flour and bread flour and cake flour, whole wheat flour and organic flour and unbleached flour. And salt—which used to be so valuable that people were paid in it (thus our word salary)—plain salt and iodized salt, sea salt and kosher salt and ice cream salt, from various makers and in various sizes.

And milk! Whole milk and cream and half-and-half and reduced fat and nonfat and organic, soy "milk" in various flavors, almond "milk" and Mocha Mix® all right there with "sell by" dates stamped on them. Fresh eggs are nearby, too—white eggs and brown eggs by the dozen, organic and low-cholesterol eggs, eggs laid by cage-free hens. The grocery store is a marvel, and even if not everything is "health food," the store has a variety of food and drink (and housewares and personal-care products, not to mention pet food and stationery) not imaginable to people a century ago (or even a few decades), or to many people today born in, say, Mali.

Which reminds me of a time when Solomon "made silver and gold as common in Jerusalem as stones" (2 Chronicles 1:15). Solomon may have been richer than anyone else in history, but a middle-class resident of Jerusalem could not have imagined what you and I can see at the neighborhood market.

But here in the United States, and in particular in Silicon Valley, we do live in a sort of Solomon's Jerusalem. The wealth and consumption (and destruction) available to some folks at a whim—astonishing. And even those of us in the "middle-class" truly are rich, as in the richest 1% or 2% of people on the planet.

So that was my gift: a reminder that by any sane measure, I'm rich and fortunate beyond what anyone has a right to deserve. And so, for some time anyway, I complained a bit less than I usually do, I experienced a bit more joy than I usually do. And I hope I also spread a bit more joy, a bit less frustration and consternation than usual.

But this morning I found things to grumble to God about. Wretched man that I am! Thanks be to God that there is no condemnation for us in Christ. And that he will, according to his promise, sanctify us (1 Thessalonians 5:23f) and complete the work he began.

Thursday, June 20, 2013

But wait, Spot-It! has only 55 cards.

In a previous post and its follow-up I described the process by which I decided that Spot-It! likely used 57 symbols and that it could have up to 57 cards in its deck. But as mentioned earlier, the deck actually has only 55 cards. Did they just leave two cards out? Do they have more symbols, or fewer?

Yes, they just left two cards out; no, they use exactly 57 symbols. To figure this out, I created a file "cards.txt", with the name of each symbol on a single line. The symbols on a given card are grouped together in an eight-line stanza; an empty line separates stanzas.

yin-yang
paint
tree
lightning
zebra
clef
ladybug
Canada

bomb
dragon
carrot
paint
ghost
question
bang
clown

spider
…etc.
The excerpt at right signifies that:
  • one card contains the yin/yang sign, the paint dots, tree, lightning bolt, zebra, (treble) clef, ladybug, maple leaf;
  • another card has the bomb, dragon, carrot, paint, ghost, question mark, exclamation point…
you get the idea.

Alert readers may notice that "cards.txt" has names consisting of only one "word" (by which I mean a contiguous sequence of non-whitespace characters) each, and that some of them don't exctly match my summary. The one-"word" thing makes them easier for shell scripts to process, and I translate these internal names into less-cryptic (I hope) things like "maple leaf" (Canada) or "exclamation point" (bang).

The format of "cards.txt" made for easy error checking, which as it turned out was necessary. (If you're thinking at this point, "How anal!" all I can say is "guilty as charged, yer honor.") It also made for some easy analysis. For example, we learn that although most of the symbols appear on 8 cards each, some don't:

% grep -v '^$' cards.txt | sort | uniq -c | sort -n | head -n16
   6 snowman
   7 Canada
   7 bang
   7 cactus
   7 daisy
   7 dinosaur
   7 dog
   7 eye
   7 ice
   7 ladybug
   7 light-bulb
   7 man
   7 question
   7 skull
   7 stop
   8 anchor
% grep -v '^$' cards.txt | sort | uniq -c | sort -n | tail -n3
   8 web
   8 yin-yang
   8 zebra
% 
The (American) English translation of the above is: only six cards have a snowman. Fourteen symbols (maple leaf, exclamation point, cactus, daisy, etc.) appear on only 7 cards each; the rest appear on 8 cards each. This suggests that two cards could be added to the deck. Each card would have a snowman and 7 of the other symbols. Which symbols? Well, any symbol that would appear with the maple leaf (aka "Canada") on a new card must not already appear with the maple leaf on another card. How to decide that?

It takes a few steps. First, a shell script converts "cards.txt" into another file, "cards-sorted.txt":

% L1=1; while [[ $L1 -lt 490 ]] ; do 
    ((L2=L1+7)); sed -n $L1,${L2}p cards.txt | sort | fmt -w222; 
    ((L1=L1+9)); done | 
    sort > cards-sorted.txt
% 
This takes, e.g., lines 1-8 of "cards.txt", sorts them, and combines them into a single line. Then it takes lines 10-17 (that's eight lines) and does the same with them. Then lines 19-26, and so on.

Each line in "cards-sorted.txt" thus corresponds to a single card, and contains the card's one-word symbol names in alphabetical order. Consequently, cards-sorted.txt starts off like this:

% head cards-sorted.txt 
Canada anchor carrot cheese clock knight stop web
Canada apple bang igloo moon scissors snowflake spider
Canada art balloon bomb drop fire lips skull
Canada bottle candle ghost light-bulb lock pencil sunglasses
Canada car cat clover clown dog ok sun
Canada clef ladybug lightning paint tree yin-yang zebra
Canada dolphin dragon eye hand heart key target
anchor apple art dinosaur dolphin ghost ladybug ok
anchor balloon clown lightning snowman spider sunglasses target
anchor bang car clef daisy key lips lock
% 
Now I'll use "cards-sorted.txt" to see which of those fourteen symbols appear together with "Canada" (maple leaf) on any card:
% f() { grep Canada.*$1 cards-sorted.txt; }
% C=; D=; grep -v '^$' cards.txt |sort | uniq -c | sort -n | grep 7 | 
    while read A B; do 
       echo === $B ===
       if f $B; then C="$C $B"; else D="$D $B"; fi 
       echo C=$C
       echo D=$D
    done
… [much output deleted]
C= bang dog eye ladybug light-bulb skull stop
D= Canada cactus daisy dinosaur ice man question
% 
The "C=" symbols already appear together with "Canada" (maple leaf), so we won't put them on the same card as a maple leaf again. Therefore, I think that the following cards, if added, would "complete" a set of 57 cards:
  • snowman, exclamation point, dog, eye, ladybug, light bulb, skull, STOP!
  • snowman, maple leaf, cactus, daisy, dinosaur, ice cube, man, question mark
I wasn't quite confident in this, so I wrote this brief shell script to check it.
% cat checkit.sh 
#!/bin/sh
# Ensure that for any pair of symbols within $C [or $D], no single card 
# contains both members of the pair.

C="bang dog eye ladybug light-bulb skull stop"
D="Canada cactus daisy dinosaur ice man question"

all_combos() {
    while [[ $# -ge 2 ]] ; do
        first=$1
        shift
        for it in $*; do
            echo checking $first $it ...
            if grep $first cards-sorted.txt | grep $it; then 
                echo ERROR $first $it 
            fi
        done
    done
}

all_combos $C
all_combos $D
% 
A visual inspection of the output revealed that we were indeed checking for "bang" and "dog" together, then "bang" and "eye"... and so on. It did all that without ever printing "ERROR", so I think the list is good.

That sure was fun! But let me check one more time. I'll add the above two cards to the list of sorted cards:

% diff cards-sorted.txt  cards-complete.txt 
55a56,57
> snowman bang dog eye ladybug light-bulb skull stop
> snowman Canada cactus daisy dinosaur ice man question
% 
Then let me ensure that given any pair of symbols on a given card, that that card is the only one containing that pair:
% cat check-complete.sh 
#!/bin/sh
# based on checkit.sh -- for every card in the 
# hypothetical "complete" Spot-It! deck (in "cards-complete.txt"),
# extract every pair of symbols. If both members of the pair appear
# on any other card in the complete deck, then print "ERROR"...
TEMPFILE=/tmp/spot-tmp.$$
all_combos() {
    while [[ $# -ge 2 ]] ; do
        first=$1
        shift
        for it in $*; do
            echo checking $first $it ...
            if grep " $first " $TEMPFILE | grep " $it "; then 
                echo ERROR $first $it 
            fi
        done
    done
}
C=cards-complete.txt 
cat $C | while read X; do
    grep -v "$X" $C | sed -e 's/^/ /' -e 's/$/ /' > $TEMPFILE
    all_combos $X
done
rm -f /tmp/spot-tmp*
% 
The above script, "check-complete.sh", completed without ever saying "ERROR".
% ./check-complete.sh > x
% grep -m5 ERROR x
ERROR Canada igloo
ERROR igloo question
ERROR igloo man
ERROR ice igloo
ERROR cactus igloo
% 
To make sure that it actually works, though, I changed the last card to say "igloo Canada…" rather than the correct "snowman Canada…". The script did in fact catch the error, as you can see at right.

Am I satisfied now? I refer you to the "anal" comment above. (In other words, No.) One more thing: the above check-complete.sh verified that we didn't have any PAIR of symbols in common between any pair of cards. What it didn't do is verify that there was in fact any symbol in common between any pair of cards. That's done by this script:

% cat overlaps.sh
#!/bin/sh
# Confirm that at least one symbol overlaps every card vs every other card.
TEMPFILE=/tmp/spot-tmp.$$

check() {
    # Given two cards ($1, $2), ensure exactly one word is in common.
    if [[ $# -ne 2 ]] ; then
        echo "check() called with wrong # of args"
        exit 1;
    fi
    ACARD="$1"
    BCARD="$2"
    NUM=0
    for asym in $ACARD; do
        for bsym in $BCARD; do
            if [[ $asym == $bsym ]] ; then
                ((NUM=NUM+1))
            fi
        done
    done
    if [[ $NUM -ne 1 ]] ; then
        echo ERROR: card1=$ACARD
        echo ERROR: card2=$BCARD
    fi
}

verify_one() {
    # Handle one card's syms (args) vs the rest of the deck ($TEMPFILE)
    ONE="$*"
    cat $TEMPFILE | while read IT; do
        echo checking $ONE ==vs== $IT
        check "$ONE" "$IT"
    done
}
C=cards-complete.txt 
cat $C | while read X; do
    grep -v "$X" $C | sed -e 's/^/ /' -e 's/$/ /' > $TEMPFILE
    verify_one $X
done
rm -f /tmp/spot-tmp*
% 
This completed without error, but just to make sure, I modified the last card to use nonexistent symbol "snowman2" and re-ran it, yielding:
% ./overlaps.sh > x; grep -m6 ERROR x
ERROR: card1=anchor balloon clown lightning snowman spider sunglasses target
ERROR: card2=snowman2 Canada cactus daisy dinosaur ice man question
ERROR: card1=apple bomb cat hand lock snowman tree web
ERROR: card2=snowman2 Canada cactus daisy dinosaur ice man question
ERROR: card1=art candle carrot key moon snowman sun yin-yang
ERROR: card2=snowman2 Canada cactus daisy dinosaur ice man question
% 
Of course, each pair of cards shown (there are lots more) should have the "snowman" in common. By tweaking the last card, I removed that sharing. So the script works, and the hypothetical deck is correct.

Wednesday, June 19, 2013

More Spot-It! Math

In a previous post, I mentioned some work with pencil and paper that led me to a certain formula. This post adds a some detail, and I hope it'll be easier to follow. As you may recall from that previous post, I use M to refer to the number of symbols on a given card, and N to refer to the number of distinct symbols (or "alphabet") from which the symbols are drawn. So, continuing on, here's my exploration of the puzzle starting with M=3.
1|ABC
2|A
D
F
E
G
3|A
4|B
DF
5|BEG
6|CDG
7|CEF
To approach the problem I decided to list each card's symbols in increasing alphabetical order. Please refer to the table at left. The progression A-B-C, A-D-E, A-F-G is perhaps obvious. What of the B-x-x cards?

After each "B", we take vertical slices of the dotted blue square, the first slice being the D F in the thin black border. The thing to note with the "B"s is that the "F" is directly below the "D" (i.e., it's down one row and zero symbols to the right).

Similarly on card 5, B-E-G, the "G" is directly below the "E" in the dotted-blue square.

Regarding the "C"s: note on card 6, C-D-G, that the G is below and to the right of the D in the dotted blue square. Card 7 (C-E-F) shows the principle (we'll expand on this later) that when "below and to the right" runs off the right-hand edge of the dotted blue square, we simply resume counting from the left-most symbol in the same row within the dotted blue square. Thus card 7's C-E-F.

So for cards matching the pattern B-x-x, the 2nd "x" is directly below (i.e., displaced zero symbols to the right) of the 1st "x". For C-x-x, the 2nd "x" is displaced one symbol to the right of the 1st "x". How did I arrive at this? Trial and error and eventual "success."

This solution is symmetric in that each of the 7 symbols appears on 4 cards; any given pair of symbols appears upon one and only one card. (Wait, does every possible pair of symbols appear on a card? Yes.)

Suppose we have M=4, N=13? Does the pattern above generalize? What will we do with cards D-x-x-x—would each subsequent "x" be one row below and two symbols to the right of its predecessor "x"?

1|ABCD
2|A
E
H
K
FG
I J
L M
3|A
4|A
5|B
EHK
6|BFIL
7|BGJM
8|CEIM
9|CFJK
10|CGHL
11|D E J L
12|DFHM
13|DGIK
In a word, "Yes." As the table at right shows, the B-x-x-x cards take vertical slices (a subsequent "x" is directly below the preceding "x"). The 3rd (4th) symbol in a C-x-x-x card is one symbol down and to the right of the 2nd (3rd) symbol within the dotted blue square, as C-E-I-M. Again, if you'd run off the right-hand end of a row, continue counting from the left, staying within the dotted blue box, as in card 9, C-F-J-K.

In the D-x-x-x cards, symbols are one down and two to the right of their predecessors within the dotted blue square, again resuming counting from the left when we run off the right-hand side of a row: thus D-E-J-L, D-F-H-M, D-G-I-K.

As in the M=3,N=7 case, we have each of the N symbols appearing on M cards each; every possible pair of symbols appearing on one card; every pair of cards having one and only one symbol in common. At least so far, then, the solution generalizes thus:

  1. one card with the first M symbols (e.g., A-B-C-D in the table at right)
  2. M-1 cards, which include the (M-1)×(M-1) blue-dotted square region
  3. (M-1)2 cards (in the table at right, three B-x-x-x cards, three C-x-x-x cards, three D-x-x-x cards). Each set of M-1 cards takes one symbol from each row of the blue-dotted square:
    • the B-x-x-x cards take vertical slices, i.e., each subsequent symbol is zero columns to the right of its predecessor in the previous row;
    • in the C-x-x-x cards, each subsequent "x" comes from one column to the right of its predecessor in the previous row (and if you're about to run off the end, resume counting at the far left of the same row);
    • in the D-x-x-x cards, each subsequent "x" comes from two columns to the right of its predecessor in the previous row (and if you're about to run off the end, resume counting at the far left of the same row);
    • … and so on
    If (M-1) is a prime number, then the above procedure yields M-1 sets of M-1 symbols in the blue-dotted square. Each set of M-1 symbols has one from each row in the blue-dotted square. And each set has at most one symbol in common with any other set. For example, E-H-K and E-I-M have only the "E" in common. And either set has only the "E" in common with E-J-L.
    Exercise: What if the (M-1)×(M-1) square is 4×4? Would the D-x-x-x-x cards run into some kind of trouble? (Answer: Yes. And I haven't figured my way out of it.)
This yields (M(M-1)+1) cards total, and the same number of symbols. As suggested in my snarky "Exercise:" comment, the above solution works unaltered only when M-1 is a prime number—which, fortunately for me, is true when M is 8.

And that is true for Spot-It!—each card has eight symbols (M=8), and the "optimum" value of N is 8*7+1=57 symbols, which allows for a deck of 57 cards. In my "ideal" (symmetric) deck, each of the 57 symbols would appear eight times (i.e., once on each of eight cards).

How does this relate to the actual Spot-It! cards? More on that in a subsequent post.

Saturday, June 15, 2013

A "Spot It!" puzzle

The game of “Spot It!” (blueorangegames.com) provided an entertaining mathematical and programming puzzle, which I have mostly (I think) figured out.

In case you haven't seen it, there's a deck of some 55 cards; on each card are eight stylized figures of various creatures or things: man, dragon, dog, cat, car, daisy, maple leaf, and so on. Any two cards in the deck have one and only one figure in common: cards 1 and 2 might both have a man, cards 2 and 3 might both have a dragon, but then card 1 won't have a dragon; card 3 won't have a man.

This strikes some people as odd. “Really?” they say. “Any two cards have exactly one thing in common?” It is indeed true. Which of course made me wonder: How does that work? If each card has M figures out of an alphabet of N, how many cards can you have?

Naturally, I started with M=2 and came upon this rather simple set of 3 cards with N=3 symbols:

A B  
A   C
  B C
If we have N=4 then one could imagine cards (A,B), (A,C), (A,D), but then what? You can't go (B,C) because that has nothing in common with (A,D). So bumping N from 3→4 actually reduces the number of cards with M=2.

Thus with M=2 there's a sort of optimum value of N, namely 3. What's the optimum value of N for M=3 or M=4? The number of cards might have something to do with the number of combinations of N things taken M at a time, but it doesn't say much about the optimum N for a given M.

After many stabs at this with pencil and paper, I came up with the following lists for M=3 and M=4:

1 |A B C        
2 |A     D E    
3 |A         F G
4 |  B   D   F  
5 |  B     E   G
6 |    C D     G
7 |    C   E F  
 
1 |A B C D                  
2 |A       E F G            
3 |A             H I J      
4 |A                   K L M
5 |  B     E     H     K    
6 |  B       F     I     L  
7 |  B         G     J     M
8 |    C   E       I       M
9 |    C     F       J K    
10 |    C       G H       L  
11 |      D E         J   L  
12 |      D   F   H         M
13 |      D     G   I   K    
The pattern that seems to emerge from this is that we have
1. card with figures {1..M}
2. card with figures {1, M+1..2M-1} (M-1) cards M sets of (M-1) cards
M. card with figures {1, (M-1)(M-1)+2..M(M-1)+1}
(M-1)(M-1)+2. card with figures {M,M+1} and (M-2) figures from the set {2M..M(M-1)+1} (M-1) cards
M(M-1)+1. card with figures {M,2M-1} and (M-2) figures from the set {2M..M(M-1)+1}
…the total being M(M-1)+1 figures in the alphabet, and M(M-1)+1 cards.

I believe the optimal N for a given M is M(M-1)+1; this gives a symmetric (I think) set of cards, with each symbol appearing on M cards. I haven't proven any of this, but I have (of course) written some code that generates a set of these things for the case where M-1 is prime—which is the case for the real Spot It! game (M=8). In particular if M=8 then the optimum N would be 8*7+1=57, and the deck could have 57 cards. Why does the real Spot It! card game have only 55 cards in the deck? You got me.

I'll add a link to the code one of these days but right now I think it's time for bed.
Update: Code is here

Sunday, June 09, 2013

The Aussie Cure For Tired Old… Bathtubs!

Short version: Accident leads to astonishing discovery: Aussie shampoo cleaned up stains on our old bathtub that scrubbing never did. Details follow.
Our bathtub is old—maybe older than I am. On its floor was (I thought) a permanent layer of dirt, maybe some sort of stain. I tried the usual chemical helps (Bon Ami, X-14®, Formula 409®) including a lot of elbow grease, all to no avail. So I gave up years ago on ever seeing the original finish. But one day, we had a happy accident: somebody stood up a bottle of Aussie shampoo, upside-down, without first securing the lid. It was a big spill.

The bottle was mostly empty, but where the pool of shampoo had been, the stains were gone, like the old gospel song: My stains were washed away / And my night was turned to day (or maybe that was "My sins were…").

In any case, I used a lot more shampoo on the bathtub floor, toward the top and right of the photo you see at right. I tried a couple of other brands, which I won't mention here. The Aussie outperformed them.

Yesterday morning I decided to take care of the remaining 40% or so of the tub. The surface was mostly dry (this is important). I squirted liberally from the giant bottle. The stuff sorta piles up, as you can see in the photo below/left.

I kept squirting, and eventually ended up with a layer over the entire stained region. This requires a dry tub surface, lest the shampoo slide off the stain and toward the drain end of the tub. When I said "liberally" I mean I might have used half the bottle. I uploaded a high-resolution photo of the result (click on the photo below/right).

The shampoo sat for maybe eight hours; after that, it was, well, shiny. I shot a photo where you can see the reflection of the camera:

How did it work? Well, I stepped into the shower (ugh! my feet were dirty!) and rinsed the shampoo away. It took a lot of rinsing, but no scrubbing. Here's what it looked like at the end:
The 50-year-old stains are gone, and the tub looks better than it has in years. I should have cleaned up the footprints before taking the picture. Next time.