On why debugging OpenSSH can be so hard

When OpenSSH works and I will say it does that most of the time, it's 
great. But when it doesn't and it has to be debugged, it can be such 
a pain.

At one point, it was not working, complaining with the message

debug3: mm_answer_keyallowed: key 0x67dc00 is disallowed

Now I just couldn't figure this out. Why did it not like my key?

It turns out it was intentionally presenting me with misinformation. 
You see, there was no key file present where it was looking. I 
thought it was looking in one place, but it was actually looking in 
another.

And when I looked at the source code, I found out why it wasn't 
telling me this.

/* Fail quietly if file does not exist */
        if (stat(file, &st) < 0) {
                /* Restore the privileged uid. */
                restore_uid();
                return 0;
        }
        /* Open the file containing the authorized keys. */
        f = fopen(file, "r");
        if (!f) {
                /* Restore the privileged uid. */
                restore_uid();
                return 0;
        }

Fail quietly, indeed! It's not simply doing this under ordinary 
operation, but even in debug operation, even under debug level 3. In 
the perfect place where it can tell us quite informatively what's 
about to go wrong--there is nothing!

And that is going to be eventually lead the program down a wrong path 
somewhere later on, which, ultimately, leads it to cough out the 
highly inaccurate error "key 0x67dc00 is disallowed".

This then looks like a great opportunity for the developers to help 
us out in the future. In fact, I even thought it was simply an 
oversight not have this logged under debugging. So I dutifully filed 
a bug report, https://bugzilla.mindrot.org/show_bug.cgi?id=1388, then 
eight months passed. And as I said earlier, the misleading error 
turns out to be intended behavior.

So in case you're wondering why debugging OpenSSH can be so hard, now you know.
--

Maurice Volaski, mvolaski@aecom.yu.edu
Computing Support, Rose F. Kennedy Center
Albert Einstein College of Medicine of Yeshiva University