Problem with AllowUsers?

I installed 3.9p1 a few weeks ago and restarted the daemon.  I had no problem.  
I then added an AllowUsers line in my sshd_config file, and reloaded (I 
believe).

Today, after I rebooted the server (RH9, 2.4.20-37.9), I could not log in as 
myself.  Thankfully, I had a different way in.

Here is what my messages say and my AllowUsers line from sshd_config:
Dec  8 17:40:21 sm sshd[2712]: Received signal 15; terminating.
Dec  8 17:40:21 sm sshd: sshd -TERM succeeded
Dec  8 17:40:33 sm sshd[4226]: Server listening on 0.0.0.0 port 22.
Dec  8 17:40:33 sm sshd:  succeeded
Dec  8 17:40:36 sm sshd[4230]: User esw not allowed because not listed in 
AllowUsers
Dec  8 17:40:36 sm sshd[4230]: Failed none for invalid user esw from 
68.126.206.89 port 51727 ssh2
Dec  8 17:40:39 sm sshd[4230]: error: Could not get shadow information for 
NOUSER
Dec  8 17:40:39 sm sshd[4230]: Failed password for invalid user esw from 
68.126.206.89 port 51727 ssh2
Dec  8 17:40:40 sm last message repeated 2 times
Dec  8 17:40:45 sm sshd[4235]: Accepted publickey for sidekick from 
68.126.206.89 port 51728 ssh2
Dec  8 17:41:07 sm su(pam_unix)[4292]: session opened for user esw by 
sidekick(uid=504)
...
AllowUsers esw, sidekick

As you can see, the other user has my public key also.  So, for the time 
being, I have a backdoor.

Now, why can't I get in as myself, esw?

Thanks