Possible SSH problems with builds 17967 and 17990

Post new topic   Reply to topic    DD-WRT Forum Index -> Broadcom SoC based Hardware
Goto page 1, 2, 3, 4  Next
Author Message
dc
DD-WRT User


Joined: 08 Jun 2006
Posts: 247
Location: Prince Edward Island - Canada

PostPosted: Thu Dec 08, 2011 16:33    Post subject: Possible SSH problems with builds 17967 and 17990 Reply with quote
*** Warning - Long and Involved ***

With build 17967, the dropbear (ssh server/client/key generator) has been updated and appears to function slightly different than before.

Previously the host key files located in the routers /tmp/root/.ssh directory were in 'openssh' format, they were 'ascii' characters that started with a header line '------BEGIN.....' and ended with a footer of '-----END.....'. These key values are also stored in the NVRAM variables sshd_dss_host_key and sshd_rsa_host_key in the same openssh format.

Now, the files in /tmp/root/.ssh are dropbear formatted binary files. All good so far.

If the NVRAM variables were not cleared after the upgrade to 17967 you are unable to connect to your router using SSH. (Yes murrkf -- we know .. we should have cleared nvram after upgrading as outlined in the Peacock Thread Smile )

I posted a quick work-around in the build thread http://www.dd-wrt.com/phpBB2/viewtopic.php?t=147962

What the workaround does is to create the two required files in the appropriate format.

Now.. if you are still with me here's were we get to narrow down the problem a bit. There is key converter as part of the dropbear set of programs. You can use this to convert from dropbear to openssh format and I expect that this is what is used behind the scenes to store the values in NVRAM as openssh keys and re-create the binary dropbear files after each boot.

It appears that the process to generate and convert the DSS key works fine. I see an NVRAM variable with a key in it. Unfortunately the RSA conversion doesn't seem to work (the dropbearconvert program segfaults) so it creates the file but does not create the appropriate NVRAM variable sshd_rsa_host_key but leaving it blank.

The downside of this is that when the router reboots it generates a new RSA key file and then (I assume) segfaults while it is trying to create the openssh key for the NVRAM variable.

Is anyone else who reset to defaults seeing this behaviour?

A quick way to tell is that after a reboot your router and start an ssh session (I use putty) it will tell you that the routers key has changed.

[edited to slightly change wording]


Last edited by dc on Mon Dec 12, 2011 14:16; edited 2 times in total
Sponsor
ErMeglio
DD-WRT User


Joined: 11 Jul 2006
Posts: 104

PostPosted: Thu Dec 08, 2011 16:55    Post subject: Reply with quote
I can confirm 100% all your findings because I experienced the same doing tests by myself after having that first -bad boy not clearing nvram- problem.
I've now done many tests in different ways:
*in my old config, clearing nvram-related variables. Problem was there, rsa was regenerated at every boot.
*after a 30/30/30, setting all again, all working except the same: rsa was new at every reboot and programs was complaining. Tested the file binary content and looks different too.
*after a 30/30/30, firmware flash, 30/30/30, problem was still the same, nvram dss is ok but rsa is not there in any test I've done.
Dark_Shadow
DD-WRT Guru


Joined: 31 Aug 2009
Posts: 2448
Location: Third Rock from the Sun

PostPosted: Thu Dec 08, 2011 17:02    Post subject: Reply with quote
http://www.dd-wrt.com/phpBB2/viewtopic.php?p=653515#653515
_________________
Peacock Thread-FAQ -- dd-wrt Wiki

Testing Multiple Routers -- Bootloader Collection Project -- My Wiki
dc
DD-WRT User


Joined: 08 Jun 2006
Posts: 247
Location: Prince Edward Island - Canada

PostPosted: Thu Dec 08, 2011 17:12    Post subject: Reply with quote
Dark_Shadow wrote:
http://www.dd-wrt.com/phpBB2/viewtopic.php?p=653515#653515


Yes.. that's my original post with a quick 'make it work' fix. After further testing it does appear to be the dropbearconvert segfaulting. If you run it from the command line it things the RSA key is a DSS key and then segfaults. I'll create a new trac ticket.
ErMeglio
DD-WRT User


Joined: 11 Jul 2006
Posts: 104

PostPosted: Thu Dec 08, 2011 17:40    Post subject: Reply with quote
http://svn.dd-wrt.com:8000/ticket/2283
dc
DD-WRT User


Joined: 08 Jun 2006
Posts: 247
Location: Prince Edward Island - Canada

PostPosted: Sat Dec 10, 2011 16:49    Post subject: Reply with quote
The problem is still there in 17990 for the K26 mega builds. Others have reported no problems with K24 builds (and perhaps others)

I'll try a few different hardware platforms here when I can get the chance and report back.
ErMeglio
DD-WRT User


Joined: 11 Jul 2006
Posts: 104

PostPosted: Sat Dec 10, 2011 17:36    Post subject: Reply with quote
Please, everyone that is experiencing similar problems, please give an help to confirm/understand and possibly fix this problem, indicating:
*hardware used/tried on
*if the upgrade was done defaulting the settings, ie clear nvram, 30/30/30, or selecting default settings in dd-wrt web ui
*what's the answer when giving these commands in telnet/ssh/web ui command page into administration:
Code:
dropbearconvert dropbear_rsa_key openssh /tmp/root/.ssh/ssh_host_rsa_key /tmp/root/.ssh/rsa

and
Code:
nvram get sshd_rsa_host_key

obviously hide your actual key data if given from the last command.
kt_haddock
DD-WRT Guru


Joined: 25 Dec 2009
Posts: 710

PostPosted: Sat Dec 10, 2011 18:28    Post subject: Reply with quote
Isn't any problem with latest dropbear (054) if that is in use ?
_________________
[I prefer answere in whole sentence]
ErMeglio
DD-WRT User


Joined: 11 Jul 2006
Posts: 104

PostPosted: Sat Dec 10, 2011 19:33    Post subject: Reply with quote
kt_haddock wrote:
Isn't any problem with latest dropbear (054) if that is in use ?

I am not understanding your question.
dropbear -V gives:
Dropbear sshd v2011.54
and it works good when connected, for exchanging commands and files, all is working there. The problem is with dropbearconvert. And with the fact a reboot / power outage or anything that lets the router restart makes it having a different identification rsa key, which is bad because it's used to verify the server you're connecting to is the right one.
PetervdM
DD-WRT User


Joined: 11 Jun 2009
Posts: 282
Location: EU

PostPosted: Sat Dec 10, 2011 20:26    Post subject: Reply with quote
don't know if it helps, but i'm running vpnkong - build 17940M with Dropbear sshd v2011.54 on E4200v1. no ssh problems here!

nvram get sshd_rsa_host_key gives:
-----BEGIN RSA PRIVATE KEY-----
blah
blah
-----END RSA PRIVATE KEY-----

dropbearconvert dropbear_rsa_key openssh /tmp/root/.ssh/ssh_host_rsa_key /tmp/root/.ssh/rsa gives:

sh/ssh_host_rsa_key /tmp/root/.ssh/rsa
Key is a RSA key
Wrote key to '/tmp/root/.ssh/rsa'

_________________
now running tomato by shibby
E4200v1 cfe 2010.09.20.0
Taomyn
DD-WRT User


Joined: 23 Oct 2011
Posts: 50

PostPosted: Sat Dec 10, 2011 21:23    Post subject: Reply with quote
Was asked in another thread to respond here:

ErMeglio wrote:
gene wrote:
Taomyn wrote:

4. I think SSH is regenerating the keys each reboot, but need to test this, just need to do some work now that I'm up and running again.
I can confirm the same result regarding SSH on reboot.

Could you do the tests I was asking for here http://www.dd-wrt.com/phpBB2/viewtopic.php?p=654142#654142 ?
I am trying to troubleshoot the problem since days and reporting any experience here http://svn.dd-wrt.com:8000/ticket/2283
Could you do it too, indicating your hardware, way you used to upgrade (cleared nvram/settings or not), results of the commands I indicate there (mainly dropbearconvert dropbear openssh /tmp/root/.ssh/sshd_host_rsa_key /tmp/root/.ssh/rsa that here gives a segmentation fault, and nvram get sshd_host_rsa_key that here gives a blank line)?
Thanks


Using "DD-WRT v24-sp2 (12/08/11) big - build 17990M NEWD-2 K2.6 Eko" on my E4200 I get:

Code:
root@monty:~# nvram get sshd_rsa_host_key

root@monty:~# dropbearconvert dropbear openssh /tmp/root/.ssh/ssh_host_rsa_key /
tmp/root/.ssh/rsa
Key is a DSS key
Segmentation fault
root@monty:~# nvram get sshd_rsa_host_key

root@monty:~#


As with all these builds I do a 30/30/30 prior to upgrade of firmware, upgrade the firmware, perform another 30/30/30, then manually re-enter all my settings. I do not use any backups, except taking one beforehand in case I want to quickly revert to the previous firmware.
Cyberian
DD-WRT User


Joined: 07 Jun 2006
Posts: 198
Location: Oregon, US

PostPosted: Sat Dec 10, 2011 22:53    Post subject: Reply with quote
dc wrote:
Others have reported no problems with K24 builds (and perhaps others)

Using K2.4 Eko OpenVPN + JFFS build on WRT54GS, I'm getting the same problem.

_________________
Michael
WRT54GS v1.1 with Eko JFFS + OpenVPN
ErMeglio
DD-WRT User


Joined: 11 Jul 2006
Posts: 104

PostPosted: Sun Dec 11, 2011 0:25    Post subject: Reply with quote
PetervdM wrote:
don't know if it helps, but i'm running vpnkong - build 17940M with Dropbear sshd v2011.54 on E4200v1. no ssh problems here!

This was VERY useful in fact!!!!
To the devs and the others, this is what I've just tried and my conclusions:
*from 17990M I reverted to Kong's 17940M without clearing nvram. In fact connection to sshd was refused. It's expected, different key kinds. So I've launched
Code:
nvram unset sshd_rsa_host_key
nvram commit
reboot

At reboot I had a newly generated rsa key that WAS into nvram variable and WAS making dropbear work. Expected, again, ok.
Here it is my rough idea that came out reading that build works: there could be a problem with the currently compiled binary of dropbearmulti (that contains server, convert, key).
*So I've saved all the binaries with an scp connection.
*Now, let's go to 17990M (EKO or BS, no difference between them here), again, without clearing nvram. Connection refused, unset variable, reboot, exactly as before. But, after reboot, nvram variable is empty as expected and dropbear is running with the new expected temp key.
*Now, let's check if there's a problem of some other kind and not only a binary problem. While running 17990M I've sent the Kong's 17940M binary I had saved to /tmp/KONGsbin with scp. I then have created a symlink called "dropbearconvert" to dropbearmulti (the only binary) to be able to launch the converter. Made them executable. Then I've tried those commands to check the difference between their acting on the SAME dropbear type key actually saved into /tmp/root/.ssh/
Code:
root@DD-WRT:~/.ssh# cd /tmp/KONGsbin
root@DD-WRT:/tmp/KONGsbin# ls
dropbearconvert  dropbearmulti
root@DD-WRT:/tmp/KONGsbin# ./dropbearconvert dropbear openssh /tmp/root/.ssh/ssh_host_rsa_key rsaKONG
Key is a RSA key
Wrote key to 'rsaKONG'
root@DD-WRT:/tmp/KONGsbin# dropbearconvert dropbear openssh /tmp/root/.ssh/ssh_host_rsa_key rsaBS
Key is a DSS key
Segmentation fault

As you can see, Kong's "old?" binary works fine!!! The actual one into the last compilation does not. Dimensions are exactly the same but the new binary doesn't seem to be correctly compiled and working as expected. Also, this could be the reason why we were not understanding why those build weren't accepting the "old" keys while they should have to, because we are speaking of the exactly same dropbear build as far as I understand:
Code:
root@DD-WRT:/tmp/KONGsbin# ./dropbear -V
Dropbear sshd v2011.54
root@DD-WRT:/tmp/KONGsbin# dropbear -V
Dropbear sshd v2011.54

So, 2011.54, same version, different behavior. I think this is not good and something has gone wrong while compiling for reasons I can't understand or due to some new compilation settings (I barely remember a compression mod of the makefile, but I don't know if it's important here).
I think it could be great for all to put out a fixed build containing the old dropbear's binaries compiled from Kong's or from an older build. I don't get how could we make the root / writable and wouldn't want to go further and make damages, I think that a new build would be simpler for everyone Wink
Sorry for the long thread. Hope it helped and doesn't contain nonsense, it could as I'm not an expert/developer or anything similar Rolling Eyes


Last edited by ErMeglio on Sun Dec 11, 2011 0:29; edited 1 time in total
dc
DD-WRT User


Joined: 08 Jun 2006
Posts: 247
Location: Prince Edward Island - Canada

PostPosted: Sun Dec 11, 2011 0:29    Post subject: Reply with quote
Looks like a possible toolchain or library issue with the Brainslayer build.

if you do a 'dropbearmulti -h' on BS's build it reports 11.52 not 11.54 but dropbearkey and dbclient report 11.54
ErMeglio
DD-WRT User


Joined: 11 Jul 2006
Posts: 104

PostPosted: Sun Dec 11, 2011 0:31    Post subject: Reply with quote
dc wrote:
Looks like a possible toolchain or library issue with the Brainslayer build.

if you do a 'dropbearmulti -h' on BS's build it reports 11.52 not 11.54 but dropbearkey and dbclient report 11.54

This IS even weirder, here it's the same too, and old binary gives the expected .54 instead.

Code:
root@DD-WRT:/tmp/KONGsbin# dropbearmulti -h
Dropbear multi-purpose version 0.52
root@DD-WRT:/tmp/KONGsbin# ./dropbearmulti -h
Dropbear multi-purpose version 2011.54
Goto page 1, 2, 3, 4  Next Display posts from previous:    Page 1 of 4
Post new topic   Reply to topic    DD-WRT Forum Index -> Broadcom SoC based Hardware All times are GMT

Navigation

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum