DD-WRT :: View topic - Kong PTB users: Are these dldd scripts useful to you?

Kong PTB users: Are these dldd scripts useful to you?

DD-WRT Forum Index -> Broadcom SoC based Hardware

View previous topic :: View next topic

Author

Message

HalfBit
DD-WRT Guru

Joined: 04 Sep 2009
Posts: 776
Location: AR, USA

Posted: Wed Jul 27, 2016 5:50 Post subject: Kong PTB users: Are these dldd scripts useful to you?

I've been keeping up with Kong's PTBs and religiously download them as new builds or compiled versions of the same builds are posted by Kong on his FTP site. I would use DDUP to download the files when available, abort the install, and then copy the files to my thumb drive mounted to /opt/.

A couple weekends ago, I got a whim to do some scripting to download the files directly to my thumb drive instead of using DDUP and then copying them to my preferred location. That evolved into me reverse engineering the DDUP script to 1) know that a new build was available, and 2) prompt the user to download the files and confirm the location to save the files.

I've since been able to automate the script even more. It can now see that a download is available, check your folder location to see if directories already exist for the build number, and then confirms that the file hasn't already been downloaded by validating that the md5sums are unique.

Hopefully, others will find these scripts useful. I'm open to any suggestions for better efficiency both for the code, and/or the cycles on the hardware. I am a Network Engineer by trade, scripting/coding is not my forte, so be gentle. Smile

I'm pretty sure the scripts are self-explanatory. I guess I could go through and add more comments. manualdldd prompts the user for each step--just change the LOCATION variable in the script to your desired location. autodldd doesn't prompt, but does give output along the way--again, update the FWLOC variable on this one to your desired location. If multiple builds with the same number are found it will append -2, -3, etc. to the filenames (the DLLOC variable).

Edit 07/27/2016:
I cleaned up the autodldd script a bunch more removing items for testing and adding some comments. Also, for anyone who wanted to peruse the script without downloading the file, the following is the autodldd script. It is geared toward the R7000 PTB filenames, but can easily be adapted for other files/hardware:

Code:

# encode util to HTML URL encoding.
encode () {
url=$(echo "$1" | sed -e 's/ /%20/g' -e 's/-/%2d/g' -e 's/\./%2e/g' -e 's/\//%2f/g' -e 's/:/%3a/g' -e 's/;/%3b/g' -e 's/\\/%5c/g' -e 's/_/%5f/g')
echo $url
}

# checksum_unique util validates that the checksums are unique.
checksum_unique () {
chksum_list=`/bin/cat $NEW_BLD_MD5 | cut -d " " -f1`
chksum_file=`/usr/bin/md5sum $1dd-wrt.v24-K3_AC_ARM_STD.bin | cut -d " " -f1`
echo -e "Checksum: "$chksum_file" found in "$FWLOC".\nChecksum: "$chksum_list" is available."
if [ "$chksum_file" != "$chksum_list" ]; then
   echo -e "Checksums did not match."
   return 1
else
   echo "Checksums match. This build has already been downloaded."
   return 0
fi
}

DD_REV=`cat /etc/release`
DD_BOARD=`nvram get DD_BOARD`
BOARD=$( encode "$DD_BOARD" )
BUILD_REV=`/usr/bin/wget -q -O - http://www.desipro.de/ddup.php?rev=$DD_REV\&board=$BOARD | grep -oE [0-9]{5}`"M"
RESULT=`/usr/bin/wget -q -O - http://www.desipro.de/ddup.php?rev=$DD_REV\&board=$BOARD`
BUILD_URL=`/usr/bin/wget -q -O - http://www.desipro.de/ddup.php?rev=$DD_REV\&up=1\&board=$BOARD`
SIG_EXT=".sig"
MD5_EXT=".md5"
NEW_BLD_MD5="/tmp/"$BUILD_REV$MD5_EXT
UPDATE=$(echo "$RESULT" | grep -o "available")
IFDUPBLD=0
DUPBUILD=1
FWLOC="/opt/Firmware/"$BUILD_REV
DLLOC=$FWLOC

# main checks for available updates based on current hardware and firmware installed then kicks off the process to check local files.
main () {
if [ "$UPDATE" == "available" ]; then
   echo -e $RESULT"\nDownloading md5sum for the available build."
   /usr/bin/wget -O $NEW_BLD_MD5 $BUILD_URL$MD5_EXT
   echo "Checking for existing firmware folders."
   checkLocalFiles
   rm $NEW_BLD_MD5
   exit 0
else
   echo $RESULT
   exit 0
fi
}

# checkLocalFiles validates if directories exist for the build, validates the checksums are unique, and if necessary downloads the files to DLLOC
checkLocalFiles (){
for f in $FWLOC*/; do
   if [ -d ${f} ]; then
      echo -e "Found "$f". Validating checksums are unique."

      if checksum_unique $f; then
         echo -e "Aborting downloads and cleaning up temporary files."
         rm $NEW_BLD_MD5
         exit 1
      else
         IFDUPBLD=1
         DUPBUILD=$((DUPBUILD+1))
      fi
   fi
done

if [ "$IFDUPBLD" == 1 ]; then
   DLLOC=$FWLOC"-"$DUPBUILD
fi

echo -e "Downloading "$BUILD_REV" files to "$DLLOC.

/bin/mkdir $DLLOC
/usr/bin/wget -O $DLLOC/dd-wrt.v24-K3_AC_ARM_STD.bin $BUILD_URL
/usr/bin/wget -O $DLLOC/dd-wrt.v24-K3_AC_ARM_STD.bin.sig $BUILD_URL$SIG_EXT
/usr/bin/wget -O $DLLOC/dd-wrt.v24-K3_AC_ARM_STD.bin.md5 $BUILD_URL$MD5_EXT

#echo "BUILD FILES DOWNLOADED TO "$DLLOC"!!!"
}

# Starts script
main

Here's some sample output from the script:

Code:

root@R7000:~# adldd
New release 30270 for Netgear R7000 available.
Downloading md5sum for the available build.
Connecting to www.desipro.de (217.160.231.132:80)
30270M.md5 100% |*******************************| 63 0:00:00 ETA
Checking for existing firmware folders.
Found /opt/FirmwareTest/30270M-2/. Validating checksums are unique.
Checksums did not match.
Found /opt/FirmwareTest/30270M/. Validating checksums are unique.
Checksums did not match.
Downloading 30270M files to /opt/FirmwareTest/30270M-3.
...I cancelled the downloads of the .bin, .md5, and .sig files in this test.

and...

Code:

root@R7000:~# adldd
New release 30270 for Netgear R7000 available.
Downloading md5sum for the available build.
Connecting to www.desipro.de (217.160.231.132:80)
30270M.md5 100% |*******************************| 63 0:00:00 ETA
Checking for existing firmware folders.
Found /opt/Firmware/30270M/. Validating checksums are unique.
Checksum: 725680c121322dacac3f0de776765d85 found in /opt/Firmware/30270M.
Checksum: 725680c121322dacac3f0de776765d85 is available.
Checksums match. This build has already been downloaded.
Aborting downloads and cleaning up temporary files.

Edit 07/31/2016:
Updated script that is a little more clean, and less hardcoded with file names. I used regex to grab the file name from end of the BUILD_URL string for use in saving the files locally. I'm still trying to figure out a more dynamic way of doing so, but am running into limitations (or lack of knowing how to format the syntax) of using regex with grep/egrep.

Edit 08/07/2016:
Updated scripts per post: http://www.dd-wrt.com/phpBB2/viewtopic.php?p=1042190#1042190

DLDD Scripts.zip

Description:

Scripts to automate downloading and archiving of Kong's DD-WRT PTBs. Last updated 08-07-2016

Download

Filename:

DLDD Scripts.zip

Filesize:

3.06 KB

Downloaded:

326 Time(s)

_________________
R7000 Nighthawk - DD-WRT v3.0-r50308
R7000 Nighthawk - DD-WRT v3.0-r50308
~~~~~~~~~~Dismantled for learning opportunities~~~~~~~~~~
WRT54Gv2
WRT54Gv8.2
~~~~~~~~~~Other Settings~~~~~~~~~
https://nextdns.io/?from=2d3sq39x
https://pi-hole.net/
https://github.com/DNSCrypt/dnscrypt-proxy

Last edited by HalfBit on Sun Aug 07, 2016 17:21; edited 3 times in total

Sponsor

HalfBit
DD-WRT Guru

Joined: 04 Sep 2009
Posts: 776
Location: AR, USA

Posted: Thu Jul 28, 2016 12:35 Post subject:

Love it! It worked beautifully this morning:

Code:

root@R7000:~# adldd
New release 30300 for Netgear R7000 available.
Downloading md5sum for the available build.
Connecting to www.desipro.de (217.160.231.132:80)
30300M.md5 100% |*******************************| 63 0:00:00 ETA
Checking for existing firmware folders.
Downloading 30300M files to /opt/Firmware/30300M.
Connecting to www.desipro.de (217.160.231.132:80)
dd-wrt.v24-K3_AC_ARM 100% |*******************************| 21192k 0:00:00 ETA
Connecting to www.desipro.de (217.160.231.132:80)
dd-wrt.v24-K3_AC_ARM 100% |*******************************| 256 0:00:00 ETA
Connecting to www.desipro.de (217.160.231.132:80)
dd-wrt.v24-K3_AC_ARM 100% |*******************************| 63 0:00:00 ETA

Also, I added a LOGFILE variable so I know when it ran after setting it up as a cronjob:

Code:

root@R7000:~# cat adldd.log
New FW version last checked on Wed Jul 27 23:03:51 GMT-5> 2016. No new version found.
New FW version last checked on Thu Jul 28 01:00:04 GMT-5> 2016. No new version found.
New FW version last checked on Thu Jul 28 07:30:37 GMT-5> 2016. New version downloaded to /opt/Firmware/30300M

slidermike
DD-WRT Guru

Joined: 11 Nov 2013
Posts: 1487
Location: USA

Posted: Thu Jul 28, 2016 23:20 Post subject:
Well done HB! I will add this thread to the tips/tricks. _________________ Router currently owned: Netgear R7800 - Router Netgear R7000 - AP mode R7000 specific Tips/Tricks. http://www.dd-wrt.com/phpBB2/viewtopic.php?t=264152

HalfBit
DD-WRT Guru

Joined: 04 Sep 2009
Posts: 776
Location: AR, USA

Posted: Fri Jul 29, 2016 1:36 Post subject:

Thanks, Mike.

In case someone wants to add the LOGFILE feature, I added the following code:

Quote:

FWLOC="/opt/Firmware/"$BUILD_REV
DLLOC=$FWLOC
LOGFILE=/opt/custom/dldd/adldd.log
...
main () {
...
checkLocalFiles
rm $NEW_BLD_MD5
echo "New FW version last checked on" $(date)". New version downloaded to "$DLLOC >> $LOGFILE
exit 0
...
checkLocalFiles (){
...
echo -e "Aborting downloads and cleaning up temporary files."
echo "New FW version last checked on" $(date)". No new version found." >> $LOGFILE
rm $NEW_BLD_MD5
exit 1
...

CloneVince
DD-WRT User

Joined: 31 May 2014
Posts: 219

Posted: Fri Jul 29, 2016 5:55 Post subject:
Thx HalfBit, it will help me to mirror Kong's releases I will test your script and modify it for me tomorrow =) _________________ Unofficial Kong's release repository : http://ddwrt-kong.clonevince.fr/

Meanybucket
DD-WRT User

Joined: 12 May 2011
Posts: 52

Posted: Sun Jul 31, 2016 3:08 Post subject:
Great Work HalfBit! Thanks for posting this. Works great. _________________ Netgear R7000 \| DD-WRT v3.0-r31520M kongac (03/01/17)

HalfBit
DD-WRT Guru

Joined: 04 Sep 2009
Posts: 776
Location: AR, USA

Posted: Sun Jul 31, 2016 15:21 Post subject:

CloneVince wrote:

Thx HalfBit, it will help me to mirror Kong's releases Wink

I will test your script and modify it for me tomorrow =)

Great! Let me know if you need help. I'm no expert, but since this is a personal project, I'm more invested/interested in getting it to work for your needs. Smile

HalfBit
DD-WRT Guru

Joined: 04 Sep 2009
Posts: 776
Location: AR, USA

Posted: Sun Jul 31, 2016 15:22 Post subject:

Meanybucket wrote:

Great Work HalfBit!
Thanks for posting this.
Works great.

You're welcome, glad it's working for you. FYI, I'm posting an updated script on the first post.
_________________
R7000 Nighthawk - DD-WRT v3.0-r50308
R7000 Nighthawk - DD-WRT v3.0-r50308
~~~~~~~~~~Dismantled for learning opportunities~~~~~~~~~~
WRT54Gv2
WRT54Gv8.2
~~~~~~~~~~Other Settings~~~~~~~~~
https://nextdns.io/?from=2d3sq39x
https://pi-hole.net/
https://github.com/DNSCrypt/dnscrypt-proxy

HalfBit
DD-WRT Guru

Joined: 04 Sep 2009
Posts: 776
Location: AR, USA

Posted: Sat Aug 06, 2016 21:32 Post subject:

I've scheduled the auto script to run every 12 hours with the following added to cron jobs on the web interface:

Code:

0 1,13 * * * root <script path>/autodldd

With the log file, I'm able to quickly check recent history:

Code:

# ckdl
New FW version last checked on Wed Jul 27 23:03:51 GMT-5> 2016. No new version found.
New FW version last checked on Thu Jul 28 01:00:04 GMT-5> 2016. No new version found.
New FW version last checked on Thu Jul 28 07:30:37 GMT-5> 2016. New version downloaded to /opt/Firmware/30300M
New FW version last checked on Thu Jul 28 13:00:04 GMT-5> 2016. No new version found.
New FW version last checked on Fri Jul 29 01:00:04 GMT-5> 2016. No new version found.
New FW version last checked on Fri Jul 29 13:00:03 GMT-5> 2016. No new version found.
New FW version last checked on Sat Jul 30 01:00:03 GMT-5> 2016. No new version found.
New FW version last checked on Sat Jul 30 13:00:06 GMT-5> 2016. No new version found.
New FW version last checked on Sun Jul 31 01:00:03 GMT-5> 2016. No new version found.
New FW version last checked on Sun Jul 31 13:00:03 GMT-5> 2016. No new version found.
New FW version last checked on Mon Aug 1 01:00:04 GMT-5> 2016. No new version found.
New FW version last checked on Mon Aug 1 13:00:04 GMT-5> 2016. No new version found.
New FW version last checked on Mon Aug 1 22:10:02 GMT-5> 2016. No new version found.
New FW version last checked on Tue Aug 2 01:00:03 GMT-5> 2016. No new version found.
New FW version last checked on Tue Aug 2 13:00:04 GMT-5> 2016. No new version found.
New FW version last checked on Wed Aug 3 01:00:04 GMT-5> 2016. No new version found.
New FW version last checked on Wed Aug 3 13:01:00 GMT-5> 2016. New version downloaded to /opt/Firmware/30325M/
New FW version last checked on Wed Aug 3 20:47:52 GMT-5> 2016. New version downloaded to /opt/Firmware/30350M/
New FW version last checked on Thu Aug 4 01:00:04 GMT-5> 2016. No new version found.
New FW version last checked on Thu Aug 4 13:00:04 GMT-5> 2016. No new version found.
New FW version last checked on Fri Aug 5 01:00:05 GMT-5> 2016. No new version found.
New FW version last checked on Fri Aug 5 13:00:04 GMT-5> 2016. No new version found.
New FW version last checked on Sat Aug 6 01:00:04 GMT-5> 2016. No new version found.
New FW version last checked on Sat Aug 6 13:00:04 GMT-5> 2016. No new version found.

Does anyone know what regex would be best to get the filename in "http://www.desipro.de/ddwrt/K3-AC-Arm/dd-wrt.v24-K3_AC_ARM_STD.bin?" Right now I am using "grep -oE dd-.*bin$" but I want to use something like "(http.*\/)" but with I think what is called a positive lookbehind. Any ideas?
_________________
R7000 Nighthawk - DD-WRT v3.0-r50308
R7000 Nighthawk - DD-WRT v3.0-r50308
~~~~~~~~~~Dismantled for learning opportunities~~~~~~~~~~
WRT54Gv2
WRT54Gv8.2
~~~~~~~~~~Other Settings~~~~~~~~~
https://nextdns.io/?from=2d3sq39x
https://pi-hole.net/
https://github.com/DNSCrypt/dnscrypt-proxy

HalfBit
DD-WRT Guru

Joined: 04 Sep 2009
Posts: 776
Location: AR, USA

Posted: Sun Aug 07, 2016 17:03 Post subject:

I've ended up using the cut command to get what I want:

Code:

cut -d / -f6

returns just

Quote:

dd-wrt.v24-K3_AC_ARM_STD.bin

from the whole URL

Quote:

http://www.desipro.de/ddwrt/K3-AC-Arm/dd-wrt.v24-K3_AC_ARM_STD.bin

Ah-Pin-Kor
DD-WRT User

Joined: 19 Sep 2015
Posts: 267

Posted: Mon Aug 08, 2016 1:14 Post subject:

HalfBit wrote:

basename should return just the filename.
_________________

Netgear R7800 kongpro 19.07 20190919 || Netgear R7000 36070M kongac (Client Bridge=5GHz, AP=2.4GHz with bridged VAP)
Linksys WRT32X davidc502 OpenWrt || Linksys WRT1200ACv1 Gargoyle 1.11.x
Linksys WRT1900ACSv2 dd-wrt 39956

HalfBit
DD-WRT Guru

Joined: 04 Sep 2009
Posts: 776
Location: AR, USA

Posted: Mon Aug 08, 2016 1:48 Post subject:

Ah-Pin-Kor wrote:

HalfBit wrote:

basename should return just the filename.

That is awesome, thank you! I modified the following variable:

Code:

FWFILENAME=$(basename `echo "$BUILD_URL"`)

My tests were successful:

Code:

root@Router:# basename `cat test.txt`
dd-wrt.v24-K3_AC_ARM_STD.bin

Edit:
Interestingly enough, I found the answer for the regex while researching basename. The following is another option:

Code:

FWFILENAME=$(echo "$BUILD_URL" | grep -Eo "[^/]*$")

Display posts from previous:

Page 1 of 1

DD-WRT Forum Index -> Broadcom SoC based Hardware

All times are GMT

Navigation

Jump to:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum

Log in

Kong PTB users: Are these dldd scripts useful to you?

Quick Links

Log in