dd-wrt - Bandwidth Usage Monitor

Post new topic   Reply to topic    DD-WRT Forum Forum Index -> Broadcom SoC based Hardware
Goto page 1, 2, 3 ... 40, 41, 42  Next
Author Message
FezJay
DD-WRT User


Joined: 11 Sep 2010
Posts: 176

PostPosted: Tue Oct 26, 2010 2:34    Post subject: dd-wrt - Bandwidth Usage Monitor Reply with quote
Current Status: VERSION 1.1
Features:

* Record Daily Usage (Can be turned on/off)
* Automatic Backup
* Automatic Periodic Reset By day of month (Can be turned on/off)
* Can use dnsmasq or custom users file
* Records usage regardless of IP address
* Low load usage
* Configurable via input parameters
* Optional use of sort (for those that do not have the sort command)
* Provides Bit32 Integer Support, via a specialised script argument
* Added use of bc command
* Allows you to change between 1000 and 1024 via Query String Parameters
* Added the ability to disable showing certain users usage via users.file

Information (setup etc): http://code.google.com/p/lal-projects/w/list

Hey guys

I have been looking extensively at the wrtbwmon for live traffic usage etc.

It does its job, but I wanted somthing more, so I have re-written it from the ground up.

It is in beta testing at the moment (on my router) and I believe it might be ready for some external user testing (purpose of this post).

It is a 2 to 3 file operation (2 files required).

If you are interested in helping me test it, you will need to have some knowledge of linux (navigation of folders) and setting up scripts etc

The main project link is http://code.google.com/p/lal-projects/

If you have any question please post them here or on the google site, as i would like to keep discussions etc in one place - http://code.google.com/p/lal-projects/wiki/bw_monitor_discussion

Please read the documentation before installing it.

The main files are bw_monitor.sh and monitor.html. The other 2 script files are optional and are mainly a guide for setting up a startup or wanup script ( iuse wanup)


Just like to say thanks to everyone that has helped test and provide feedback on this project. It has been very much appreciated.

I would like to say thank you to the following, bpsmicro (dd-wrt forumns), Dark_Shadow (dd-wrt forumn), Luniz2k1 (dd-wrt forumn), mojso (dd-wrt forumn), Wolf Kodi (dd-wrt forumn) and arthu (dd-wrt forumn).

Cheers Everyone



bw_monitiro_pic1_194.png
 Description:
 Filesize:  84 KB
 Viewed:  69530 Time(s)

bw_monitiro_pic1_194.png



bw_monitiro_pic2_141.png
 Description:
 Filesize:  44.87 KB
 Viewed:  69530 Time(s)

bw_monitiro_pic2_141.png



bw_monitiro_pic3_171.png
 Description:
 Filesize:  81.6 KB
 Viewed:  69530 Time(s)

bw_monitiro_pic3_171.png



_________________
dd-wrt eko/V24-K26 15508
Belkin F7D4301
8MB Flash
64MB RAM

Optware (jffs/mnt) 4GB Flash Drive


Last edited by FezJay on Sat Oct 22, 2011 22:23; edited 8 times in total
Sponsor
gobbledigook
DD-WRT Guru


Joined: 06 Apr 2009
Posts: 727

PostPosted: Tue Oct 26, 2010 14:42    Post subject: Reply with quote
hey there!

looks very smooth Smile i would still be interested in per user live data... my apologies but i'm not great with bash and got lost when looking at this before! for me it is the most useful feature, i can see who exactly is caning my net and politely ask then to stop Wink
FezJay
DD-WRT User


Joined: 11 Sep 2010
Posts: 176

PostPosted: Wed Oct 27, 2010 3:39    Post subject: Reply with quote
gobbledigook wrote:
hey there!

looks very smooth Smile i would still be interested in per user live data... my apologies but i'm not great with bash and got lost when looking at this before! for me it is the most useful feature, i can see who exactly is caning my net and politely ask then to stop Wink


Thats why I created it :-)

I have a few boarders that exceed my limit by 20 to 50 GBs per month, and I need to start recovering the costs lol.

The idea behind this was to group users to machines as many people have more than 1 pc/laptop.

As you can see from the shots, I have several displays. 2 of which shows a breakdown of each user and there machine usage, and one that purely display user usage (over all machines).

The daily one is a 'nice to have'. It is optional. I put it on so people can see what they can do with the data.

If I had to explain how to install it I would do t the following way.

1. Create folder in persistant storage area and add bw_monitor.sh and monitor.html (users.file if not using dnsmasq)

2. Set up a CRON or Startup script to, copy over the html file to a web directory /tmp/www/, and another line that calls the script

cp /mnt/monitor/setup/monitor.html /tmp/www/monitor.html

/mnt/monitor/setup/bw_monitor 30 4 30 4 /tmp/dnsmasq.config /mnt/monitor/usage.backup /mnt/monitor/history/ #Leave rest as defauit

Thats it :-)

If you dont hv a storage area, use wget to download them to tmp.

What the above call does is, update every 30 seconds, every 4th update do a publish, every 30 updates do a backup, reset and archive on the 4th of every month, use the dnsmasq user file, store backups as /mnt/monitor/usage.backup, store history files in /mnt/monitor/history/.

There are optional options after this that control other things.

echo 'Arg 1 - Interval Update (Mandatory): This indicates in seconds how often an update is performed.'
echo 'Arg 2 - Publish Iteration (Mandatory): Indicates how many update iterations must past before a publish is done.'
echo 'Arg 3 Iteration Backup (Mandatory): Indicates how many update iterations must past before a backup is done.'
echo 'Arg 4 Reset Day (Mandatory): Indicates what day at midnight should the usage data be reset to 0 and backed up to history. If no reset value is stated, no reset will occur'
echo 'Arg 5 User File Path (Mandatory): The path and file name for where user data is saved. Format MUST be MAC,IP,USER,MAC TYPE'
echo 'Arg 6 Backup Usage File Path (Mandatory): The path and file name for the interval back up of the usage file.'
echo 'Arg 7 Backup History Path (Mandatory): The path for where history files will be kept. Used when reset occurs.'
echo 'Arg 8 MAC Usage File Path (Optional): Indicates the path and file name for usage data. If none is defined a default will be used. A Value of 1 will keep the usage in memory. Any other value will force the values to be written to disk'
echo 'Arg 9 Javascript Output Path (Optional): The path for where the user_details.js output file should be saved. If none is defined the deafult /tmp/www folder will be used'
echo 'Arg 10 Do Usage File Restore (Optional): Indicates if the usage file is restored from the backed up location when the monitor is started. Values are 1 (true) and 0(false). Default is 1'
echo 'Arg 11 Use Sort (Optional): Indicates if we can use the sort command or not. Values are 1 (true) and 0(false). Default is 1'
echo 'Arg 12 Keep Last Javascript File(Optional): Indicates if the last js file should be kept for showing last months usage. Default 0 (false)'
echo 'Arg 13 History Javascript File(Optional): Indicates if the javascript file should also be backed up to history. Default 0 (false)'
echo 'Arg 14 Record Daily Usage(Optional): Indicates if you would like to record daily usage or not. Default is 0 (false)'

This information is on my Wiki Site

http://code.google.com/p/lal-projects/wiki/bw_monitor_setup

_________________
dd-wrt eko/V24-K26 15508
Belkin F7D4301
8MB Flash
64MB RAM

Optware (jffs/mnt) 4GB Flash Drive


Last edited by FezJay on Thu Oct 28, 2010 12:55; edited 2 times in total
yzy-oui-fi
DD-WRT Guru


Joined: 03 Mar 2009
Posts: 2826
Location: France

PostPosted: Wed Oct 27, 2010 7:07    Post subject: Reply with quote
very nice stuff!

did you thought of exporting data to an external website (such rflowcollector does)?

I'm very interested in replacing rflow which do not work as service.

_________________
DD-WRT WDS MESH + DASHBOARD (fr), DD-WRT network setting tool (tools.yzy-oui-fi.com), Wifi Business and IT guy After hours, My Blog, Free DD-WRT VPN Community(www.wrt-pptp-ww.com), DD-WRT pré-réglés pour réseau outdoor(hotspot.yzy-oui-fi.com), Nouveau Forum DD-WRT francophone
FezJay
DD-WRT User


Joined: 11 Sep 2010
Posts: 176

PostPosted: Wed Oct 27, 2010 9:30    Post subject: Reply with quote
yzy-oui-fi wrote:
very nice stuff!

did you thought of exporting data to an external website (such rflowcollector does)?

I'm very interested in replacing rflow which do not work as service.


Interesting that you bring this up.

One of the reasons why I allow a user to save the usage file to a directory (often /tmp/) and generate an external javascript file (default is stored in /tmp/www/) is so, if a user wishes they can move the files over to another web server, and in the case of the usage file, manipulate it in what way feels best (I chose javascript).

If you look at an example of the javascript, you can see it is OO in design.

I always thought it would go vice versa, create a web site that points to the js (on the router), but there is nothing preventing you from creating a script/cron job that copies/moves the files over to a web location.

Currently, you can tell the system where to copy the js file to (and the usage file).

Depending on what you have set up, you could just say copy files to an external CIFS location (this is an assumption at this point)

e.g /cifs/website/scripts/

_________________
dd-wrt eko/V24-K26 15508
Belkin F7D4301
8MB Flash
64MB RAM

Optware (jffs/mnt) 4GB Flash Drive
yzy-oui-fi
DD-WRT Guru


Joined: 03 Mar 2009
Posts: 2826
Location: France

PostPosted: Wed Oct 27, 2010 10:06    Post subject: Reply with quote
ok, i do not want to use a directory like jffs, tmp or smbshare, because the idea is that deices are remote.

actualy we use rflowcollector server between remote device and mysql server. I would like to remove the rflow collector from this procedure. It seems that your project could be the base of what i'd like to do.

I just asked you if you tried to work at it because i'm little bit lazy today Wink.

I will have a look and try modifyed scripts in order it do not store data on the device but send it to my mysql server.

anyway, clap hands, this is a nice job!

_________________
DD-WRT WDS MESH + DASHBOARD (fr), DD-WRT network setting tool (tools.yzy-oui-fi.com), Wifi Business and IT guy After hours, My Blog, Free DD-WRT VPN Community(www.wrt-pptp-ww.com), DD-WRT pré-réglés pour réseau outdoor(hotspot.yzy-oui-fi.com), Nouveau Forum DD-WRT francophone
FezJay
DD-WRT User


Joined: 11 Sep 2010
Posts: 176

PostPosted: Wed Oct 27, 2010 11:19    Post subject: Reply with quote
yzy-oui-fi wrote:
ok, i do not want to use a directory like jffs, tmp or smbshare, because the idea is that deices are remote.

actualy we use rflowcollector server between remote device and mysql server. I would like to remove the rflow collector from this procedure. It seems that your project could be the base of what i'd like to do.

I just asked you if you tried to work at it because i'm little bit lazy today Wink.

I will have a look and try modifyed scripts in order it do not store data on the device but send it to my mysql server.

anyway, clap hands, this is a nice job!



Ahhhhhhhhh, I see what you are hinting at.

Instead of saving the data in a flat file, can I send the data to a database?

Well, buddy, anything is possible Wink.

This project doesnt do that at present, as one of the goals was to have it self contained and easy to setup on the router.

But, the script that I have can be easily modified to call a SQL statement to insert/update the data into a database.

I already have the statements for retrieving data, all you would need to do is

1. Connect to DB
2. Invoke SQL statement(s)

Instead of writing to internal objects or a file.

Very do-able. If you have an external server running all the time.

_________________
dd-wrt eko/V24-K26 15508
Belkin F7D4301
8MB Flash
64MB RAM

Optware (jffs/mnt) 4GB Flash Drive
yzy-oui-fi
DD-WRT Guru


Joined: 03 Mar 2009
Posts: 2826
Location: France

PostPosted: Wed Oct 27, 2010 15:12    Post subject: Reply with quote
yes! Smile
_________________
DD-WRT WDS MESH + DASHBOARD (fr), DD-WRT network setting tool (tools.yzy-oui-fi.com), Wifi Business and IT guy After hours, My Blog, Free DD-WRT VPN Community(www.wrt-pptp-ww.com), DD-WRT pré-réglés pour réseau outdoor(hotspot.yzy-oui-fi.com), Nouveau Forum DD-WRT francophone
bpsmicro
DD-WRT User


Joined: 07 Oct 2010
Posts: 160

PostPosted: Wed Oct 27, 2010 16:09    Post subject: Reply with quote
Couple of newbie questions:

- in your setup instructions above, step 2, you mention setting up a startup script. Can you (or somebody) post the *exact* text I'd stick in my startup script to make that work? Is it just those two lines?

- On my router, I'm using the Buffalo-specific build of DD-WRT. I don't have a /tmp/dnsmasq.config, but I do have a /tmp/dnsmasq.conf. Is that the same thing?

I think once I get past the above, I can figure out the rest. I have a USB key set up as a cifs drive, and also a samba share set up to a Linux Virtual Machine on my network (I don't "know" Linux, but I can keep a VM running).

Brad.
FezJay
DD-WRT User


Joined: 11 Sep 2010
Posts: 176

PostPosted: Wed Oct 27, 2010 20:28    Post subject: Reply with quote
bpsmicro wrote:
Couple of newbie questions:

- in your setup instructions above, step 2, you mention setting up a startup script. Can you (or somebody) post the *exact* text I'd stick in my startup script to make that work? Is it just those two lines?

- On my router, I'm using the Buffalo-specific build of DD-WRT. I don't have a /tmp/dnsmasq.config, but I do have a /tmp/dnsmasq.conf. Is that the same thing?

I think once I get past the above, I can figure out the rest. I have a USB key set up as a cifs drive, and also a samba share set up to a Linux Virtual Machine on my network (I don't "know" Linux, but I can keep a VM running).

Brad.


Lol, yeah. I should stop guessing the extensions of files.

You are right, it is the dnsmasq file in tmp Wink.

The Wiki Sites I have created several pages that look at setup, http://code.google.com/p/lal-projects/wiki/bw_monitor_setup

My current setup, In use 2 scripts, a startup and a wanup. The startup does a quick set up of rules, while the wanup starts and keeps the script going.

A simple startup/wanup script would be this:

while [ ! -f /tmp/monitor-started.lock ] && [ ! -f /tmp/monitor-stop ]; do
cp /mnt/monitor/setup/monitor.html /tmp/www/monitor.html
/mnt/monitor/setup/bw_monitor.sh 30 3 30 4 /mnt/monitor/setup/users.file /mnt/monitor/mac_usage.backup /mnt/monitor/history/ 1 '' 1 1 1 1 1
if [ ! -f /tmp/monitor-started.lock ]; then
sleep 10
fi
done &

Dont worry about the while statement, it will end when the lock file is created or you manually create the monitor-stop file.

From my above example, I have all options enabled.

Also, you dont need a VM running. Once everything is configured, it will run in RAM (whole system uses 400KB and I am monitoring 20 MACS).

All you need is a valid place to store the usage backup (and archive if enabled), and the JS file (which is about 40 to 100KB, very small).

_________________
dd-wrt eko/V24-K26 15508
Belkin F7D4301
8MB Flash
64MB RAM

Optware (jffs/mnt) 4GB Flash Drive


Last edited by FezJay on Thu Oct 28, 2010 12:56; edited 2 times in total
yzy-oui-fi
DD-WRT Guru


Joined: 03 Mar 2009
Posts: 2826
Location: France

PostPosted: Wed Oct 27, 2010 22:42    Post subject: Reply with quote
I corrected your syntax, because with router, this could be bothering issue(end line syntax).

Code:
while [ ! -f /tmp/monitor-started.lock ] && [ ! -f /tmp/monitor-stop ]; do
cp /mnt/monitor/setup/monitor.html /tmp/www/monitor.html ;
/mnt/monitor/setup/bw_monitor.sh 30 3 30 4 /mnt/monitor/setup/users.file /mnt/monitor/mac_usage.backup /mnt/monitor/history 1 '' 1 1 1 1 1 ;
if [ ! -f /tmp/monitor-started.lock ]; then
sleep 10 ;
fi
done &


you could also use an httpd statement to create a second webserver on your router on another port than port 80 used for webUI.

Code:
httpd -p 81 -h /mnt/setup


then directly run this from the mnt directory
you will have to call http://yourrouterIP:81 to get the stats

_________________
DD-WRT WDS MESH + DASHBOARD (fr), DD-WRT network setting tool (tools.yzy-oui-fi.com), Wifi Business and IT guy After hours, My Blog, Free DD-WRT VPN Community(www.wrt-pptp-ww.com), DD-WRT pré-réglés pour réseau outdoor(hotspot.yzy-oui-fi.com), Nouveau Forum DD-WRT francophone
FezJay
DD-WRT User


Joined: 11 Sep 2010
Posts: 176

PostPosted: Wed Oct 27, 2010 23:32    Post subject: Reply with quote
yzy-oui-fi wrote:
I corrected your syntax, because with router, this could be bothering issue(end line syntax).

Code:
while [ ! -f /tmp/monitor-started.lock ] && [ ! -f /tmp/monitor-stop ]; do
cp /mnt/monitor/setup/monitor.html /tmp/www/monitor.html ;
/mnt/monitor/setup/bw_monitor.sh 30 3 30 4 /mnt/monitor/setup/users.file /mnt/monitor/mac_usage.backup /mnt/monitor/history 1 '' 1 1 1 1 1 ;
if [ ! -f /tmp/monitor-started.lock ]; then
sleep 10 ;
fi
done &


you could also use an httpd statement to create a second webserver on your router on another port than port 80 used for webUI.

Code:
httpd -p 81 -h /mnt/setup


then directly run this from the mnt directory
you will have to call http://yourrouterIP:81 to get the stats


Thanks man

I Didnt realise that end line/statement sytax was even used lol. Im a c# dev, so initially I had ; every where but I thought shell complained about them so I forced myself to not put them in Smile.


That webserver statement is very useful. Do you know if you can apply basic authentication (simple username password) to such a web server?

_________________
dd-wrt eko/V24-K26 15508
Belkin F7D4301
8MB Flash
64MB RAM

Optware (jffs/mnt) 4GB Flash Drive
FezJay
DD-WRT User


Joined: 11 Sep 2010
Posts: 176

PostPosted: Thu Oct 28, 2010 0:07    Post subject: Reply with quote
I have some future ideas once I have released the first version of this.

These are on my googldecode home page and found in the googlecode wiki pages as well.


Future Work/Ideas

    * Allow users to force a backup
    * Allow users to force a publish
    * Provide the ability to stop/pause the refresh (workaround, enter a large number Wink
    * Allow users to set the bytes base value (1000 or 1024 etc)
    * Add Graphs
    * Use JQuery - Have to make this optional as jquery takes up space
    * Add automatic or warning security. If you specify an IP to a MAC or vice versa, you expect this to always occur. What happens if it doesnt? Someone is trying to get around it. Well theoritically, the script could add rules that prevent users that do this from downloading/uploading. I could also create a warnings section, that displays users that have had an incorrect IP assigned to them etc.
    * Maybe I should create a custom config file instead of having everything passed via command line? What do you guys think? Maybe give users the option....yes much better idea.
    * Clean UP the UI. This will be ongoing. Make it more stylie so to speak. I wish I could use JQuery with this lol.
    * Store Usage by MONTH and DAY. Mmmmmm. This will definately make the JS file way bigger lol. Imagine seeing on the UI, current month, then prev, next, that show other months. Ooooo me likey. If you are going to use this, I would recommend getting a USB drive an creating a public server of it. May run into issues with YEAR. This needs alot more thinking


What do you guys think?

If you have any ideas, please post them.

The plan is, I will select the best 4 and release them in the next version after Version 1.0 Wink.

_________________
dd-wrt eko/V24-K26 15508
Belkin F7D4301
8MB Flash
64MB RAM

Optware (jffs/mnt) 4GB Flash Drive
bpsmicro
DD-WRT User


Joined: 07 Oct 2010
Posts: 160

PostPosted: Thu Oct 28, 2010 14:47    Post subject: Reply with quote
So here's what I did:

- Renamed 'interface.html' in your download section to 'monitor.html', since your samples use monitor.
- Modified my copy of monitor.html to use the right IP for my router (it's not 192.168.1.1)
- Added to my startup script:

while [ ! -f /tmp/monitor-started.lock ] && [ ! -f /tmp/monitor-stop ]; do
cp /mnt/mmc/monitor.html /tmp/www/monitor.html ;
/mnt/mmc/bw_monitor.sh 30 3 30 15 /tmp/dnsmasq.conf /mnt/mmc/mac_usage.backup /mnt/mmc/history ;
if [ ! -f /tmp/monitor-started.lock ]; then
sleep 10 ;
fi
done &

with paths tweaked to reflect my directory structure. Confirmed location of dnsmasq.conf name & location.
- created a /mnt/mmc/history folder (I assume the folder needs to be there).
- rebooted the router

Upon reboot, I see that monitor.html is copied to /tmp/www. I can bring up that page, but there's no actual data there.

"mac_usage.backup" isn't in my /mnt/mmc folder. Is the script supposed to be creating that?

- the html file refers to a "user_details.js". Where's that supposed to come from?

- I don't have a 'wanup' (or at least nothing by that name is in the dd-wrt interface), so I added the wanup lines to my startup script. But your posted bw_monitor.startup has a bunch of other stuff that you don't reference in this thread. Does that also need to go in my startup? Before or after the wanup stuff?

Seriously, when I say I'm a newbie at this, I mean it. You can't skip *any* steps in the setup instructions. Smile
bpsmicro
DD-WRT User


Joined: 07 Oct 2010
Posts: 160

PostPosted: Thu Oct 28, 2010 15:50    Post subject: Reply with quote
Hmm, a clue. If I telnet into my router and run the bw_monitor.sh line directly, I get:

-sh: /mnt/mmc/bw_monitor.sh: not found

Despite the fact I can look in my /mnt/mmc folder and *clearly* see that it's there, and marked to execute (I actually did a chmod 777 to make sure).

If I run "sh bw_monitor.sh ...", I get some obscure warning about line 37 ("{"), and a syntax error on line 57 about it expecting 'then' instead of 'fi'.

And the last thing created the /tmp/monitor-started.lock, and I can't remove it (I expect it'll go away with the next reboot, which I'm doing next).

Also, on my router the dnsmasq.conf file looks nothing like what's described in the wiki. dnsmasq.leases looks sorta like what you need. For now I made my own MACtoNAME.txt, since I have that data handy anyway.

Debugging tips?

Brad.
Goto page 1, 2, 3 ... 40, 41, 42  Next Display posts from previous:    Page 1 of 42
Post new topic   Reply to topic    DD-WRT Forum Forum Index -> Broadcom SoC based Hardware All times are GMT

Navigation

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum