YAMon 2.2 Usage Reporting: by device per hr/day/mo or live

Post new topic   Reply to topic    DD-WRT Forum Forum Index -> Broadcom SoC based Hardware
Goto page 1, 2, 3 ... 107, 108, 109  Next
Author Message
al_c
DD-WRT Guru


Joined: 13 Apr 2013
Posts: 1520
Location: Ottawa Canada

PostPosted: Wed Mar 26, 2014 5:11    Post subject: YAMon 2.2 Usage Reporting: by device per hr/day/mo or live Reply with quote
**** May 31, 2016 - FINAL UPDATE to YAMon 2.2.6 ****
Update from Al wrote:
NB: All new development will occur in the version 3 code base. See the YAMon web site for the latest installer or the YAMon 3 discussion thread in the DD-WRT forum.


Al wrote:
If you want to get YAMon but are intimidated by the notion of `flashing` your router yourself (i.e., replacing the stock firmware with DD-WRT), checkout the `Can I get a router with DD-WRT pre-installed?` help topic at the YAMon web site.


EDIT - prior to launching the 2.2 version, I recommend that you clear the contents of /tmp/www/ and/or reboot your router to ensure that any old symlinks are removed.


**** Lots of new features... see `What's new in 2.2`

Yet Another Monitor (YAMon) records and reports on the traffic (downloads and uploads) for all of the devices connecting to your router. The data is aggregated by hour, day and month (within your ISP billing interval) and can be rolled-up into arbitrary groups (e.g., by family member or by any other logical grouping of devices).

In short, YAMon gives you an unprecedented view of the traffic on your router.
- The `Daily Usage` and `Hourly Usage` reports allow you to see which devices used how much bandwidth when (did your kids really really go to school and did they really shut off their devices at bedtime?).
- The `Monthly Usage` report shows who is consuming the most bandwidth on which device and projects your total usage for your ISP billing interval (allowing you to throttle consumption before you get hit with a large overage fee).
- The `Live Usage` reports allow you to see which addresses each device is connecting to and includes geo-location lookups (you can find out or at least ask why your kid's device is connecting to a server in Lithunia or elsewhere).
- Usage history allows you to view and compare results across billing intervals.
Further, you can optionally include upload/download figures from your ISP to compare totals recorded at your router with theirs. Other options in the configuration file allow you to define an `unlimited` usage interval (e.g., some ISPs do not count usage between 2AM and 7AM towards your bandwidth cap).

YAMon runs on routers that have been flashed with one of the *WRT firmware variants (e.g. DD-WRT, OpenWRT and also AsusWRT).

New in Version 2.2.6 -
- in v2.2.6, there is a minor fix to a regular expression to allow version 2 and version 3 to run concurrently
You must now stop yamon2.sh prior to editting users.js!

Changed Files in Version 2.2 - at a minimum, the following files must be updated on your router:
- yamon2.sh (in /Setup)

Fresh Installation Instructions

killmasta93 created an installation tutorial. See http://www.dd-wrt.com/phpBB2/viewtopic.php?p=954980#954980 for the forum thread or for https://www.mediafire.com/?b488czq0xg242cg for the actual tutorial.

For new users, you can also use the readme.txt in the zip file and the attached installation guide (`Configuring YAMon 2`).

Upgrade Instructions If you are upgrading from an earlier version of YAMon:
Restart Required: YES - If you are currently running an earlier version of YAMon 2.x, you will have to stop the script, replace files, and restart it (see below)
Changes to the Configuration file: No

Upgrade Steps
If you are running an earlier version of YAMon, 2.x:
- stop the current script (enter `/opt/YAMon2/Setup/yamon2.sh --stop`)**
- unzip the archive
- copy the files to your router
***
- if you have been using a startup script to make a local copy of the JavaScript & CSS files normally hosted at my usage-monitoring.com domain, change the instructions on the DD-WRT GUI -->Administration-->Commands tab to reflect the updates in yamon.startup... that local copy functionality is now part of the main yamon2.sh script
***
- check `default config.file` to set the new `_doLocalFiles` parameter as applicable to your installation and make changes as appropriate
NB - if you do not add this parameter to your config.file, yamon2.sh will assume the default value of 0 (no copy)
- if you plan to use the new `_organizeData`, you must first run `organize.sh` (included in the zip file) to move your existing data files into the appropriate structure.

** enter that command in Putty (or the command line interface for your router); the path must change if you did not install YAMon in the default location

Viewing the reports
A number of people have told me that they've had trouble determining the URL for the reports... I guess putting it in the `readme.txt` was too subtle. The URL is
Code:
http://<router IP>/user/yamon2.html
e.g., http://192.168.1.1/user/yamon2.html


Related (but disjoint) thoughts

*** THE FOLLOWING REPORT FEATURES ARE [still] NOT WORKING ***
- IPv6 data is [still] not detected by yamon2.sh... but I've gotten a new router that supports IPv6 so I have no excuses now. NB - This is one of the major new features in YAMon3

Bugs and Feature Requests
Please report all bugs to me and I will try to get them fixed ASAP... if possible send screenshots and/or any error message (in particular those in the JavaScript console (which you can open by typing `crtl`+`shift`+`J`)). For you MSIE users, type `F12` and change to the `console` tab. Often, it is very helpful if you send the log files to me as well.
NEW - I have added a new _loglevel of `-1` that will include even more debugging info.

If you're having difficulties, post a question to this thread (or PM me through this forum or at al@caughey.ca).

As always, your feedback (and encouragement) is greatly appreciated!

Al

Prior to the 2.0.12 update on Jul 30 2014, YAMon had been downloaded 1085 times.
Prior to the 2.0.13 update on Aug 13 2014, YAMon had been downloaded 1292 times.
Prior to the 2.0.15 update on Oct 17 2014, YAMon had been downloaded 1835 times.
Prior to the 2.0.16 update on Oct 17 2014, YAMon had been downloaded 2014 times.
Prior to the 2.0.17 update on Nov 20 2014, YAMon had been downloaded 2252 times.
Prior to the 2.0.18 update on Mar 20 2015, YAMon had been downloaded 3588 times.
Prior to the 2.1.0 update on May 31 2015, YAMon had been downloaded 4272 times.
As of Aug 28, 2015, YAMon has been downloaded 5344 times
As of Sep 9, 2015, YAMon has been downloaded 5641 times
As of May 31, 2016, YAMon 2 has been downloaded 9408 times!



install (2016-10-05).zip
 Description:
See http://usage-monitoring.com/download.php for the latest v3 installer

Download
 Filename:  install (2016-10-05).zip
 Filesize:  9.29 KB
 Downloaded:  11076 Time(s)


Configuring YAMon 2.pdf
 Description:

Download
 Filename:  Configuring YAMon 2.pdf
 Filesize:  434.38 KB
 Downloaded:  9259 Time(s)



Last edited by al_c on Fri Dec 30, 2016 18:28; edited 58 times in total
Sponsor
al_c
DD-WRT Guru


Joined: 13 Apr 2013
Posts: 1520
Location: Ottawa Canada

PostPosted: Wed Mar 26, 2014 16:34    Post subject: Re: cp: invalid option -- u Reply with quote
andyrav wrote:
Getting the following message
cp: invalid option -- u
cp: invalid option -- u
root@Linksys E4200:/opt/YAMon2/Setup# cp: invalid option -- u

any ideas what is wrong
cheers


andyrav - Thanks for the feedback; sorry for the error message.

The copyfiles function is crapping out...
In particular, the problem is on line #107 $(echo "cp -au $src $dst")

I included the -u parameter so (in theory) the `cp` command only replaces files that have been changed. This works with my firmware but apparently not yours. (NUTS!)

Change the line to: $(echo "cp -a $src $dst")
That should make those messages go away.

What router & what firmware build are you using? Did the report files get copied to the `/tmp/www` directory (in spite of the error message)?

(In a future update, I might add an option in the config.file that allows you to specify the `cp` parameters so that *smarter* firmware builds can use the -u parameter... just a thought)

Al
andyrav
DD-WRT Novice


Joined: 04 Mar 2014
Posts: 21

PostPosted: Wed Mar 26, 2014 16:52    Post subject: Reply with quote
Linksys E3400
Firmware: DD-WRT v24-sp2 (12/05/13) mega

/tmp/YAMon-running is empty
/tmp/www only contains folder this itis missing yamon2.html

there is no data in daily usage

Error #4 reading the devices data file: `data/users.js`
Not Found
Error #4 reading the hourly data file: `data/2014-03-26-hourly_data.js`
Not Found
Error #4 reading the monthly data file: `data/2014-03-01-mac_data.js`
Not Found
Mizehra
DD-WRT Novice


Joined: 10 Oct 2013
Posts: 47

PostPosted: Wed Mar 26, 2014 17:25    Post subject: Reply with quote
Looks like some pretty hefty issues going on with mine. grep is unhappy with it, as well as cp.

root@DD-WRT:/tmp/smbshare# ./yamon.startup

2014-03-26 13:42:39 0 === Checking the script parameters ===
[NB - this message is being shown on the screen because
the path to the log file wont't be known until
the config file has been read.]

cp: illegal option -- u
cp: illegal option -- u
root@DD-WRT:/tmp/smbshare# grep: illegal option -- w
root@DD-WRT:/tmp/smbshare# grep: illegal option -- w
grep: illegal option -- w
grep: illegal option -- w
grep: illegal option -- w
grep: illegal option -- w
grep: illegal option -- w
grep: illegal option -- w
grep: illegal option -- w
grep: illegal option -- w
grep: illegal option -- w
grep: illegal option -- w
grep: illegal option -- w
grep: illegal option -- w
grep: illegal option -- w
grep: illegal option -- w
grep: illegal option -- w
grep: illegal option -- w
grep: illegal option -- w
grep: illegal option -- w
grep: illegal option -- w
grep: illegal option -- w
grep: illegal option -- w
grep: illegal option -- w
grep: illegal option -- w
grep: illegal option -- w
grep: illegal option -- w
grep: illegal option -- w
grep: illegal option -- w

root@DD-WRT:/tmp/smbshare# ./yamon.shutdown

root@DD-WRT:/tmp/smbshare# cp: illegal option -- u


root@DD-WRT:/tmp/smbshare#
al_c
DD-WRT Guru


Joined: 13 Apr 2013
Posts: 1520
Location: Ottawa Canada

PostPosted: Wed Mar 26, 2014 19:14    Post subject: Reply with quote
Mizehra wrote:
Looks like some pretty hefty issues going on with mine. grep is unhappy with it, as well as cp.

root@DD-WRT:/tmp/smbshare# ./yamon.startup

2014-03-26 13:42:39 0 === Checking the script parameters ===
[NB - this message is being shown on the screen because
the path to the log file wont't be known until
the config file has been read.]

cp: illegal option -- u
cp: illegal option -- u
root@DD-WRT:/tmp/smbshare# grep: illegal option -- w
...
grep: illegal option -- w

root@DD-WRT:/tmp/smbshare# ./yamon.shutdown

root@DD-WRT:/tmp/smbshare# cp: illegal option -- u


root@DD-WRT:/tmp/smbshare#


As described earlier, the first problem is in the copyfiles function... remove the -u from line #107


The second problem is another example of inconsistencies between firmware versions... it appears that your grep version does not support the -w (whole word match)... this is going to be a little more problematic... That option was included to prevent partial matching of IP addresses -e.g., without this option, it could be that 192.168.1.1 would also match 192.168.1.13 or 192.168.1.119, etc.

There is a relatively simple fix to this one as well but there are several instances in yamon2.sh that will require tweaks:

the first occurrence is found in the function `checkIPChain()`
Line #330 can be changed to
Code:
foundRule=$(iptables -L FORWARD | grep -c "$chain")

(i.e., the options for grep changed from `-cw` to just `-c`)

the next occurrences are in the function `updateUsage()`
Lines #631 & #632 can be changed to
Code:
ndown=$(echo "$iptablesData" | grep -i "0\s*$cIP " | cut -d' ' -f1)
    nup=$(echo "$iptablesData" | grep -i "$cIP \s*0" | cut -d ' ' -f1)

(i.e., the options for grep changed from `-w` to `-i` *and* note that a space has been added after `$cIP` (inside the double quotes) on both lines)

And finally , the changes as above must be made in the function `doliveUpdates()`... it is an entire coincidence that these lines are #731 & #732
Code:
ndown=$(echo "$iptablesData" | grep -i "0\s*$cIP " | cut -d' ' -f1)
    nup=$(echo "$iptablesData" | grep -i "$cIP \s*0" | cut -d ' ' -f1)

(again, the options for grep changed from `-w` to `-i` *and* a space has been added after `$cIP` in both lines)

NB - you will have to stop and restart the script for these changes to take effect.

Thanks for bringing this to my attention.

Al


Last edited by al_c on Wed Mar 26, 2014 19:17; edited 1 time in total
al_c
DD-WRT Guru


Joined: 13 Apr 2013
Posts: 1520
Location: Ottawa Canada

PostPosted: Wed Mar 26, 2014 19:16    Post subject: Reply with quote
andyrav wrote:
Linksys E3400
Firmware: DD-WRT v24-sp2 (12/05/13) mega

/tmp/YAMon-running is empty
/tmp/www only contains folder this itis missing yamon2.html

there is no data in daily usage

Error #4 reading the devices data file: `data/users.js`
Not Found
Error #4 reading the hourly data file: `data/2014-03-26-hourly_data.js`
Not Found
Error #4 reading the monthly data file: `data/2014-03-01-mac_data.js`
Not Found


Have you stopped the script, made the changes as suggested above and then restarted the script?

Please ensure that logging is turned on and the loglevel is set to 0 in config.file


Al
andyrav
DD-WRT Novice


Joined: 04 Mar 2014
Posts: 21

PostPosted: Wed Mar 26, 2014 21:19    Post subject: build Reply with quote
any chance u can u do another build with the changes in
NiTrus
DD-WRT User


Joined: 25 Dec 2010
Posts: 261
Location: Twin Cities, MN

PostPosted: Wed Mar 26, 2014 22:59    Post subject: Reply with quote
yamon2.html isnt getting copied to tmp/www directory..i see the yamon.html..ive tryed stopping/starting a few times..it never gets copied??

http://192.168.1.1/user/yamon2.html doesnt work?

yes i'm running both versions..

_________________
NETGEAR R9000 | RT | 32170
NETGEAR R7800 | AP | 32170
NETGEAR R7000 | AP | 32170

al_c
DD-WRT Guru


Joined: 13 Apr 2013
Posts: 1520
Location: Ottawa Canada

PostPosted: Thu Mar 27, 2014 4:32    Post subject: Re: build Reply with quote
andyrav wrote:
any chance u can u do another build with the changes in


Done... see the attachments on the first message of this thread. I've also included a PDF to hopefully help people get over the hump.

Al
al_c
DD-WRT Guru


Joined: 13 Apr 2013
Posts: 1520
Location: Ottawa Canada

PostPosted: Thu Mar 27, 2014 4:34    Post subject: Reply with quote
NiTrus wrote:
yamon2.html isnt getting copied to tmp/www directory..i see the yamon.html..ive tryed stopping/starting a few times..it never gets copied??

http://192.168.1.1/user/yamon2.html doesnt work?

yes i'm running both versions..


try 2.0.1 to see if you were getting the copy error. Otherwise make sure logging is turned on (and set to 0). Then send me the log.
Thanks!
andyrav
DD-WRT Novice


Joined: 04 Mar 2014
Posts: 21

PostPosted: Thu Mar 27, 2014 9:12    Post subject: Problems Reply with quote
Cool it is loading now. But...
there is no values in the Daily Usage > Downloads column.
and i copied the data from the old Yamon and ran the convert.sh. but it is now showing any old data.
andyrav
DD-WRT Novice


Joined: 04 Mar 2014
Posts: 21

PostPosted: Thu Mar 27, 2014 9:13    Post subject: Problems Reply with quote
Cool it is loading now. But...

there is no values in the Daily Usage > Downloads column.
i copied the data from the old Yamon and ran the convert.sh.
I have data in Monthly Totals and Devices
Nothing in Monthly breakdown and live update is not show anything.

Log file attached

ps great work!!!
al_c
DD-WRT Guru


Joined: 13 Apr 2013
Posts: 1520
Location: Ottawa Canada

PostPosted: Thu Mar 27, 2014 12:32    Post subject: Re: Problems Reply with quote
andyrav wrote:
Cool it is loading now. But...

there is no values in the Daily Usage > Downloads column.
i copied the data from the old Yamon and ran the convert.sh.
I have data in Monthly Totals and Devices
Nothing in Monthly breakdown and live update is not show anything.

Log file attached

ps great work!!!

Thanks for the encouragement and the feedback

The error you were seeing was occuring in my debugtrace function which is meant to show where errors are occuring rather than cause them... the irony of that is well... frustrating.

Oddly, my version of Firefox did not die but Chrome and Safari did. I've disabled that function and things seem to be working now... except, I just noticed that Safari doesn't like something about the pie charts... I'll fix that later in the day.

Thanks for the feedback!

Al
howard30
DD-WRT Novice


Joined: 07 Mar 2014
Posts: 30

PostPosted: Thu Mar 27, 2014 15:01    Post subject: Reply with quote
YAMon 2.0.1.zip doesn't have Setup folder. Should we mkdir Setup and move strings and www into it, along all sh script files?
andyrav
DD-WRT Novice


Joined: 04 Mar 2014
Posts: 21

PostPosted: Thu Mar 27, 2014 16:08    Post subject: Reply with quote
howard30 wrote:
YAMon 2.0.1.zip doesn't have Setup folder. Should we mkdir Setup and move strings and www into it, along all sh script files?


i did yes and just moved all files into it
Goto page 1, 2, 3 ... 107, 108, 109  Next Display posts from previous:    Page 1 of 109
Post new topic   Reply to topic    DD-WRT Forum Forum Index -> Broadcom SoC based Hardware All times are GMT

Navigation

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum