YAMon 3.0... Usage Reporting by device per hr/day/mo

Post new topic   This topic is locked: you cannot edit posts or make replies.    DD-WRT Forum Index -> General Questions
Goto page Previous  1, 2, 3 ... 104, 105, 106 ... 149, 150, 151  Next
Author Message
al_c
DD-WRT Guru


Joined: 13 Apr 2013
Posts: 2134
Location: Ottawa Canada

PostPosted: Mon Jul 17, 2017 5:08    Post subject: Re: Losing data on local devices Reply with quote
racerx1234 wrote:
Yeah, I see it too. I am d/l at least a few gigs a day. I will reboot the router later tonite and see if it fixes anything.


When did you install YAMon? Recently or should there be data going back to the start of this interval? Does this help? Scheduled Router Reboots at 12AM

Did you just update your firmware?

If YAMon goes south again, enter the following command in a PuTTY window:

Code:
iptables -L | grep Chain


It'll give just the chain names without all of the details - e.g., on mine I get the following results:

Code:
root@DD-WRT:~# iptables -t mangle -L | grep Chain
Chain PREROUTING (policy ACCEPT)
Chain INPUT (policy ACCEPT)
Chain FORWARD (policy ACCEPT)
Chain OUTPUT (policy ACCEPT)
Chain POSTROUTING (policy ACCEPT)
Chain YAMON33v4 (1 references) ***
Chain YAMON33v4Entry (3 references) ***
Chain YAMON33v4Local (9 references) ***
Chain YAMON_DBv4 (1 references)
Chain YAMON_DBv4Entry (3 references)
Chain YAMON_DBv4Local (9 references)
Chain YAMONv4 (3 references)

You want to see the ones with *** above
NB - for testing purposes, I have _useTMangle=1 (so I have to include the `-t mangle` in my command, you likely do not need it). Ignore the YAMON_DBv4 entries... that is a new version that I'm working on. And, YAMONv4 is my instance of YAMon3.2 that I'm running in parallel for testing purposes.

When YAMon stops collecting, I think that one or more of the YAMON33v4 chains get dropped somehow (I have no idea how or why). I'm close to having 3.3.1 ready and have added some code that should address this issue.

In the meantime, set _loglevel=0 and let things run for a bit then send the zipped log file to questions@usage-monitoring.com please.

Sorry - no answers just questions.

Al
Sponsor
ddaniel51
DD-WRT Guru


Joined: 19 Feb 2013
Posts: 1464

PostPosted: Mon Jul 17, 2017 12:19    Post subject: Reply with quote
I suggest NOT scheduling a router reboot at 12 AM as this will interfere with YAMon end of day activities.
Schedule a reboot at an odd time like 00:13 so YAMon is finished with it's new day functions.

_________________
Segment 1 XR700 10Gb LAN, 1Gb WAN ISP BS
Wired AP 1 Unifi Wifi 6 LR US 1Gb LAN
Wired AP 2 Unifi Wifi 6 LR US 1Gb LAN
Wired AP 3 Unifi Wifi 6 LR US 1Gb LAN
Syslog Services Asustor 7110T NAS 10GB
NetGear XS716T 10GB Switch
download1.dd-wrt.com/dd-wrtv2/downloads/betas/ (Brain Slayer)
YAMon https://usage-monitoring.com/index.php
al_c
DD-WRT Guru


Joined: 13 Apr 2013
Posts: 2134
Location: Ottawa Canada

PostPosted: Mon Jul 17, 2017 13:34    Post subject: Reply with quote
ddaniel51 wrote:
I suggest NOT scheduling a router reboot at 12 AM as this will interfere with YAMon end of day activities.
Schedule a reboot at an odd time like 00:13 so YAMon is finished with it's new day functions.


Setup.sh warns if the router is scheduled to reboot at midnight. I suppose I could add a check after the fact but it seems like a lot of overhead when a DDT (Don't Do That) could be sufficient.

Al
racerx1234
DD-WRT Novice


Joined: 11 Mar 2017
Posts: 7

PostPosted: Mon Jul 17, 2017 15:52    Post subject: Re: Losing data on local devices Reply with quote
al_c wrote:
racerx1234 wrote:
Yeah, I see it too. I am d/l at least a few gigs a day. I will reboot the router later tonite and see if it fixes anything.


When did you install YAMon? Recently or should there be data going back to the start of this interval? Does this help? Scheduled Router Reboots at 12AM

Did you just update your firmware?

If YAMon goes south again, enter the following command in a PuTTY window:

Code:
iptables -L | grep Chain


It'll give just the chain names without all of the details - e.g., on mine I get the following results:

Code:
root@DD-WRT:~# iptables -t mangle -L | grep Chain
Chain PREROUTING (policy ACCEPT)
Chain INPUT (policy ACCEPT)
Chain FORWARD (policy ACCEPT)
Chain OUTPUT (policy ACCEPT)
Chain POSTROUTING (policy ACCEPT)
Chain YAMON33v4 (1 references) ***
Chain YAMON33v4Entry (3 references) ***
Chain YAMON33v4Local (9 references) ***
Chain YAMON_DBv4 (1 references)
Chain YAMON_DBv4Entry (3 references)
Chain YAMON_DBv4Local (9 references)
Chain YAMONv4 (3 references)

You want to see the ones with *** above
NB - for testing purposes, I have _useTMangle=1 (so I have to include the `-t mangle` in my command, you likely do not need it). Ignore the YAMON_DBv4 entries... that is a new version that I'm working on. And, YAMONv4 is my instance of YAMon3.2 that I'm running in parallel for testing purposes.

When YAMon stops collecting, I think that one or more of the YAMON33v4 chains get dropped somehow (I have no idea how or why). I'm close to having 3.3.1 ready and have added some code that should address this issue.

In the meantime, set _loglevel=0 and let things run for a bit then send the zipped log file to questions@usage-monitoring.com please.

Sorry - no answers just questions.

Al



I did a clean install July 3rd. Deleted all the files, loaded a new install.sh. Had a few issues with the 3.3 files served but when you fixed that, everything seemed fine. No firmware update, Firmware: DD-WRT v24-sp2 (06/07/14) kingkong Linksys E4200

Just rebooted the router, it looks like it is picking up data again. I will keep you posted.
al_c
DD-WRT Guru


Joined: 13 Apr 2013
Posts: 2134
Location: Ottawa Canada

PostPosted: Tue Jul 18, 2017 5:23    Post subject: Just posted YAMon 3.3.1 Reply with quote
I posted YAMon v3.3.1 to http://usage-monitoring.com last night.

Lots of changed files with a bunch of housekeeping
* removed several blocks of unused code
* refactored things a bit
* changed from ip_conntrack to nf_conntrack for the live tab
* added code that should hopefully better handle odd router issues where iptable entries go away?!?
* added a count index to all new devices (so you should see New Device-1, New Device-2, ... rather than all of the appearing with the same default name)

To update, re-run install.sh or go to the manual install page

It's late, I really hope this works for everyone else too... it does at my end.

-----------------
NB - I recommend that you do *not* upgrade to the dd-wrt firmware build r32597... a number of issues have been reported against this build.

I just noticed that there is another more recent build now available today. I will update to it shortly (I made the mistake of updating to 32597)

I'll let you know if it fixes the busybox issue which is causing the download problem in install.sh

-----------------

Thanks to everyone for their assistance with this little project.

Al


Last edited by al_c on Tue Jul 18, 2017 15:35; edited 1 time in total
gainestr
DD-WRT User


Joined: 06 Jul 2017
Posts: 78

PostPosted: Tue Jul 18, 2017 15:32    Post subject: Re: Just posted YAMon 3.3.1 Reply with quote
al_c wrote:
a bunch of housekeeping
removed several blocks of unused code
refactored things a bit
changed from ip_conntrack to nf_conntrack for the live tab
added code that should hopefully better handle odd router issues where iptable entries go away?!?
add a count index to all new devices (so you should see New Device-1, New Device-2, ... rather than all of the appearing with the same default name)

re-run install.sh or go to the manual install page

it's late, I really hope this works for everyone else too... it does at my end.


Only issue I'm having with this build is this hourly usage alert. Any ideas?



Screen Shot 2.png
 Description:
 Filesize:  71.08 KB
 Viewed:  5389 Time(s)

Screen Shot 2.png


al_c
DD-WRT Guru


Joined: 13 Apr 2013
Posts: 2134
Location: Ottawa Canada

PostPosted: Tue Jul 18, 2017 15:39    Post subject: Re: Just posted YAMon 3.3.1 Reply with quote
gainestr wrote:
al_c wrote:
a bunch of housekeeping
removed several blocks of unused code
refactored things a bit
changed from ip_conntrack to nf_conntrack for the live tab
added code that should hopefully better handle odd router issues where iptable entries go away?!?
add a count index to all new devices (so you should see New Device-1, New Device-2, ... rather than all of the appearing with the same default name)

re-run install.sh or go to the manual install page

it's late, I really hope this works for everyone else too... it does at my end.


Only issue I'm having with this build is this hourly usage alert. Any ideas?


Was YAMon running properly at that time? Are there entries in the hourly file for those time intervals? If yes to either of those questions, there is an issue.

If you just installed YAMon for the first time at 9AM, then there is no issue. The message is an alert not an error... it is intended to let you know there are some holes in the data file. OK?

Al
gainestr
DD-WRT User


Joined: 06 Jul 2017
Posts: 78

PostPosted: Tue Jul 18, 2017 17:39    Post subject: Re: Just posted YAMon 3.3.1 Reply with quote
al_c wrote:


Was YAMon running properly at that time? Are there entries in the hourly file for those time intervals? If yes to either of those questions, there is an issue.

If you just installed YAMon for the first time at 9AM, then there is no issue. The message is an alert not an error... it is intended to let you know there are some holes in the data file. OK?

Al


Yes YAMon was just installed at 9AM. Thank you for clearing that up. I was going to just let it run and see if the alert would go away the following day but I thought I would post anyway. I appreciate your help with this and I will be donating so this work can continue. So far everything is working well on my WRT1900ACSv2 with DD-WRT r32597.
al_c
DD-WRT Guru


Joined: 13 Apr 2013
Posts: 2134
Location: Ottawa Canada

PostPosted: Tue Jul 18, 2017 18:27    Post subject: Re: Just posted YAMon 3.3.1 Reply with quote
gainestr wrote:
al_c wrote:


Was YAMon running properly at that time? Are there entries in the hourly file for those time intervals? If yes to either of those questions, there is an issue.

If you just installed YAMon for the first time at 9AM, then there is no issue. The message is an alert not an error... it is intended to let you know there are some holes in the data file. OK?

Al


Yes YAMon was just installed at 9AM. Thank you for clearing that up. I was going to just let it run and see if the alert would go away the following day but I thought I would post anyway. I appreciate your help with this and I will be donating so this work can continue. So far everything is working well on my WRT1900ACSv2 with DD-WRT r32597.


You're not the first to get fooled by the message. I've added a to-do to clarify the text.

DD-WRT r32597 has known issues - see http://www.dd-wrt.com/phpBB2/viewtopic.php?t=310059&highlight=
ddaniel51
DD-WRT Guru


Joined: 19 Feb 2013
Posts: 1464

PostPosted: Wed Jul 19, 2017 4:59    Post subject: Reply with quote
In the latest dd-wrt builds with busybox 1.27 the -e option
has been replaced with --full-time.

This requires a change to line 141 in yamon3.3.1.sh

local ulmt=$(ls -e "$_usersFile" | tr -s ' ' ' ' )
to
local ulmt=$(ls --full-time "$_usersFile" | tr -s ' ' ' ' )

Al, would it help to put both lines in?

_________________
Segment 1 XR700 10Gb LAN, 1Gb WAN ISP BS
Wired AP 1 Unifi Wifi 6 LR US 1Gb LAN
Wired AP 2 Unifi Wifi 6 LR US 1Gb LAN
Wired AP 3 Unifi Wifi 6 LR US 1Gb LAN
Syslog Services Asustor 7110T NAS 10GB
NetGear XS716T 10GB Switch
download1.dd-wrt.com/dd-wrtv2/downloads/betas/ (Brain Slayer)
YAMon https://usage-monitoring.com/index.php
al_c
DD-WRT Guru


Joined: 13 Apr 2013
Posts: 2134
Location: Ottawa Canada

PostPosted: Wed Jul 19, 2017 20:55    Post subject: Reply with quote
ddaniel51 wrote:
In the latest dd-wrt builds with busybox 1.27 the -e option
has been replaced with --full-time.

This requires a change to line 141 in yamon3.3.1.sh

local ulmt=$(ls -e "$_usersFile" | tr -s ' ' ' ' )
to
local ulmt=$(ls --full-time "$_usersFile" | tr -s ' ' ' ' )

Al, would it help to put both lines in?

Thanks for the update. I'll fix that ASAP

Al
al_c
DD-WRT Guru


Joined: 13 Apr 2013
Posts: 2134
Location: Ottawa Canada

PostPosted: Thu Jul 20, 2017 5:17    Post subject: possible bad symlinks Reply with quote
Thanks to Adrian for bringing this one to my attention.

If you enter a different path to the routers (rather than /opt/YAMon3), the symlinks to the reports will likely be incorrect and you will see 404 errors in the browser console window...
see http://usage-monitoring.com/help/?t=bad-symlinks for images and the work around
(which is to delete and recreate the symlinks)

This problem will only appear until your first router reboot... when the symlinks get re-created by the script, they will be correct.

Sorry if this has caused any confusion or grief.
al_c
DD-WRT Guru


Joined: 13 Apr 2013
Posts: 2134
Location: Ottawa Canada

PostPosted: Thu Jul 20, 2017 5:44    Post subject: Reply with quote
al_c wrote:
ddaniel51 wrote:
In the latest dd-wrt builds with busybox 1.27 the -e option
has been replaced with --full-time.

This requires a change to line 141 in yamon3.3.1.sh

local ulmt=$(ls -e "$_usersFile" | tr -s ' ' ' ' )
to
local ulmt=$(ls --full-time "$_usersFile" | tr -s ' ' ' ' )

Al, would it help to put both lines in?

Thanks for the update. I'll fix that ASAP

Al

Dave - the only reason I had the `ls -e` was so that I could get the last modified date for users.js
And had to do stupid gyrations to really get it into a format I wanted...

I've replace that call with `_usersLastMod=$(date -r "$_usersFile" "+%Y-%d-%m %T")` which seems to be a much cleaner way to get the info.
(the things that you learn over time... thank goodness for stackoverflow).

I'll roll this into v3.3.2... is there anyone else out there encountering the same issue?

Al
gainestr
DD-WRT User


Joined: 06 Jul 2017
Posts: 78

PostPosted: Thu Jul 20, 2017 13:36    Post subject: Reply with quote
al_c wrote:


I'll roll this into v3.3.2... is there anyone else out there encountering the same issue?

Al


Yes, I too noticed the error with the -e option on my WRT1900ACSv2 with BS build r32753. However, I was having so many issues with that build(GUI lockups, OpenVPN Server not working, inability to even reboot from interface...) that I reverted back to r32597.

It seems to me that there are so many routers that are supported with DD-WRT that it would be really easy for something to go wrong with a build every time a simple driver or new code is introduced. I'm seeing one build work great for one group of users/devices and the next build not. I'm starting to wonder if I should be looking for a build that is tested exclusively on the WRT brand of devices? I've heard of good things from the DavidC502 LEDE builds.

I know that I shouldn't be talking about such things on this thread because it isn't necessarily related to YAMon but it does show how from one version to the next things change and how it makes it difficult for Al to incorporate changes for each brand of router and each build.

Thanks Al for all your hard work on this.
NiTrus
DD-WRT User


Joined: 25 Dec 2010
Posts: 295
Location: Twin Cities, MN

PostPosted: Thu Jul 20, 2017 15:08    Post subject: Reply with quote
just a fyi, in newer fw builds with SFE(Shortcut Forwarding Engine) enabled, yamon doesnt record download/upload usage correctly, so i turned sfe off & everything started working correctly..anyone else experience this?
_________________
NETGEAR R9000 | RT | 40134
NETGEAR R7800 | AP | 40134

Goto page Previous  1, 2, 3 ... 104, 105, 106 ... 149, 150, 151  Next Display posts from previous:    Page 105 of 151
Post new topic   This topic is locked: you cannot edit posts or make replies.    DD-WRT Forum Index -> General Questions All times are GMT

Navigation

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum