Ok, I was able to fix it. I edited the install.sh and added a sleep right before the validation for 20 seconds. Then I replaced those files, it validated and everything was good. I'm still not sure how these two files got corrupted every time I tried to install (more than 10 times). Could it be a result of an old symlink?
Hmmm? One of the APs just wrote a lower case MAC entry into
users.js which already had an upper case entry for the same device??
DD-WRT Ver 31780 R8500
Dave
Dave - just saw this... I thought that I always converted mac's to lower case (or upper... not sure which)...
I think I *used* to use the ignore case option when grep`ing mac (but that is less efficient and I thought I was converting all of them. I missed one... I guess.
/opt/YAMon3/yamon3.2.5.sh: line 1028: syntax error: unterminated quoted string
I've been checking it for a while and it looks fine to me, so I'm not sure what's up. You might want to have a look into it. It doesn't run with this error, it stops after about 5 seconds.
I thought I created a help topic for this... apparently not.
The `unterminated quoted string` error message is a sure fire indication that
a) your firmware only includes the `wget` function, and
b) `yamon3.2.5.sh` did not download properly when you ran `install.sh` (I expect that if you edit the file on your router, you'll likely see that ends abruptly in the middle of line 1028).
Unfortunately, re-running `install.sh` likely won't work because `wget` is going to download the same cached version of that file - i.e., the incomplete one that stops in line 1028. There is no way - that I know of - that forces `wget` to clear the cache.
The best bet is to go to the manual install page (http://usage-monitoring.com/manualInstall.php), get a fresh copy of `yamon3.2.5.sh` and copy it to your router via WinSCP.
Hmmm... see my response to your earlier post... it looks like both `yamon3.2.5.sh` & `www/yamon3.2.html` did not download properly when you ran install.sh
Again, sorry for the grief... I blame it on the version of `wget` that comes with busybox in your firmware.
The `curl` function provides the improved functionality over `wget` (like the ability to bypass cached versions of the file)... but mmany DD-WRT variants do not include `curl`. BTW, install.sh will use `curl` if it is available and fails over to `wget` if not.
Ok, I was able to fix it. I edited the install.sh and added a sleep right before the validation for 20 seconds. Then I replaced those files, it validated and everything was good. I'm still not sure how these two files got corrupted every time I tried to install (more than 10 times). Could it be a result of an old symlink?
Is it possible to see a deeper view than what YAMon shows by default?
What I mean is, can I see what applications or websites are using all this bandwidth? It shows me at around 30 GB / day. But i'm working most of the day, other than that just watching videos or livestreams. I'm thinking it could be an app in the background but have no way of checking. If YAMon can not achieve this, are there other ways/scripts to accomplish this?
Thanks,
evac
The live connections tab shows the current number of connections and devices with connections, as well as source/destination address, port and protocol (tcp or udp). You can start to partially infer what the devices are doing by checking the addresses (lots normally with traffic to apple, amazon, facebook, google, microsoft and the big server farm companies, etc...) but it is not automatically all done for you.
By collating a days worth of livedata file contents (copying the file's contents each time a new version is updated) it wouldn't be too hard to start making conclusions as to top addresses/ports per device or per top devices. Although lots of apps use secure http and http nowadays and that would hinder knowing precisely what the guilty apps are (except for when/if you see obviously identifiable stuff like ftp, telnet, smtp ports, etc.. ), one could still learn a lot just by where the traffic is being exchanged with, for the remote websites/addresses.
I've been meaning to see if I could mess around with something like that (I can't imagine it being very hard to write some scripts to do all that for someone that has the right skills). I don't have the right skills. Trying to see if I can teach myself some windows batch file programming right now, since my Yamon is CIFS hosted from a windows share. Also, trying to get a sym link from the livedata file back to my yamon directory, since that would then be on my CIFS. Not having much success with either (like I said, I really don't have the correct skills).
I won't ask Al to do this because he already has his hands full with other priorities and I already earlier mentioned some other possible changes to the live data tab that I think would be better to have first.
Also, not sure about additional load on the router for some of this (maybe my setup would be an advantage since windows batch file processing would be done on a different machine). One could theoretically get the same information from log messages if one setup the router to log all dropped, accepted and rejected traffic (to a local syslog server for example). However, I don't think that is advisable due to possible impact it could have on router loads. Setting up the router with rflow would also get the same type of info, but I can't find an open source rflow server.
Overall, the possibilities are awesome with where Al is going with the Yamon thing.
@artyddwrt - are you trying to keep me focussed and on task? that's not gonna happen
@evac - I deliberately overwrite the live usage file every 30 or so seconds (because it is supposed to just show the current state of affairs) and also, keeping all of them would require a fair bit of disk space.
To see the current connections on your router, open PuTTY and enter `cat /proc/net/ip_conntrack`... the live usage file contains a culled and formatted version of the information from that string.
As you can see from the string, (and as artyddwrt pointed out), ip_conntrack shows the IP addresses of the connections, not the actual URLs. TBH, I don't know where to find the URLs... and trust me, I looked (but not really really hard).
It would not be hard to adapt yamon3.2.5.sh (on or about line #890
Code:
echo "$_liveusage" >> $_liveFilePath
Add a line to either append the `$_liveusage` to another file (on your USB drive) or also save the results to another time-stamped (rather than date-stamped) file... I leave that to you as an exercise Adding a parameter in config.file to retain the live data is an option but comes with *much* more overhead.
One of your easiest options might be to simply set your router to use opendns.org. That'll give you a list of sites accessed but you won't get which device accessed the site.
Al with this new version I'm unable to reboot the router without pulling the physical plug because I'm getting this:
/opt/YAMon3/shutdown.sh: line 64: did not kill process: 21103 ?!? try rebooting your router: not found
Any idea what might be causing this?
What is causing it? A crappy software developer (aka me).
I made some changes to shutdown.sh that should prevent the endless loop... I've seen what you reported once... a yamon process but a `kill pid` does not actually get rid of it.
Shutdown now better detects this oddball situation. Rerun install.sh to get the latest files.
Al
I am still getting this error, it loops and keeps the router from restarting. I tried re-running setup.sh with no luck, and compare.sh states all my files match.
@artyddwrt - are you trying to keep me focussed and on task? that's not gonna happen
It was worth a try!
Just joking of course, since your priorities (not our desires) are what benefit all of us and got things to where they are. Very useful insight as always. Thanks very much.
Something I've noticed is when visiting the webpage for the first time it turns the setup, which is fine. But then each new device visiting the webpage for the first time also runs the first time setup. I think it's an issue because if I make a change to the settings then visit the webpage on a new device that seeing gets reset.
Something I've noticed is when visiting the webpage for the first time it turns the setup, which is fine. But then each new device visiting the webpage for the first time also runs the first time setup. I think it's an issue because if I make a change to the settings then visit the webpage on a new device that seeing gets reset.
Posted: Fri Apr 07, 2017 17:53 Post subject: Re: YAMon on AsusMerlin
al_c wrote:
John - I use the command command to see whether the sort command exists... I think many (if not all) of your issues have been addressed in my latest dev build. Can you send your setup.log and perhaps screenshots showing the errors on setup.log. Please send those to questions@usage-monitoring.com.
I picked up an Asus router a couple of weeks ago so that I'd have my own Merlin test environment... but haven't gotten to it yet... I have to continue to rely on patient souls like you for help...
Thx
Al
Hi Al,
Sorry for the delay, I have sent over the setup.log as you had asked. Thanks!
Posted: Sat Apr 08, 2017 23:18 Post subject: Yamon looking for non-existent days?
Hi Al and all,
I installed YAMon on an Archer C7 v2 running LEDE recently and it's been working great--I bought the router and installed LEDE just so I could use YAMon. It's been working great until today when we had a lot of power outages (I am in Senegal where the power can be iffy).
I installed YAMon on 27 March. In monthly view for April, it is working just fine. When I go to March, it is giving me the yellow triangle error saying:
Code:
Error #4 reading the hourly data file: `data3/2017/03/2017-03-01-hourly_data.js` Not Found
I do not have data from that day, YAMON wasn't installed yet but YAMon is looking for it and not finding it.
When I scroll through the days of March the data is there for the 27th and following but throws me a similar yellow triangle error on the 28th saying:
Code:
Error #4 reading the hourly data file: `data3/2017/03/2017-03-28-hourly_data.js` SyntaxError: Unexpected number
So far I have restored the backup of that day and restarted and also reinstalled YAMon (without wiping settings) but the error persists.
Are there any steps I could take to recover the data or any other backup steps I could take? I am in testing mode right now, so this particular data isn't that important, but I am preparing it for use in a school here where they need to maintain this data.
Many thanks for the tool and the work you and others have done.
None of the spit and polish of the current YAMon reports but I'm pleased to report that I'm making significant progress on the direct to db front... The screenshot below shows that I'm now capturing traffic and device info into to my external database.
So far, I'm encouraged to see that the traffic totals in the database appear to be consistent with those in my regular instance of YAMon.
Without the need to manage everything within the constrained capabilities of the router means that the code is a lot smaller and a lot less complicated - e.g., I no longer have recalculate per device traffic using my home-grown string-based addition function (which was required on some routers to avoid integer overflows when totals exceeded 4GB).
I messed up last evening and none of the overnight results were saved to the database... but I had captured all of the traffic locally in a log file. So this AM, I wrote a little ~20 line script that recovered everything... YES!!! (OK, not so proud about missing the error which prevented the updates but impress with how easy it was to recover the data).
A *lot* of testing and development is still required but I'm pretty excited with the way things are coming along.
I'll be looking for early alpha testers to help shake out the details... (but TBH it'll probably take a couple of weeks still to build a lot of infrastructure to support that).
I expect that it'll be possible to run YAMon almost fully from /tmp with a small startup script which downloads all of the necessary files & info...
Posted: Sun Apr 09, 2017 18:51 Post subject: Re: Yamon looking for non-existent days?
coreygarrett wrote:
Hi Al and all,
I installed YAMon on an Archer C7 v2 running LEDE recently and it's been working great--I bought the router and installed LEDE just so I could use YAMon. It's been working great until today when we had a lot of power outages (I am in Senegal where the power can be iffy).
I installed YAMon on 27 March. In monthly view for April, it is working just fine. When I go to March, it is giving me the yellow triangle error saying:
Code:
Error #4 reading the hourly data file: `data3/2017/03/2017-03-01-hourly_data.js` Not Found
I do not have data from that day, YAMON wasn't installed yet but YAMon is looking for it and not finding it.
When I scroll through the days of March the data is there for the 27th and following but throws me a similar yellow triangle error on the 28th saying:
Code:
Error #4 reading the hourly data file: `data3/2017/03/2017-03-28-hourly_data.js` SyntaxError: Unexpected number
So far I have restored the backup of that day and restarted and also reinstalled YAMon (without wiping settings) but the error persists.
Are there any steps I could take to recover the data or any other backup steps I could take? I am in testing mode right now, so this particular data isn't that important, but I am preparing it for use in a school here where they need to maintain this data.
Many thanks for the tool and the work you and others have done.
Senegal... cool!
When you load the previous month/billing interval, YAMon tries to open the first day of that interval... in your case, the data file does not exist and that causes the message. So it's normal, but not necessarily the most friendly, behaviour.
Can you send me the data file for the 28th? If you experience frequent power outages, it could be that one coincided with a write to disk and the file might contain incomplete data. Also, if you have logging turned on, it might be helpful if you can send the log as well...
pls send them to questions@usage-monitoring.com
What file system did you use for your USB drive? I would recommend against `ext2` if you are subject to random power outages. I initially used `ext2` but found that files could become corrupted following an unscheduled shutdown... since moving to `ext4`, I've never had that sort of corruption. (However, that said, I've recently learned that not all router chipsets and firmware combinations support `ext4`.)
