ip_conntrack with QoS and bandwidth use reporting

Post new topic   Reply to topic    DD-WRT Forum Forum Index -> Contributions Upload
Goto page 1, 2, 3, 4, 5, 6, 7  Next
Author Message
alexatkinuk
DD-WRT User


Joined: 07 Dec 2010
Posts: 130
Location: Sheffield, UK

PostPosted: Sun Dec 19, 2010 1:38    Post subject: ip_conntrack with QoS and bandwidth use reporting Reply with quote
What is it?
I have created an alternative to the LAN page which shows the active IP connections including their QoS rating as well as the Internet bandwidth use per machine on the LAN.

At its core it is based on the second command from the QoS Wiki page and wrtbwmon, for the iptables rules needed to make traffic accounting work. It does not use wrtbwmon itself though, just the commands from setup and some bits from update.

It uses very little resources on the router as most of the calculations are offloaded onto the web browser on the PC you view the router pages on.

Hostnames are loaded from the /tmp/hosts file when your first load the page so any machines in your Static DHCP list should be identified automatically.

How to get it working:

You need to extract the contents of the zip into the /tmp/ directory and then run
Code:
chmod +x /tmp/MyPage/*.sh
ln -s /tmp/MyPage/www/qos_conntrack.js /tmp/www/
so the scripts are executable.

To activate the new page run:
Code:
nvram set mypage_scripts="/tmp/MyPage/qos_conntrack.sh"
nvram commit
It should then appear as My Page on the DD-WRT Web Control Panel.

For the bandwidth monitoring to work you also need to run
Code:
/tmp/MyPage/traffic_monitor.sh&
either from the command line via telnet/ssh or run it from the Administration, Command page on the router. You can then visit the My Page link on your router control panel to see it working (hopefully).

To make My Page work permanently you will need some storage on your router, either enable jffs2 (easiest) or any storage connected to the USB port should work fine.

Assuming you use jffs then to make sure the script works after a reboot simply copy the directory to jffs:
Code:
cp -r /tmp/MyPage /jffs/


Then put the following in your Startup Script:
Code:
cp -r /jffs/MyPage /tmp/
ln -s /tmp/MyPage/www/qos_conntrack.js /tmp/www/
chmod +x /tmp/MyPage/*.sh
/tmp/MyPage/traffic_monitor.sh&


This will make sure the scripts get copied to RAM after every reboot and start up the traffic monitoring automatically.

NOTE
The chmod line is not strictly necessary if you are using a Linux filesystem to store the MyPage such as jffs or ext2/3 as it should keep the permissions, but its safe to leave it either way.

_________________
2xWZR-HP-G300NH(B) (B0 B0) DD-WRT v24-sp2 (06/14/11) std 17201
One antenna swapped for an RP-SMA connector and 14dB external Yagi.
http://csdprojects.co.uk/ddwrt/


Last edited by alexatkinuk on Mon Feb 07, 2011 15:13; edited 20 times in total
Sponsor
lordlinus
DD-WRT Novice


Joined: 24 Dec 2010
Posts: 9

PostPosted: Fri Dec 24, 2010 1:19    Post subject: Reply with quote
Great work, is it possible to include the real time graph similar to bandwidth page (for the total traffic)?
alexatkinuk
DD-WRT User


Joined: 07 Dec 2010
Posts: 130
Location: Sheffield, UK

PostPosted: Fri Dec 24, 2010 6:01    Post subject: Reply with quote
lordlinus wrote:
Great work, is it possible to include the real time graph similar to bandwidth page (for the total traffic)?


Do you mean just plotting on the graph how much bandwidth each machine is currently using?

At the moment I am just trying to make sure its not putting too much load on the router doing what it does right now as I notice the load average seem to drift between 0.3 and 0.6 plus my router has randomly rebooted several times. I think it might be related to the constant telnet connection, scp the files over and starting/stopping the script for testing though. Its just a bit confusing as the actual CPU usage stays low and its not IO that I can see, neither does it seem to be directly related to how much traffic is being routed at the time or if I have the web page open or not.

Also I have been measuring and it seems my router can do 200Mbit routing stock, but it drops to just under 100Mbit with QoS enabled. Although there is nothing to stop you using the script with QoS disabled, although I have not tested it.

The beauty of a graph though, it wouldn't increase the load as we already have all the information available so it can be done pure JavaScript. But at the moment I am tweaking and wanting to get a button on there to refresh the ip_conntrack table.

I was thinking of some sort of graph that shows how much each client is using out of the total bandwidth (taken from the QoS upstream/downstream value) but I'm not sure how yet.

_________________
2xWZR-HP-G300NH(B) (B0 B0) DD-WRT v24-sp2 (06/14/11) std 17201
One antenna swapped for an RP-SMA connector and 14dB external Yagi.
http://csdprojects.co.uk/ddwrt/
amitd
DD-WRT Novice


Joined: 16 Dec 2010
Posts: 8

PostPosted: Fri Dec 24, 2010 11:28    Post subject: Reply with quote
Very nice and smooth........I am new to all this, how can i get this to publish on 192.168.xx.xx/user/xxxx.html instead of my page just interested in upstream and downstream per user ( want to make it easier for house mates to check their Live usage (as they use too much torrent), without giving details about the router) and also can we save this to a some some.db file?????
alexatkinuk
DD-WRT User


Joined: 07 Dec 2010
Posts: 130
Location: Sheffield, UK

PostPosted: Fri Dec 24, 2010 21:55    Post subject: Reply with quote
amitd wrote:
Very nice and smooth........I am new to all this, how can i get this to publish on 192.168.xx.xx/user/xxxx.html instead of my page just interested in upstream and downstream per user ( want to make it easier for house mates to check their Live usage (as they use too much torrent), without giving details about the router) and also can we save this to a some some.db file?????


The original version I based this one did all the things you want, however I never managed to get it working. Search the forum for wrtbwmon if you want to try.

It shouldn't be too much of an issue to get it working in /user/, the only reason I used MyPage is it was neater and allowed me to reuse the built-in DD-WRT JavaScript rather than reinventing the wheel. But I have to admit, it is annoying having to login just to see the stats so maybe I will do a summary page in /user/ at some point.

This is all quite new to me as previously if I needed to do anything interesting I would just use PHP. So learning the limitations and tricks of plain shell scripting takes some time, especially trying not to nuke the routers CPU cycles.

As an example I tested plain Buffalo DD-WRT as being able to do NAT at 20.4MiB/s, this halved once QoS was enabled and dropped as low at 7MiB/s with the bandwidth script running and an ext3 formatted USB stick mounted. I can only imagine how bad that could be with someone running it on a slower router as 400Mhz is fairly high-end for a router, most are less with only a few being faster than that.

_________________
2xWZR-HP-G300NH(B) (B0 B0) DD-WRT v24-sp2 (06/14/11) std 17201
One antenna swapped for an RP-SMA connector and 14dB external Yagi.
http://csdprojects.co.uk/ddwrt/
qualif
DD-WRT Novice


Joined: 06 Jul 2008
Posts: 46

PostPosted: Sat Dec 25, 2010 10:06    Post subject: Reply with quote
hello

Perfect Script thk
for wrt610Nv2
alexatkinuk
DD-WRT User


Joined: 07 Dec 2010
Posts: 130
Location: Sheffield, UK

PostPosted: Sat Dec 25, 2010 19:17    Post subject: Reply with quote
qualif wrote:
hello

Perfect Script thk
for wrt610Nv2


Thanks.

I have now uploaded v0.7 which is a major improvement over previous versions. It now offloads almost all calculations to your web browser so that there should be little risk of the script causing your router to slow down now.

Of course the sheer fact of enabling QoS itself will still halve the speed your router can deal with, but theoretically this script will still work with QoS turned off should you wish.

To Do

Add real-time sorting, so you can sort the column and it still be sorted when it refreshes. I will need to figure out how the DD-WRT sorting Javascript works first and adapt it.

Add ability to click on an IP address and have it only display the ip_conntrack results for that IP address. Probably initially just while paused with the ability for it to do so while refreshing later. The lack of associative arrays in Javascript is a real PITA for some of this stuff.

Figure out why its reporting my download speed as faster than it possibly could be. Its a good guide right now, but seems to be off by quite a large margin that can't simply be explained by overhead.

_________________
2xWZR-HP-G300NH(B) (B0 B0) DD-WRT v24-sp2 (06/14/11) std 17201
One antenna swapped for an RP-SMA connector and 14dB external Yagi.
http://csdprojects.co.uk/ddwrt/
alexatkinuk
DD-WRT User


Joined: 07 Dec 2010
Posts: 130
Location: Sheffield, UK

PostPosted: Mon Dec 27, 2010 17:09    Post subject: Reply with quote
Uploaded 0.8 which includes some fixes for silly mistakes and a new protocol summary table.

There is a blank space where I will probably add a pie chart to show connection percentages and spare capacity. Basically, look at the screenshot to know what on earth I am going on about.

Uploaded 0.8b which fixes a REALLY silly mistake that broke the javascript entirely.

_________________
2xWZR-HP-G300NH(B) (B0 B0) DD-WRT v24-sp2 (06/14/11) std 17201
One antenna swapped for an RP-SMA connector and 14dB external Yagi.
http://csdprojects.co.uk/ddwrt/
lordlinus
DD-WRT Novice


Joined: 24 Dec 2010
Posts: 9

PostPosted: Sun Jan 02, 2011 3:13    Post subject: total traffic Reply with quote
Good work alex, can you pls tell me how to get the live total upload/total download traffic ?
alexatkinuk
DD-WRT User


Joined: 07 Dec 2010
Posts: 130
Location: Sheffield, UK

PostPosted: Sun Jan 02, 2011 11:59    Post subject: Re: total traffic Reply with quote
lordlinus wrote:
Good work alex, can you pls tell me how to get the live total upload/total download traffic ?


Depends, get it using what?

The traffic_monitor.sh script does it by keeping the bandwidth monitoring tables updated with the arp table so that any PC just switched on will get monitored and also pulls out the stats for what has been transferred over the NAT since the last time we looked (1 second ago) then resets the counters to 0. That much is a direct copy from wrtbwmon.

The difference is I then just dump the values out into the /tmp/www/traffic.asp so that the javascript can then do all the processing on the results, which is why its able to handle 1 second updates.

The total bandwidth used then it figured out by just adding all the individual connection totals together while its generating the active connections table.

If you are just wanting the data and are running the traffic_monitor.sh script then just pull it from /tmp/www/traffic.asp if you need it for a script on the router itself, or read it from http://router/user/traffic.asp using javascript if you want it for a web page hosted on the router.

One catch though, DD-WRT checks your referrer so you cannot just pull data from its web pages into a page hosted on another web server - you would have to spoof the referrer. I found that a little annoying at first but quickly recovered when I realised having full root access means you can just run scripts to put the data on the server you need it on, or just enable snmp (haven't tested that yet).

If on the other hand you want the cumulative bandwidth used today, this week, this month, well I haven't looked into that yet. Its hard to do when you are constantly testing scripts, qos, rebooting the router, so the data would be inaccurate anyway.

One possibility there is if you have ttraf running you could steal its data from nvram such as:
Code:
nvram get traff-MM-YYYY

Where MM is the months data you want, YYYY is year. I believe it updates daily so its not realtime, the realtime data I would suspect is somewhere in /tmp/ but I haven't looked.

_________________
2xWZR-HP-G300NH(B) (B0 B0) DD-WRT v24-sp2 (06/14/11) std 17201
One antenna swapped for an RP-SMA connector and 14dB external Yagi.
http://csdprojects.co.uk/ddwrt/


Last edited by alexatkinuk on Tue Jan 04, 2011 0:49; edited 1 time in total
lordlinus
DD-WRT Novice


Joined: 24 Dec 2010
Posts: 9

PostPosted: Mon Jan 03, 2011 1:06    Post subject: Reply with quote
hi alex, i am trying to graph the bandwidth usage for individual connections (similar to http://router/Status_Bandwidth.asp) in real time and total usage all in one real time svg graph.

i am wondering if i could use your script (to get individual host traffic) and graph_if.svg?br0 script (to get the total bandwidth) to achieve this. but am still a n00b and struggling.
alexatkinuk
DD-WRT User


Joined: 07 Dec 2010
Posts: 130
Location: Sheffield, UK

PostPosted: Mon Jan 03, 2011 13:13    Post subject: Reply with quote
Interesting, the SVG script pulls in the data via http://router/fetchif.cgi?br0 which in turn seems to pull the data from /proc/net/dev.

So the SVG code is calculating the current bandwidth by comparing the traffic over br0 since last reboot, compared to what it was a second ago.

What bandwidth are you looking for? If its WAN bandwidth then you should be looking at eth1 as br0 will pickup all traffic to the router.

Now what I would like to do personally is generate a graph that shows the total available bandwidth (taken from the QoS settings, I already have done this much in the latest version I am working on) and then segment that into how much of that different machines are using. The way I was trying to do it was a pie chart, but I have not had much luck so far.

_________________
2xWZR-HP-G300NH(B) (B0 B0) DD-WRT v24-sp2 (06/14/11) std 17201
One antenna swapped for an RP-SMA connector and 14dB external Yagi.
http://csdprojects.co.uk/ddwrt/
dannyellis1987
DD-WRT Novice


Joined: 03 Jan 2011
Posts: 6

PostPosted: Tue Jan 04, 2011 22:19    Post subject: Reply with quote
Could this conflict at all with wrtbwmon? I have that installed as well and I just went to try this out and after all the steps minus the startup commands, I wasn't able to access my web gui anymore. I did an nvram erase and reboot as I'm remoted in from work at the moment and now I can't ping my house at all :S. I hope I didn't brick it, but I guess I will find out in an hour from now... lol
dannyellis1987
DD-WRT Novice


Joined: 03 Jan 2011
Posts: 6

PostPosted: Tue Jan 04, 2011 23:47    Post subject: Reply with quote
Update to my post, not bricked thankfully. Anyways, I realized I didn't have QoS enabled and I saw in a post that you hadn't tested it. This might have actually caused the problem but I have yet to try it again.
alexatkinuk
DD-WRT User


Joined: 07 Dec 2010
Posts: 130
Location: Sheffield, UK

PostPosted: Wed Jan 05, 2011 1:20    Post subject: Reply with quote
At worst having QoS disabled will break the javascript so the page doesn't work. You absolutely should not have wrtbwmon running at the same time though.

It wont hurt the router, we only ever write to /tmp and read from nvram. However neither will get the correct results as both scripts reset the bandwidth counters after reading the values.

I also had to make a small change to the traffic_monitor.sh when I upgraded to the latest version of DD-WRT, as the /proc/net/ip_conntrack table suddenly had an extra column before the QoS column. I need to look into that too.

_________________
2xWZR-HP-G300NH(B) (B0 B0) DD-WRT v24-sp2 (06/14/11) std 17201
One antenna swapped for an RP-SMA connector and 14dB external Yagi.
http://csdprojects.co.uk/ddwrt/
Goto page 1, 2, 3, 4, 5, 6, 7  Next Display posts from previous:    Page 1 of 7
Post new topic   Reply to topic    DD-WRT Forum Forum Index -> Contributions Upload All times are GMT

Navigation

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum