ip_conntrack with QoS and bandwidth use reporting

Post new topic   Reply to topic    DD-WRT Forum Index -> Contributions Upload
Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8  Next
Author Message
lordlinus
DD-WRT Novice


Joined: 24 Dec 2010
Posts: 9

PostPosted: Wed Jan 05, 2011 5:48    Post subject: bw_table Reply with quote
Alex, i am trying to grab the download speed from the table using awk but having touble with one line code

grep "bw_table" /tmp/traffic.dat| awk -f example.awk

where example.awk has the below code WORKS

#!/usr/bin/awk -f
{
FS="[,'']";{for(i=8;i<=NF;i=i+9)print$i;}
}

but the single liner is not working

grep "bw_table" /tmp/traffic.dat | awk '{FS="[,'']";{for(i=8;i<=NF;i=i+9)print $i;} }'

can you help?
Sponsor
alexatkinuk
DD-WRT User


Joined: 07 Dec 2010
Posts: 131
Location: Sheffield, UK

PostPosted: Wed Jan 05, 2011 15:38    Post subject: Reply with quote
Unfortunately I am a pure novice with awk myself. I did have a look and try a few things, but haven't a clue what the error message is all about.

Worst case, you could always edit traffic_monitor.sh to output the data in a format more suited to your needs - or even hack your own script into there itself.

Incidentally, you should be getting the information from /tmp/www/traffic.asp as /tmp/traffic.dat is a temporary file and could be in an incomplete state depending on how long it takes the script to run, which a sudden burst of load on the router could easily slow it right down.

In fact in the latest version /tmp/traffic.dat is moved to /tmp/www/traffic.asp once its finished being created as simply copying it was still causing the web page to randomly stop updating. I had forgotten the rule that moving a file is really fast because you are just changing the pointer not actually rewriting the file, so it should never freeze now as a request for the file will at worst return the complete old version if the pointer is not updated fast enough. (I think Wink)

_________________
2xWZR-HP-G300NH(B) (B0 B0) DD-WRT v24-sp2 (06/14/11) std 17201
One antenna swapped for an RP-SMA connector and 14dB external Yagi.
http://csdprojects.co.uk/ddwrt/
dannyellis1987
DD-WRT Novice


Joined: 03 Jan 2011
Posts: 6

PostPosted: Wed Jan 05, 2011 18:18    Post subject: Reply with quote
Thanks for the info. I guess unfortunately I need to monitor the total bandwidth used by each user more than I need to know real time stats. Maybe if I get bored one day I'll dig into the code and see what I can do. You're stuff looks good though, just need something a bit different Smile
alexatkinuk
DD-WRT User


Joined: 07 Dec 2010
Posts: 131
Location: Sheffield, UK

PostPosted: Wed Jan 05, 2011 20:05    Post subject: Reply with quote
dannyellis1987 wrote:
Thanks for the info. I guess unfortunately I need to monitor the total bandwidth used by each user more than I need to know real time stats. Maybe if I get bored one day I'll dig into the code and see what I can do. You're stuff looks good though, just need something a bit different Smile


I would love to monitor that too as theoretically its not that much more work, until you consider the stats wipe every reboot.

So once you start monitoring an ongoing total you need to make sure you regularly save that data in case the router is rebooted, which is not a good idea to internal flash. I wonder how often ttraff saves?

Chances are at some point the data will be inaccurate as the only time its safe to save real time stats is if you have a HDD plugged into the USB port. For anything else, you need to keep saving to a minimum to avoid wearing out the internal flash memory. So even if you saved the stats every hour (which I would be wary of doing to jffs but might risk it on a microSD as you can easily replace it if its breaks) you risk potentially losing a lot of information if the router reboots before saving. What makes it harder is the fact you rarely reboot a router the "proper" way, you will usually just pull the plug, so you can't even rely on saving in the shutdown script.

Now stats since last reboot, that would be easy to do, but is it really useful?

If you want to try then replace traffic_monitor.sh with this updated code:
Code:
#!/bin/sh
# Bandwidth Download/Upload Rate Counter
LAN_IFACE=$(nvram get lan_ifname)
LAN_TYPE=$(nvram get lan_ipaddr | awk ' { FS="."; print $1"."$2 }')

if [ -f /tmp/traffic_monitor.lock ];
then
  if [ ! -d /proc/$(cat /tmp/traffic_monitor.lock) ]; then
    echo "WARNING : Lockfile detected but process $(cat /tmp/traffic_monitor.lock) does not exist. Reinitialising lock file!"
    rm -f /tmp/traffic_monitor.lock
  else
    echo "WARNING : Process is already running as $(cat /tmp/traffic_monitor.lock), aborting!"
    exit
  fi
fi

echo $$ > /tmp/traffic_monitor.lock
echo "Monitoring network ${LAN_TYPE}.x.255"

while :
do
  #Create the RRDIPT CHAIN (it doesn't matter if it already exists).       
  iptables -N RRDIPT 2> /dev/null                                         
                                                                                 
  #Add the RRDIPT CHAIN to the FORWARD chain (if non existing).               
  iptables -L FORWARD --line-numbers -n | grep "RRDIPT" | grep "1" > /dev/null
  if [ $? -ne 0 ]; then                                                       
    iptables -L FORWARD -n | grep "RRDIPT" > /dev/null                 
    if [ $? -eq 0 ]; then                                               
      iptables -D FORWARD -j RRDIPT                               
    fi                                                                 
  iptables -I FORWARD -j RRDIPT                                       
  fi                                                                         
                                                                                   
  #For each host in the ARP table                                             
  grep ${LAN_TYPE} /proc/net/arp | while read IP TYPE FLAGS MAC MASK IFACE   
  do                                                                           
    #Add iptable rules (if non existing).                               
    iptables -nL RRDIPT | grep "${IP}[[:space:]]" > /dev/null                     
    if [ $? -ne 0 ]; then                                               
      iptables -I RRDIPT -d ${IP} -j RETURN                       
      iptables -I RRDIPT -s ${IP} -j RETURN                       
    fi                                                                 
  done                                                                       

  # The following code is related to qos_conntrack and can be removed if you just want to enable traffic monitoring and then pull those values periodically with a cron job
  grep ${LAN_TYPE} /proc/net/arp | awk 'BEGIN { printf "{arp::"} { printf "'\''%s'\'','\''%s'\'',",$1,$4; } END { print "'\''-'\''}"}' >> /tmp/traffic.dat
  awk 'BEGIN { printf "{hosts::"} { printf "'\''%s'\'','\''%s'\'',",$1,$2; } END { print "'\''<% show_wanipinfo(); %>'\''}"}' /tmp/hosts >> /tmp/traffic.dat
  awk 'BEGIN { printf "{ip_conntrack::"} { gsub(/(src|dst|sport|dport|mark)=/, ""); printf "'\''%s'\'','\''%s'\'','\''%s'\'','\''%s'\'','\''%s'\'',%s,",$1,$1 == "tcp" ? $5 : $4,$1 == "tcp" ? $7 : $6,$1 == "tcp" ? $6 : $5,$1 == "tcp" ? $8 : $7,$(NF-1); } END { print "'\''-'\''}"}' /proc/net/ip_conntrack >> /tmp/traffic.dat
  #
  # This is the line that pulls the bandwidth data, change this if you want the data in a different format
  iptables -L RRDIPT -vnx -t filter | grep ${LAN_TYPE} | awk 'BEGIN { printf "{bw_table::" } { if (NR % 2 == 1) printf "'\''%s'\'','\''%s'\'',",$8,$2; else printf "'\''%s'\'',",$2;} END { print "'\''-'\''}"}' >> /tmp/traffic.dat
  #
  uptime | awk 'BEGIN { printf "{uptime::" } { printf "%s", $0 } END { print  "}"; print "{ipinfo::<% show_wanipinfo(); %>}" }' >> /tmp/traffic.dat
  mv -f /tmp/traffic.dat /tmp/www/traffic.asp
  # END OF qos_conntrack section

  sleep 1
done


Now instead of showing you the data transfer per second, MyPage will actually be showing you the total data transfer since you made the change.

Now the problem is the script only outputs machines currently in the ARP table, so you switch a machine off and its stats are no longer displayed. They are still tracked though as the tracking rules are not removed.

Backup the stats periodically:
Here is a quick hack I setup to log bandwidth stats to /jffs which is actually a microSD card. Simply execute it with a cron job however often you need to keep records, I will try it at 5 minutes I think as that its as losing 5 minutes of data would not be the end of the world, but losing say half an hour could be many megabytes - not good if you are trying to keep track of a bandwidth quota.

The idea is if your router reboots then you can recalculate the missing data as the next result after a reboot will be lower than the one before it. It is a simple comma delimited file but you could easily change that, its basically identical to traffic_monitor.sh but with the bits at the beginning and end of the line removed and a date/timestamp added.

Code:
#!/bin/sh
LAN_TYPE=$(nvram get lan_ipaddr | awk ' { FS="."; print $1"."$2 }')
DATE=$(date -I)
DATESTAMP=$(date -I'minutes')
echo -n "${DATESTAMP}," >> /jffs/stats-${DATE}.txt
iptables -L RRDIPT -vnx -t filter | grep ${LAN_TYPE} | awk '{ if (NR % 2 == 1) printf "'\''%s'\'','\''%s'\'',",$8,$2; else printf "'\''%s'\'',",$2;} END { print "'\''-'\''"}' >> /jffs/stats-${DATE}.txt
alexatkinuk
DD-WRT User


Joined: 07 Dec 2010
Posts: 131
Location: Sheffield, UK

PostPosted: Sat Jan 08, 2011 2:03    Post subject: Reply with quote
I found another snag to gathering cumulative statistics, at least if you often change settings on your router. Basically, if you use the Apply Settings button on the control panel, you lose all statistics.

I'm guessing this is DD-WRT playing it safe and wiping all iptables rules and reinitialising them, but its damned annoying in this case and I wonder if there is any way to stop it?

_________________
2xWZR-HP-G300NH(B) (B0 B0) DD-WRT v24-sp2 (06/14/11) std 17201
One antenna swapped for an RP-SMA connector and 14dB external Yagi.
http://csdprojects.co.uk/ddwrt/


Last edited by alexatkinuk on Thu Jan 13, 2011 0:08; edited 1 time in total
lordlinus
DD-WRT Novice


Joined: 24 Dec 2010
Posts: 9

PostPosted: Sun Jan 09, 2011 23:42    Post subject: Bytes to Bits Reply with quote
Hi, i am having trouble comparing the web test results with the values from the IPtables.

Speedtest normally gives me 32 Mbites/sec (i.e 4 MBYTES/sec) but the "MyPages" values have gone as high as 120 MBites/sec (i.e 15 MBytes/sec).

How can i calibrated to represent the approx. device usage as show by the speedtest?
alexatkinuk
DD-WRT User


Joined: 07 Dec 2010
Posts: 131
Location: Sheffield, UK

PostPosted: Mon Jan 10, 2011 22:24    Post subject: Re: Bytes to Bits Reply with quote
lordlinus wrote:
Hi, i am having trouble comparing the web test results with the values from the IPtables.

Speedtest normally gives me 32 Mbites/sec (i.e 4 MBYTES/sec) but the "MyPages" values have gone as high as 120 MBites/sec (i.e 15 MBytes/sec).

How can i calibrated to represent the approx. device usage as show by the speedtest?


Try the latest version 0.9 where its calculated in your browser.

Oddly I had to drop the refresh rate to 5 seconds due to some strange issues where its reporting burst speeds that do not seem remotely likely (slow then faster than my maximum, then slow, repeat) but it seems to average down accurate over 5 seconds.

_________________
2xWZR-HP-G300NH(B) (B0 B0) DD-WRT v24-sp2 (06/14/11) std 17201
One antenna swapped for an RP-SMA connector and 14dB external Yagi.
http://csdprojects.co.uk/ddwrt/
lordlinus
DD-WRT Novice


Joined: 24 Dec 2010
Posts: 9

PostPosted: Tue Jan 11, 2011 0:15    Post subject: usage Reply with quote
Hi, can you please advise on how to calculate using shell. (instead of the browser) . Reason i ask is if i can could get the values i can pipe them into a graphing application data format.

Cheers

edit: i am getting the bytes values from iptables -L RRDIPT -vnx -t filter and subtracting latest with previous one every 1 sec and it seems to be fairly working.

is there a better/accurate way to calculate.
alexatkinuk
DD-WRT User


Joined: 07 Dec 2010
Posts: 131
Location: Sheffield, UK

PostPosted: Tue Jan 11, 2011 14:46    Post subject: Reply with quote
lordlinus wrote:
Hi, can you please advise on how to calculate using shell. (instead of the browser) . Reason i ask is if i can could get the values i can pipe them into a graphing application data format.

Cheers

edit: i am getting the bytes values from iptables -L RRDIPT -vnx -t filter and subtracting latest with previous one every 1 sec and it seems to be fairly working.

is there a better/accurate way to calculate.


With stock DD-WRT I think that is all you can do and was what I was doing in 0.8c. The problem with doing it on the router is that it fails if the router lags getting the results as it might actually have been 1.9 seconds between probes which means your speed will report 1.9 times what it should do.

The pain is that the date command in busybox is cut down and does not support the nanosecond time reporting that it does on desktop Linux. So at best if you use date to work out the time period between the last probe and current one, that still only prevents your results being more than 1.9 times (possibly 1.4, I'm not sure if the seconds are rounded up/down or not) overestimated which is still pretty poor accuracy.

You might be able to pull down a more complete version of date if you setup opkg, but when I did that last the OpenWRT libraries were causing DD-WRT to crash and reboot. It might work for your purposes though as plenty of people seem to have it setup without causing problems.

What would be better IMO would be setting up a DD-WRT build environment on your PC compiling your own fully DD-WRT compatible version of the date command. Personally, I would be tempted to go all out for a complete C version of the probing script for the speed improvements by not having to call lots of external commands, but then my C skills have never been tested as I am pretty much a copy/paste guy when it comes to coding, learning from other peoples code snippets, and scripting is sooo much easier than C.

Now what DOES puzzle me is that I am still getting unreliable results using javascript to calculate the speed every second or even every 4 seconds. In theory it should be pretty accurate as there is only the lag between requesting the current data and the web browser pulling it in that might cause some offset between the actual current values and what you are getting in the browser. But I am getting very odd results of it under-estimating on one refresh then over-estimating on the next. That might make sense for every second, but even every 2 or 3 seconds it still does it which really is odd as it should average out. I had to drop the refresh down to 5 seconds before it averaged out properly.

Its not the end of the world, I will just adapt the code to only update the speeds every 5 second while refreshing everything else every second as before. But its kinda annoying.

_________________
2xWZR-HP-G300NH(B) (B0 B0) DD-WRT v24-sp2 (06/14/11) std 17201
One antenna swapped for an RP-SMA connector and 14dB external Yagi.
http://csdprojects.co.uk/ddwrt/
lordlinus
DD-WRT Novice


Joined: 24 Dec 2010
Posts: 9

PostPosted: Wed Jan 12, 2011 3:52    Post subject: Reply with quote
Thanks Alex. How can i upgarde to full version of date?

ipkg list has no "date" package. Would you kindly give me where to get the full date package?
i install opkg package too but i get "opkg: symbol 'stdout': can't handle reloc type 0x7e" error

When i get the date package. should get a time stamp before the while loop and divide..

e.g.

<- get initial bytes for each IP from iptables ->

while :
do
<- get the time stamp in nano seconds ->
< - get bytes for each IP from iptables ->
<- get the time stamp in nano seconds ->
calculate speed = number of bytes/(difference in seconds)
sleep 1 (???)


Really appreciate your help

------------------------------------
Router Model:Linksys WRT320N
Firmware Version:DD-WRT v24-sp2 (08/07/10) mega - build 14896
alexatkinuk
DD-WRT User


Joined: 07 Dec 2010
Posts: 131
Location: Sheffield, UK

PostPosted: Wed Jan 12, 2011 15:29    Post subject: Reply with quote
I have no idea if date IS available in a package. I did not look into it as I knew I could just switch the calculations out into the browser which saves router CPU cycles too, so a win win.

As for how to use it, I just did:

Code:
#!/bin/sh
# Bandwidth Download/Upload Rate Counter
LAN_IFACE=$(nvram get lan_ifname)
LAN_TYPE=$(nvram get lan_ipaddr | awk ' { FS="."; print $1"."$2 }')
LASTTIME=$(date +%s)

while :
do
  #Create the RRDIPT CHAIN (it doesn't matter if it already exists).       
  iptables -N RRDIPT 2> /dev/null                                         
                                                                                 
  #Add the RRDIPT CHAIN to the FORWARD chain (if non existing).               
  iptables -L FORWARD --line-numbers -n | grep "RRDIPT" | grep "1" > /dev/null
  if [ $? -ne 0 ]; then                                                       
    iptables -L FORWARD -n | grep "RRDIPT" > /dev/null                 
    if [ $? -eq 0 ]; then                                               
      iptables -D FORWARD -j RRDIPT                               
    fi                                                                 
  iptables -I FORWARD -j RRDIPT                                       
  fi                                                                         
                                                                                   
  #For each host in the ARP table                                             
  grep ${LAN_IFACE} /proc/net/arp | while read IP TYPE FLAGS MAC MASK IFACE   
  do                                                                           
    #Add iptable rules (if non existing).                               
    iptables -nL RRDIPT | grep "${IP}[[:space:]]" > /dev/null                     
    if [ $? -ne 0 ]; then                                               
      iptables -I RRDIPT -d ${IP} -j RETURN                       
      iptables -I RRDIPT -s ${IP} -j RETURN                       
    fi                                                                 
  done                                                                       

  awk 'BEGIN { printf "{hosts::"} { printf "'\''%s'\'','\''%s'\'',",$1,$2; } END { print "'\''<% show_wanipinfo(); %>'\''}"}' /tmp/hosts >> /tmp/traffic.dat
  awk 'BEGIN { printf "{ip_conntrack::"} { gsub(/(src|dst|sport|dport|mark)=/, ""); printf "'\''%s'\'','\''%s'\'','\''%s'\'','\''%s'\'','\''%s'\'',%s,",$1,$1 == "tcp" ? $5 : $4,$1 == "tcp" ? $7 : $6,$1 == "tcp" ? $6 : $5,$1 == "tcp" ? $8 : $7,$(NF-1); } END { print "'\''-'\''}"}' /proc/net/ip_conntrack >> /tmp/traffic.dat
  TIMESTAMP=$(date +%s)
  DELAY=$((TIMESTAMP-LASTTIME))                                                                                                                                                                                                                                                             
  iptables -L RRDIPT -vnxZ -t filter | grep ${LAN_TYPE} | awk 'BEGIN { printf "{bw_table::" } { if (NR % 2 == 1) printf "'\''%s'\'','\''%s'\'',",$8,$2/'${DELAY}'; else printf "'\''%s'\'',",$2;} END { print "'\''-'\''}"}' >> /tmp/traffic.dat
  LASTTIME=$TIMESTAMP
  uptime | awk 'BEGIN { printf "{uptime::" } { printf "%s", $0 } END { print  "}"; print "{ipinfo::<% show_wanipinfo(); %>}" }' >> /tmp/traffic.dat
  mv -f /tmp/traffic.dat /tmp/www/traffic.asp
  sleep 1
done

_________________
2xWZR-HP-G300NH(B) (B0 B0) DD-WRT v24-sp2 (06/14/11) std 17201
One antenna swapped for an RP-SMA connector and 14dB external Yagi.
http://csdprojects.co.uk/ddwrt/
cyberde
DD-WRT Guru


Joined: 07 Jun 2006
Posts: 1488
Location: the Netherlands

PostPosted: Mon Jan 24, 2011 18:31    Post subject: Reply with quote
Looks really awesome, only I'm getting a javascript error and the counters stay at "CALCULATING...".

Quote:
Error: sec0 is not defined
Source File: http://192.168.0.1/user/qos_conntrack.js
Line: 377


I did follow your instructions on how to install Wink

_________________
Firmware: DD-WRT v24-sp2 (latest available) mega
WRT320N

Donater
alexatkinuk
DD-WRT User


Joined: 07 Dec 2010
Posts: 131
Location: Sheffield, UK

PostPosted: Mon Jan 24, 2011 19:00    Post subject: Reply with quote
Which web browser and can you try on Firefox or Chrome (preferable, as it has good debugging) as admittedly, those are all I have tested it on so far.

Known potential issues are:

It seems to struggle on Firefox 3.6 eating a ton of CPU power for no apparent reason. (a bug in the javascript?)
That said, I had to ditch 3.6 on my desktop and use 4 beta due to how poorly it ran in general, not just on this.

Another issue is that it depends on prototype but uses whichever version is included with DD-WRT, so there is a slight chance of incompatibility. Although it seems unlikely as most of what we are using prototype for is the DD-WRT event handler/refresh script itself. My scripts are only using very basic prototype functionality which is unlikely to differ in newer/older versions.

_________________
2xWZR-HP-G300NH(B) (B0 B0) DD-WRT v24-sp2 (06/14/11) std 17201
One antenna swapped for an RP-SMA connector and 14dB external Yagi.
http://csdprojects.co.uk/ddwrt/
cyberde
DD-WRT Guru


Joined: 07 Jun 2006
Posts: 1488
Location: the Netherlands

PostPosted: Tue Jan 25, 2011 7:50    Post subject: Reply with quote
I am using Firefox 3.6, also happens in IE8. I do see my hosts appearing in the source of the page but no additional data. So it could be a DD-WRT issue... I'm using the latest DD-WRT v24SP2 12/24/10.

IE8 error wrote:
Webpage error details

User Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET CLR 1.1.4322; .NET4.0C; .NET4.0E)
Timestamp: Tue, 25 Jan 2011 07:48:25 UTC


Message: 'sec0' is undefined
Line: 377
Char: 5
Code: 0
URI: http://192.168.0.1/user/qos_conntrack.js


_________________
Firmware: DD-WRT v24-sp2 (latest available) mega
WRT320N

Donater
alexatkinuk
DD-WRT User


Joined: 07 Dec 2010
Posts: 131
Location: Sheffield, UK

PostPosted: Tue Jan 25, 2011 22:53    Post subject: Reply with quote
cyberde wrote:
I am using Firefox 3.6, also happens in IE8. I do see my hosts appearing in the source of the page but no additional data. So it could be a DD-WRT issue... I'm using the latest DD-WRT v24SP2 12/24/10.

IE8 error wrote:
Webpage error details

User Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET CLR 1.1.4322; .NET4.0C; .NET4.0E)
Timestamp: Tue, 25 Jan 2011 07:48:25 UTC


Message: 'sec0' is undefined
Line: 377
Char: 5
Code: 0
URI: http://192.168.0.1/user/qos_conntrack.js



Can you run
Code:
cat /proc/net/ip_conntrack > /tmp/www/conntrack.html
via telnet/SSH or Administration, Commands on the router?

Then provide a copy of http://router/user/conntrack.html and http://router/user/traffic.asp as I think I know what is going on but I need data from an affected router to know exactly where and how its going wrong.

Bear in mind that at least one of those files, potentially both, DO include your WAN IP, so I understand if you do not wish to provide them. At the very least I would not recommend attaching them to the thread. You can probably copy/paste their contents in a private message to me though.

I may be able to fix it without the entire files, but its going to need a lot more work from you that way.

_________________
2xWZR-HP-G300NH(B) (B0 B0) DD-WRT v24-sp2 (06/14/11) std 17201
One antenna swapped for an RP-SMA connector and 14dB external Yagi.
http://csdprojects.co.uk/ddwrt/
Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8  Next Display posts from previous:    Page 2 of 8
Post new topic   Reply to topic    DD-WRT Forum Index -> Contributions Upload All times are GMT

Navigation

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum