Router Slowdown

From DD-WRT Wiki

(Difference between revisions)
Jump to: navigation, search
Revision as of 20:34, 24 February 2007 (edit)
B.Gates (Talk | contribs)

← Previous diff
Revision as of 12:18, 22 February 2013 (edit) (undo)
Sash (Talk | contribs)
(Known issues with Applications/Operating Systems)
Next diff →
(46 intermediate revisions not shown.)
Line 1: Line 1:
-__TOC__+<!-- Just putting in this comment: the recommendation is to use == instead of 1 = when making the sections. It makes the article a little more read-able. -->
 +<!-- __TOC__: The TOC is automatically placed in the beginning, so this markup is not needed-->
-=Slowdown=+==Introduction==
-A lot of people have issues with their router, some symptoms can be+Issues occur with many routers, ''including routers running DD-WRT'', when using the router with heavy P2P applications. The router becomes 'slow' over time, and restarting helps for a short time. Some symptoms can be:
-* Slow web-interface+* Slow web-interface, or cannot connect at all to web interface
-* Slow transfer of data, e.g. browsing+* Slowing transfer of data, e.g. browsing, after a reboot
* Not responding to ping * Not responding to ping
-* Crash or even rebooting+* Router Crash or even rebooting
-Usually these people are running P2P software like Emule, Bittorrent, Shareaza or something else. These programs require a lot of connections which could cause the ip_conntrack-table to get full. Especially BitTorrent DHT feature sends thousands of UDP packets that quickly overflow this table.+Usually the culprits are heavy P2P software like Emule, Bittorrent, uTorrent, Azureus, Shareaza or something similar. These programs, by default, can require a lot of connections which could cause the routers' ip_conntrack table to get full.
 +*Especially BitTorrent's DHT feature sends thousands of UDP packets that quickly overflow this table.
-Routers affected with this issue are Linksys WRT54G, Linksys WRT54GL, Linksys WRT54GS or just any other machine running DD-WRT.+Routers affected with this issue are the most common types of routers running DD-WRT.
 +==Definitions==
 +;P2P:Peer to Peer applications. Could mean eMule, uTorrent, Azureus. Within the context of this article does not mean Skype, or other 'lite' P2P applications.
 +;DD-WRT Device:A Router running third-party, DD-WRT firmware. Although configuration in other methods besides 'router' is possible, such a 'bridge', this article and phenomena only occurs in the default router configuration of DD-WRT.
 +;Web Interface:The default management method, open a browser to http://192.168.1.1 or your custom DD-WRT Device's LAN IP.
-=Diagnosis=+==Diagnosis==
-Even if this occurs and the router doesn't respond to pings nor HTTP, you still can check what's going on: +When this 'slowdown' occurs and the router doesn't respond to pings nor Web Interface requests, you still can check what's going on:
* Close all your P2P/network applications and wait a few minutes for connections to be freed. * Close all your P2P/network applications and wait a few minutes for connections to be freed.
-* Try to [[Telnet/SSH and the Command Line|telnet or SSH]] into your router. If that's not possible, reboot.+* Try to use the [[CLI]] to communicate with your router.
-* Check to see if your problem is caused by TCP or UDP connections:+**If that's not possible, reboot the DD-WRT Device.
- +** Check to see if your problem is caused by TCP or UDP connections. From the CLI:
<pre> <pre>
cat /proc/sys/net/ipv4/netfilter/ip_conntrack_max cat /proc/sys/net/ipv4/netfilter/ip_conntrack_max
-4096 # your max space+4096 # Here the router displays the maximum number of possible connections.
-grep ^udp /proc/net/ip_conntrack | wc+grep -c ^udp /proc/net/ip_conntrack
-3693 50338 215245 # here UDP uses 3693 entries+3693 # Here UDP is using 3693 entries
-grep ^tcp /proc/net/ip_conntrack | wc+grep -c ^tcp /proc/net/ip_conntrack
-115 1836 20098 # and TCP uses only 115 entries+115 # and TCP uses only 115 entries
</pre> </pre>
- +:* You may also view syslog messages:
-* You may also view syslog messages+
- +
<pre> <pre>
cat /var/log/messages cat /var/log/messages
</pre> </pre>
- 
# First, you'll see 'full, dropping packet.' messages followed by 'messages suppressed.' # First, you'll see 'full, dropping packet.' messages followed by 'messages suppressed.'
# Eventually, it will get verbose and you'll see logs similar to the following : # Eventually, it will get verbose and you'll see logs similar to the following :
Line 42: Line 45:
<4>kernel: NET: 12 messages suppressed. <4>kernel: NET: 12 messages suppressed.
-=Solution 1=+==Solution==
-*Go to the Web-Admin and log in+After you have rebooted and turned off any heavy P2P applications:
 +*Go to the Web Interface of the DD-WRT Device and log in
*Go to 'Administration' *Go to 'Administration'
*Go to 'Management' *Go to 'Management'
*Enter the following values at 'IP Filter Settings' *Enter the following values at 'IP Filter Settings'
-**'''Maximum Ports''': 4096+**'''Maximum Ports''': 4096 (For an 8MB RAM model set it no higher than 1024)
-**'''TCP Timeout (s)''': 90 to 3600 (decrease if you have many TCP connections)+**'''TCP Timeout (s)''': between 120 to 600 (higher is safer, lower can forget connections too quickly)
-**'''UDP Timeout (s)''': 90 to 3600 (decrease if you have many UDP connections)+**'''UDP Timeout (s)''': 120
-[[Image:Ipfiltersettings.JPG|alt IP Filter Settings at DD-WRT v23]]+[[Image:IPFilterSettings.png|IP Filter Settings on DD-WRT v24 svn 12548]]
 +::Image: IP Filter Settings on DD-WRT v24 svn 12548
*Save Settings and then Reboot Router *Save Settings and then Reboot Router
-*Check via command line if it worked: 
 +The above settings control how long it takes before inactive TCP & UDP connections are forgotten by the router. If you set them too low then the router will forget connections too quickly and active connections will be dropped.
 +*'''If you start getting frequent disconnections in particular programs but not others (commonly Instant Messenger programs) then raise your TCP timeout to at least 300 or more.'''
 +
 +If you want about 15% more RAM available, which can help the router reach the theoretical maximum of 4096 ports:
 +* First, make sure the [[CLI]] is enabled. This way, it is possible to connect to the DD-WRT Device and turn the Web Interface back on when needed.
 +* Then turn off the Web Interface. This is done through the Web Interface. Apply your changes.
 +* When starting the Web Interface on an as-needed basis, use the command 'httpd'. To close it later, use the command 'killall httpd'.
 +
 +At this point, you are finished. If you want to look at more technical information, and the technical results, more information is presented below.
 +
 +====Increasing Max Connection count beyond limits of GUI====
 +Please note that this is more or less experimental and may compromise your router's stability.
 +
 +From the thread:
 +>>Here is a solution for increasing the maximum number of connections in Kernel 2.6:
 +K2.6 is very different regarding ip_conntrack_max (IP Filter Maximum Ports in the GUI) and it's hash table compared to the older kernel.
 +
 +ip_conntrack_max sets the maximum number of connections that can be kept at one time. Most people here seems to decrease the default timeout (3600s) for these connections (which results in other problems) instead of increasing the maximum value.
 +
 +On routers with large amount of RAM (32 MB for example) one could increase this by 10 times from the default max of 4096 without any problem.
 +
 +From what I understand you also have to increase the hash table where these are stored in order to benefit from this increase. In the older kernel this was not possible since one would have to set this value before booting (and there was no nvram setting for that?).
 +
 +Although in K2.6 one can change booth of these in realtime dynamically without restarting any process. A hashsize equaly large to the conntrack_max, has the best performance as I understand. Also since K2.4.21 the hashsize performs best with a value that is a power of 2 ex. 2^14 = 16384. (I have also used this as an example)
 +
 +
 +Since K2.4.23 (and newer), to change ip_conntrack_max:
 + echo "16384" > /proc/sys/net/ipv4/netfilter/ip_conntrack_max
 +
 +Since K2.6.20 (and newer), to change the hashsize:
 + echo "16384" > /sys/module/nf_conntrack/parameters/hashsize
 +
 +But none of these will be stored after a reboot, since the values are stored in RAM.
 +
 +
 +Permanently change ip_conntrack_max, ether use the webgui or write:
 + nvram set ip_conntrack_max=16384
 + nvram commit
 + reboot
 +
 +I haven't found any way to permanently change the hashsize parameter. Although since you now in K2.6 can change it in realtime, you can just add it to your startup script and it should do the job.
 + echo "16384" > /sys/module/nf_conntrack/parameters/hashsize
 +
 +
 +==Checking the Results==
 +Besides an obvious improvement in performance, there is a way to check via [[CLI]] the technical results:
<pre> <pre>
cat /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established cat /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established
Line 60: Line 110:
90 90
</pre> </pre>
-*This causes your inactive connections to be dropped after 90 seconds. 
-==Popular, but outdated solution==+==Commonly mentioned out-dated solution & The modern equivalent==
-The magical line used to correct timeouts manually +The following command is often recommended in forums, but only applies to older versions of the DD-WRT firmware up to V23SP1:
- echo '600 1800 120 60 120 120 10 60 30 120' > /proc/sys/net/ipv4/ip_conntrack_tcp_timeouts+ <code>echo '600 1800 120 60 120 120 10 60 30 120' > /proc/sys/net/ipv4/ip_conntrack_tcp_timeouts</code>
-is often recommended in forums, but it works '''only on old firmwares''' (e.g. it does nothing on DD-WRT v23 SP2)+ <code>echo '30 60' > /proc/sys/net/ipv4/ip_conntrack_udp_timeouts</code>
-and '''only''' for TCP traffic, not for UDP. Starting from kernel 2.4.23 this file is ignored and proper one is in /proc/sys/net/ipv4/netfilter/ as shown above.+
-BTW precious few know what this line really does. Here is the explanation. It is a list for all the possible TCP states and their timeouts in seconds. See my table below (they are in the correct order):+'''Note:''' The method above, for resolving slowdown issues, does not apply for DD-WRT v23 SP2 firmware and beyond. Furthermore, this command only applies to TCP traffic issues only, not UDP issues.
 + 
 +The newer firmwares' (V23SP2 and newer) 2.4.23 kernel ignores both of these files. For this method to work in later kernels, the timeout values must be placed in individual files. The correct files are located in /proc/sys/net/ipv4/netfilter/ as shown below.
 + 
 +===How It Works===
 +Few people know what this line actually does. It is a list for all possible TCP states and their timeouts in seconds. See the table below (they are in the correct order):
State Timeout value State Timeout value
Line 84: Line 137:
LISTEN 2 minutes LISTEN 2 minutes
 +The full list of files in the /proc/sys/net/ipv4/netfilter/ directory that one can populate to get the same effect on v23 SP2 and later firmwares is:
-==AIM, ICQ and other IM programs==+<pre>
-If you're using certain IM clients such as ICQ or AIM (or other applications which do not send frequent keep-alive packets), you should set TCP timeout to 300 or 600 to help them stay connected. As this may cause hangups when you're dealing with very heavy P2P traffic, you might want to experiment with these values.+/proc/sys/net/ipv4/netfilter/ip_conntrack_generic_timeout
 +/proc/sys/net/ipv4/netfilter/ip_conntrack_icmp_timeout
 +/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_close
 +/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_close_wait
 +/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established
 +/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_syn_sent
 +/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_syn_recv
 +/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_fin_wait
 +/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_time_wait
 +/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_last_ack
 +/proc/sys/net/ipv4/netfilter/ip_conntrack_udp_timeout
 +/proc/sys/net/ipv4/netfilter/ip_conntrack_udp_timeout_stream
 +</pre>
 +===Script===
 +The following script should set these values automatically. It can be installed by copying it into Administration -> Commands and clicking Save Startup.
 +
 +If anyone's wondering, the 'prefix' var is there to save space in nvram.
 +
 +<pre>
 +prefix=/proc/sys/net/ipv4/netfilter/ip_conntrack
 +
 +echo 600 > ${prefix}_generic_timeout
 +echo 30 > ${prefix}_udp_timeout
 +echo 60 > ${prefix}_udp_timeout_stream
 +echo 1800 > ${prefix}_tcp_timeout_established
 +echo 120 > ${prefix}_tcp_timeout_syn_sent
 +echo 60 > ${prefix}_tcp_timeout_syn_recv
 +echo 120 > ${prefix}_tcp_timeout_fin_wait
 +echo 120 > ${prefix}_tcp_timeout_time_wait
 +echo 10 > ${prefix}_tcp_timeout_close
 +echo 60 > ${prefix}_tcp_timeout_close_wait
 +echo 30 > ${prefix}_tcp_timeout_last_ack
 +
 +</pre>
 +
 +==AIM, ICQ and other IM programs==
 +If you're using certain IM clients such as ICQ or AIM (or other applications which do not send frequent keep-alive packets), you should set TCP timeout to 300 or 600 to help them stay connected. As higher values for TCP may cause hangups when you're dealing with very heavy P2P traffic, you might want to experiment with these values.
-=Solution 2=+==More help==
DD-WRT has an inbuild proxy feature that allows rewriting of HTML content to filter ActiveX cookies, etc. DD-WRT has an inbuild proxy feature that allows rewriting of HTML content to filter ActiveX cookies, etc.
-As this is load-intensive, you may want to disable this feature.+As this is load-intensive, you may want to disable this feature, to give the DD-WRT Device more time to deal with open port requests.
This should be done under Security->Firewall. This should be done under Security->Firewall.
Unfortunately this function has 3 bugs: Unfortunately this function has 3 bugs:
-*Though all 4 checkboxes are unchecked the rewriting proxy is still running (you might have to reboot)+*When unchecking all 4 checkboxes the rewriting proxy may still be running (you might have to reboot)
-*It drops connections if under heavy load+*The proxy drops connections if under heavy load
-*It probably runs haywire checking traffic on other ports than 80, hence the slowdown with P2P applications (unconfirmed)+*One user notes: It probably runs haywire checking traffic on other ports than 80, hence the slowdown with P2P applications (unconfirmed)
-The only solution is to totally disable the firewall (which I do '''''not''''' recommend) in+A solution is to disable the firewall (which is '''''not''''' recommended):
Security->Firewall. Security->Firewall.
-This fixes all slowdowns.+This can also fix slow-downs.
-This behavior has been seen in v22 and v23.+(This behavior has been seen in v22 and v23.
-Please report this to the maintainer and put it in the bug database.+Please report this to the maintainer and put it in the bug database.)
-=Solution 3=+==Bug in older WRT54G/S==
Apparently there is a bug in the WRT54GS v2.2. The bug involves a fatal memory access error due to a difference in the CPU clock speed and the clock speed on the memory bus. This bug causes connections to be dropped when transferring large files. New versions of the Linksys firmware overclock the CPU to 216 MHz. This can be fixed by logging into the router via [[Telnet/SSH and the Command Line|SSH or Telnet]] and executing the following commands: Apparently there is a bug in the WRT54GS v2.2. The bug involves a fatal memory access error due to a difference in the CPU clock speed and the clock speed on the memory bus. This bug causes connections to be dropped when transferring large files. New versions of the Linksys firmware overclock the CPU to 216 MHz. This can be fixed by logging into the router via [[Telnet/SSH and the Command Line|SSH or Telnet]] and executing the following commands:
Line 116: Line 206:
</pre> </pre>
-When Rrunning DD-WRT micro v23sp2 on a WRT54G v5 the clock speed can be adjusted by logging into the router via [[Telnet/SSH and the Command Line|SSH or Telnet]] and executing the following commands:+When running DD-WRT micro v23sp2 on a WRT54G v5 the clock speed can be adjusted by logging into the router via [[Telnet/SSH and the Command Line|SSH or Telnet]] and executing the following commands:
<pre> <pre>
-nvram set overclock=216+nvram set overclocking=216
nvram commit nvram commit
reboot reboot
Line 128: Line 218:
<pre> <pre>
nvram get clkfreq nvram get clkfreq
-</pre> +</pre>
- +
- +
----2/20/07-- +
-[[Category:English documentation]]+
- +
- +
- +
- +
- +
- +
- +
 +==More help: Adjust your P2P applications==
 +For P2P applications, and especially Bittorrent, the problem can be an overwhelming number of incoming connections. Try disabling DHT and/or setting a limit to the number of peers/incoming connections to around 100 or so. See the [http://www.azureuswiki.com/index.php/Network_connection_dying Azureus page] regarding this.
 +==More help: Heat-sinks==
 +Consumer routers are not designed for high load applications, but instead for low cost. They often lack heatsinks on the main chips as a cost and space saving measure. Unfortunately, they can overheat when dealing with the load of continuous, heavy traffic.
 +Installing heat sinks, like those used for desktop system chipsets but smaller, does help this problem. A small amount of heat-transfer solution and some cyanoacrylate [the chemical name for SuperGlue-style glues, although much better quality versions can be found at hobby and electronics stores and online]. Clearance [distance to the inside-top of the router case] is a consideration.
 +Note that some Linksys routers in particular are intended to be assembled, not disassembled. You may need to use glue or other ugly ways to get the case back together after opening.
 +==More Help: Web Traffic Daemon on v24x==
 +If you are unable to load the WebUI and have already tried the TCP timeout tweaks, try turning off the WAN Traffic Daemon. This has been known to cause problems up to v24 for the D-Link DIR-300.
 +==Known issues with Applications/Operating Systems==
 +:'''Applications'''
 +* When using Steam (of Valve) and when you refresh your server list, it might cause your router in the default setup, to reboot since it opens many connections. You can remedy this issue by following the steps on this Steam suport page: https://support.steampowered.com/kb_article.php?ref=4487-YUOC-6922
 +==p2p protocoll info==
 + applejuice
 + audiogalaxy tcp
 + ares tcp
 + bearshare tcp
 + bittorrent tcp/udp
 + directconnect tcp
 + edonkey tcp/udp
 + fasttrak/kazaa tcp/udp
 + freenet
 + gnucleuslan
 + gnutella tcp
 + goboogy
 + hotline
 + imesh
 + kugoo
 + mute tcp
 + napster tcp
 + openft tcp
 + pando tcp
 + soribada tcp
 + soulseek tcp
 + tesla
 + waste
 + winmx tcp
 + xdcc
 + xunlei tcp/udp
-[http://abcsearchdirect.info abcsearchdirect] Charlotte Home For Sale [http://charlotte-home-for-sale.abcsearchdirect.info/charlotte-home-for-sale.html charlotte home for sale] [http://online-stock-market-trading.abcsearchdirect.info/index.html online stock market trading] Charlotte Home Builder [http://charlotte-home-builder.abcsearchdirect.info/charlotte-home-builder.html charlotte home builder] [http://online-stock-trading.abcsearchdirect.info/index.html online stock trading] Charlotte Diamond [http://charlotte-diamond.abcsearchdirect.info/charlotte-diamond.html charlotte diamond] [http://online-store.abcsearchdirect.info/index.html online store] Charlie Christian [http://charlie-christian.abcsearchdirect.info/charlie-christian.html charlie christian] [http://online-survey.abcsearchdirect.info/index.html online survey] Charleston Real Estate [http://charleston-real-estate.abcsearchdirect.info/charleston-real-estate.html charleston real estate] [http://online-tax.abcsearchdirect.info/index.html online tax] Charleston Home For Sale [http://charleston-home-for-sale.abcsearchdirect.info/charleston-home-for-sale.html charleston home for sale] [http://online-toy-store.abcsearchdirect.info/index.html online toy store] Charity Donation [http://charity-donation.abcsearchdirect.info/charity-donation.html charity donation] [http://on-line-trading.abcsearchdirect.info/index.html on line trading] Character Education [http://character-education.abcsearchdirect.info/character-education.html character education] [http://online-trading.abcsearchdirect.info/index.html online trading] Chaparral Boat [http://chaparral-boat.abcsearchdirect.info/chaparral-boat.html chaparral boat] [http://online-travel-agency.abcsearchdirect.info/index.html online travel agency] Channel Science [http://channel-science.abcsearchdirect.info/channel-science.html channel science] [http://online-travel-agent.abcsearchdirect.info/index.html online travel agent] Chandler Arizona [http://chandler-arizona.abcsearchdirect.info/chandler-arizona.html chandler arizona] [http://online-universities-and-colleges.abcsearchdirect.info/index.html online universities and colleges] Chandeliers [http://chandeliers.abcsearchdirect.info/chandeliers.html chandeliers] [http://online-valium.abcsearchdirect.info/index.html online valium] Champion Nutrition [http://champion-nutrition.abcsearchdirect.info/champion-nutrition.html champion nutrition] [http://online-viagra.abcsearchdirect.info/index.html online viagra] Chair Massage [http://chair-massage.abcsearchdirect.info/chair-massage.html chair massage] [http://online-vicodin.abcsearchdirect.info/index.html online vicodin] Chair Dentist [http://chair-dentist.abcsearchdirect.info/chair-dentist.html chair dentist] [http://online-video-poker.abcsearchdirect.info/index.html online video poker] Chair [http://chair.abcsearchdirect.info/chair.html chair] [http://online-wager.abcsearchdirect.info/index.html online wager] Chadwick [http://chadwick.abcsearchdirect.info/chadwick.html chadwick] [http://online-wagering.abcsearchdirect.info/index.html online wagering] Cfd Trading [http://cfd-trading.abcsearchdirect.info/cfd-trading.html cfd trading] [http://online-watcher-weight.abcsearchdirect.info/index.html online watcher weight] Cervical Cancer [http://cervical-cancer.abcsearchdirect.info/cervical-cancer.html cervical cancer] [http://ontario-classifieds.abcsearchdirect.info/index.html ontario classifieds] Certificate Lien Tax [http://certificate-lien-tax.abcsearchdirect.info/certificate-lien-tax.html certificate lien tax] [http://ontario-education.abcsearchdirect.info/index.html ontario education] Cephalexin [http://cephalexin.abcsearchdirect.info/cephalexin.html cephalexin] [http://ontario-home-builder.abcsearchdirect.info/index.html ontario home builder] Center Treatment [http://center-treatment.abcsearchdirect.info/center-treatment.html center treatment] [http://ontario-travel.abcsearchdirect.info/index.html ontario travel] Center Pacific Science [http://center-pacific-science.abcsearchdirect.info/center-pacific-science.html center pacific science] [http://ontario-travel-agent.abcsearchdirect.info/index.html ontario travel agent] Center Orlando Science [http://center-orlando-science.abcsearchdirect.info/center-orlando-science.html center orlando science] [http://oprah-and-tom-cruise.abcsearchdirect.info/index.html oprah and tom cruise] Center Ontario Science [http://center-ontario-science.abcsearchdirect.info/center-ontario-science.html center ontario science] [http://optimum-nutrition.abcsearchdirect.info/index.html optimum nutrition] Center Drug Treatment [http://center-drug-treatment.abcsearchdirect.info/center-drug-treatment.html center drug treatment] [http://option-stock-trading.abcsearchdirect.info/index.html option stock trading] Center Diabetes [http://center-diabetes.abcsearchdirect.info/center-diabetes.html center diabetes] [http://option-trading.abcsearchdirect.info/index.html option trading] Cellulite Treatment [http://cellulite-treatment.abcsearchdirect.info/cellulite-treatment.html cellulite treatment] [http://orange.abcsearchdirect.info/index.html orange] Cellulite [http://cellulite.abcsearchdirect.info/cellulite.html cellulite] [http://orange-county-home-for-sale.abcsearchdirect.info/index.html orange county home for sale] Cellular Phone [http://cellular-phone.abcsearchdirect.info/cellular-phone.html cellular phone] [http://orbitz-travel.abcsearchdirect.info/index.html orbitz travel] Cell Phone Batteries [http://cell-phone-batteries.abcsearchdirect.info/cell-phone-batteries.html cell phone batteries] [http://orchard-bank-credit-card.abcsearchdirect.info/index.html orchard bank credit card] Cell Phone Accessories [http://cell-phone-accessories.abcsearchdirect.info/cell-phone-accessories.html cell phone accessories] [http://order-check.abcsearchdirect.info/index.html order check] Cell Phone [http://cell-phone.abcsearchdirect.info/cell-phone.html cell phone] [http://order-check-online.abcsearchdirect.info/index.html order check online] Cellini Rolex [http://cellini-rolex.abcsearchdirect.info/cellini-rolex.html cellini rolex] [http://order-checks.abcsearchdirect.info/index.html order checks] Celexa [http://celexa.abcsearchdirect.info/celexa.html celexa] [http://order-contact-lens.abcsearchdirect.info/index.html order contact lens] Celebrity Diet [http://celebrity-diet.abcsearchdirect.info/celebrity-diet.html celebrity diet] [http://order-contact-lens-online.abcsearchdirect.info/index.html order contact lens online] Celebrity Cruises [http://celebrity-cruises.abcsearchdirect.info/celebrity-cruises.html celebrity cruises] [http://order-personal-check.abcsearchdirect.info/index.html order personal check] Celebrity Cruise Line [http://celebrity-cruise-line.abcsearchdirect.info/celebrity-cruise-line.html celebrity cruise line] [http://order-viagra.abcsearchdirect.info/index.html order viagra] Celebrex Lawyer [http://celebrex-lawyer.abcsearchdirect.info/celebrex-lawyer.html celebrex lawyer] [http://oregano.abcsearchdirect.info/index.html oregano] Celebrex [http://celebrex.abcsearchdirect.info/celebrex.html celebrex] [http://oregon-coast-vacation-rental.abcsearchdirect.info/index.html oregon coast vacation rental] Celebraties [http://celebraties.abcsearchdirect.info/celebraties.html celebraties] [http://oregon-home-equity-loan.abcsearchdirect.info/index.html oregon home equity loan] Ceiling Lighting [http://ceiling-lighting.abcsearchdirect.info/ceiling-lighting.html ceiling lighting] [http://oregon-real-estate.abcsearchdirect.info/index.html oregon real estate] Ceiling Fans [http://ceiling-fans.abcsearchdirect.info/ceiling-fans.html ceiling fans] [http://oregon-refinance.abcsearchdirect.info/index.html oregon refinance] Ceiling Fan Lighting [http://ceiling-fan-lighting.abcsearchdirect.info/ceiling-fan-lighting.html ceiling fan lighting] [http://orexis.abcsearchdirect.info/index.html orexis] Cd Storage [http://cd-storage.abcsearchdirect.info/cd-storage.html cd storage] [http://organic-coffee.abcsearchdirect.info/index.html organic coffee] Cd Printer [http://cd-printer.abcsearchdirect.info/cd-printer.html cd printer] [http://orgy-party.abcsearchdirect.info/index.html orgy party] Cd Poker [http://cd-poker.abcsearchdirect.info/cd-poker.html cd poker] [http://oriental-massage.abcsearchdirect.info/index.html oriental massage] Cd Labels [http://cd-labels.abcsearchdirect.info/cd-labels.html cd labels] [http://oriental-trading.abcsearchdirect.info/index.html oriental trading] Cd Duplication [http://cd-duplication.abcsearchdirect.info/cd-duplication.html cd duplication] [http://oriental-trading-company.abcsearchdirect.info/index.html oriental trading company] Cd Covers [http://cd-covers.abcsearchdirect.info/cd-covers.html cd covers] [http://orlando.abcsearchdirect.info/index.html orlando] Cd Cases [http://cd-cases.abcsearchdirect.info/cd-cases.html cd cases] [http://orlando-discount-hotel.abcsearchdirect.info/index.html orlando discount hotel] Cctv Cameras [http://cctv-cameras.abcsearchdirect.info/cctv-cameras.html cctv cameras] [http://orlando-dodge.abcsearchdirect.info/index.html orlando dodge] Cctv [http://cctv.abcsearchdirect.info/cctv.html cctv] [http://orlando-florida.abcsearchdirect.info/index.html orlando florida] Cause Stroke [http://cause-stroke.abcsearchdirect.info/cause-stroke.html cause stroke] [http://orlando-home-builder.abcsearchdirect.info/index.html orlando home builder] Cause Of The Great Depression [http://cause-of-the-great-depression.abcsearchdirect.info/cause-of-the-great-depression.html cause of the great depression] [http://orlando-home-for-sale.abcsearchdirect.info/index.html orlando home for sale] Cause Of High Cholesterol [http://cause-of-high-cholesterol.abcsearchdirect.info/cause-of-high-cholesterol.html cause of high cholesterol] [http://orlando-hotel.abcsearchdirect.info/index.html orlando hotel] Cause Of Depression [http://cause-of-depression.abcsearchdirect.info/cause-of-depression.html cause of depression] [http://orlando-package-vacation.abcsearchdirect.info/index.html orlando package vacation] Cause Diabetes [http://cause-diabetes.abcsearchdirect.info/cause-diabetes.html cause diabetes] [http://orlando-rental-vacation.abcsearchdirect.info/index.html orlando rental vacation] Cat Quote [http://cat-quote.abcsearchdirect.info/cat-quote.html cat quote] [http://orlando-resorts.abcsearchdirect.info/index.html orlando resorts] Cat Nutrition [http://cat-nutrition.abcsearchdirect.info/cat-nutrition.html cat nutrition] [http://orlando-used-car.abcsearchdirect.info/index.html orlando used car] Catnip What Is [http://catnip-what-is.abcsearchdirect.info/catnip-what-is.html catnip what is] [http://orlando-vacation.abcsearchdirect.info/index.html orlando vacation] Ca Ticketmaster [http://ca-ticketmaster.abcsearchdirect.info/ca-ticketmaster.html ca ticketmaster] [http://orlando-vacation-home.abcsearchdirect.info/index.html orlando vacation home] Catholic Singles [http://catholic-singles.abcsearchdirect.info/catholic-singles.html catholic singles] [http://orlando-vacation-home-rental.abcsearchdirect.info/index.html orlando vacation home rental] Catholic Single [http://catholic-single.abcsearchdirect.info/catholic-single.html catholic single] [http://orlando-vacation-rental.abcsearchdirect.info/index.html orlando vacation rental] Cat Furniture [http://cat-furniture.abcsearchdirect.info/cat-furniture.html cat furniture] [http://orleans.abcsearchdirect.info/index.html orleans] Catalog Clothing Womens [http://catalog-clothing-womens.abcsearchdirect.info/catalog-clothing-womens.html catalog clothing womens] [http://orleans-home-builder.abcsearchdirect.info/index.html orleans home builder] Casting [http://casting.abcsearchdirect.info/casting.html casting] [http://orvis.abcsearchdirect.info/index.html orvis] Cast Die Nascar [http://cast-die-nascar.abcsearchdirect.info/cast-die-nascar.html cast die nascar] [http://osteoporosis-treatment.abcsearchdirect.info/index.html osteoporosis treatment] Casio World Open [http://casio-world-open.abcsearchdirect.info/casio-world-open.html casio world open] [http://outdoor-christmas-lighting.abcsearchdirect.info/index.html outdoor christmas lighting] Casio Waveceptor [http://casio-waveceptor.abcsearchdirect.info/casio-waveceptor.html casio waveceptor] [http://outdoor-furniture.abcsearchdirect.info/index.html outdoor furniture] Casio Wave Ceptor [http://casio-wave-ceptor.abcsearchdirect.info/casio-wave-ceptor.html casio wave ceptor] [http://outdoor-heaters.abcsearchdirect.info/index.html outdoor heaters] Casio Watch Solar [http://casio-watch-solar.abcsearchdirect.info/casio-watch-solar.html casio watch solar] [http://outdoor-lighting.abcsearchdirect.info/index.html outdoor lighting] Casio Watches [http://casio-watches.abcsearchdirect.info/casio-watches.html casio watches] [http://outdoor-lighting-fixture.abcsearchdirect.info/index.html outdoor lighting fixture] Casio Watch Band [http://casio-watch-band.abcsearchdirect.info/casio-watch-band.html casio watch band] [http://outdoor-water-fountain.abcsearchdirect.info/index.html outdoor water fountain] Casio Watch [http://casio-watch.abcsearchdirect.info/casio-watch.html casio watch] [http://outer-banks-vacation-rental.abcsearchdirect.info/index.html outer banks vacation rental] Casio Tv [http://casio-tv.abcsearchdirect.info/casio-tv.html casio tv] [http://outer-wear-womens.abcsearchdirect.info/index.html outer wear womens] Casio Privia [http://casio-privia.abcsearchdirect.info/casio-privia.html casio privia] [http://outlander.abcsearchdirect.info/index.html outlander] Casio Piano [http://casio-piano.abcsearchdirect.info/casio-piano.html casio piano] [http://outlet-ticketmaster.abcsearchdirect.info/index.html outlet ticketmaster] Casio Pathfinder Watch [http://casio-pathfinder-watch.abcsearchdirect.info/casio-pathfinder-watch.html casio pathfinder watch] [http://outline-resume.abcsearchdirect.info/index.html outline resume] Casio Pathfinder [http://casio-pathfinder.abcsearchdirect.info/casio-pathfinder.html casio pathfinder] [http://outsourcing.abcsearchdirect.info/index.html outsourcing] Casio Man Watch [http://casio-man-watch.abcsearchdirect.info/casio-man-watch.html casio man watch] [http://overhead-projector.abcsearchdirect.info/index.html overhead projector] Casio Manual [http://casio-manual.abcsearchdirect.info/casio-manual.html casio manual] [http://over-stock.abcsearchdirect.info/index.html over stock] Casio Keyboard [http://casio-keyboard.abcsearchdirect.info/casio-keyboard.html casio keyboard] [http://overstock.abcsearchdirect.info/index.html overstock] Casio G Shock Watch [http://casio-g-shock-watch.abcsearchdirect.info/casio-g-shock-watch.html casio g shock watch] [http://overstock-coupon.abcsearchdirect.info/index.html overstock coupon] Casio G Shock [http://casio-g-shock.abcsearchdirect.info/casio-g-shock.html casio g shock] [http://over-weight.abcsearchdirect.info/index.html over weight] Casio Exlim [http://casio-exlim.abcsearchdirect.info/casio-exlim.html casio exlim] [http://overweight.abcsearchdirect.info/index.html overweight] Casio Exilim Digital Camera [http://casio-exilim-digital-camera.abcsearchdirect.info/casio-exilim-digital-camera.html casio exilim digital camera] [http://owned-pre-rolex.abcsearchdirect.info/index.html owned pre rolex] Casio Exilim Camera [http://casio-exilim-camera.abcsearchdirect.info/casio-exilim-camera.html casio exilim camera] [http://oxycontin.abcsearchdirect.info/index.html oxycontin] Casio Exilim [http://casio-exilim.abcsearchdirect.info/casio-exilim.html casio exilim] [http://oyster-perpetual-rolex.abcsearchdirect.info/index.html oyster perpetual rolex] Casio Ex [http://casio-ex.abcsearchdirect.info/casio-ex.html casio ex] [http://oyster-rolex.abcsearchdirect.info/index.html oyster rolex] Casio Elixim [http://casio-elixim.abcsearchdirect.info/casio-elixim.html casio elixim] [http://oz-diet.abcsearchdirect.info/index.html oz diet] Casio Digital Watch [http://casio-digital-watch.abcsearchdirect.info/casio-digital-watch.html casio digital watch] [http://pa-bankruptcy.abcsearchdirect.info/index.html pa bankruptcy] Casio Digital Piano [http://casio-digital-piano.abcsearchdirect.info/casio-digital-piano.html casio digital piano] [http://pacer-bankruptcy.abcsearchdirect.info/index.html pacer bankruptcy] Casio Digital Camera [http://casio-digital-camera.abcsearchdirect.info/casio-digital-camera.html casio digital camera] [http://pacific-poker.abcsearchdirect.info/index.html pacific poker] Casio Digital [http://casio-digital.abcsearchdirect.info/casio-digital.html casio digital] [http://package-vacation.abcsearchdirect.info/index.html package vacation] Casio Cash Register [http://casio-cash-register.abcsearchdirect.info/casio-cash-register.html casio cash register] [http://package-vacation-vegas.abcsearchdirect.info/index.html package vacation vegas] Casio Camera [http://casio-camera.abcsearchdirect.info/casio-camera.html casio camera] [http://packers.abcsearchdirect.info/index.html packers] Casio Calculator [http://casio-calculator.abcsearchdirect.info/casio-calculator.html casio calculator] [http://paddle-boat.abcsearchdirect.info/index.html paddle boat] Casio Baby G [http://casio-baby-g.abcsearchdirect.info/casio-baby-g.html casio baby g] [http://pagemaker.abcsearchdirect.info/index.html pagemaker] Casio Atomic Watch [http://casio-atomic-watch.abcsearchdirect.info/casio-atomic-watch.html casio atomic watch] [http://pager.abcsearchdirect.info/index.html pager] Casino Slot [http://casino-slot.abcsearchdirect.info/casino-slot.html casino slot] [http://paid-for-surveys.abcsearchdirect.info/index.html paid for surveys] Casinos Bonuses [http://casinos-bonuses.abcsearchdirect.info/casinos-bonuses.html casinos bonuses] [http://paid-surveys.abcsearchdirect.info/index.html paid surveys] Casinos [http://casinos.abcsearchdirect.info/casinos.html casinos] [http://pain-medications.abcsearchdirect.info/index.html pain medications] Casino Roulette [http://casino-roulette.abcsearchdirect.info/casino-roulette.html casino roulette] [http://pain-relief.abcsearchdirect.info/index.html pain relief] Casino Poker Chips [http://casino-poker-chips.abcsearchdirect.info/casino-poker-chips.html casino poker chips] [http://paintball.abcsearchdirect.info/index.html paintball] Casino Poker [http://casino-poker.abcsearchdirect.info/casino-poker.html casino poker] [http://pajamas-party.abcsearchdirect.info/index.html pajamas party] Casino On The Net [http://casino-on-the-net.abcsearchdirect.info/casino-on-the-net.html casino on the net] [http://pajamas-womens.abcsearchdirect.info/index.html pajamas womens] Casino Online Slot [http://casino-online-slot.abcsearchdirect.info/casino-online-slot.html casino online slot] [http://palm.abcsearchdirect.info/index.html palm] Casino Online Game [http://casino-online-game.abcsearchdirect.info/casino-online-game.html casino online game] [http://palm-springs-hotel.abcsearchdirect.info/index.html palm springs hotel] Casino Online [http://casino-online.abcsearchdirect.info/casino-online.html casino online] [http://pa-mortgage-rate.abcsearchdirect.info/index.html pa mortgage rate] Casino Louisiana [http://casino-louisiana.abcsearchdirect.info/casino-louisiana.html casino louisiana] [http://panama-canal-cruise.abcsearchdirect.info/index.html panama canal cruise] Casino Internet Poker [http://casino-internet-poker.abcsearchdirect.info/casino-internet-poker.html casino internet poker] [http://panama-city-beach.abcsearchdirect.info/index.html panama city beach] Casino Hotel Island Treasure [http://casino-hotel-island-treasure.abcsearchdirect.info/casino-hotel-island-treasure.html casino hotel island treasure] [http://panama-cruise.abcsearchdirect.info/index.html panama cruise] Casino Games [http://casino-games.abcsearchdirect.info/casino-games.html casino games] [http://panasonic-home-theater-system.abcsearchdirect.info/index.html panasonic home theater system] Casino Gambling Online [http://casino-gambling-online.abcsearchdirect.info/casino-gambling-online.html casino gambling online] [http://panther-party.abcsearchdirect.info/index.html panther party] Casino Gambling [http://casino-gambling.abcsearchdirect.info/casino-gambling.html casino gambling] [http://pantie-womens.abcsearchdirect.info/index.html pantie womens] Casino Craps [http://casino-craps.abcsearchdirect.info/casino-craps.html casino craps] [http://papa-johns-coupon.abcsearchdirect.info/index.html papa johns coupon] Casino Chips [http://casino-chips.abcsearchdirect.info/casino-chips.html casino chips] [http://papa-johns-pizza-coupon.abcsearchdirect.info/index.html papa johns pizza coupon] Casino Bonus [http://casino-bonus.abcsearchdirect.info/casino-bonus.html casino bonus] [http://paper-shredders.abcsearchdirect.info/index.html paper shredders] Casino Black Jack [http://casino-black-jack.abcsearchdirect.info/casino-black-jack.html casino black jack] [http://paper-truck.abcsearchdirect.info/index.html paper truck] Casino Betting [http://casino-betting.abcsearchdirect.info/casino-betting.html casino betting] [http://paradise-poker.abcsearchdirect.info/index.html paradise poker] Casino Arizona [http://casino-arizona.abcsearchdirect.info/casino-arizona.html casino arizona] [http://parenteral-nutrition.abcsearchdirect.info/index.html parenteral nutrition] Casino Affiliate Program [http://casino-affiliate-program.abcsearchdirect.info/casino-affiliate-program.html casino affiliate program] [http://paris-hotel.abcsearchdirect.info/index.html paris hotel] Cash Payday Loan [http://cash-payday-loan.abcsearchdirect.info/cash-payday-loan.html cash payday loan] [http://paris-hotel-las-vegas.abcsearchdirect.info/index.html paris hotel las vegas] Cash Out Refinance [http://cash-out-refinance.abcsearchdirect.info/cash-out-refinance.html cash out refinance] [http://paris-vacation.abcsearchdirect.info/index.html paris vacation] Cash Need [http://cash-need.abcsearchdirect.info/cash-need.html cash need] [http://park-city-hotel.abcsearchdirect.info/index.html park city hotel] Cash Loan Online [http://cash-loan-online.abcsearchdirect.info/cash-loan-online.html cash loan online] [http://parking-at-gatwick-airport.abcsearchdirect.info/index.html parking at gatwick airport] Cash Loan Fast [http://cash-loan-fast.abcsearchdirect.info/cash-loan-fast.html cash loan fast] [http://parkinsons-disease.abcsearchdirect.info/index.html parkinsons disease] Cash Loan [http://cash-loan.abcsearchdirect.info/cash-loan.html cash loan] [http://part-racing.abcsearchdirect.info/index.html part racing] Cashing Check [http://cashing-check.abcsearchdirect.info/cashing-check.html cashing check] [http://part-rolex.abcsearchdirect.info/index.html part rolex] Cash In Advance [http://cash-in-advance.abcsearchdirect.info/cash-in-advance.html cash in advance] [http://part-truck.abcsearchdirect.info/index.html part truck] Cashier Check [http://cashier-check.abcsearchdirect.info/cashier-check.html cashier check] [http://party-america.abcsearchdirect.info/index.html party america] Cash For Settlement [http://cash-for-settlement.abcsearchdirect.info/cash-for-settlement.html cash for settlement] [http://party-boy.abcsearchdirect.info/index.html party boy] Cash For Annuity [http://cash-for-annuity.abcsearchdirect.info/cash-for-annuity.html cash for annuity] [http://party-bus.abcsearchdirect.info/index.html party bus] Cash Check Into [http://cash-check-into.abcsearchdirect.info/cash-check-into.html cash check into] [http://party-city.abcsearchdirect.info/index.html party city] Cash Check [http://cash-check.abcsearchdirect.info/cash-check.html cash check] [http://party-cove.abcsearchdirect.info/index.html party cove] Cash Advance Payday Loan [http://cash-advance-payday-loan.abcsearchdirect.info/cash-advance-payday-loan.html cash advance payday loan] [http://party-decoration.abcsearchdirect.info/index.html party decoration] Cash Advance Credit Card [http://cash-advance-credit-card.abcsearchdirect.info/cash-advance-credit-card.html cash advance credit card] [http://party-dress.abcsearchdirect.info/index.html party dress] Cash Advance America [http://cash-advance-america.abcsearchdirect.info/cash-advance-america.html cash advance america] [http://party-favor.abcsearchdirect.info/index.html party favor] Cash Advance [http://cash-advance.abcsearchdirect.info/cash-advance.html cash advance] [http://party-favors.abcsearchdirect.info/index.html party favors] Casa De Campo [http://casa-de-campo.abcsearchdirect.info/casa-de-campo.html casa de campo] [http://party-game.abcsearchdirect.info/index.html party game] Car Wax [http://car-wax.abcsearchdirect.info/car-wax.html car wax] [http://party-girl.abcsearchdirect.info/index.html party girl] Car Warranty [http://car-warranty.abcsearchdirect.info/car-warranty.html car warranty] [http://party-halls.abcsearchdirect.info/index.html party halls] Car Undefined Used [http://car-undefined-used.abcsearchdirect.info/car-undefined-used.html car undefined used] [http://party-hardcore.abcsearchdirect.info/index.html party hardcore] Car Truck Used Van [http://car-truck-used-van.abcsearchdirect.info/car-truck-used-van.html car truck used van] [http://party-idea.abcsearchdirect.info/index.html party idea] Car Truck Part [http://car-truck-part.abcsearchdirect.info/car-truck-part.html car truck part] [http://party-invitation.abcsearchdirect.info/index.html party invitation] Car Truck [http://car-truck.abcsearchdirect.info/car-truck.html car truck] [http://party-monster.abcsearchdirect.info/index.html party monster] Cartridge Refills [http://cartridge-refills.abcsearchdirect.info/cartridge-refills.html cartridge refills] [http://party-music.abcsearchdirect.info/index.html party music] Cartridge [http://cartridge.abcsearchdirect.info/cartridge.html cartridge] [http://party-place.abcsearchdirect.info/index.html party place] Car Trading [http://car-trading.abcsearchdirect.info/car-trading.html car trading] [http://party-planner.abcsearchdirect.info/index.html party planner] Cartier Watch [http://cartier-watch.abcsearchdirect.info/cartier-watch.html cartier watch] [http://party-planning.abcsearchdirect.info/index.html party planning] Cartier [http://cartier.abcsearchdirect.info/cartier.html cartier] [http://party-poker.abcsearchdirect.info/index.html party poker] Cart Go Racing [http://cart-go-racing.abcsearchdirect.info/cart-go-racing.html cart go racing] [http://partypoker.abcsearchdirect.info/index.html partypoker] Car Stereo [http://car-stereo.abcsearchdirect.info/car-stereo.html car stereo] [http://party-poker-bonus.abcsearchdirect.info/index.html party poker bonus] Cars Rental [http://cars-rental.abcsearchdirect.info/cars-rental.html cars rental] [http://party-poker-chips.abcsearchdirect.info/index.html party poker chips] Car Spare Part [http://car-spare-part.abcsearchdirect.info/car-spare-part.html car spare part] [http://party-rental.abcsearchdirect.info/index.html party rental] Cars New [http://cars-new.abcsearchdirect.info/cars-new.html cars new] [http://party-store.abcsearchdirect.info/index.html party store] Car Slot Track [http://car-slot-track.abcsearchdirect.info/car-slot-track.html car slot track] [http://party-supplies.abcsearchdirect.info/index.html party supplies] Car Slot [http://car-slot.abcsearchdirect.info/car-slot.html car slot] [http://party-supply.abcsearchdirect.info/index.html party supply] Car Show [http://car-show.abcsearchdirect.info/car-show.html car show] [http://party-supply-store.abcsearchdirect.info/index.html party supply store] Car Seat [http://car-seat.abcsearchdirect.info/car-seat.html car seat] [http://party-tent.abcsearchdirect.info/index.html party tent] Cars Buy New [http://cars-buy-new.abcsearchdirect.info/cars-buy-new.html cars buy new] [http://party-theme.abcsearchdirect.info/index.html party theme] Car Satellite Tv [http://car-satellite-tv.abcsearchdirect.info/car-satellite-tv.html car satellite tv] [http://party-world.abcsearchdirect.info/index.html party world] Car Sales [http://car-sales.abcsearchdirect.info/car-sales.html car sales] [http://passing-drug-test.abcsearchdirect.info/index.html passing drug test] Carriage House [http://carriage-house.abcsearchdirect.info/carriage-house.html carriage house] [http://passion-party.abcsearchdirect.info/index.html passion party] Car Repair [http://car-repair.abcsearchdirect.info/car-repair.html car repair] [http://patent.abcsearchdirect.info/index.html patent] Car Rental Vancouver [http://car-rental-vancouver.abcsearchdirect.info/car-rental-vancouver.html car rental vancouver] [http://patio-awning.abcsearchdirect.info/index.html patio awning] Car Rental Uk [http://car-rental-uk.abcsearchdirect.info/car-rental-uk.html car rental uk] [http://patio-furniture.abcsearchdirect.info/index.html patio furniture] Car Rental Toronto [http://car-rental-toronto.abcsearchdirect.info/car-rental-toronto.html car rental toronto] [http://patio-heater.abcsearchdirect.info/index.html patio heater] Car Rental Thrifty [http://car-rental-thrifty.abcsearchdirect.info/car-rental-thrifty.html car rental thrifty] [http://patio-umbrellas.abcsearchdirect.info/index.html patio umbrellas] Car Rental Sports [http://car-rental-sports.abcsearchdirect.info/car-rental-sports.html car rental sports] [http://paul-mitchell.abcsearchdirect.info/index.html paul mitchell] Car Rental Special [http://car-rental-special.abcsearchdirect.info/car-rental-special.html car rental special] [http://paulson-poker-chips.abcsearchdirect.info/index.html paulson poker chips] Car Rental South Africa [http://car-rental-south-africa.abcsearchdirect.info/car-rental-south-africa.html car rental south africa] [http://paxil.abcsearchdirect.info/index.html paxil] Car Rental Seattle [http://car-rental-seattle.abcsearchdirect.info/car-rental-seattle.html car rental seattle] [http://payday.abcsearchdirect.info/index.html payday] Car Rental Philadelphia [http://car-rental-philadelphia.abcsearchdirect.info/car-rental-philadelphia.html car rental philadelphia] [http://payday-advance.abcsearchdirect.info/index.html payday advance] Car Rental Nj [http://car-rental-nj.abcsearchdirect.info/car-rental-nj.html car rental nj] [http://payday-cash-advance.abcsearchdirect.info/index.html payday cash advance] Car Rental Europe [http://car-rental-europe.abcsearchdirect.info/car-rental-europe.html car rental europe] [http://pay-day-cash-loan.abcsearchdirect.info/index.html pay day cash loan] Car Rental Coupon [http://car-rental-coupon.abcsearchdirect.info/car-rental-coupon.html car rental coupon] [http://payday-loan.abcsearchdirect.info/index.html payday loan] Car Rental Companies [http://car-rental-companies.abcsearchdirect.info/car-rental-companies.html car rental companies] [http://payday-loan-no-fax.abcsearchdirect.info/index.html payday loan no fax] Car Rental Chicago [http://car-rental-chicago.abcsearchdirect.info/car-rental-chicago.html car rental chicago] [http://payment-processing.abcsearchdirect.info/index.html payment processing] Car Rental [http://car-rental.abcsearchdirect.info/car-rental.html car rental] [http://pay-pal-coupon.abcsearchdirect.info/index.html pay pal coupon] Car Rate Rental [http://car-rate-rental.abcsearchdirect.info/car-rate-rental.html car rate rental] [http://payroll.abcsearchdirect.info/index.html payroll] Car Racing Street [http://car-racing-street.abcsearchdirect.info/car-racing-street.html car racing street] [http://payroll-accounting.abcsearchdirect.info/index.html payroll accounting] Car Racing Stock [http://car-racing-stock.abcsearchdirect.info/car-racing-stock.html car racing stock] [http://payroll-accounting-software.abcsearchdirect.info/index.html payroll accounting software] Car Racing Slot [http://car-racing-slot.abcsearchdirect.info/car-racing-slot.html car racing slot] [http://payroll-software.abcsearchdirect.info/index.html payroll software] Car Racing [http://car-racing.abcsearchdirect.info/car-racing.html car racing] [http://payroll-tax.abcsearchdirect.info/index.html payroll tax] Car Quote [http://car-quote.abcsearchdirect.info/car-quote.html car quote] [http://pay-sprint-bill.abcsearchdirect.info/index.html pay sprint bill] Car Pricing [http://car-pricing.abcsearchdirect.info/car-pricing.html car pricing] [http://pay-tax.abcsearchdirect.info/index.html pay tax] Car Price [http://car-price.abcsearchdirect.info/car-price.html car price] [http://pcanywhere.abcsearchdirect.info/index.html pcanywhere] Car Payless Rental [http://car-payless-rental.abcsearchdirect.info/car-payless-rental.html car payless rental] [http://pc-satellite-tv.abcsearchdirect.info/index.html pc satellite tv] Car Part Wholesale [http://car-part-wholesale.abcsearchdirect.info/car-part-wholesale.html car part wholesale] [http://pc-store.abcsearchdirect.info/index.html pc store] Car Part Truck [http://car-part-truck.abcsearchdirect.info/car-part-truck.html car part truck] [http://pda.abcsearchdirect.info/index.html pda] Car Part Store [http://car-part-store.abcsearchdirect.info/car-part-store.html car part store] [http://peachtree-accounting.abcsearchdirect.info/index.html peachtree accounting] Car Part Online [http://car-part-online.abcsearchdirect.info/car-part-online.html car part online] [http://peachtree-accounting-software.abcsearchdirect.info/index.html peachtree accounting software] Car Part New York [http://car-part-new-york.abcsearchdirect.info/car-part-new-york.html car part new york] [http://pearls.abcsearchdirect.info/index.html pearls] Car Part For Sale [http://car-part-for-sale.abcsearchdirect.info/car-part-for-sale.html car part for sale] [http://pearson-education.abcsearchdirect.info/index.html pearson education] Car Part Catalog [http://car-part-catalog.abcsearchdirect.info/car-part-catalog.html car part catalog] [http://pedal-boat.abcsearchdirect.info/index.html pedal boat] Car Part Canada [http://car-part-canada.abcsearchdirect.info/car-part-canada.html car part canada] [http://pedal-car-part.abcsearchdirect.info/index.html pedal car part] Car Part And Accessory [http://car-part-and-accessory.abcsearchdirect.info/car-part-and-accessory.html car part and accessory] [http://pedicure-spa.abcsearchdirect.info/index.html pedicure spa] Car Ontario Rental [http://car-ontario-rental.abcsearchdirect.info/car-ontario-rental.html car ontario rental] [http://pee-party.abcsearchdirect.info/index.html pee party] Car Online [http://car-online.abcsearchdirect.info/car-online.html car online] [http://peets-coffee.abcsearchdirect.info/index.html peets coffee] Car One Rental Way [http://car-one-rental-way.abcsearchdirect.info/car-one-rental-way.html car one rental way] [http://pen.abcsearchdirect.info/index.html pen] Carolina North Rental Vacation [http://carolina-north-rental-vacation.abcsearchdirect.info/carolina-north-rental-vacation.html carolina north rental vacation] [http://pendant-lighting.abcsearchdirect.info/index.html pendant lighting]+[[Category:Monitoring]]
 +[[Category:Applications settings]]

Revision as of 12:18, 22 February 2013


Contents

Introduction

Issues occur with many routers, including routers running DD-WRT, when using the router with heavy P2P applications. The router becomes 'slow' over time, and restarting helps for a short time. Some symptoms can be:

  • Slow web-interface, or cannot connect at all to web interface
  • Slowing transfer of data, e.g. browsing, after a reboot
  • Not responding to ping
  • Router Crash or even rebooting

Usually the culprits are heavy P2P software like Emule, Bittorrent, uTorrent, Azureus, Shareaza or something similar. These programs, by default, can require a lot of connections which could cause the routers' ip_conntrack table to get full.

  • Especially BitTorrent's DHT feature sends thousands of UDP packets that quickly overflow this table.

Routers affected with this issue are the most common types of routers running DD-WRT.

Definitions

P2P
Peer to Peer applications. Could mean eMule, uTorrent, Azureus. Within the context of this article does not mean Skype, or other 'lite' P2P applications.
DD-WRT Device
A Router running third-party, DD-WRT firmware. Although configuration in other methods besides 'router' is possible, such a 'bridge', this article and phenomena only occurs in the default router configuration of DD-WRT.
Web Interface
The default management method, open a browser to http://192.168.1.1 or your custom DD-WRT Device's LAN IP.

Diagnosis

When this 'slowdown' occurs and the router doesn't respond to pings nor Web Interface requests, you still can check what's going on:

  • Close all your P2P/network applications and wait a few minutes for connections to be freed.
  • Try to use the CLI to communicate with your router.
    • If that's not possible, reboot the DD-WRT Device.
    • Check to see if your problem is caused by TCP or UDP connections. From the CLI:
cat /proc/sys/net/ipv4/netfilter/ip_conntrack_max
4096 # Here the router displays the maximum number of possible connections.
grep -c ^udp /proc/net/ip_conntrack
3693 # Here UDP is using 3693 entries
grep -c ^tcp /proc/net/ip_conntrack
115   # and TCP uses only 115 entries
  • You may also view syslog messages:
cat /var/log/messages
  1. First, you'll see 'full, dropping packet.' messages followed by 'messages suppressed.'
  2. Eventually, it will get verbose and you'll see logs similar to the following :
<4>kernel: ip_conntrack: table full, dropping packet.
<4>kernel: NET: 15 messages suppressed.
<4>kernel: ip_conntrack: table full, dropping packet.
<4>kernel: NET: 12 messages suppressed.

Solution

After you have rebooted and turned off any heavy P2P applications:

  • Go to the Web Interface of the DD-WRT Device and log in
  • Go to 'Administration'
  • Go to 'Management'
  • Enter the following values at 'IP Filter Settings'
    • Maximum Ports: 4096 (For an 8MB RAM model set it no higher than 1024)
    • TCP Timeout (s): between 120 to 600 (higher is safer, lower can forget connections too quickly)
    • UDP Timeout (s): 120

IP Filter Settings on DD-WRT v24 svn 12548

Image: IP Filter Settings on DD-WRT v24 svn 12548
  • Save Settings and then Reboot Router

The above settings control how long it takes before inactive TCP & UDP connections are forgotten by the router. If you set them too low then the router will forget connections too quickly and active connections will be dropped.

  • If you start getting frequent disconnections in particular programs but not others (commonly Instant Messenger programs) then raise your TCP timeout to at least 300 or more.

If you want about 15% more RAM available, which can help the router reach the theoretical maximum of 4096 ports:

  • First, make sure the CLI is enabled. This way, it is possible to connect to the DD-WRT Device and turn the Web Interface back on when needed.
  • Then turn off the Web Interface. This is done through the Web Interface. Apply your changes.
  • When starting the Web Interface on an as-needed basis, use the command 'httpd'. To close it later, use the command 'killall httpd'.

At this point, you are finished. If you want to look at more technical information, and the technical results, more information is presented below.

Increasing Max Connection count beyond limits of GUI

Please note that this is more or less experimental and may compromise your router's stability.

From the thread: >>Here is a solution for increasing the maximum number of connections in Kernel 2.6: K2.6 is very different regarding ip_conntrack_max (IP Filter Maximum Ports in the GUI) and it's hash table compared to the older kernel.

ip_conntrack_max sets the maximum number of connections that can be kept at one time. Most people here seems to decrease the default timeout (3600s) for these connections (which results in other problems) instead of increasing the maximum value.

On routers with large amount of RAM (32 MB for example) one could increase this by 10 times from the default max of 4096 without any problem.

From what I understand you also have to increase the hash table where these are stored in order to benefit from this increase. In the older kernel this was not possible since one would have to set this value before booting (and there was no nvram setting for that?).

Although in K2.6 one can change booth of these in realtime dynamically without restarting any process. A hashsize equaly large to the conntrack_max, has the best performance as I understand. Also since K2.4.21 the hashsize performs best with a value that is a power of 2 ex. 2^14 = 16384. (I have also used this as an example)


Since K2.4.23 (and newer), to change ip_conntrack_max:

echo "16384" > /proc/sys/net/ipv4/netfilter/ip_conntrack_max

Since K2.6.20 (and newer), to change the hashsize:

echo "16384" > /sys/module/nf_conntrack/parameters/hashsize

But none of these will be stored after a reboot, since the values are stored in RAM.


Permanently change ip_conntrack_max, ether use the webgui or write:

nvram set ip_conntrack_max=16384 
nvram commit 
reboot

I haven't found any way to permanently change the hashsize parameter. Although since you now in K2.6 can change it in realtime, you can just add it to your startup script and it should do the job.

echo "16384" > /sys/module/nf_conntrack/parameters/hashsize


Checking the Results

Besides an obvious improvement in performance, there is a way to check via CLI the technical results:

cat /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established
90
cat /proc/sys/net/ipv4/netfilter/ip_conntrack_udp_timeout_stream
90


Commonly mentioned out-dated solution & The modern equivalent

The following command is often recommended in forums, but only applies to older versions of the DD-WRT firmware up to V23SP1:

echo '600 1800 120 60 120 120 10 60 30 120' > /proc/sys/net/ipv4/ip_conntrack_tcp_timeouts
echo '30 60' > /proc/sys/net/ipv4/ip_conntrack_udp_timeouts

Note: The method above, for resolving slowdown issues, does not apply for DD-WRT v23 SP2 firmware and beyond. Furthermore, this command only applies to TCP traffic issues only, not UDP issues.

The newer firmwares' (V23SP2 and newer) 2.4.23 kernel ignores both of these files. For this method to work in later kernels, the timeout values must be placed in individual files. The correct files are located in /proc/sys/net/ipv4/netfilter/ as shown below.

How It Works

Few people know what this line actually does. It is a list for all possible TCP states and their timeouts in seconds. See the table below (they are in the correct order):

State		Timeout value

NONE		10 minutes
ESTABLISHED	30 minutes (default is 5 days)
SYN_SENT	2 minutes
SYN_RECV	60 seconds
FIN_WAIT	2 minutes
TIME_WAIT	2 minutes
CLOSE		10 seconds
CLOSE_WAIT	60 seconds (default is 12 hours)
LAST_ACK	30 seconds
LISTEN		2 minutes

The full list of files in the /proc/sys/net/ipv4/netfilter/ directory that one can populate to get the same effect on v23 SP2 and later firmwares is:

/proc/sys/net/ipv4/netfilter/ip_conntrack_generic_timeout
/proc/sys/net/ipv4/netfilter/ip_conntrack_icmp_timeout
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_close
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_close_wait
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_syn_sent
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_syn_recv
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_fin_wait
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_time_wait
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_last_ack
/proc/sys/net/ipv4/netfilter/ip_conntrack_udp_timeout
/proc/sys/net/ipv4/netfilter/ip_conntrack_udp_timeout_stream

Script

The following script should set these values automatically. It can be installed by copying it into Administration -> Commands and clicking Save Startup.

If anyone's wondering, the 'prefix' var is there to save space in nvram.

prefix=/proc/sys/net/ipv4/netfilter/ip_conntrack

echo 600 > ${prefix}_generic_timeout
echo 30 > ${prefix}_udp_timeout
echo 60 > ${prefix}_udp_timeout_stream
echo 1800 > ${prefix}_tcp_timeout_established
echo 120 > ${prefix}_tcp_timeout_syn_sent
echo 60 > ${prefix}_tcp_timeout_syn_recv
echo 120 > ${prefix}_tcp_timeout_fin_wait
echo 120 > ${prefix}_tcp_timeout_time_wait
echo 10 > ${prefix}_tcp_timeout_close
echo 60 > ${prefix}_tcp_timeout_close_wait
echo 30 > ${prefix}_tcp_timeout_last_ack

AIM, ICQ and other IM programs

If you're using certain IM clients such as ICQ or AIM (or other applications which do not send frequent keep-alive packets), you should set TCP timeout to 300 or 600 to help them stay connected. As higher values for TCP may cause hangups when you're dealing with very heavy P2P traffic, you might want to experiment with these values.

More help

DD-WRT has an inbuild proxy feature that allows rewriting of HTML content to filter ActiveX cookies, etc. As this is load-intensive, you may want to disable this feature, to give the DD-WRT Device more time to deal with open port requests. This should be done under Security->Firewall.

Unfortunately this function has 3 bugs:

  • When unchecking all 4 checkboxes the rewriting proxy may still be running (you might have to reboot)
  • The proxy drops connections if under heavy load
  • One user notes: It probably runs haywire checking traffic on other ports than 80, hence the slowdown with P2P applications (unconfirmed)

A solution is to disable the firewall (which is not recommended): Security->Firewall. This can also fix slow-downs.

(This behavior has been seen in v22 and v23. Please report this to the maintainer and put it in the bug database.)


Bug in older WRT54G/S

Apparently there is a bug in the WRT54GS v2.2. The bug involves a fatal memory access error due to a difference in the CPU clock speed and the clock speed on the memory bus. This bug causes connections to be dropped when transferring large files. New versions of the Linksys firmware overclock the CPU to 216 MHz. This can be fixed by logging into the router via SSH or Telnet and executing the following commands:

nvram set clkfreq=216
nvram commit
reboot

When running DD-WRT micro v23sp2 on a WRT54G v5 the clock speed can be adjusted by logging into the router via SSH or Telnet and executing the following commands:

nvram set overclocking=216
nvram commit
reboot

After rebooting the router you can confirm that the new clock speed has applied by checking the Status page of the DD-WRT GUI or by logging in via Telnet and executing the following:

nvram get clkfreq

More help: Adjust your P2P applications

For P2P applications, and especially Bittorrent, the problem can be an overwhelming number of incoming connections. Try disabling DHT and/or setting a limit to the number of peers/incoming connections to around 100 or so. See the Azureus page regarding this.

More help: Heat-sinks

Consumer routers are not designed for high load applications, but instead for low cost. They often lack heatsinks on the main chips as a cost and space saving measure. Unfortunately, they can overheat when dealing with the load of continuous, heavy traffic.

Installing heat sinks, like those used for desktop system chipsets but smaller, does help this problem. A small amount of heat-transfer solution and some cyanoacrylate [the chemical name for SuperGlue-style glues, although much better quality versions can be found at hobby and electronics stores and online]. Clearance [distance to the inside-top of the router case] is a consideration.

Note that some Linksys routers in particular are intended to be assembled, not disassembled. You may need to use glue or other ugly ways to get the case back together after opening.

More Help: Web Traffic Daemon on v24x

If you are unable to load the WebUI and have already tried the TCP timeout tweaks, try turning off the WAN Traffic Daemon. This has been known to cause problems up to v24 for the D-Link DIR-300.

Known issues with Applications/Operating Systems

Applications

p2p protocoll info

applejuice
audiogalaxy tcp
ares        tcp
bearshare   tcp
bittorrent    tcp/udp
directconnect tcp
edonkey       tcp/udp
fasttrak/kazaa tcp/udp
freenet
gnucleuslan
gnutella   tcp
goboogy
hotline
imesh
kugoo
mute       tcp
napster     tcp
openft    tcp
pando     tcp
soribada   tcp
soulseek    tcp
tesla
waste
winmx      tcp
xdcc
xunlei     tcp/udp