Squid Proxy running inside DD with HD usb like a big cache

Post new topic   Reply to topic    DD-WRT Forum Forum Index -> Broadcom SoC based Hardware
Goto page Previous  1, 2, 3  Next
Author Message
speedbump
DD-WRT Novice


Joined: 26 Sep 2008
Posts: 6

PostPosted: Sat Jan 01, 2011 0:34    Post subject: no go for me Reply with quote
Hello

I have followed this thread step by step and I have a 160GB hd plugged into rt-n16 with ext3.

I have added cache and logs dirs in the squid dir and also did chmod 777 -R on /mnt/squid yet I still get

and I did run the nobody commands...

-----------------
root@zooo:~# squid -z
2010/12/31 19:21:16| Creating Swap Directories
FATAL: Failed to make swap directory /mnt/squid/cache/: (13) Permission denied
Squid Cache (Version 2.6.STABLE23): Terminated abnormally.
CPU Usage: 0.000 seconds = 0.000 user + 0.000 sys
Maximum Resident Size: 0 KB
Page faults with physical i/o: 0
root@zooo:~#
------------------

it's weird:
root@zooo:~# ls -la /mnt/squid
drwxrwxrwx 4 root root 4096 Dec 31 19:12 .
drwx------ 4 1000 1000 4096 Dec 30 22:39 ..
drwxrwxrwx 2 root root 4096 Dec 31 19:12 cache
drwxrwxrwx 2 root root 4096 Dec 31 19:12 logs


Any ideas would be great, thanks!

Also once I get this going do I have to set my browser proxy to ip of my router ... ?

greatly appreciated
Sponsor
Thireus
DD-WRT User


Joined: 12 Mar 2009
Posts: 84

PostPosted: Tue Mar 01, 2011 15:41    Post subject: Reply with quote
Hi everyone!

I managed to install Squid on my Asus WL-500W, once started as a transparent proxy it was running very very very slow!!!

For example, Google loads terribly slowly (more than 45sec...) oO

Does anyone know why?

CPU load was about 30%
I use a very fast USB drive mounted on /opt/ where squid is installed.

I used the exact same configuration as shown here + the script provited here for iptables: Proxy Server on the LAN Subnet (http://www.dd-wrt.com/wiki/index.php/Transparent_web_proxy)

_________________
www.DareYourMind.net
oloth
DD-WRT Novice


Joined: 08 Dec 2010
Posts: 18

PostPosted: Wed Mar 16, 2011 14:10    Post subject: Reply with quote
I am no Linux guru or even expert.

My device is Buffalo WZR-HP-G300NH (64Mb Ram) and I successfully installed and configured squid inside the device.


What I did is:

- OPKG install squid (the box is opt-ware enabled)

- Had to manually create the directories for cache and logs

- had to chmod 777 to the cache and log dirs

- created nobody user as this guide suggests


The actual working squid.conf came to be a compilation the post author's with some pieces from other threads in the forum:

Code:
cache_effective_user nobody
cache_mgr sorry@nomail.ru
visible_hostname your-router-name

#cache_replacement_policy LFUDA
#memory_replacement_policy LFUDA

ipcache_size 2048
cache_swap_low 90
cache_swap_high 95
maximum_object_size_in_memory 100 KB

#If you have 64/8MB Router you can use 16MB cache_mem if smaler router use 8MB
cache_mem 16 MB

#this is cachedir you can change if you want. 400 meams 400MB cache. size you can change what you want
cache_dir ufs /opt/var/squid/cache 400 16 256

logfile_rotate 10
fqdncache_size 2048
memory_pools off
maximum_object_size 16384 KB
quick_abort_min 0 KB
quick_abort_max 0 KB
log_icp_queries off
client_db off
buffered_logs on
half_closed_clients off
negative_dns_ttl 10 second
connect_timeout 60 second
read_timeout 80 second
request_timeout 80 second

#
dns_testnames none
mime_table /opt/etc/squid/mime.conf
unlinkd_program /opt/usr/lib/squid/unlinkd
logfile_daemon /opt/usr/lib/squid/logfile-daemon
icon_directory /opt/usr/share/squid/icons
error_directory /opt/usr/share/squid/errors/English

#if you don't direct your logs to nul it will be even
#slower than the usual slow

#Logs you can change dir what you want
#cache_access_log /opt/var/squid/logs/access.log
cache_access_log /dev/nul
cache_log /opt/var/squid/logs/debug
cache_store_log /dev/nul
#cache_store_log /opt/var/squid/logs/storage

hierarchy_stoplist on

#Proxy Ip same as ddwrt RuterIp. change how you want
http_port 10.10.14.1:3128 transparent

# Global ACL-Definitions (Access control lists)
acl idents ident REQUIRED
acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255
acl all src 0.0.0.0/0.0.0.0
acl intern dst 10.10.14.0/255.255.255.0
acl FTP proto FTP
always_direct allow FTP

#Allowed working outgoing Ports and LanIps
acl Allowed_Ports port 80 99 443 21 563 488 777 210 1025-65535
acl yourLAN src 192.168.1.0/255.255.255.0

#http_access definition
http_access allow idents
http_access allow all
http_access allow intern
http_access deny manager all
http_access allow yourLAN
http_access deny all
icp_access deny all
miss_access allow all
always_direct allow intern



The result was:

  • It works but unbearably slow.

  • I had to manually specify -f <path_to_cfg> option to point to /opt/etc/squid.conf, otherwise it would seek /etc/squid

  • Had to specify -D option to skip the initial DNS check or it won't start. (though DNS resolution works fine).
icehacker
DD-WRT Novice


Joined: 16 May 2011
Posts: 45

PostPosted: Mon Oct 24, 2011 19:05    Post subject: Reply with quote
how do i setup squid to cache my windows, ubuntu, redhat, & mac os x updates. would love if i can save some b/w TIA
zdenekji
DD-WRT Novice


Joined: 05 Feb 2010
Posts: 13

PostPosted: Sat Dec 24, 2011 20:42    Post subject: Reply with quote
icehacker wrote:
how do i setup squid to cache my windows, ubuntu, redhat, & mac os x updates. would love if i can save some b/w TIA
Windows - you will need a so-called WSUS.
Linux - it should work with your proxy as expected.
darice
DD-WRT Novice


Joined: 08 Oct 2010
Posts: 3

PostPosted: Mon May 07, 2012 9:32    Post subject: File path defaults change. Reply with quote
Had a few problems with path names. The following I put in the squid.conf file. It now works but is not consistent. Occasionally it just won't fine the web page and times out. Running on a TP-WR1043ND.

#unlinkd_program /usr/lib/squid/unlinkd
unlinkd_program /opt/usr/lib/squid/unlinkd

# logfile_daemon /usr/lib/squid/logfile-daemon
logfile_daemon /opt/usr/lib/squid/logfile-daemon

# icon_directory /usr/share/squid/icons
icon_directory /opt/usr/share/squid/icons

# error_directory /usr/share/squid/errors/English
error_directory /opt/usr/share/squid/errors/English

# pinger_program /usr/lib/squid/pinger
pinger_program /opt/usr/lib/squid/pinger
GiJOr33
DD-WRT Novice


Joined: 07 Aug 2012
Posts: 3

PostPosted: Thu Aug 16, 2012 6:53    Post subject: Reply with quote
Hi guys,

I know this thread is a little old but was just wondering if anyone had discovered a fix for the slowness of squid on the DD-WRT platform?

I've got a Linksys WRT610N v2 which is pretty much serving as a wireless access point with a few other services such as DHCP and DNSMasq as I have a Cisco Router for my ADSL/firewall.

I have a 4gb USB stick formatted to EXT2 and Optware The Right Way installed on that. Squid is also installed with it's cache location on the USB stick. It should be fast enough as it can read/write at 5MB/s and my ADSL connection is only syncing at 16mbit. Plus I have assigned 16mb RAM for caching as well so a page like google.com should appear very quickly, but it takes 10's of seconds. I have Pixelserv also installed on the DD-WRT to combat ads. I removed the blacklist from the DNS config to make sure it's not that causing the slowness and still get slow browsing.

Any help would be much appreciated.

This is my config (mildly edited)

Code:
#       WELCOME TO SQUID 2.7.STABLE9

cache_effective_user nobody
cache_mgr sorry@nomail.ru
visible_hostname WRT610N

ipcache_size 2048
cache_swap_low 90
cache_swap_high 95
cache_mem 32 MB
maximum_object_size 16384 KB
maximum_object_size_in_memory 70 KB
memory_pools off
hosts_file /etc/hosts
cache_dir ufs /opt/var/squid/cache 1024 24 256
quick_abort_min 0 KB
quick_abort_max 0 KB
log_icp_queries off
client_db off
buffered_logs on
half_closed_clients off
negative_dns_ttl 10 second
connect_timeout 1 minute
read_timeout 80 second
request_timeout 80 second

#Hide Proxy Headers
httpd_suppress_version_string on
via off
forwarded_for off

mime_table /opt/etc/squid/mime.conf
unlinkd_program /opt/libexec/unlinkd
logfile_daemon /opt/libexec/logfile-daemon
icon_directory /opt/share/squid/icons
error_directory /opt/share/squid/errors/English

#if you don't direct your logs to null it will be even
#slower than the usual slow

#Logs you can change dir what you want
#cache_access_log /opt/var/squid/logs/access.log
cache_access_log /dev/null
cache_log /opt/var/squid/logs/debug
cache_store_log /dev/null
#cache_store_log /opt/var/squid/logs/storage

dns_nameservers <dd-wrt ip>
dns_testnames www.google.com
hierarchy_stoplist on

http_port <dd-wrt ip>:3128

# Recommended minimum configuration:
# Global ACL-Definitions (Access control lists)
acl idents ident REQUIRED
acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl all src 0.0.0.0/0.0.0.0
acl to_internal dst <internal private subnet>
acl FTP proto FTP
always_direct allow FTP

#Allowed working outgoing Ports and LanIps
acl Allowed_Ports port 80 99 443 21 563 488 777 210 1025-65535
acl insideLAN src <internal private subnet>

# Andy's http_access rules
http_access allow idents
# http_access allow all
http_access allow to_internal
http_access allow manager localhost
http_access deny manager all
http_access allow insideLAN
http_access allow localhost
# And finally deny all other access to this proxy
http_access deny all
icp_access deny all
miss_access allow all
always_direct allow insideLAN

# END
6wheels
DD-WRT Novice


Joined: 02 Sep 2009
Posts: 10

PostPosted: Sun Jan 13, 2013 18:29    Post subject: found this Reply with quote
this guy is claiming to see improvements in browsing speed using squid in dd-wrt.

http://sq4ind.eu/?p=59

I wish this thread gets active again Rolling Eyes
zdenekji
DD-WRT Novice


Joined: 05 Feb 2010
Posts: 13

PostPosted: Mon Jun 24, 2013 18:28    Post subject: Re: found this Reply with quote
6wheels wrote:
this guy is claiming to see improvements in browsing speed using squid in dd-wrt.

http://sq4ind.eu/?p=59

I wish this thread gets active again Rolling Eyes
Short: Don't.
Long answer: The claim is not even made that squid did actually speed up his browsing; it's just a conjecture. I say that it won't work for these reasons:
* Modern browsers have a massive 2-layer cache (RAM, hard drive) that exceeds the router's resources by several orders.
* Today's HTTP uses many advanced cache controls and a single missing thing in the proxy server will (a) slow things down, or (b) ruin the workings of the website.
* Your thumb drive is very limited in its number of writes. You will wear it down by writing everything that ever goes to your browser.
* Today's websites use AJAX extensively and they need many things fresh. Unlike websites 15 years ago when there was a boom of these proxies - with deliberate ignoring of no-cache directives by default!
* HTTPS will avoid the caching altogether - perhaps for its own good, so Google will work and you will be able to marvel at your snake oil.
dan.searle
DD-WRT Novice


Joined: 24 Jul 2013
Posts: 3

PostPosted: Wed Jul 24, 2013 9:21    Post subject: Building custom squid 3.3 sources into an opt package Reply with quote
Hi not sure if this is the correct place, I'm a complete DD-WRT noob, but have lots of experience cross-compiling and building embedded linux firmware.

I want to use the latest squid 3.3, no caching, but implementing some features of squid 3.3 on a 64MB RAM, 32MB flash, MIPS AR9 based Buffalo router which I have already flashed with the latest build of DD-WRT.

I was planning to build the entire DD-WRT from sources, but after reading up, it seems I would be best modifying an existing ROM (.bin) as per: http://www.dd-wrt.com/wiki/index.php/Development#Extracting_the_firmware_.28extract_firmware.sh.29

However, I will still need to build squid 3.3 from sources and create an "opt ipkg" somehow. I will also need to eventually modify the GUI to allow custom configuration of my new squid 3.3 package.

Please advise, regards, Dan...
threehappypenguins
DD-WRT Novice


Joined: 17 Dec 2013
Posts: 38

PostPosted: Mon Feb 03, 2014 14:25    Post subject: Reply with quote
Maybe I'm not understanding this right, but is this a thread for plugging a USB stick in a router running DD-WRT firmware and running Squid off of the USB stick?

If so, how do you install Squid on the USB stick if I am operating out of Windows to access the Web GUI from? I can telnet the router... is that what I need to do? If so, what commands do I put in? Are there any tutorials for this?

I successfully installed Squid on Windows, and am attempting to run a transparent proxy (I think?). I just want to make Squid run where all computers and devices in the network go through Squid for logging purposes.

I tried to do that explanation here: http://www.dd-wrt.com/phpBB2/viewtopic.php?t=257503&highlight=) but I can't get it to work.

So if I can do it through a USB stick plugged in, then I'll do that!
magick777
DD-WRT User


Joined: 06 Jan 2014
Posts: 75
Location: United Kingdom

PostPosted: Mon Feb 03, 2014 17:13    Post subject: Reply with quote
threehappypenguins wrote:
Maybe I'm not understanding this right, but is this a thread for plugging a USB stick in a router running DD-WRT firmware and running Squid off of the USB stick?

If so, how do you install Squid on the USB stick if I am operating out of Windows to access the Web GUI from? I can telnet the router... is that what I need to do? If so, what commands do I put in? Are there any tutorials for this?


What you're looking for in running packages off a USB stick is (generally) known as Optware. Whether it exists and works for your router depends upon your architecture, build, version, etc. Research that first, and look for results that specifically mention your router; optware is not generic or common across all DD-WRT releases. You should expect a very steep learning curve if you are not familiar with the Linux command line, and (unless you are very keen) the embedded environment might not be the best place to learn.

If and when you have a working optware setup on your particular router, you should find packages available for squid; note that many users report that it is slow and a resource hog so you'd want a router with a fair bit of grunt before this is a sensible proposition. You might also look at tinyproxy and polipo, but I suspect if you're familiar with squid then you'll want to stick with it. This is in the territory of "some people have got it to work, some of the time, on some models". Don't expect that any tutorial will work out-of-the-box; this is not yet a very common use case for DD-WRT because until recently most routers didn't have the clout to run it. See if there's a thread for your router; asking users who have the same model may throw some more light on the subject than asking in a generic thread on running squid.

So, start by familiarising yourself with optware for your router, and if that's in order, have a go at squid. Good luck...

_________________
Netgear R7000, build 23655M (kongac)
Fonera 2100 A/B/C, build 21286
threehappypenguins
DD-WRT Novice


Joined: 17 Dec 2013
Posts: 38

PostPosted: Mon Feb 03, 2014 17:46    Post subject: Reply with quote
Thanks for your reply! I am actually learning as I'm going, so I think you're right; this might not be right for me.

I am simply trying to log all websites accessed, search queries and which images are being looked at (all from specific IP addresses on the network). This is for a home network.

I did more Google searching and I ran into something called "Raspberry Pi." I am assuming that maybe something like that might be my best bet? I'm not familiar with the command line at all; so I would have to follow tutorials for installing Squid on Linux (aka 'copy and paste' method). I do the same when using Command Prompt in Windows. I only have a few commands memorized (you know... the typical stuff like ipconfig).

I really tried to get all traffic to go through Squid on my PC by putting that IP table command in DD-WRT... but... it won't work. I'm just afraid of buying the Raspberry Pi and getting the same problem!

I wish there was a simple solution for in depth network logging! Arrgggg!!!
magick777
DD-WRT User


Joined: 06 Jan 2014
Posts: 75
Location: United Kingdom

PostPosted: Mon Feb 03, 2014 18:28    Post subject: Reply with quote
It sounds like a hell of a lot of work in order to log (some part of) who accesses what. What's your need/want to do this? Hint: if you're aiming to check up on what your kids/users are browsing, a common use case, this isn't really the way, because:

a) traffic that goes over HTTPS cannot properly be monitored
b) it can easily be circumvented by using another proxy, a forwarding web proxy, Tor, Opera Link (downloads through their servers) or any number of other mechanisms you can think of.

At best, you can use a combination of web proxy logs and firewall logs to get an idea of what someone is up to, but catch them once and it's easily circumvented. Describe what you need to achieve and maybe someone can suggest a better way (although, if you don't control/restrict the devices your users use, they'll be able to circumvent anything you can do at the router.

As for the Raspberry Pi... never used one, but I understand it to be an embedded computer on a board, probably not what you want either. If you want to play with installing Squid under Linux, make some space on whatever machine you're using and install your Linux distro of choice on it. LiveCDs work, as do installs to a USB key (if your BIOS will boot from one). Best way to learn Linux is to have it in front of you.

Back to what DD-WRT can achieve, it certainly can transparently forward all traffic for a given port onto a transparent proxy, and we can probably figure out the iptables incantations to do that if you have a correctly working proxy in the first place. No reason you can't run it on a Windows machine and have DD-WRT route web traffic through it. As a rough hint, here's how DD-WRT forwards all HTTP traffic to privoxy:

root@router:~# iptables -t nat -n -L PREROUTING
Chain PREROUTING (policy ACCEPT)
DNAT tcp -- 0.0.0.0/0 !82.35.162.79 tcp dpt:80 to:192.168.1.1:8118

You'd only have to DNAT the relevant traffic over to your Squid host, which is almost certainly a lesser task than installing Squid anew. Where ever you install it, you're still going to have to conquer iptables if you want to route traffic to it, so might as well get it working that way first, then decide if it's worth running it on an embedded system.

You may also like to have a read of 2nd post at http://www.dd-wrt.com/phpBB2/viewtopic.php?t=177443&postdays=0&postorder=asc&start=1320 which describes how to use iptables, so as to exempt things all but one MAC address from forwarding to privoxy. You can probably extrapolate from there as to how to forward the traffic you want.

NB You may do well to create a separate thread for that under Generic questions, it'll probably get more views and responses than a topic on squid per se. Many more people use iptables than use squid. Smile

_________________
Netgear R7000, build 23655M (kongac)
Fonera 2100 A/B/C, build 21286
threehappypenguins
DD-WRT Novice


Joined: 17 Dec 2013
Posts: 38

PostPosted: Tue Feb 04, 2014 16:30    Post subject: Reply with quote
I think I'm way over my head. I installed Privoxy, and read the documentation... but I just don't understand its purpose. I finally figured out that with all these config files and whatever, I am dealing with Perl (which I know nothing about). So I was having trouble figuring out what to edit in order to make the logfile work (I think I want that working in Privoxy? It logs the same things as Squid? I don't know...). I don't even know whether I am supposed to use a forward slash or backslash. I think forward slash because of the example that it gives for the Unix directory.

Still a blank logfile.

I'm just ready to cry. I just want to log where a teenager is going (for a friend) what he is searching for (whether in Google or in Instagram), and what images he is looking at. My friend wants to gather information so he can decide what to do. He has 8 kids, so of course things can get really busy. But he needs to know what a particular teenager is doing online so he can deal with it.

I see Tomato has logging capabilities: http://tomato.groov.pl/wp-content/gallery/screenshots/web_usage.png

I'll have to drive over there again and flash ANOTHER firmware on it. My friend is already computer illiterate, so it will be difficult to explain what I am doing.
Goto page Previous  1, 2, 3  Next Display posts from previous:    Page 2 of 3
Post new topic   Reply to topic    DD-WRT Forum Forum Index -> Broadcom SoC based Hardware All times are GMT

Navigation

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum