[GTER] SQUID squid-3.1.0.14 + TPROXY

Marcelus Trojahn mtrojahn at gmail.com
Mon Nov 23 09:54:09 -02 2009


acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8
acl localnet src XXX.XXX.XXX.XXX/28
acl localnet src XXX.XXX.XXX.XXX/28
acl clientes src XXX.XXX.XXX.XXX/20

acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl Safe_ports port 901         # SWAT
acl purge method PURGE
acl CONNECT method CONNECT

http_access allow manager localhost
http_access allow manager localnet
http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localnet
http_access allow clientes
http_access allow localhost
http_access deny all

http_port 3128
http_port 3129 tproxy

acl nocache dstdomain camarariodasostras.com.br
acl nocache dstdomain ostras.net
acl nocache dstdomain talkgadget.google.com
acl nocache dstdomain ceptro.br
acl nocache dstdomain batepapo.uol.com.br
acl nocache dstdomain facebook.com
acl nocachereg url_regex (get_video\?|videoplayback\?|videodownload\?)
acl nocachereg url_regex (xvideos\.com.*\.flv\??)
acl nocachereg url_regex (tube8\.com.*\.flv\??)
acl nocachereg url_regex (redtube\.com.*\.flv\??)
acl nocachereg url_regex (pornhub\.com.*\.flv\??)
acl nocachereg url_regex (hardsextube\.com.*\.flv\??)
acl nocachereg url_regex \.(mpg|mpeg)\??$
no_cache deny nocache
no_cache deny nocachereg

refresh_pattern -i \.(js|css|swf)$
    2880  200% 43200
refresh_pattern -i \.txt$
    1440  100% 20160 reload-into-ims
refresh_pattern -i \.(html|htm)$
    10    150% 43200
refresh_pattern .   0 20% 4320


quick_abort_min 0 KB
quick_abort_max 0 KB
request_timeout 60 seconds
connect_timeout 60 seconds
read_timeout 60 seconds

hierarchy_stoplist cgi-bin ?

cache_mem 5500 MB

maximum_object_size_in_memory 512 KB


memory_replacement_policy heap GDSF


max_open_disk_fds 0

access_log /var/log/squid/access.log squid
cache_log /var/log/squid/cache.log

logfile_rotate 3

emulate_httpd_log off

pid_filename /var/run/squid/squid-1.pid

debug_options ALL,1

buffered_logs on

unique_hostname squid1

log_icp_queries off

error_directory /usr/src/squid-3.1.0.9/errors/pt-br
error_default_language pt-br

client_db on
coredump_dir /var/cache/squid




--
Marcelus Trojahn
Administrador de Rede
I-Conecta Redes de Telecomunicação Ltda
Rua Araruama, 132 - Sobreloja, Centro
Rio das Ostras - RJ - CEP: 28.890-000
Tel: (22) 2764.4500 - Fax: (22) 2764.4510



2009/11/20 Arlindo F. Neto <lopan.eti at gmail.com>:
> Marcelus,
>
> Poderia postar seu squid.conf? Estou acompanhando a thread pois até então
> todos relatavam o mesmo problema que tenho. Cheguei a mesma conclusão que
> você mas ainda não testei sem cache. O que você fez? Um RAMFS ou
> simplesmente descartou o cache_disk?
>
> []'s
>
> 2009/11/19 Marcelus Trojahn <mtrojahn at gmail.com>
>
>> Sim, tproxy...
>>
>> Em horarios entre 17h ate as 23h passa por ele cerca de 80Mbps de
>> trafego HTTP... Por este motivo, se voce prestar atencao abaixo, ele
>> nao tem cache em disco...
>>
>> O servidor tem 8GB de memoria (que pretendo aumentar para 16) e
>> trabalha apenas com cache_mem com objetos com maximo de 500K.
>>
>> Com este trafego todo, foi impossivel fazer o Squid nao ter problema
>> de I/O de disco... Depois de testar as mais variadas tecnicas (raid0,
>> filesystems diferentes, caches grandes, caches pequenos, varios
>> caches, etc) optamos por simplesmente nao fazer nada em disco... O
>> Squid nao dava conta de fazer a rotacao dos arquivos em disco... Assim
>> que o cache enchia e ele tinha que comecar a deletar objetos antigos
>> para armazenar novos ele ficava lento... Arquivos eram criados muito
>> mais rapido do que ele podia apagar ate o ponto onde ele parava de
>> fazer proxy e se dedicava apenas a limpar o cache...
>>
>> Obviamente minha economia de banda nao eh tanta desta forma, mas mesmo
>> assim se ecomiza mais de 10Mbs no meu link... Em um link de 100Mbs, eu
>> diria que 10% de economia ja esta de bom tamanho.
>>
>>
>> Squid Object Cache: Version 3.1.0.13-20090807
>> Connection information for squid:
>>        Number of clients accessing cache:      1445
>>        Number of HTTP requests received:       7247571
>>        Number of ICP messages received:        0
>>        Number of ICP messages sent:    0
>>        Number of queued ICP replies:   0
>>        Number of HTCP messages received:       0
>>        Number of HTCP messages sent:   0
>>        Request failure ratio:   0.00
>>        Average HTTP requests per minute since start:   17079.7
>>        Average ICP messages per minute since start:    0.0
>>        Select loop called: 243685394 times, 0.104 ms avg
>> Cache information for squid:
>>        Hits as % of all requests:      5min: 25.6%, 60min: 27.4%
>>        Hits as % of bytes sent:        5min: 8.0%, 60min: 8.6%
>>        Memory hits as % of hit requests:       5min: 64.7%, 60min: 65.0%
>>        Disk hits as % of hit requests: 5min: 0.1%, 60min: 0.1%
>>        Storage Swap size:      0 KB
>>        Storage Swap capacity:   0.0% used,  0.0% free
>>        Storage Mem size:       5631744 KB
>>        Storage Mem capacity:   100.0% used,  0.0% free
>>        Mean Object Size:       0.00 KB
>>        Requests given to unlinkd:      0
>> Median Service Times (seconds)  5 min    60 min:
>>        HTTP Requests (All):   0.14252  0.12106
>>        Cache Misses:          0.22004  0.22004
>>        Cache Hits:            0.00000  0.00000
>>        Near Hits:             0.03622  0.02899
>>        Not-Modified Replies:  0.00000  0.00000
>>        DNS Lookups:           0.00000  0.00000
>>        ICP Queries:           0.00000  0.00000
>> Resource usage for squid:
>>        UP Time:        25460.275 seconds
>>        CPU Time:       4958.640 seconds
>>        CPU Usage:      19.48%
>>        CPU Usage, 5 minute avg:        23.70%
>>        CPU Usage, 60 minute avg:       25.00%
>>        Process Data Segment Size via sbrk(): 6917456 KB
>>        Maximum Resident Size: 0 KB
>>        Page faults with physical i/o: 21
>> Memory usage for squid via mallinfo():
>>        Total space in arena:  -1470876 KB
>>        Ordinary blocks:       -1474849 KB  43100 blks
>>        Small blocks:               0 KB      0 blks
>>        Holding blocks:        335460 KB   2048 blks
>>        Free Small blocks:          0 KB
>>        Free Ordinary blocks:    3972 KB
>>        Total in use:          -1139389 KB 100%
>>        Total free:              3972 KB 0%
>>        Total size:            -1135416 KB
>> Memory accounted for:
>>        Total accounted:       -1643262 KB 145%
>>        memPool accounted:     -1643262 KB 145%
>>        memPool unaccounted:   507845 KB -44%
>>        memPoolAlloc calls: 1451417810
>>        memPoolFree calls:  1433098664
>> File descriptor usage for squid:
>>         Maximum number of file descriptors:   102400
>>        Largest file desc currently in use:   12099
>>        Number of file desc currently in use: 10563
>>        Files queued for open:                   0
>>        Available number of file descriptors: 91837
>>         Reserved number of file descriptors:   100
>>         Store Disk files open:                   0
>> Internal Data Structures:
>>        439312 StoreEntries
>>        439312 StoreEntries with MemObjects
>>        438576 Hot Object Cache Items
>>             0 on-disk objects
>>
>> --
>> Marcelus Trojahn
>>
>>
>> 2009/11/19 Luzivan <luzivan at gmail.com>:
>> > Prezado Marcelus, perguntas quanto ao seu squid:
>> >
>> > 1) Trabalha junto com o tproxy, fazendo spoofing do IP dos clientes ?
>> >
>> > 2) Tem quantos acessos simultaneos ?
>> >
>> > 2009/11/17 Marcelus Trojahn <mtrojahn at gmail.com>
>> >
>> >> Da uma monitorada no syslog desta maquina... Eu lembro de ter o mesmo
>> >> problema que voce e, na realidade, era relacionado com os valores
>> >> padroes de TCP do kernel... Meu syslog ficava cheio de erros de TCP
>> >> enquanto eu quebrava a cabeca apenas lendo o log do Squdi... Tive que
>> >> dar uma boa aumentada nas variaveis abaixo para parar com estes erros
>> >> e deixar o squid estavel...
>> >>
>> >> Aumentando estas variaveis, uma coisa levou a outra... Tive mais
>> >> problemas de memoria tambem relacionado ao TCP, VM, etc... No final
>> >> das contas, minhas configuracoes em /etc/sysctl.conf ficaram as
>> >> seguintes:
>> >>
>> >> net.core.rmem_default = 65536
>> >> net.core.rmem_max = 8388608
>> >> net.core.wmem_default = 65536
>> >> net.core.wmem_max = 8388608
>> >> net.ipv4.tcp_rmem = 4096 87380 8388608
>> >> net.ipv4.tcp_wmem = 4096 65536 8388608
>> >> net.ipv4.tcp_mem = 8388608 8388608 8388608
>> >> net.ipv4.tcp_low_latency = 1
>> >> net.core.netdev_max_backlog = 4000
>> >> net.ipv4.ip_local_port_range = 1024 65000
>> >> net.ipv4.tcp_max_syn_backlog = 1024
>> >> vm.min_free_kbytes = 65536
>> >>
>> >> Sinceramente, nao lembro mais o que elas fazem... Sei que tem um
>> >> grande potencial pra deixar pior ;) Tome cuidado, salve os defaults e
>> >> leia um pouco sobre elas antes de aplicar isto ai...
>> >>
>> >> --
>> >> Marcelus Trojahn
>> >>
>> >>
>> >>
>> >> On Sat, Nov 7, 2009 at 2:58 AM, Luzivan <luzivan at gmail.com> wrote:
>> >> > Após 1 minuto acontecer este erro... já pesquisei e seguir vários
>> >> > procedimentos encontrados na internet mas nenhum deles resolveu.
>> >> >
>> >> > commBind: Cannot bind socket FD 987 to xxx.xxx.xxx.xxx: (22) Invalid
>> >> > argument
>> >> > WARNING: Reset of FD 987 for xxx.xxx.xxx.xxx failed to bind: (22)
>> Invalid
>> >> > argument
>> >> > commBind: Cannot bind socket FD 987 to xxx.xxx.xxx.xxx: (22) Invalid
>> >> > argument
>> >> > WARNING: Reset of FD 987 for xxx.xxx.xxx.xxx failed to bind: (22)
>> Invalid
>> >> > argument
>> >> > commBind: Cannot bind socket FD 1059 to xxx.xxx.xxx.xxx: (22) Invalid
>> >> > argument
>> >> > WARNING: Reset of FD 1059 for xxx.xxx.xxx.xxx failed to bind: (22)
>> >> Invalid
>> >> > argument
>> >> > commBind: Cannot bind socket FD 1059 to xxx.xxx.xxx.xxx: (22) Invalid
>> >> > argument
>> >> > WARNING: Reset of FD 1059 for xxx.xxx.xxx.xxx failed to bind: (22)
>> >> Invalid
>> >> > argument
>> >> > commBind: Cannot bind socket FD 1094 to xxx.xxx.xxx.xxx: (22) Invalid
>> >> > argument
>> >> > WARNING: Reset of FD 1094 for xxx.xxx.xxx.xxx failed to bind: (22)
>> >> Invalid
>> >> > argument
>> >> > commBind: Cannot bind socket FD 1094 to xxx.xxx.xxx.xxx: (22) Invalid
>> >> > argument
>> >> > WARNING: Reset of FD 1094 for xxx.xxx.xxx.xxx failed to bind: (22)
>> >> Invalid
>> >> > argument
>> >> > commBind: Cannot bind socket FD 1109 to xxx.xxx.xxx.xxx: (22) Invalid
>> >> > argument
>> >> > WARNING: Reset of FD 1109 for xxx.xxx.xxx.xxx failed to bind: (22)
>> >> Invalid
>> >> > argument
>> >> > commBind: Cannot bind socket FD 1109 to xxx.xxx.xxx.xxx: (22) Invalid
>> >> > argument
>> >> > WARNING: Reset of FD 1109 for xxx.xxx.xxx.xxx failed to bind: (22)
>> >> Invalid
>> >> > argument
>> >> >
>> >> >
>> >> > Parametros de configução para compilacao do squid
>> >> > ./configure --prefix=/usr/local/squid --enable-linux-netfilter
>> >> > --with-default-user=squid --build=x86_64-linux-gnu --with-pthreads
>> >> > --enable-storeio=ufs,aufs,diskd --with-filedescriptors=65536
>> >> >
>> >> > http_port 3129 tproxy
>> >> > --
>> >> > gter list    https://eng.registro.br/mailman/listinfo/gter
>> >> >
>> >> --
>> >> gter list    https://eng.registro.br/mailman/listinfo/gter
>> >>
>> >
>> >
>> >
>> > --
>> > [Luzivan ;]
>> > "O caminho do sucesso está sempre em construção"
>> > --
>> > gter list    https://eng.registro.br/mailman/listinfo/gter
>> >
>> --
>> gter list    https://eng.registro.br/mailman/listinfo/gter
>>
> --
> gter list    https://eng.registro.br/mailman/listinfo/gter



More information about the gter mailing list