[GTER] SQUID squid-3.1.0.14 + TPROXY
Renato Murilo Langona
renato at linuxsecurity.com.br
Mon Nov 23 10:00:27 -02 2009
Voce vai criar 4 caches em disco, porem eles podem ser compartilhados
entre si e podem ter objetos distintos, por exemplo, voce pode usar como
ponto de partida as confs desse artigo que esta postado em alguns forums
e portais:
http://www.jorgefernando.eti.br/index.php?option=com_content&view=article&id=44:squid-instancias&catid=36:servicos&Itemid=55
Abs!
Marcelus Trojahn escreveu:
> Isto eu ja fiz... Mas com cache de disco, isso acaba criando 4 caches
> pois voce nao pode usar o mesmo cache_dir para processos diferentes do
> Squid... No final das contas a economia de banda cai dastricamente...
>
> --
> Marcelus Trojahn
> Administrador de Rede
> I-Conecta Redes de Telecomunicação Ltda
> Rua Araruama, 132 - Sobreloja, Centro
> Rio das Ostras - RJ - CEP: 28.890-000
> Tel: (22) 2764.4500 - Fax: (22) 2764.4510
>
>
>
> 2009/11/20 Marcus Almeida <marcus at isprj.com.br>:
>
>> Abre "X" instancias do squid (onde X é o numero de processadores) e coloca a
>> regra no iptables pra redirecionar para o squid conforme o bloco de ip
>> destino que você deseja.
>>
>> Atenciosamente,
>> Marcus Roberto.
>>
>>
>> 2009/11/19 Luzivan <luzivan at gmail.com>
>>
>>
>>> Prezado Marcus Almeida, como fazer o cache por dominio de destino ?
>>>
>>> qual config do squid precisa alterar ou criar ?
>>>
>>> 2009/11/19 Marcus Almeida <marcus at isprj.com.br>
>>>
>>>
>>>> Faça o cache por dominio de destino, assim você podera distribuir melhor
>>>>
>>> a
>>>
>>>> carga sobre o squid.
>>>>
>>>> O problema não é a quantidade de usuarios, espaço em disco ou memoria
>>>>
>>> ram,
>>>
>>>> e
>>>> sim a arquitetura que você esta utilizando que impede o crescimento
>>>>
>>> sadio.
>>>
>>>> Atenciosamente,
>>>> Marcus Roberto.
>>>>
>>>>
>>>> 2009/11/19 Marcelus Trojahn <mtrojahn at gmail.com>
>>>>
>>>>
>>>>> Sim, tproxy...
>>>>>
>>>>> Em horarios entre 17h ate as 23h passa por ele cerca de 80Mbps de
>>>>> trafego HTTP... Por este motivo, se voce prestar atencao abaixo, ele
>>>>> nao tem cache em disco...
>>>>>
>>>>> O servidor tem 8GB de memoria (que pretendo aumentar para 16) e
>>>>> trabalha apenas com cache_mem com objetos com maximo de 500K.
>>>>>
>>>>> Com este trafego todo, foi impossivel fazer o Squid nao ter problema
>>>>> de I/O de disco... Depois de testar as mais variadas tecnicas (raid0,
>>>>> filesystems diferentes, caches grandes, caches pequenos, varios
>>>>> caches, etc) optamos por simplesmente nao fazer nada em disco... O
>>>>> Squid nao dava conta de fazer a rotacao dos arquivos em disco... Assim
>>>>> que o cache enchia e ele tinha que comecar a deletar objetos antigos
>>>>> para armazenar novos ele ficava lento... Arquivos eram criados muito
>>>>> mais rapido do que ele podia apagar ate o ponto onde ele parava de
>>>>> fazer proxy e se dedicava apenas a limpar o cache...
>>>>>
>>>>> Obviamente minha economia de banda nao eh tanta desta forma, mas mesmo
>>>>> assim se ecomiza mais de 10Mbs no meu link... Em um link de 100Mbs, eu
>>>>> diria que 10% de economia ja esta de bom tamanho.
>>>>>
>>>>>
>>>>> Squid Object Cache: Version 3.1.0.13-20090807
>>>>> Connection information for squid:
>>>>> Number of clients accessing cache: 1445
>>>>> Number of HTTP requests received: 7247571
>>>>> Number of ICP messages received: 0
>>>>> Number of ICP messages sent: 0
>>>>> Number of queued ICP replies: 0
>>>>> Number of HTCP messages received: 0
>>>>> Number of HTCP messages sent: 0
>>>>> Request failure ratio: 0.00
>>>>> Average HTTP requests per minute since start: 17079.7
>>>>> Average ICP messages per minute since start: 0.0
>>>>> Select loop called: 243685394 times, 0.104 ms avg
>>>>> Cache information for squid:
>>>>> Hits as % of all requests: 5min: 25.6%, 60min: 27.4%
>>>>> Hits as % of bytes sent: 5min: 8.0%, 60min: 8.6%
>>>>> Memory hits as % of hit requests: 5min: 64.7%, 60min:
>>>>>
>>> 65.0%
>>>
>>>>> Disk hits as % of hit requests: 5min: 0.1%, 60min: 0.1%
>>>>> Storage Swap size: 0 KB
>>>>> Storage Swap capacity: 0.0% used, 0.0% free
>>>>> Storage Mem size: 5631744 KB
>>>>> Storage Mem capacity: 100.0% used, 0.0% free
>>>>> Mean Object Size: 0.00 KB
>>>>> Requests given to unlinkd: 0
>>>>> Median Service Times (seconds) 5 min 60 min:
>>>>> HTTP Requests (All): 0.14252 0.12106
>>>>> Cache Misses: 0.22004 0.22004
>>>>> Cache Hits: 0.00000 0.00000
>>>>> Near Hits: 0.03622 0.02899
>>>>> Not-Modified Replies: 0.00000 0.00000
>>>>> DNS Lookups: 0.00000 0.00000
>>>>> ICP Queries: 0.00000 0.00000
>>>>> Resource usage for squid:
>>>>> UP Time: 25460.275 seconds
>>>>> CPU Time: 4958.640 seconds
>>>>> CPU Usage: 19.48%
>>>>> CPU Usage, 5 minute avg: 23.70%
>>>>> CPU Usage, 60 minute avg: 25.00%
>>>>> Process Data Segment Size via sbrk(): 6917456 KB
>>>>> Maximum Resident Size: 0 KB
>>>>> Page faults with physical i/o: 21
>>>>> Memory usage for squid via mallinfo():
>>>>> Total space in arena: -1470876 KB
>>>>> Ordinary blocks: -1474849 KB 43100 blks
>>>>> Small blocks: 0 KB 0 blks
>>>>> Holding blocks: 335460 KB 2048 blks
>>>>> Free Small blocks: 0 KB
>>>>> Free Ordinary blocks: 3972 KB
>>>>> Total in use: -1139389 KB 100%
>>>>> Total free: 3972 KB 0%
>>>>> Total size: -1135416 KB
>>>>> Memory accounted for:
>>>>> Total accounted: -1643262 KB 145%
>>>>> memPool accounted: -1643262 KB 145%
>>>>> memPool unaccounted: 507845 KB -44%
>>>>> memPoolAlloc calls: 1451417810
>>>>> memPoolFree calls: 1433098664
>>>>> File descriptor usage for squid:
>>>>> Maximum number of file descriptors: 102400
>>>>> Largest file desc currently in use: 12099
>>>>> Number of file desc currently in use: 10563
>>>>> Files queued for open: 0
>>>>> Available number of file descriptors: 91837
>>>>> Reserved number of file descriptors: 100
>>>>> Store Disk files open: 0
>>>>> Internal Data Structures:
>>>>> 439312 StoreEntries
>>>>> 439312 StoreEntries with MemObjects
>>>>> 438576 Hot Object Cache Items
>>>>> 0 on-disk objects
>>>>>
>>>>> --
>>>>> Marcelus Trojahn
>>>>>
>>>>>
>>>>> 2009/11/19 Luzivan <luzivan at gmail.com>:
>>>>>
>>>>>> Prezado Marcelus, perguntas quanto ao seu squid:
>>>>>>
>>>>>> 1) Trabalha junto com o tproxy, fazendo spoofing do IP dos clientes ?
>>>>>>
>>>>>> 2) Tem quantos acessos simultaneos ?
>>>>>>
>>>>>> 2009/11/17 Marcelus Trojahn <mtrojahn at gmail.com>
>>>>>>
>>>>>>
>>>>>>> Da uma monitorada no syslog desta maquina... Eu lembro de ter o
>>>>>>>
>>> mesmo
>>>
>>>>>>> problema que voce e, na realidade, era relacionado com os valores
>>>>>>> padroes de TCP do kernel... Meu syslog ficava cheio de erros de TCP
>>>>>>> enquanto eu quebrava a cabeca apenas lendo o log do Squdi... Tive
>>>>>>>
>>> que
>>>
>>>>>>> dar uma boa aumentada nas variaveis abaixo para parar com estes
>>>>>>>
>>> erros
>>>
>>>>>>> e deixar o squid estavel...
>>>>>>>
>>>>>>> Aumentando estas variaveis, uma coisa levou a outra... Tive mais
>>>>>>> problemas de memoria tambem relacionado ao TCP, VM, etc... No final
>>>>>>> das contas, minhas configuracoes em /etc/sysctl.conf ficaram as
>>>>>>> seguintes:
>>>>>>>
>>>>>>> net.core.rmem_default = 65536
>>>>>>> net.core.rmem_max = 8388608
>>>>>>> net.core.wmem_default = 65536
>>>>>>> net.core.wmem_max = 8388608
>>>>>>> net.ipv4.tcp_rmem = 4096 87380 8388608
>>>>>>> net.ipv4.tcp_wmem = 4096 65536 8388608
>>>>>>> net.ipv4.tcp_mem = 8388608 8388608 8388608
>>>>>>> net.ipv4.tcp_low_latency = 1
>>>>>>> net.core.netdev_max_backlog = 4000
>>>>>>> net.ipv4.ip_local_port_range = 1024 65000
>>>>>>> net.ipv4.tcp_max_syn_backlog = 1024
>>>>>>> vm.min_free_kbytes = 65536
>>>>>>>
>>>>>>> Sinceramente, nao lembro mais o que elas fazem... Sei que tem um
>>>>>>> grande potencial pra deixar pior ;) Tome cuidado, salve os defaults
>>>>>>>
>>> e
>>>
>>>>>>> leia um pouco sobre elas antes de aplicar isto ai...
>>>>>>>
>>>>>>> --
>>>>>>> Marcelus Trojahn
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Nov 7, 2009 at 2:58 AM, Luzivan <luzivan at gmail.com> wrote:
>>>>>>>
>>>>>>>> Após 1 minuto acontecer este erro... já pesquisei e seguir vários
>>>>>>>> procedimentos encontrados na internet mas nenhum deles resolveu.
>>>>>>>>
>>>>>>>> commBind: Cannot bind socket FD 987 to xxx.xxx.xxx.xxx: (22)
>>>>>>>>
>>> Invalid
>>>
>>>>>>>> argument
>>>>>>>> WARNING: Reset of FD 987 for xxx.xxx.xxx.xxx failed to bind: (22)
>>>>>>>>
>>>>> Invalid
>>>>>
>>>>>>>> argument
>>>>>>>> commBind: Cannot bind socket FD 987 to xxx.xxx.xxx.xxx: (22)
>>>>>>>>
>>> Invalid
>>>
>>>>>>>> argument
>>>>>>>> WARNING: Reset of FD 987 for xxx.xxx.xxx.xxx failed to bind: (22)
>>>>>>>>
>>>>> Invalid
>>>>>
>>>>>>>> argument
>>>>>>>> commBind: Cannot bind socket FD 1059 to xxx.xxx.xxx.xxx: (22)
>>>>>>>>
>>>> Invalid
>>>>
>>>>>>>> argument
>>>>>>>> WARNING: Reset of FD 1059 for xxx.xxx.xxx.xxx failed to bind: (22)
>>>>>>>>
>>>>>>> Invalid
>>>>>>>
>>>>>>>> argument
>>>>>>>> commBind: Cannot bind socket FD 1059 to xxx.xxx.xxx.xxx: (22)
>>>>>>>>
>>>> Invalid
>>>>
>>>>>>>> argument
>>>>>>>> WARNING: Reset of FD 1059 for xxx.xxx.xxx.xxx failed to bind: (22)
>>>>>>>>
>>>>>>> Invalid
>>>>>>>
>>>>>>>> argument
>>>>>>>> commBind: Cannot bind socket FD 1094 to xxx.xxx.xxx.xxx: (22)
>>>>>>>>
>>>> Invalid
>>>>
>>>>>>>> argument
>>>>>>>> WARNING: Reset of FD 1094 for xxx.xxx.xxx.xxx failed to bind: (22)
>>>>>>>>
>>>>>>> Invalid
>>>>>>>
>>>>>>>> argument
>>>>>>>> commBind: Cannot bind socket FD 1094 to xxx.xxx.xxx.xxx: (22)
>>>>>>>>
>>>> Invalid
>>>>
>>>>>>>> argument
>>>>>>>> WARNING: Reset of FD 1094 for xxx.xxx.xxx.xxx failed to bind: (22)
>>>>>>>>
>>>>>>> Invalid
>>>>>>>
>>>>>>>> argument
>>>>>>>> commBind: Cannot bind socket FD 1109 to xxx.xxx.xxx.xxx: (22)
>>>>>>>>
>>>> Invalid
>>>>
>>>>>>>> argument
>>>>>>>> WARNING: Reset of FD 1109 for xxx.xxx.xxx.xxx failed to bind: (22)
>>>>>>>>
>>>>>>> Invalid
>>>>>>>
>>>>>>>> argument
>>>>>>>> commBind: Cannot bind socket FD 1109 to xxx.xxx.xxx.xxx: (22)
>>>>>>>>
>>>> Invalid
>>>>
>>>>>>>> argument
>>>>>>>> WARNING: Reset of FD 1109 for xxx.xxx.xxx.xxx failed to bind: (22)
>>>>>>>>
>>>>>>> Invalid
>>>>>>>
>>>>>>>> argument
>>>>>>>>
>>>>>>>>
>>>>>>>> Parametros de configução para compilacao do squid
>>>>>>>> ./configure --prefix=/usr/local/squid --enable-linux-netfilter
>>>>>>>> --with-default-user=squid --build=x86_64-linux-gnu --with-pthreads
>>>>>>>> --enable-storeio=ufs,aufs,diskd --with-filedescriptors=65536
>>>>>>>>
>>>>>>>> http_port 3129 tproxy
>>>>>>>> --
>>>>>>>> gter list https://eng.registro.br/mailman/listinfo/gter
>>>>>>>>
>>>>>>>>
>>>>>>> --
>>>>>>> gter list https://eng.registro.br/mailman/listinfo/gter
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> [Luzivan ;]
>>>>>> "O caminho do sucesso está sempre em construção"
>>>>>> --
>>>>>> gter list https://eng.registro.br/mailman/listinfo/gter
>>>>>>
>>>>>>
>>>>> --
>>>>> gter list https://eng.registro.br/mailman/listinfo/gter
>>>>>
>>>>>
>>>> --
>>>> gter list https://eng.registro.br/mailman/listinfo/gter
>>>>
>>>>
>>>
>>> --
>>> [Luzivan ;]
>>> "O caminho do sucesso está sempre em construção"
>>> --
>>> gter list https://eng.registro.br/mailman/listinfo/gter
>>>
>>>
>> --
>> gter list https://eng.registro.br/mailman/listinfo/gter
>>
>>
> --
> gter list https://eng.registro.br/mailman/listinfo/gter
>
More information about the gter
mailing list