[caiu] Indisponibilidade Lúmen

Leonardo Porto leonardo.porto em iw.net.br
Ter Maio 4 15:29:13 -03 2021 

Aqui também, mesmo horário.

Leonardo

Em 04/05/2021 15:23, Edinilson J. Santos escreveu:
> Aqui sim, ontem, 03/05/2021 por volta das 19:00hs (antes disso, ficava 
> indo e voltando).
>
> Edinilson
>
>
> Em 04/05/2021 08:52, noc em consuldata.com.br escreveu:
>> Bom dia. O link normalizou para vocês?
>>
>> Att,
>>
>>
>> Everton Araujo
>>
>> On 5/3/21 5:24 PM, Eduardo Duarte wrote:
>>> Aqui igual
>>>
>>> Jundiaí SP
>>>
>>> O link hoje parece pisca pisca de árvore de Natal
>>>
>>> Em seg., 3 de mai. de 2021 17:00, Alexandre Chaves Fonceca por (caiu) <
>>> caiu em eng.registro.br> escreveu:
>>>
>>>> essas sao as informações oficiais deles... em ordem cronologica (de
>>>> baixo para cima), e em horario GMT 0
>>>>
>>>> 2021-05-03 19:53:47 GMT - The IP NOC observed another reboot,
>>>> investigation and troubleshooting efforts remain ongoing.
>>>>
>>>> 2021-05-03 19:37:11 GMT - The IP NOC is reporting all affected alarms
>>>> remain clear and will continue to monitor and investigate the fault 
>>>> for the
>>>> next 4 hours. As such, the next notification will be provided at 
>>>> 23:45 GMT
>>>> or as additional information is obtained.
>>>>
>>>> 2021-05-03 18:49:21 GMT - The IP NOC reports that the Sao Paulo, 
>>>> Brazil
>>>> equipment has been restored to a full redundancy condition. Traffic 
>>>> was
>>>> migrated off the redundant uplink and remains stable. The IP NOC will
>>>> continue to monitor the services. IP Engineering and the equipment 
>>>> vendor
>>>> are still collaboratively investigating the cause for the service 
>>>> impact.
>>>>
>>>> 2021-05-03 17:53:10 GMT - The IP NOC reports that the impacted 
>>>> equipment
>>>> in Sao Paulo, Brazil is still stable since the redundant routing 
>>>> path was
>>>> taken out of service. Services continue to remain stable as well.
>>>> Cooperative investigations by multiple support groups along with the
>>>> equipment vendor remain underway.
>>>>
>>>> 2021-05-03 16:52:08 GMT - The IP NOC remotely migrated impacted 
>>>> services
>>>> to a redundant uplink, however some services have continued to 
>>>> bounce in
>>>> and out of service. The IP NOC, Engineering teams, Field Operations 
>>>> and the
>>>> equipment vendor continue investigating the the cause for the service
>>>> impact.
>>>>
>>>> 2021-05-03 15:55:43 GMT - Collaborative troubleshooting with the 
>>>> equipment
>>>> vendor, Field Operations and the IP NOC remain ongoing.
>>>>
>>>> 2021-05-03 15:07:56 GMT - Field Operations is onsite at the Sao Paulo
>>>> fault site. Collaborative troubleshooting with the equipment vendor 
>>>> remains
>>>> ongoing.
>>>>
>>>> 2021-05-03 14:11:40 GMT - The IP NOC reported a device has restarted
>>>> multiple times. Collaborative troubleshooting with the equipment 
>>>> vendor
>>>> remains ongoing. Field Operations are being engaged as well.
>>>>
>>>> 2021-05-03 13:51:28 GMT - The IP NOC continues to work in 
>>>> cooperation with
>>>> the equipment vendor to troubleshoot the issue. We understand how 
>>>> important
>>>> these services are to our customers and the issue has been 
>>>> escalated to the
>>>> highest levels within Lumen Service Assurance Leadership.
>>>>
>>>> 2021-05-03 13:48:56 GMT - The IP NOC continues to work in 
>>>> cooperation with
>>>> the equipment vendor to troubleshoot the issue.
>>>>
>>>> 2021-05-03 13:16:06 GMT - Cooperative investigations by multiple 
>>>> support
>>>> groups remain underway at this time.
>>>>
>>>> 2021-05-03 13:13:13 GMT - The IP NOC is working in cooperation with 
>>>> Field
>>>> Operations and the equipment vendor to troubleshoot the issue.
>>>>
>>>> 2021-05-03 12:50:33 GMT - Cooperative investigations by multiple 
>>>> support
>>>> groups remain underway at this time.
>>>>
>>>> 2021-05-03 12:02:37 GMT - On May 03, 2021 at 11:46 GMT, Lumen 
>>>> identifieda
>>>> service impact in Sao Paulo, Brazil. As this network fault is 
>>>> impacting
>>>> multiple clients, the event has increased visibility with Lumen 
>>>> leadership.
>>>> As such, client trouble tickets associated to this fault have been
>>>> automatically escalated to higher priority.
>>>> The IP Network Operation Center (NOC) is engaged and investigating in
>>>> order to isolate the cause. Please be advised that updates for this 
>>>> event
>>>> will be relayed at a minimum of hourly unless otherwise noted. The
>>>> information conveyed hereafter is associated to live 
>>>> troubleshooting effort
>>>> and as the discovery process evolves through to service resolution, 
>>>> ticket
>>>> closure, or post incident review, details may evolve.
>>>>
>>>> 2021-05-03 10:27:08 GMT - Final Notification
>>>>
>>>> Start time: May 03, 2021 05:10 GMT
>>>> Stop Time: May 03, 2021 05:43 GMT
>>>> Second Start Time: May 03, 2021 06:07 GMT
>>>> Second Stop Time: May 03, 2021 07:23 GMT
>>>> Root Cause: An equipment issue in Sao Paulo, Brazil impacted client
>>>> services.
>>>> Fix Action: The self-restarted, thus restoring services to a stable
>>>> condition.
>>>>
>>>> This service impact has concluded. Due to the nature of this 
>>>> outage, it
>>>> may be necessary to reset your services locally at your equipment.  If
>>>> after that action has been performed a service issue prevails, please
>>>> contact the Lumen Repair Center for troubleshooting assistance.
>>>>
>>>> Customers with trouble tickets associated to this network event do not
>>>> need to open a separate Reason for Outage (RFO) ticket to request 
>>>> formal
>>>> documentation. Please allow 3-5 business days for post incident 
>>>> analysisto
>>>> conclude. Once final analysis has been completed, all customers 
>>>> associated
>>>> to this network event will receive a formal RFO document via email 
>>>> from the
>>>> Lumen RFO Team. Additionally, a satisfaction survey link may be 
>>>> available.
>>>> We strive to provide thorough communications containing the available
>>>> information during a service disruption. Please let us know if the 
>>>> updates
>>>> you received during this event were satisfactory.
>>>>
>>>> 2021-05-03 10:22:43 GMT - Additional support was engaged to assist 
>>>> with
>>>> investigations inconjunction with the equipment vendor. The equipment
>>>> vendor has advised that services have remained stable since the device
>>>> self-restarted for the second time. Field Operations verified the 
>>>> equipment
>>>> and the vendor has advised that an on-going investigation into the 
>>>> reason
>>>> for outage will continue; however, as services have remained stable, a
>>>> final notification for the Event will be provided momentarily and the
>>>> reason for outage will be provided in the Formal RFO document 
>>>> following the
>>>> outage.
>>>>
>>>> 2021-05-03 09:28:18 GMT - Field Operations has been engaged to 
>>>> dispatch to
>>>> a site in Cotia, Brazil to perform additional troubleshooting and to
>>>> attempt a cold rest on the routing engine. No ETA has been provided 
>>>> at this
>>>> time.
>>>>
>>>> 2021-05-03 08:43:22 GMT - The equipment vendor continues 
>>>> investigations
>>>> into a suspected software issue at this time. The IP NOC continues to
>>>> monitor for service stability as investigations progress.
>>>>
>>>> 2021-05-03 07:53:20 GMT - The equipment vendor continues isolation 
>>>> efforts
>>>> at this time. The IP NOC has advised that services have self-restored
>>>> again, but remain at risk of impact due to the equipment issue. The 
>>>> IP NOC
>>>> will continue monitoring services for stability as isolation efforts
>>>> continue.
>>>>
>>>> 2021-05-03 07:10:28 GMT - Investigations have determined that an 
>>>> equipment
>>>> issue is the root cause of both outages. The equipment 
>>>> self-rebooted and
>>>> restored; however, shortly thereafter the equipment failed again. The
>>>> equipment vendor has been engaged and is investigating the cause of 
>>>> the
>>>> equipment failure at this time.
>>>>
>>>> 2021-05-03 06:16:38 GMT - On May 03, 2021 at 06:07 GMT, Lumen 
>>>> identifieda
>>>> service impact in Sao Paulo, Brazil. As this network fault is 
>>>> impacting
>>>> multiple clients, the event has increased visibility with Lumen 
>>>> leadership.
>>>> As such, client trouble tickets associated to this fault have been
>>>> automatically escalated to higher priority.
>>>>
>>>> The IP Network Operation Center (NOC) is engaged and investigating in
>>>> order to isolate the cause. Please be advised that updates for this 
>>>> event
>>>> will be relayed at a minimum of hourly unless otherwise noted. The
>>>> information conveyed hereafter is associated to live 
>>>> troubleshooting effort
>>>> and as the discovery process evolves through to service resolution, 
>>>> ticket
>>>> closure, or post incident review, details may evolve.
>>>>
>>>> 2021-05-03 05:54:41 GMT - Final Notification
>>>>
>>>> Start time: May 03, 2021 05:10 GMT
>>>> Stop Time: May 03, 2021 05:43 GMT
>>>>
>>>> Root Cause: An equipment issue in Sao Paulo, Brazil impacted client
>>>> services.
>>>> Fix Action: The self-restarted, thus restoring services to a stable
>>>> condition.
>>>>
>>>> This service impact has concluded. Due to the nature of this 
>>>> outage, it
>>>> may be necessary to reset your services locally at your equipment.  If
>>>> after that action has been performed a service issue prevails, please
>>>> contact the Lumen Repair Center for troubleshooting assistance.
>>>>
>>>> Customers with trouble tickets associated to this network event do not
>>>> need to open a separate Reason for Outage (RFO) ticket to request 
>>>> formal
>>>> documentation. Please allow 3-5 business days for post incident 
>>>> analysisto
>>>> conclude. Once final analysis has been completed, all customers 
>>>> associated
>>>> to this network event will receive a formal RFO document via email 
>>>> from the
>>>> Lumen RFO Team. Additionally, a satisfaction survey link may be 
>>>> available.
>>>> We strive to provide thorough communications containing the available
>>>> information during a service disruption. Please let us know if the 
>>>> updates
>>>> you received during this event were satisfactory.
>>>>
>>>> 2021-05-03 05:27:56 GMT - On May 03, 2021 at 05:10 GMT, Lumen 
>>>> identifieda
>>>> service impact in Sao Paulo, Brazil. As this network fault is 
>>>> impacting
>>>> multiple clients, the event has increased visibility with Lumen 
>>>> leadership.
>>>> As such, client trouble tickets associated to this fault have been
>>>> automatically escalated to higher priority.
>>>>
>>>> The IP Network Operation Center (NOC) is engaged and investigating in
>>>> order to isolate the cause. Please be advised that updates for this 
>>>> event
>>>> will be relayed at a minimum of hourly unless otherwise noted. The
>>>> information conveyed hereafter is associated to live 
>>>> troubleshooting effort
>>>> and as the discovery process evolves through to service resolution, 
>>>> ticket
>>>> closure, or post incident review, details may evolve.
>>>>
>>>>
>>>>
>>>>
>>>> att,
>>>>
>>>> On 03/05/2021 16:51, Alexandre Chaves Fonceca por (caiu) wrote:
>>>>> sim, mesma coisa aqui..
>>>>>
>>>>> desde as 2 da madrugada.. chamado aberto e aguardando... pelas
>>>>> informações estao com problemas em equipamentos mesmo.. e nao sabem
>>>>> "ainda" o que ocorre e/ou pq..
>>>>>
>>>>> att,
>>>>> Specialist Linux Solutions
>>>>>
>>>>> facebook <https://www.facebook.com/specialistlinux/>instagram
>>>>> <https://www.instagram.com/specialistlinuxsolutions/>linkedin
>>>>> <https://www.linkedin.com/company/specialist-linux-solutions/>whatsapp 
>>>>>
>>>>> <https://api.whatsapp.com/send?phone=5511976158584>
>>>>>
>>>>> Alexandre Chaves Fonceca
>>>>> Diretor,
>>>>> Specialist Linux Solutions
>>>>> E-mail:     alexandre em specialist.srv.br
>>>>> <mailto:alexandre em specialist.srv.br>
>>>>> Site:     www.specialist.srv.br <http://www.specialist.srv.br>
>>>>> Telefone:     (11) 3317-5042
>>>>>
>>>>> On 03/05/2021 16:16, Edinilson J. Santos wrote:
>>>>>> Está abrindo o chamado pelo Portal na URL:
>>>>>> https://latam.my.level3.com/portal
>>>>>> ?
>>>>>>
>>>>>> Pergunto pois, ultimamente, é praticamente a única coisa que 
>>>>>> funciona
>>>>>> direito nessa empresa.
>>>>>> Pelo menos envia emails regularmente informando a situação do 
>>>>>> ticket.
>>>>>>
>>>>>> Edinilson
>>>>>>
>>>>>>
>>>>>> Em 03/05/2021 15:05, Leonardo Porto por (caiu) escreveu:
>>>>>>> Sofrendo aqui com mesmo problema, mas sem nenhuma informação 
>>>>>>> adicional.
>>>>>>>
>>>>>>> Leonardo
>>>>>>>
>>>>>>> Em 03/05/2021 15:02, Luiz Fernando Mizael Meier escreveu:
>>>>>>>> Tarde!
>>>>>>>>
>>>>>>>> Com certeza alguém mais deve estar sofrendo com o tombo da
>>>>>>>> Lumen/CenturyLink/Level3/GlobalCrossing/Impsat.
>>>>>>>>
>>>>>>>> Circuito inoperante desde 8:50 e me informaram a pouco que a única
>>>>>>>> coisa
>>>>>>>> que sabem é que foi em Curitiba.
>>>>>>>>
>>>>>>>> Alguém tem mais alguma informação?
>>>>>>>>
>>>>>>>> Luiz
>>>>>>>> _______________________________________________
>>>>>>>> caiu mailing list
>>>>>>>> caiu em eng.registro.br
>>>>>>>> https://eng.registro.br/mailman/listinfo/caiu
>>>>>>>>
>>>>>>>>
>>>>>>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>>>>>>>>
>>>>>>>> https://eng.registro.br/mailman/options/caiu
>>>>>>> _______________________________________________
>>>>>>> caiu mailing list
>>>>>>> caiu em eng.registro.br
>>>>>>> https://eng.registro.br/mailman/listinfo/caiu
>>>>>>>
>>>>>>>
>>>>>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>>>>>>>
>>>>>>> https://eng.registro.br/mailman/options/caiu
>>>>>>>
>>>>>> _______________________________________________
>>>>>> caiu mailing list
>>>>>> caiu em eng.registro.br
>>>>>> https://eng.registro.br/mailman/listinfo/caiu
>>>>>>
>>>>>>
>>>>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>>>>>>
>>>>>> https://eng.registro.br/mailman/options/caiu
>>>>> _______________________________________________
>>>>> caiu mailing list
>>>>> caiu em eng.registro.br
>>>>> https://eng.registro.br/mailman/listinfo/caiu
>>>>>
>>>>>
>>>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>>>>>
>>>>> https://eng.registro.br/mailman/options/caiu
>>>> _______________________________________________
>>>> caiu mailing list
>>>> caiu em eng.registro.br
>>>> https://eng.registro.br/mailman/listinfo/caiu
>>>>
>>>>
>>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>>>>
>>>> https://eng.registro.br/mailman/options/caiu
>>>>
>>> _______________________________________________
>>> caiu mailing list
>>> caiu em eng.registro.br
>>> https://eng.registro.br/mailman/listinfo/caiu
>>>
>>>
>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>>>
>>> https://eng.registro.br/mailman/options/caiu
>> _______________________________________________
>> caiu mailing list
>> caiu em eng.registro.br
>> https://eng.registro.br/mailman/listinfo/caiu
>>
>>
>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>>
>> https://eng.registro.br/mailman/options/caiu
>>
>
> _______________________________________________
> caiu mailing list
> caiu em eng.registro.br
> https://eng.registro.br/mailman/listinfo/caiu
>
>
> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>
> https://eng.registro.br/mailman/options/caiu

Mais detalhes sobre a lista de discussão caiu