[caiu] Indisponibilidade Lúmen

Edinilson J. Santos edinilson em atinet.com.br
Ter Maio 4 15:23:34 -03 2021


Aqui sim, ontem, 03/05/2021 por volta das 19:00hs (antes disso, ficava indo e voltando).

Edinilson


Em 04/05/2021 08:52, noc em consuldata.com.br escreveu:
> Bom dia. O link normalizou para vocês?
>
> Att,
>
>
> Everton Araujo
>
> On 5/3/21 5:24 PM, Eduardo Duarte wrote:
>> Aqui igual
>>
>> Jundiaí SP
>>
>> O link hoje parece pisca pisca de árvore de Natal
>>
>> Em seg., 3 de mai. de 2021 17:00, Alexandre Chaves Fonceca por (caiu) <
>> caiu em eng.registro.br> escreveu:
>>
>>> essas sao as informações oficiais deles... em ordem cronologica (de
>>> baixo para cima), e em horario GMT 0
>>>
>>> 2021-05-03 19:53:47 GMT - The IP NOC observed another reboot,
>>> investigation and troubleshooting efforts remain ongoing.
>>>
>>> 2021-05-03 19:37:11 GMT - The IP NOC is reporting all affected alarms
>>> remain clear and will continue to monitor and investigate the fault for the
>>> next 4 hours. As such, the next notification will be provided at 23:45 GMT
>>> or as additional information is obtained.
>>>
>>> 2021-05-03 18:49:21 GMT - The IP NOC reports that the Sao Paulo, Brazil
>>> equipment has been restored to a full redundancy condition. Traffic was
>>> migrated off the redundant uplink and remains stable. The IP NOC will
>>> continue to monitor the services. IP Engineering and the equipment vendor
>>> are still collaboratively investigating the cause for the service impact.
>>>
>>> 2021-05-03 17:53:10 GMT - The IP NOC reports that the impacted equipment
>>> in Sao Paulo, Brazil is still stable since the redundant routing path was
>>> taken out of service. Services continue to remain stable as well.
>>> Cooperative investigations by multiple support groups along with the
>>> equipment vendor remain underway.
>>>
>>> 2021-05-03 16:52:08 GMT - The IP NOC remotely migrated impacted services
>>> to a redundant uplink, however some services have continued to bounce in
>>> and out of service. The IP NOC, Engineering teams, Field Operations and the
>>> equipment vendor continue investigating the the cause for the service
>>> impact.
>>>
>>> 2021-05-03 15:55:43 GMT - Collaborative troubleshooting with the equipment
>>> vendor, Field Operations and the IP NOC remain ongoing.
>>>
>>> 2021-05-03 15:07:56 GMT - Field Operations is onsite at the Sao Paulo
>>> fault site. Collaborative troubleshooting with the equipment vendor remains
>>> ongoing.
>>>
>>> 2021-05-03 14:11:40 GMT - The IP NOC reported a device has restarted
>>> multiple times. Collaborative troubleshooting with the equipment vendor
>>> remains ongoing. Field Operations are being engaged as well.
>>>
>>> 2021-05-03 13:51:28 GMT - The IP NOC continues to work in cooperation with
>>> the equipment vendor to troubleshoot the issue. We understand how important
>>> these services are to our customers and the issue has been escalated to the
>>> highest levels within Lumen Service Assurance Leadership.
>>>
>>> 2021-05-03 13:48:56 GMT - The IP NOC continues to work in cooperation with
>>> the equipment vendor to troubleshoot the issue.
>>>
>>> 2021-05-03 13:16:06 GMT - Cooperative investigations by multiple support
>>> groups remain underway at this time.
>>>
>>> 2021-05-03 13:13:13 GMT - The IP NOC is working in cooperation with Field
>>> Operations and the equipment vendor to troubleshoot the issue.
>>>
>>> 2021-05-03 12:50:33 GMT - Cooperative investigations by multiple support
>>> groups remain underway at this time.
>>>
>>> 2021-05-03 12:02:37 GMT - On May 03, 2021 at 11:46 GMT, Lumen identifieda
>>> service impact in Sao Paulo, Brazil. As this network fault is impacting
>>> multiple clients, the event has increased visibility with Lumen leadership.
>>> As such, client trouble tickets associated to this fault have been
>>> automatically escalated to higher priority.
>>> The IP Network Operation Center (NOC) is engaged and investigating in
>>> order to isolate the cause. Please be advised that updates for this event
>>> will be relayed at a minimum of hourly unless otherwise noted. The
>>> information conveyed hereafter is associated to live troubleshooting effort
>>> and as the discovery process evolves through to service resolution, ticket
>>> closure, or post incident review, details may evolve.
>>>
>>> 2021-05-03 10:27:08 GMT - Final Notification
>>>
>>> Start time: May 03, 2021 05:10 GMT
>>> Stop Time: May 03, 2021 05:43 GMT
>>> Second Start Time: May 03, 2021 06:07 GMT
>>> Second Stop Time: May 03, 2021 07:23 GMT
>>> Root Cause: An equipment issue in Sao Paulo, Brazil impacted client
>>> services.
>>> Fix Action: The self-restarted, thus restoring services to a stable
>>> condition.
>>>
>>> This service impact has concluded. Due to the nature of this outage, it
>>> may be necessary to reset your services locally at your equipment.  If
>>> after that action has been performed a service issue prevails, please
>>> contact the Lumen Repair Center for troubleshooting assistance.
>>>
>>> Customers with trouble tickets associated to this network event do not
>>> need to open a separate Reason for Outage (RFO) ticket to request formal
>>> documentation. Please allow 3-5 business days for post incident analysisto
>>> conclude. Once final analysis has been completed, all customers associated
>>> to this network event will receive a formal RFO document via email from the
>>> Lumen RFO Team. Additionally, a satisfaction survey link may be available.
>>> We strive to provide thorough communications containing the available
>>> information during a service disruption. Please let us know if the updates
>>> you received during this event were satisfactory.
>>>
>>> 2021-05-03 10:22:43 GMT - Additional support was engaged to assist with
>>> investigations inconjunction with the equipment vendor. The equipment
>>> vendor has advised that services have remained stable since the device
>>> self-restarted for the second time. Field Operations verified the equipment
>>> and the vendor has advised that an on-going investigation into the reason
>>> for outage will continue; however, as services have remained stable, a
>>> final notification for the Event will be provided momentarily and the
>>> reason for outage will be provided in the Formal RFO document following the
>>> outage.
>>>
>>> 2021-05-03 09:28:18 GMT - Field Operations has been engaged to dispatch to
>>> a site in Cotia, Brazil to perform additional troubleshooting and to
>>> attempt a cold rest on the routing engine. No ETA has been provided at this
>>> time.
>>>
>>> 2021-05-03 08:43:22 GMT - The equipment vendor continues investigations
>>> into a suspected software issue at this time. The IP NOC continues to
>>> monitor for service stability as investigations progress.
>>>
>>> 2021-05-03 07:53:20 GMT - The equipment vendor continues isolation efforts
>>> at this time. The IP NOC has advised that services have self-restored
>>> again, but remain at risk of impact due to the equipment issue. The IP NOC
>>> will continue monitoring services for stability as isolation efforts
>>> continue.
>>>
>>> 2021-05-03 07:10:28 GMT - Investigations have determined that an equipment
>>> issue is the root cause of both outages. The equipment self-rebooted and
>>> restored; however, shortly thereafter the equipment failed again. The
>>> equipment vendor has been engaged and is investigating the cause of the
>>> equipment failure at this time.
>>>
>>> 2021-05-03 06:16:38 GMT - On May 03, 2021 at 06:07 GMT, Lumen identifieda
>>> service impact in Sao Paulo, Brazil. As this network fault is impacting
>>> multiple clients, the event has increased visibility with Lumen leadership.
>>> As such, client trouble tickets associated to this fault have been
>>> automatically escalated to higher priority.
>>>
>>> The IP Network Operation Center (NOC) is engaged and investigating in
>>> order to isolate the cause. Please be advised that updates for this event
>>> will be relayed at a minimum of hourly unless otherwise noted. The
>>> information conveyed hereafter is associated to live troubleshooting effort
>>> and as the discovery process evolves through to service resolution, ticket
>>> closure, or post incident review, details may evolve.
>>>
>>> 2021-05-03 05:54:41 GMT - Final Notification
>>>
>>> Start time: May 03, 2021 05:10 GMT
>>> Stop Time: May 03, 2021 05:43 GMT
>>>
>>> Root Cause: An equipment issue in Sao Paulo, Brazil impacted client
>>> services.
>>> Fix Action: The self-restarted, thus restoring services to a stable
>>> condition.
>>>
>>> This service impact has concluded. Due to the nature of this outage, it
>>> may be necessary to reset your services locally at your equipment.  If
>>> after that action has been performed a service issue prevails, please
>>> contact the Lumen Repair Center for troubleshooting assistance.
>>>
>>> Customers with trouble tickets associated to this network event do not
>>> need to open a separate Reason for Outage (RFO) ticket to request formal
>>> documentation. Please allow 3-5 business days for post incident analysisto
>>> conclude. Once final analysis has been completed, all customers associated
>>> to this network event will receive a formal RFO document via email from the
>>> Lumen RFO Team. Additionally, a satisfaction survey link may be available.
>>> We strive to provide thorough communications containing the available
>>> information during a service disruption. Please let us know if the updates
>>> you received during this event were satisfactory.
>>>
>>> 2021-05-03 05:27:56 GMT - On May 03, 2021 at 05:10 GMT, Lumen identifieda
>>> service impact in Sao Paulo, Brazil. As this network fault is impacting
>>> multiple clients, the event has increased visibility with Lumen leadership.
>>> As such, client trouble tickets associated to this fault have been
>>> automatically escalated to higher priority.
>>>
>>> The IP Network Operation Center (NOC) is engaged and investigating in
>>> order to isolate the cause. Please be advised that updates for this event
>>> will be relayed at a minimum of hourly unless otherwise noted. The
>>> information conveyed hereafter is associated to live troubleshooting effort
>>> and as the discovery process evolves through to service resolution, ticket
>>> closure, or post incident review, details may evolve.
>>>
>>>
>>>
>>>
>>> att,
>>>
>>> On 03/05/2021 16:51, Alexandre Chaves Fonceca por (caiu) wrote:
>>>> sim, mesma coisa aqui..
>>>>
>>>> desde as 2 da madrugada.. chamado aberto e aguardando... pelas
>>>> informações estao com problemas em equipamentos mesmo.. e nao sabem
>>>> "ainda" o que ocorre e/ou pq..
>>>>
>>>> att,
>>>> Specialist Linux Solutions
>>>>
>>>> facebook <https://www.facebook.com/specialistlinux/>instagram
>>>> <https://www.instagram.com/specialistlinuxsolutions/>linkedin
>>>> <https://www.linkedin.com/company/specialist-linux-solutions/>whatsapp
>>>> <https://api.whatsapp.com/send?phone=5511976158584>
>>>>
>>>> Alexandre Chaves Fonceca
>>>> Diretor,
>>>> Specialist Linux Solutions
>>>> E-mail:     alexandre em specialist.srv.br
>>>> <mailto:alexandre em specialist.srv.br>
>>>> Site:     www.specialist.srv.br <http://www.specialist.srv.br>
>>>> Telefone:     (11) 3317-5042
>>>>
>>>> On 03/05/2021 16:16, Edinilson J. Santos wrote:
>>>>> Está abrindo o chamado pelo Portal na URL:
>>>>> https://latam.my.level3.com/portal
>>>>> ?
>>>>>
>>>>> Pergunto pois, ultimamente, é praticamente a única coisa que funciona
>>>>> direito nessa empresa.
>>>>> Pelo menos envia emails regularmente informando a situação do ticket.
>>>>>
>>>>> Edinilson
>>>>>
>>>>>
>>>>> Em 03/05/2021 15:05, Leonardo Porto por (caiu) escreveu:
>>>>>> Sofrendo aqui com mesmo problema, mas sem nenhuma informação adicional.
>>>>>>
>>>>>> Leonardo
>>>>>>
>>>>>> Em 03/05/2021 15:02, Luiz Fernando Mizael Meier escreveu:
>>>>>>> Tarde!
>>>>>>>
>>>>>>> Com certeza alguém mais deve estar sofrendo com o tombo da
>>>>>>> Lumen/CenturyLink/Level3/GlobalCrossing/Impsat.
>>>>>>>
>>>>>>> Circuito inoperante desde 8:50 e me informaram a pouco que a única
>>>>>>> coisa
>>>>>>> que sabem é que foi em Curitiba.
>>>>>>>
>>>>>>> Alguém tem mais alguma informação?
>>>>>>>
>>>>>>> Luiz
>>>>>>> _______________________________________________
>>>>>>> caiu mailing list
>>>>>>> caiu em eng.registro.br
>>>>>>> https://eng.registro.br/mailman/listinfo/caiu
>>>>>>>
>>>>>>>
>>>>>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>>>>>>>
>>>>>>> https://eng.registro.br/mailman/options/caiu
>>>>>> _______________________________________________
>>>>>> caiu mailing list
>>>>>> caiu em eng.registro.br
>>>>>> https://eng.registro.br/mailman/listinfo/caiu
>>>>>>
>>>>>>
>>>>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>>>>>>
>>>>>> https://eng.registro.br/mailman/options/caiu
>>>>>>
>>>>> _______________________________________________
>>>>> caiu mailing list
>>>>> caiu em eng.registro.br
>>>>> https://eng.registro.br/mailman/listinfo/caiu
>>>>>
>>>>>
>>>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>>>>>
>>>>> https://eng.registro.br/mailman/options/caiu
>>>> _______________________________________________
>>>> caiu mailing list
>>>> caiu em eng.registro.br
>>>> https://eng.registro.br/mailman/listinfo/caiu
>>>>
>>>>
>>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>>>>
>>>> https://eng.registro.br/mailman/options/caiu
>>> _______________________________________________
>>> caiu mailing list
>>> caiu em eng.registro.br
>>> https://eng.registro.br/mailman/listinfo/caiu
>>>
>>>
>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>>>
>>> https://eng.registro.br/mailman/options/caiu
>>>
>> _______________________________________________
>> caiu mailing list
>> caiu em eng.registro.br
>> https://eng.registro.br/mailman/listinfo/caiu
>>
>>
>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>>
>> https://eng.registro.br/mailman/options/caiu
> _______________________________________________
> caiu mailing list
> caiu em eng.registro.br
> https://eng.registro.br/mailman/listinfo/caiu
>
>
> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>
> https://eng.registro.br/mailman/options/caiu
>



Mais detalhes sobre a lista de discussão caiu