[caiu] Indisponibilidade Lúmen

Ter Maio 4 08:52:32 -03 2021

Bom dia. O link normalizou para vocês?

Att,

Everton Araujo

On 5/3/21 5:24 PM, Eduardo Duarte wrote:
> Aqui igual
>
> Jundiaí SP
>
> O link hoje parece pisca pisca de árvore de Natal
>
> Em seg., 3 de mai. de 2021 17:00, Alexandre Chaves Fonceca por (caiu) <
> caiu em eng.registro.br> escreveu:
>
>> essas sao as informações oficiais deles... em ordem cronologica (de
>> baixo para cima), e em horario GMT 0
>>
>> 2021-05-03 19:53:47 GMT - The IP NOC observed another reboot,
>> investigation and troubleshooting efforts remain ongoing.
>>
>> 2021-05-03 19:37:11 GMT - The IP NOC is reporting all affected alarms
>> remain clear and will continue to monitor and investigate the fault for the
>> next 4 hours. As such, the next notification will be provided at 23:45 GMT
>> or as additional information is obtained.
>>
>> 2021-05-03 18:49:21 GMT - The IP NOC reports that the Sao Paulo, Brazil
>> equipment has been restored to a full redundancy condition. Traffic was
>> migrated off the redundant uplink and remains stable. The IP NOC will
>> continue to monitor the services. IP Engineering and the equipment vendor
>> are still collaboratively investigating the cause for the service impact.
>>
>> 2021-05-03 17:53:10 GMT - The IP NOC reports that the impacted equipment
>> in Sao Paulo, Brazil is still stable since the redundant routing path was
>> taken out of service. Services continue to remain stable as well.
>> Cooperative investigations by multiple support groups along with the
>> equipment vendor remain underway.
>>
>> 2021-05-03 16:52:08 GMT - The IP NOC remotely migrated impacted services
>> to a redundant uplink, however some services have continued to bounce in
>> and out of service. The IP NOC, Engineering teams, Field Operations and the
>> equipment vendor continue investigating the the cause for the service
>> impact.
>>
>> 2021-05-03 15:55:43 GMT - Collaborative troubleshooting with the equipment
>> vendor, Field Operations and the IP NOC remain ongoing.
>>
>> 2021-05-03 15:07:56 GMT - Field Operations is onsite at the Sao Paulo
>> fault site. Collaborative troubleshooting with the equipment vendor remains
>> ongoing.
>>
>> 2021-05-03 14:11:40 GMT - The IP NOC reported a device has restarted
>> multiple times. Collaborative troubleshooting with the equipment vendor
>> remains ongoing. Field Operations are being engaged as well.
>>
>> 2021-05-03 13:51:28 GMT - The IP NOC continues to work in cooperation with
>> the equipment vendor to troubleshoot the issue. We understand how important
>> these services are to our customers and the issue has been escalated to the
>> highest levels within Lumen Service Assurance Leadership.
>>
>> 2021-05-03 13:48:56 GMT - The IP NOC continues to work in cooperation with
>> the equipment vendor to troubleshoot the issue.
>>
>> 2021-05-03 13:16:06 GMT - Cooperative investigations by multiple support
>> groups remain underway at this time.
>>
>> 2021-05-03 13:13:13 GMT - The IP NOC is working in cooperation with Field
>> Operations and the equipment vendor to troubleshoot the issue.
>>
>> 2021-05-03 12:50:33 GMT - Cooperative investigations by multiple support
>> groups remain underway at this time.
>>
>> 2021-05-03 12:02:37 GMT - On May 03, 2021 at 11:46 GMT, Lumen identified a
>> service impact in Sao Paulo, Brazil. As this network fault is impacting
>> multiple clients, the event has increased visibility with Lumen leadership.
>> As such, client trouble tickets associated to this fault have been
>> automatically escalated to higher priority.
>> The IP Network Operation Center (NOC) is engaged and investigating in
>> order to isolate the cause. Please be advised that updates for this event
>> will be relayed at a minimum of hourly unless otherwise noted. The
>> information conveyed hereafter is associated to live troubleshooting effort
>> and as the discovery process evolves through to service resolution, ticket
>> closure, or post incident review, details may evolve.
>>
>> 2021-05-03 10:27:08 GMT - Final Notification
>>
>> Start time: May 03, 2021 05:10 GMT
>> Stop Time: May 03, 2021 05:43 GMT
>> Second Start Time: May 03, 2021 06:07 GMT
>> Second Stop Time: May 03, 2021 07:23 GMT
>> Root Cause: An equipment issue in Sao Paulo, Brazil impacted client
>> services.
>> Fix Action: The self-restarted, thus restoring services to a stable
>> condition.
>>
>> This service impact has concluded. Due to the nature of this outage, it
>> may be necessary to reset your services locally at your equipment.  If
>> after that action has been performed a service issue prevails, please
>> contact the Lumen Repair Center for troubleshooting assistance.
>>
>> Customers with trouble tickets associated to this network event do not
>> need to open a separate Reason for Outage (RFO) ticket to request formal
>> documentation. Please allow 3-5 business days for post incident analysis to
>> conclude. Once final analysis has been completed, all customers associated
>> to this network event will receive a formal RFO document via email from the
>> Lumen RFO Team. Additionally, a satisfaction survey link may be available.
>> We strive to provide thorough communications containing the available
>> information during a service disruption. Please let us know if the updates
>> you received during this event were satisfactory.
>>
>> 2021-05-03 10:22:43 GMT - Additional support was engaged to assist with
>> investigations inconjunction with the equipment vendor. The equipment
>> vendor has advised that services have remained stable since the device
>> self-restarted for the second time. Field Operations verified the equipment
>> and the vendor has advised that an on-going investigation into the reason
>> for outage will continue; however, as services have remained stable, a
>> final notification for the Event will be provided momentarily and the
>> reason for outage will be provided in the Formal RFO document following the
>> outage.
>>
>> 2021-05-03 09:28:18 GMT - Field Operations has been engaged to dispatch to
>> a site in Cotia, Brazil to perform additional troubleshooting and to
>> attempt a cold rest on the routing engine. No ETA has been provided at this
>> time.
>>
>> 2021-05-03 08:43:22 GMT - The equipment vendor continues investigations
>> into a suspected software issue at this time. The IP NOC continues to
>> monitor for service stability as investigations progress.
>>
>> 2021-05-03 07:53:20 GMT - The equipment vendor continues isolation efforts
>> at this time. The IP NOC has advised that services have self-restored
>> again, but remain at risk of impact due to the equipment issue. The IP NOC
>> will continue monitoring services for stability as isolation efforts
>> continue.
>>
>> 2021-05-03 07:10:28 GMT - Investigations have determined that an equipment
>> issue is the root cause of both outages. The equipment self-rebooted and
>> restored; however, shortly thereafter the equipment failed again. The
>> equipment vendor has been engaged and is investigating the cause of the
>> equipment failure at this time.
>>
>> 2021-05-03 06:16:38 GMT - On May 03, 2021 at 06:07 GMT, Lumen identified a
>> service impact in Sao Paulo, Brazil. As this network fault is impacting
>> multiple clients, the event has increased visibility with Lumen leadership.
>> As such, client trouble tickets associated to this fault have been
>> automatically escalated to higher priority.
>>
>> The IP Network Operation Center (NOC) is engaged and investigating in
>> order to isolate the cause. Please be advised that updates for this event
>> will be relayed at a minimum of hourly unless otherwise noted. The
>> information conveyed hereafter is associated to live troubleshooting effort
>> and as the discovery process evolves through to service resolution, ticket
>> closure, or post incident review, details may evolve.
>>
>> 2021-05-03 05:54:41 GMT - Final Notification
>>
>> Start time: May 03, 2021 05:10 GMT
>> Stop Time: May 03, 2021 05:43 GMT
>>
>> Root Cause: An equipment issue in Sao Paulo, Brazil impacted client
>> services.
>> Fix Action: The self-restarted, thus restoring services to a stable
>> condition.
>>
>> This service impact has concluded. Due to the nature of this outage, it
>> may be necessary to reset your services locally at your equipment.  If
>> after that action has been performed a service issue prevails, please
>> contact the Lumen Repair Center for troubleshooting assistance.
>>
>> Customers with trouble tickets associated to this network event do not
>> need to open a separate Reason for Outage (RFO) ticket to request formal
>> documentation. Please allow 3-5 business days for post incident analysis to
>> conclude. Once final analysis has been completed, all customers associated
>> to this network event will receive a formal RFO document via email from the
>> Lumen RFO Team. Additionally, a satisfaction survey link may be available.
>> We strive to provide thorough communications containing the available
>> information during a service disruption. Please let us know if the updates
>> you received during this event were satisfactory.
>>
>> 2021-05-03 05:27:56 GMT - On May 03, 2021 at 05:10 GMT, Lumen identified a
>> service impact in Sao Paulo, Brazil. As this network fault is impacting
>> multiple clients, the event has increased visibility with Lumen leadership.
>> As such, client trouble tickets associated to this fault have been
>> automatically escalated to higher priority.
>>
>> The IP Network Operation Center (NOC) is engaged and investigating in
>> order to isolate the cause. Please be advised that updates for this event
>> will be relayed at a minimum of hourly unless otherwise noted. The
>> information conveyed hereafter is associated to live troubleshooting effort
>> and as the discovery process evolves through to service resolution, ticket
>> closure, or post incident review, details may evolve.
>>
>>
>>
>>
>> att,
>>
>> On 03/05/2021 16:51, Alexandre Chaves Fonceca por (caiu) wrote:
>>> sim, mesma coisa aqui..
>>>
>>> desde as 2 da madrugada.. chamado aberto e aguardando...  pelas
>>> informações estao com problemas em equipamentos mesmo.. e nao sabem
>>> "ainda" o que ocorre e/ou pq..
>>>
>>> att,
>>> Specialist Linux Solutions
>>>
>>> facebook <https://www.facebook.com/specialistlinux/>instagram
>>> <https://www.instagram.com/specialistlinuxsolutions/>linkedin
>>> <https://www.linkedin.com/company/specialist-linux-solutions/>whatsapp
>>> <https://api.whatsapp.com/send?phone=5511976158584>
>>>
>>> Alexandre Chaves Fonceca
>>> Diretor,
>>> Specialist Linux Solutions
>>> E-mail:     alexandre em specialist.srv.br
>>> <mailto:alexandre em specialist.srv.br>
>>> Site:     www.specialist.srv.br <http://www.specialist.srv.br>
>>> Telefone:     (11) 3317-5042
>>>
>>> On 03/05/2021 16:16, Edinilson J. Santos wrote:
>>>> Está abrindo o chamado pelo Portal na URL:
>>>> https://latam.my.level3.com/portal
>>>> ?
>>>>
>>>> Pergunto pois, ultimamente, é praticamente a única coisa que funciona
>>>> direito nessa empresa.
>>>> Pelo menos envia emails regularmente informando a situação do ticket.
>>>>
>>>> Edinilson
>>>>
>>>>
>>>> Em 03/05/2021 15:05, Leonardo Porto por (caiu) escreveu:
>>>>> Sofrendo aqui com mesmo problema, mas sem nenhuma informação adicional.
>>>>>
>>>>> Leonardo
>>>>>
>>>>> Em 03/05/2021 15:02, Luiz Fernando Mizael Meier escreveu:
>>>>>> Tarde!
>>>>>>
>>>>>> Com certeza alguém mais deve estar sofrendo com o tombo da
>>>>>> Lumen/CenturyLink/Level3/GlobalCrossing/Impsat.
>>>>>>
>>>>>> Circuito inoperante desde 8:50 e me informaram a pouco que a única
>>>>>> coisa
>>>>>> que sabem é que foi em Curitiba.
>>>>>>
>>>>>> Alguém tem mais alguma informação?
>>>>>>
>>>>>> Luiz
>>>>>> _______________________________________________
>>>>>> caiu mailing list
>>>>>> caiu em eng.registro.br
>>>>>> https://eng.registro.br/mailman/listinfo/caiu
>>>>>>
>>>>>>
>>>>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>>>>>>
>>>>>> https://eng.registro.br/mailman/options/caiu
>>>>> _______________________________________________
>>>>> caiu mailing list
>>>>> caiu em eng.registro.br
>>>>> https://eng.registro.br/mailman/listinfo/caiu
>>>>>
>>>>>
>>>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>>>>>
>>>>> https://eng.registro.br/mailman/options/caiu
>>>>>
>>>> _______________________________________________
>>>> caiu mailing list
>>>> caiu em eng.registro.br
>>>> https://eng.registro.br/mailman/listinfo/caiu
>>>>
>>>>
>>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>>>>
>>>> https://eng.registro.br/mailman/options/caiu
>>> _______________________________________________
>>> caiu mailing list
>>> caiu em eng.registro.br
>>> https://eng.registro.br/mailman/listinfo/caiu
>>>
>>>
>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>>>
>>> https://eng.registro.br/mailman/options/caiu
>> _______________________________________________
>> caiu mailing list
>> caiu em eng.registro.br
>> https://eng.registro.br/mailman/listinfo/caiu
>>
>>
>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>>
>> https://eng.registro.br/mailman/options/caiu
>>
> _______________________________________________
> caiu mailing list
> caiu em eng.registro.br
> https://eng.registro.br/mailman/listinfo/caiu
>
>
> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>
> https://eng.registro.br/mailman/options/caiu