[caiu] Indisponibilidade Lúmen

Seg Maio 3 15:59:53 -03 2021

Problema em um roteador de borda deles em Cotia/SP, desde as 02:00hs de hoje, 03/05/2021.

Edinilson

The IP NOC reports that the Sao Paulo, Brazil equipment has been restored to a full redundancy condition. Traffic was migrated off the redundant uplink and remains stable. The IP NOC will continue to monitor the services. IP Engineering and the equipment vendor are still collaboratively investigating the cause for the service impact.

Próxima atualização em 2021-05-03 19:50 GMT

Histórico de Observações:

2021-05-03 18:49:21 GMT - The IP NOC reports that the Sao Paulo, Brazil equipment has been restored to a full redundancy condition. Traffic was migrated off the redundant uplink and remains stable. The IP NOC will continue to monitor the services. IP Engineering and the equipment vendor are still collaboratively investigating the cause for the service impact.

2021-05-03 17:53:10 GMT - The IP NOC reports that the impacted equipment in Sao Paulo, Brazil is still stable since the redundant routing path was taken out of service. Services continue to remain stable as well. Cooperative investigations by multiple support groups along with the equipment vendor remain underway.

2021-05-03 16:52:08 GMT - The IP NOC remotely migrated impacted services to a redundant uplink, however some services have continued to bounce in and out of service. The IP NOC, Engineering teams, Field Operations and the equipment vendor continue investigating the the cause for the service impact.

2021-05-03 15:55:43 GMT - Collaborative troubleshooting with the equipment vendor, Field Operations and the IP NOC remain ongoing.

2021-05-03 15:07:56 GMT - Field Operations is onsite at the Sao Paulo fault site. Collaborative troubleshooting with the equipment vendor remains ongoing.

2021-05-03 14:11:40 GMT - The IP NOC reported a device has restarted multiple times. Collaborative troubleshooting with the equipment vendor remains ongoing. Field Operations are being engaged as well.

2021-05-03 13:51:28 GMT - The IP NOC continues to work in cooperation with the equipment vendor to troubleshoot the issue. We understand how important these services are to our customers and the issue has been escalated to the highest levels within Lumen Service Assurance Leadership.

2021-05-03 13:48:56 GMT - The IP NOC continues to work in cooperation with the equipment vendor to troubleshoot the issue.

2021-05-03 13:16:06 GMT - Cooperative investigations by multiple support groups remain underway at this time.

2021-05-03 13:13:13 GMT - The IP NOC is working in cooperation with Field Operations and the equipment vendor to troubleshoot the issue.

2021-05-03 12:50:33 GMT - Cooperative investigations by multiple support groups remain underway at this time.

2021-05-03 12:02:37 GMT - On May 03, 2021 at 11:46 GMT, Lumen identified a service impact in Sao Paulo, Brazil. As this network fault is impacting multiple clients, the event has increased visibility with Lumen leadership. As such, client trouble tickets associated to this fault have been automatically escalated to higher priority.
The IP Network Operation Center (NOC) is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve.

2021-05-03 10:27:08 GMT - Final Notification

Start time: May 03, 2021 05:10 GMT
Stop Time: May 03, 2021 05:43 GMT
Second Start Time: May 03, 2021 06:07 GMT
Second Stop Time: May 03, 2021 07:23 GMT
Root Cause: An equipment issue in Sao Paulo, Brazil impacted client services.
Fix Action: The self-restarted, thus restoring services to a stable condition.

This service impact has concluded. Due to the nature of this outage, it may be necessary to reset your services locally at your equipment.  If after that action has been performed a service issue prevails, please contact the Lumen Repair Center for troubleshooting assistance.

Customers with trouble tickets associated to this network event do not need to open a separate Reason for Outage (RFO) ticket to request formal documentation. Please allow 3-5 business days for post incident analysis to conclude. Once final analysis has been completed, all customers associated to this network event will receive a formal RFO document via email from the Lumen RFO Team. Additionally, a satisfaction survey link may be available. We strive to provide thorough communications containing the available information during a service disruption. Please let us know if the updates you received during this event were satisfactory.

2021-05-03 10:22:43 GMT - Additional support was engaged to assist with investigations inconjunction with the equipment vendor. The equipment vendor has advised that services have remained stable since the device self-restarted for the second time. Field Operations verified the equipment and the vendor has advised that an on-going investigation into the reason for outage will continue; however, as services have remained stable, a final notification for the Event will be provided momentarily and the reason for outage will be provided in the Formal RFO document following the outage.

2021-05-03 09:28:18 GMT - Field Operations has been engaged to dispatch to a site in Cotia, Brazil to perform additional troubleshooting and to attempt a cold rest on the routing engine. No ETA has been provided at this time.

2021-05-03 08:43:22 GMT - The equipment vendor continues investigations into a suspected software issue at this time. The IP NOC continues to monitor for service stability as investigations progress.

2021-05-03 07:53:20 GMT - The equipment vendor continues isolation efforts at this time. The IP NOC has advised that services have self-restored again, but remain at risk of impact due to the equipment issue. The IP NOC will continue monitoring services for stability as isolation efforts continue.

2021-05-03 07:10:28 GMT - Investigations have determined that an equipment issue is the root cause of both outages. The equipment self-rebooted and restored; however, shortly thereafter the equipment failed again. The equipment vendor has been engaged and is investigating the cause of the equipment failure at this time.

2021-05-03 06:16:38 GMT - On May 03, 2021 at 06:07 GMT, Lumen identified a service impact in Sao Paulo, Brazil. As this network fault is impacting multiple clients, the event has increased visibility with Lumen leadership. As such, client trouble tickets associated to this fault have been automatically escalated to higher priority.

The IP Network Operation Center (NOC) is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve.

2021-05-03 05:54:41 GMT - Final Notification

Start time: May 03, 2021 05:10 GMT
Stop Time: May 03, 2021 05:43 GMT

Root Cause: An equipment issue in Sao Paulo, Brazil impacted client services.
Fix Action: The self-restarted, thus restoring services to a stable condition.

This service impact has concluded. Due to the nature of this outage, it may be necessary to reset your services locally at your equipment.  If after that action has been performed a service issue prevails, please contact the Lumen Repair Center for troubleshooting assistance.

Customers with trouble tickets associated to this network event do not need to open a separate Reason for Outage (RFO) ticket to request formal documentation. Please allow 3-5 business days for post incident analysis to conclude. Once final analysis has been completed, all customers associated to this network event will receive a formal RFO document via email from the Lumen RFO Team. Additionally, a satisfaction survey link may be available. We strive to provide thorough communications containing the available information during a service disruption. Please let us know if the updates you received during this event were satisfactory.

2021-05-03 05:27:56 GMT - On May 03, 2021 at 05:10 GMT, Lumen identified a service impact in Sao Paulo, Brazil. As this network fault is impacting multiple clients, the event has increased visibility with Lumen leadership. As such, client trouble tickets associated to this fault have been automatically escalated to higher priority.

The IP Network Operation Center (NOC) is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve.

Em 03/05/2021 15:02, Luiz Fernando Mizael Meier escreveu:
> Tarde!
>
> Com certeza alguém mais deve estar sofrendo com o tombo da
> Lumen/CenturyLink/Level3/GlobalCrossing/Impsat.
>
> Circuito inoperante desde 8:50 e me informaram a pouco que a única coisa
> que sabem é que foi em Curitiba.
>
> Alguém tem mais alguma informação?
>
> Luiz
> _______________________________________________
> caiu mailing list
> caiu em eng.registro.br
> https://eng.registro.br/mailman/listinfo/caiu
>
>
> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>
> https://eng.registro.br/mailman/options/caiu
>