[caiu] Indisponibilidade Lúmen
Eduardo Duarte
eduduarteneves em gmail.com
Seg Maio 3 17:24:36 -03 2021
Aqui igual
Jundiaí SP
O link hoje parece pisca pisca de árvore de Natal
Em seg., 3 de mai. de 2021 17:00, Alexandre Chaves Fonceca por (caiu) <
caiu em eng.registro.br> escreveu:
> essas sao as informações oficiais deles... em ordem cronologica (de
> baixo para cima), e em horario GMT 0
>
> 2021-05-03 19:53:47 GMT - The IP NOC observed another reboot,
> investigation and troubleshooting efforts remain ongoing.
>
> 2021-05-03 19:37:11 GMT - The IP NOC is reporting all affected alarms
> remain clear and will continue to monitor and investigate the fault for the
> next 4 hours. As such, the next notification will be provided at 23:45 GMT
> or as additional information is obtained.
>
> 2021-05-03 18:49:21 GMT - The IP NOC reports that the Sao Paulo, Brazil
> equipment has been restored to a full redundancy condition. Traffic was
> migrated off the redundant uplink and remains stable. The IP NOC will
> continue to monitor the services. IP Engineering and the equipment vendor
> are still collaboratively investigating the cause for the service impact.
>
> 2021-05-03 17:53:10 GMT - The IP NOC reports that the impacted equipment
> in Sao Paulo, Brazil is still stable since the redundant routing path was
> taken out of service. Services continue to remain stable as well.
> Cooperative investigations by multiple support groups along with the
> equipment vendor remain underway.
>
> 2021-05-03 16:52:08 GMT - The IP NOC remotely migrated impacted services
> to a redundant uplink, however some services have continued to bounce in
> and out of service. The IP NOC, Engineering teams, Field Operations and the
> equipment vendor continue investigating the the cause for the service
> impact.
>
> 2021-05-03 15:55:43 GMT - Collaborative troubleshooting with the equipment
> vendor, Field Operations and the IP NOC remain ongoing.
>
> 2021-05-03 15:07:56 GMT - Field Operations is onsite at the Sao Paulo
> fault site. Collaborative troubleshooting with the equipment vendor remains
> ongoing.
>
> 2021-05-03 14:11:40 GMT - The IP NOC reported a device has restarted
> multiple times. Collaborative troubleshooting with the equipment vendor
> remains ongoing. Field Operations are being engaged as well.
>
> 2021-05-03 13:51:28 GMT - The IP NOC continues to work in cooperation with
> the equipment vendor to troubleshoot the issue. We understand how important
> these services are to our customers and the issue has been escalated to the
> highest levels within Lumen Service Assurance Leadership.
>
> 2021-05-03 13:48:56 GMT - The IP NOC continues to work in cooperation with
> the equipment vendor to troubleshoot the issue.
>
> 2021-05-03 13:16:06 GMT - Cooperative investigations by multiple support
> groups remain underway at this time.
>
> 2021-05-03 13:13:13 GMT - The IP NOC is working in cooperation with Field
> Operations and the equipment vendor to troubleshoot the issue.
>
> 2021-05-03 12:50:33 GMT - Cooperative investigations by multiple support
> groups remain underway at this time.
>
> 2021-05-03 12:02:37 GMT - On May 03, 2021 at 11:46 GMT, Lumen identified a
> service impact in Sao Paulo, Brazil. As this network fault is impacting
> multiple clients, the event has increased visibility with Lumen leadership.
> As such, client trouble tickets associated to this fault have been
> automatically escalated to higher priority.
> The IP Network Operation Center (NOC) is engaged and investigating in
> order to isolate the cause. Please be advised that updates for this event
> will be relayed at a minimum of hourly unless otherwise noted. The
> information conveyed hereafter is associated to live troubleshooting effort
> and as the discovery process evolves through to service resolution, ticket
> closure, or post incident review, details may evolve.
>
> 2021-05-03 10:27:08 GMT - Final Notification
>
> Start time: May 03, 2021 05:10 GMT
> Stop Time: May 03, 2021 05:43 GMT
> Second Start Time: May 03, 2021 06:07 GMT
> Second Stop Time: May 03, 2021 07:23 GMT
> Root Cause: An equipment issue in Sao Paulo, Brazil impacted client
> services.
> Fix Action: The self-restarted, thus restoring services to a stable
> condition.
>
> This service impact has concluded. Due to the nature of this outage, it
> may be necessary to reset your services locally at your equipment. If
> after that action has been performed a service issue prevails, please
> contact the Lumen Repair Center for troubleshooting assistance.
>
> Customers with trouble tickets associated to this network event do not
> need to open a separate Reason for Outage (RFO) ticket to request formal
> documentation. Please allow 3-5 business days for post incident analysis to
> conclude. Once final analysis has been completed, all customers associated
> to this network event will receive a formal RFO document via email from the
> Lumen RFO Team. Additionally, a satisfaction survey link may be available.
> We strive to provide thorough communications containing the available
> information during a service disruption. Please let us know if the updates
> you received during this event were satisfactory.
>
> 2021-05-03 10:22:43 GMT - Additional support was engaged to assist with
> investigations inconjunction with the equipment vendor. The equipment
> vendor has advised that services have remained stable since the device
> self-restarted for the second time. Field Operations verified the equipment
> and the vendor has advised that an on-going investigation into the reason
> for outage will continue; however, as services have remained stable, a
> final notification for the Event will be provided momentarily and the
> reason for outage will be provided in the Formal RFO document following the
> outage.
>
> 2021-05-03 09:28:18 GMT - Field Operations has been engaged to dispatch to
> a site in Cotia, Brazil to perform additional troubleshooting and to
> attempt a cold rest on the routing engine. No ETA has been provided at this
> time.
>
> 2021-05-03 08:43:22 GMT - The equipment vendor continues investigations
> into a suspected software issue at this time. The IP NOC continues to
> monitor for service stability as investigations progress.
>
> 2021-05-03 07:53:20 GMT - The equipment vendor continues isolation efforts
> at this time. The IP NOC has advised that services have self-restored
> again, but remain at risk of impact due to the equipment issue. The IP NOC
> will continue monitoring services for stability as isolation efforts
> continue.
>
> 2021-05-03 07:10:28 GMT - Investigations have determined that an equipment
> issue is the root cause of both outages. The equipment self-rebooted and
> restored; however, shortly thereafter the equipment failed again. The
> equipment vendor has been engaged and is investigating the cause of the
> equipment failure at this time.
>
> 2021-05-03 06:16:38 GMT - On May 03, 2021 at 06:07 GMT, Lumen identified a
> service impact in Sao Paulo, Brazil. As this network fault is impacting
> multiple clients, the event has increased visibility with Lumen leadership.
> As such, client trouble tickets associated to this fault have been
> automatically escalated to higher priority.
>
> The IP Network Operation Center (NOC) is engaged and investigating in
> order to isolate the cause. Please be advised that updates for this event
> will be relayed at a minimum of hourly unless otherwise noted. The
> information conveyed hereafter is associated to live troubleshooting effort
> and as the discovery process evolves through to service resolution, ticket
> closure, or post incident review, details may evolve.
>
> 2021-05-03 05:54:41 GMT - Final Notification
>
> Start time: May 03, 2021 05:10 GMT
> Stop Time: May 03, 2021 05:43 GMT
>
> Root Cause: An equipment issue in Sao Paulo, Brazil impacted client
> services.
> Fix Action: The self-restarted, thus restoring services to a stable
> condition.
>
> This service impact has concluded. Due to the nature of this outage, it
> may be necessary to reset your services locally at your equipment. If
> after that action has been performed a service issue prevails, please
> contact the Lumen Repair Center for troubleshooting assistance.
>
> Customers with trouble tickets associated to this network event do not
> need to open a separate Reason for Outage (RFO) ticket to request formal
> documentation. Please allow 3-5 business days for post incident analysis to
> conclude. Once final analysis has been completed, all customers associated
> to this network event will receive a formal RFO document via email from the
> Lumen RFO Team. Additionally, a satisfaction survey link may be available.
> We strive to provide thorough communications containing the available
> information during a service disruption. Please let us know if the updates
> you received during this event were satisfactory.
>
> 2021-05-03 05:27:56 GMT - On May 03, 2021 at 05:10 GMT, Lumen identified a
> service impact in Sao Paulo, Brazil. As this network fault is impacting
> multiple clients, the event has increased visibility with Lumen leadership.
> As such, client trouble tickets associated to this fault have been
> automatically escalated to higher priority.
>
> The IP Network Operation Center (NOC) is engaged and investigating in
> order to isolate the cause. Please be advised that updates for this event
> will be relayed at a minimum of hourly unless otherwise noted. The
> information conveyed hereafter is associated to live troubleshooting effort
> and as the discovery process evolves through to service resolution, ticket
> closure, or post incident review, details may evolve.
>
>
>
>
> att,
>
> On 03/05/2021 16:51, Alexandre Chaves Fonceca por (caiu) wrote:
> > sim, mesma coisa aqui..
> >
> > desde as 2 da madrugada.. chamado aberto e aguardando... pelas
> > informações estao com problemas em equipamentos mesmo.. e nao sabem
> > "ainda" o que ocorre e/ou pq..
> >
> > att,
> > Specialist Linux Solutions
> >
> > facebook <https://www.facebook.com/specialistlinux/>instagram
> > <https://www.instagram.com/specialistlinuxsolutions/>linkedin
> > <https://www.linkedin.com/company/specialist-linux-solutions/>whatsapp
> > <https://api.whatsapp.com/send?phone=5511976158584>
> >
> > Alexandre Chaves Fonceca
> > Diretor,
> > Specialist Linux Solutions
> > E-mail: alexandre em specialist.srv.br
> > <mailto:alexandre em specialist.srv.br>
> > Site: www.specialist.srv.br <http://www.specialist.srv.br>
> > Telefone: (11) 3317-5042
> >
> > On 03/05/2021 16:16, Edinilson J. Santos wrote:
> >> Está abrindo o chamado pelo Portal na URL:
> >> https://latam.my.level3.com/portal
> >> ?
> >>
> >> Pergunto pois, ultimamente, é praticamente a única coisa que funciona
> >> direito nessa empresa.
> >> Pelo menos envia emails regularmente informando a situação do ticket.
> >>
> >> Edinilson
> >>
> >>
> >> Em 03/05/2021 15:05, Leonardo Porto por (caiu) escreveu:
> >>> Sofrendo aqui com mesmo problema, mas sem nenhuma informação adicional.
> >>>
> >>> Leonardo
> >>>
> >>> Em 03/05/2021 15:02, Luiz Fernando Mizael Meier escreveu:
> >>>> Tarde!
> >>>>
> >>>> Com certeza alguém mais deve estar sofrendo com o tombo da
> >>>> Lumen/CenturyLink/Level3/GlobalCrossing/Impsat.
> >>>>
> >>>> Circuito inoperante desde 8:50 e me informaram a pouco que a única
> >>>> coisa
> >>>> que sabem é que foi em Curitiba.
> >>>>
> >>>> Alguém tem mais alguma informação?
> >>>>
> >>>> Luiz
> >>>> _______________________________________________
> >>>> caiu mailing list
> >>>> caiu em eng.registro.br
> >>>> https://eng.registro.br/mailman/listinfo/caiu
> >>>>
> >>>>
> >>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
> >>>>
> >>>> https://eng.registro.br/mailman/options/caiu
> >>> _______________________________________________
> >>> caiu mailing list
> >>> caiu em eng.registro.br
> >>> https://eng.registro.br/mailman/listinfo/caiu
> >>>
> >>>
> >>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
> >>>
> >>> https://eng.registro.br/mailman/options/caiu
> >>>
> >>
> >> _______________________________________________
> >> caiu mailing list
> >> caiu em eng.registro.br
> >> https://eng.registro.br/mailman/listinfo/caiu
> >>
> >>
> >> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
> >>
> >> https://eng.registro.br/mailman/options/caiu
> >
> > _______________________________________________
> > caiu mailing list
> > caiu em eng.registro.br
> > https://eng.registro.br/mailman/listinfo/caiu
> >
> >
> > --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
> >
> > https://eng.registro.br/mailman/options/caiu
>
> _______________________________________________
> caiu mailing list
> caiu em eng.registro.br
> https://eng.registro.br/mailman/listinfo/caiu
>
>
> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>
> https://eng.registro.br/mailman/options/caiu
>
Mais detalhes sobre a lista de discussão caiu