[caiu] Indisponibilidade Lúmen
Thiago Brandão Leiras
thiago.leiras em solutiontech.com.br
Qua Maio 5 17:20:54 -03 2021
Percebi queda as 16:30 hs.
-----Mensagem original-----
De: caiu [mailto:caiu-bounces em eng.registro.br] Em nome de André Osaki
Enviada: quarta-feira, 5 de maio de 2021 17:00
Para: Lista das indisponibilidades da Internet brasileira
<caiu em eng.registro.br>
Assunto: Re: [caiu] Indisponibilidade Lúmen
Várias oscilações hoje com a Lumen na Região de Campinas e São Paulo. Mais
alguém?
--
*André Kiyohiko Osaki*
On Tue, May 4, 2021 at 3:29 PM Leonardo Porto por (caiu) <
caiu em eng.registro.br> wrote:
> Aqui também, mesmo horário.
>
> Leonardo
>
> Em 04/05/2021 15:23, Edinilson J. Santos escreveu:
> > Aqui sim, ontem, 03/05/2021 por volta das 19:00hs (antes disso,
> > ficava indo e voltando).
> >
> > Edinilson
> >
> >
> > Em 04/05/2021 08:52, noc em consuldata.com.br escreveu:
> >> Bom dia. O link normalizou para vocês?
> >>
> >> Att,
> >>
> >>
> >> Everton Araujo
> >>
> >> On 5/3/21 5:24 PM, Eduardo Duarte wrote:
> >>> Aqui igual
> >>>
> >>> Jundiaí SP
> >>>
> >>> O link hoje parece pisca pisca de árvore de Natal
> >>>
> >>> Em seg., 3 de mai. de 2021 17:00, Alexandre Chaves Fonceca por
> >>> (caiu) < caiu em eng.registro.br> escreveu:
> >>>
> >>>> essas sao as informações oficiais deles... em ordem cronologica
> >>>> (de baixo para cima), e em horario GMT 0
> >>>>
> >>>> 2021-05-03 19:53:47 GMT - The IP NOC observed another reboot,
> >>>> investigation and troubleshooting efforts remain ongoing.
> >>>>
> >>>> 2021-05-03 19:37:11 GMT - The IP NOC is reporting all affected
> >>>> alarms remain clear and will continue to monitor and investigate
> >>>> the fault for the next 4 hours. As such, the next notification
> >>>> will be provided at
> >>>> 23:45 GMT
> >>>> or as additional information is obtained.
> >>>>
> >>>> 2021-05-03 18:49:21 GMT - The IP NOC reports that the Sao Paulo,
> >>>> Brazil equipment has been restored to a full redundancy
> >>>> condition. Traffic was migrated off the redundant uplink and
> >>>> remains stable. The IP NOC will continue to monitor the services.
> >>>> IP Engineering and the equipment vendor are still collaboratively
> >>>> investigating the cause for the service impact.
> >>>>
> >>>> 2021-05-03 17:53:10 GMT - The IP NOC reports that the impacted
> >>>> equipment in Sao Paulo, Brazil is still stable since the
> >>>> redundant routing path was taken out of service. Services
> >>>> continue to remain stable as well.
> >>>> Cooperative investigations by multiple support groups along with
> >>>> the equipment vendor remain underway.
> >>>>
> >>>> 2021-05-03 16:52:08 GMT - The IP NOC remotely migrated impacted
> >>>> services to a redundant uplink, however some services have
> >>>> continued to bounce in and out of service. The IP NOC,
> >>>> Engineering teams, Field Operations and the equipment vendor
> >>>> continue investigating the the cause for the service impact.
> >>>>
> >>>> 2021-05-03 15:55:43 GMT - Collaborative troubleshooting with the
> >>>> equipment vendor, Field Operations and the IP NOC remain ongoing.
> >>>>
> >>>> 2021-05-03 15:07:56 GMT - Field Operations is onsite at the Sao
> >>>> Paulo fault site. Collaborative troubleshooting with the
> >>>> equipment vendor remains ongoing.
> >>>>
> >>>> 2021-05-03 14:11:40 GMT - The IP NOC reported a device has
> >>>> restarted multiple times. Collaborative troubleshooting with the
> >>>> equipment vendor remains ongoing. Field Operations are being
> >>>> engaged as well.
> >>>>
> >>>> 2021-05-03 13:51:28 GMT - The IP NOC continues to work in
> >>>> cooperation with the equipment vendor to troubleshoot the issue.
> >>>> We understand how important these services are to our customers
> >>>> and the issue has been escalated to the highest levels within
> >>>> Lumen Service Assurance Leadership.
> >>>>
> >>>> 2021-05-03 13:48:56 GMT - The IP NOC continues to work in
> >>>> cooperation with the equipment vendor to troubleshoot the issue.
> >>>>
> >>>> 2021-05-03 13:16:06 GMT - Cooperative investigations by multiple
> >>>> support groups remain underway at this time.
> >>>>
> >>>> 2021-05-03 13:13:13 GMT - The IP NOC is working in cooperation
> >>>> with Field Operations and the equipment vendor to troubleshoot
> >>>> the issue.
> >>>>
> >>>> 2021-05-03 12:50:33 GMT - Cooperative investigations by multiple
> >>>> support groups remain underway at this time.
> >>>>
> >>>> 2021-05-03 12:02:37 GMT - On May 03, 2021 at 11:46 GMT, Lumen
> >>>> identifieda service impact in Sao Paulo, Brazil. As this network
> >>>> fault is impacting multiple clients, the event has increased
> >>>> visibility with Lumen leadership.
> >>>> As such, client trouble tickets associated to this fault have
> >>>> been automatically escalated to higher priority.
> >>>> The IP Network Operation Center (NOC) is engaged and
> >>>> investigating in order to isolate the cause. Please be advised
> >>>> that updates for this event will be relayed at a minimum of
> >>>> hourly unless otherwise noted. The information conveyed hereafter
> >>>> is associated to live troubleshooting effort and as the discovery
> >>>> process evolves through to service resolution, ticket closure, or
> >>>> post incident review, details may evolve.
> >>>>
> >>>> 2021-05-03 10:27:08 GMT - Final Notification
> >>>>
> >>>> Start time: May 03, 2021 05:10 GMT Stop Time: May 03, 2021 05:43
> >>>> GMT Second Start Time: May 03, 2021 06:07 GMT Second Stop Time:
> >>>> May 03, 2021 07:23 GMT Root Cause: An equipment issue in Sao
> >>>> Paulo, Brazil impacted client services.
> >>>> Fix Action: The self-restarted, thus restoring services to a
> >>>> stable condition.
> >>>>
> >>>> This service impact has concluded. Due to the nature of this
> >>>> outage, it may be necessary to reset your services locally at
> >>>> your equipment. If after that action has been performed a
> >>>> service issue prevails, please contact the Lumen Repair Center
> >>>> for troubleshooting assistance.
> >>>>
> >>>> Customers with trouble tickets associated to this network event
> >>>> do not need to open a separate Reason for Outage (RFO) ticket to
> >>>> request formal documentation. Please allow 3-5 business days for
> >>>> post incident analysisto conclude. Once final analysis has been
> >>>> completed, all customers associated to this network event will
> >>>> receive a formal RFO document via email from the Lumen RFO Team.
> >>>> Additionally, a satisfaction survey link may be available.
> >>>> We strive to provide thorough communications containing the
> >>>> available information during a service disruption. Please let us
> >>>> know if the updates you received during this event were
> >>>> satisfactory.
> >>>>
> >>>> 2021-05-03 10:22:43 GMT - Additional support was engaged to
> >>>> assist with investigations inconjunction with the equipment
> >>>> vendor. The equipment vendor has advised that services have
> >>>> remained stable since the device self-restarted for the second
> >>>> time. Field Operations verified the equipment and the vendor has
> >>>> advised that an on-going investigation into the reason for outage
> >>>> will continue; however, as services have remained stable, a final
> >>>> notification for the Event will be provided momentarily and the
> >>>> reason for outage will be provided in the Formal RFO document
> >>>> following the outage.
> >>>>
> >>>> 2021-05-03 09:28:18 GMT - Field Operations has been engaged to
> >>>> dispatch to a site in Cotia, Brazil to perform additional
> >>>> troubleshooting and to attempt a cold rest on the routing engine.
> >>>> No ETA has been provided at this time.
> >>>>
> >>>> 2021-05-03 08:43:22 GMT - The equipment vendor continues
> >>>> investigations into a suspected software issue at this time. The
> >>>> IP NOC continues to monitor for service stability as
> >>>> investigations progress.
> >>>>
> >>>> 2021-05-03 07:53:20 GMT - The equipment vendor continues
> >>>> isolation efforts at this time. The IP NOC has advised that
> >>>> services have self-restored again, but remain at risk of impact
> >>>> due to the equipment issue. The IP NOC will continue monitoring
> >>>> services for stability as isolation efforts continue.
> >>>>
> >>>> 2021-05-03 07:10:28 GMT - Investigations have determined that an
> >>>> equipment issue is the root cause of both outages. The equipment
> >>>> self-rebooted and restored; however, shortly thereafter the
> >>>> equipment failed again. The equipment vendor has been engaged and
> >>>> is investigating the cause of the equipment failure at this time.
> >>>>
> >>>> 2021-05-03 06:16:38 GMT - On May 03, 2021 at 06:07 GMT, Lumen
> >>>> identifieda service impact in Sao Paulo, Brazil. As this network
> >>>> fault is impacting multiple clients, the event has increased
> >>>> visibility with Lumen leadership.
> >>>> As such, client trouble tickets associated to this fault have
> >>>> been automatically escalated to higher priority.
> >>>>
> >>>> The IP Network Operation Center (NOC) is engaged and
> >>>> investigating in order to isolate the cause. Please be advised
> >>>> that updates for this event will be relayed at a minimum of
> >>>> hourly unless otherwise noted. The information conveyed hereafter
> >>>> is associated to live troubleshooting effort and as the discovery
> >>>> process evolves through to service resolution, ticket closure, or
> >>>> post incident review, details may evolve.
> >>>>
> >>>> 2021-05-03 05:54:41 GMT - Final Notification
> >>>>
> >>>> Start time: May 03, 2021 05:10 GMT Stop Time: May 03, 2021 05:43
> >>>> GMT
> >>>>
> >>>> Root Cause: An equipment issue in Sao Paulo, Brazil impacted
> >>>> client services.
> >>>> Fix Action: The self-restarted, thus restoring services to a
> >>>> stable condition.
> >>>>
> >>>> This service impact has concluded. Due to the nature of this
> >>>> outage, it may be necessary to reset your services locally at
> >>>> your equipment. If after that action has been performed a
> >>>> service issue prevails, please contact the Lumen Repair Center
> >>>> for troubleshooting assistance.
> >>>>
> >>>> Customers with trouble tickets associated to this network event
> >>>> do not need to open a separate Reason for Outage (RFO) ticket to
> >>>> request formal documentation. Please allow 3-5 business days for
> >>>> post incident analysisto conclude. Once final analysis has been
> >>>> completed, all customers associated to this network event will
> >>>> receive a formal RFO document via email from the Lumen RFO Team.
> >>>> Additionally, a satisfaction survey link may be available.
> >>>> We strive to provide thorough communications containing the
> >>>> available information during a service disruption. Please let us
> >>>> know if the updates you received during this event were
> >>>> satisfactory.
> >>>>
> >>>> 2021-05-03 05:27:56 GMT - On May 03, 2021 at 05:10 GMT, Lumen
> >>>> identifieda service impact in Sao Paulo, Brazil. As this network
> >>>> fault is impacting multiple clients, the event has increased
> >>>> visibility with Lumen leadership.
> >>>> As such, client trouble tickets associated to this fault have
> >>>> been automatically escalated to higher priority.
> >>>>
> >>>> The IP Network Operation Center (NOC) is engaged and
> >>>> investigating in order to isolate the cause. Please be advised
> >>>> that updates for this event will be relayed at a minimum of
> >>>> hourly unless otherwise noted. The information conveyed hereafter
> >>>> is associated to live troubleshooting effort and as the discovery
> >>>> process evolves through to service resolution, ticket closure, or
> >>>> post incident review, details may evolve.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> att,
> >>>>
> >>>> On 03/05/2021 16:51, Alexandre Chaves Fonceca por (caiu) wrote:
> >>>>> sim, mesma coisa aqui..
> >>>>>
> >>>>> desde as 2 da madrugada.. chamado aberto e aguardando... pelas
> >>>>> informações estao com problemas em equipamentos mesmo.. e nao
> >>>>> sabem "ainda" o que ocorre e/ou pq..
> >>>>>
> >>>>> att,
> >>>>> Specialist Linux Solutions
> >>>>>
> >>>>> facebook <https://www.facebook.com/specialistlinux/>instagram
> >>>>> <https://www.instagram.com/specialistlinuxsolutions/>linkedin
> >>>>> <https://www.linkedin.com/company/specialist-linux-solutions/>wh
> >>>>> atsapp
>
> >>>>>
> >>>>> <https://api.whatsapp.com/send?phone=5511976158584>
> >>>>>
> >>>>> Alexandre Chaves Fonceca
> >>>>> Diretor,
> >>>>> Specialist Linux Solutions
> >>>>> E-mail: alexandre em specialist.srv.br
> >>>>> <mailto:alexandre em specialist.srv.br>
> >>>>> Site: www.specialist.srv.br <http://www.specialist.srv.br>
> >>>>> Telefone: (11) 3317-5042
> >>>>>
> >>>>> On 03/05/2021 16:16, Edinilson J. Santos wrote:
> >>>>>> Está abrindo o chamado pelo Portal na URL:
> >>>>>> https://latam.my.level3.com/portal
> >>>>>> ?
> >>>>>>
> >>>>>> Pergunto pois, ultimamente, é praticamente a única coisa que
> >>>>>> funciona direito nessa empresa.
> >>>>>> Pelo menos envia emails regularmente informando a situação do
> >>>>>> ticket.
> >>>>>>
> >>>>>> Edinilson
> >>>>>>
> >>>>>>
> >>>>>> Em 03/05/2021 15:05, Leonardo Porto por (caiu) escreveu:
> >>>>>>> Sofrendo aqui com mesmo problema, mas sem nenhuma informação
> >>>>>>> adicional.
> >>>>>>>
> >>>>>>> Leonardo
> >>>>>>>
> >>>>>>> Em 03/05/2021 15:02, Luiz Fernando Mizael Meier escreveu:
> >>>>>>>> Tarde!
> >>>>>>>>
> >>>>>>>> Com certeza alguém mais deve estar sofrendo com o tombo da
> >>>>>>>> Lumen/CenturyLink/Level3/GlobalCrossing/Impsat.
> >>>>>>>>
> >>>>>>>> Circuito inoperante desde 8:50 e me informaram a pouco que a
> >>>>>>>> única coisa que sabem é que foi em Curitiba.
> >>>>>>>>
> >>>>>>>> Alguém tem mais alguma informação?
> >>>>>>>>
> >>>>>>>> Luiz
> >>>>>>>> _______________________________________________
> >>>>>>>> caiu mailing list
> >>>>>>>> caiu em eng.registro.br
> >>>>>>>> https://eng.registro.br/mailman/listinfo/caiu
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
> >>>>>>>>
> >>>>>>>> https://eng.registro.br/mailman/options/caiu
> >>>>>>> _______________________________________________
> >>>>>>> caiu mailing list
> >>>>>>> caiu em eng.registro.br
> >>>>>>> https://eng.registro.br/mailman/listinfo/caiu
> >>>>>>>
> >>>>>>>
> >>>>>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
> >>>>>>>
> >>>>>>> https://eng.registro.br/mailman/options/caiu
> >>>>>>>
> >>>>>> _______________________________________________
> >>>>>> caiu mailing list
> >>>>>> caiu em eng.registro.br
> >>>>>> https://eng.registro.br/mailman/listinfo/caiu
> >>>>>>
> >>>>>>
> >>>>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
> >>>>>>
> >>>>>> https://eng.registro.br/mailman/options/caiu
> >>>>> _______________________________________________
> >>>>> caiu mailing list
> >>>>> caiu em eng.registro.br
> >>>>> https://eng.registro.br/mailman/listinfo/caiu
> >>>>>
> >>>>>
> >>>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
> >>>>>
> >>>>> https://eng.registro.br/mailman/options/caiu
> >>>> _______________________________________________
> >>>> caiu mailing list
> >>>> caiu em eng.registro.br
> >>>> https://eng.registro.br/mailman/listinfo/caiu
> >>>>
> >>>>
> >>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
> >>>>
> >>>> https://eng.registro.br/mailman/options/caiu
> >>>>
> >>> _______________________________________________
> >>> caiu mailing list
> >>> caiu em eng.registro.br
> >>> https://eng.registro.br/mailman/listinfo/caiu
> >>>
> >>>
> >>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
> >>>
> >>> https://eng.registro.br/mailman/options/caiu
> >> _______________________________________________
> >> caiu mailing list
> >> caiu em eng.registro.br
> >> https://eng.registro.br/mailman/listinfo/caiu
> >>
> >>
> >> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
> >>
> >> https://eng.registro.br/mailman/options/caiu
> >>
> >
> > _______________________________________________
> > caiu mailing list
> > caiu em eng.registro.br
> > https://eng.registro.br/mailman/listinfo/caiu
> >
> >
> > --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
> >
> > https://eng.registro.br/mailman/options/caiu
> _______________________________________________
> caiu mailing list
> caiu em eng.registro.br
> https://eng.registro.br/mailman/listinfo/caiu
>
>
> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>
> https://eng.registro.br/mailman/options/caiu
>
_______________________________________________
caiu mailing list
caiu em eng.registro.br
https://eng.registro.br/mailman/listinfo/caiu
--> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
https://eng.registro.br/mailman/options/caiu
Mais detalhes sobre a lista de discussão caiu