[caiu] Indisponibilidade Lúmen

Qua Maio 5 18:29:09 -03 2021

Aqui falha as 17:41hrs

Em qua., 5 de mai. de 2021 17:21, Thiago Brandão Leiras <
thiago.leiras em solutiontech.com.br> escreveu:

> Percebi queda as 16:30 hs.
>
>
> -----Mensagem original-----
> De: caiu [mailto:caiu-bounces em eng.registro.br] Em nome de André Osaki
> Enviada: quarta-feira, 5 de maio de 2021 17:00
> Para: Lista das indisponibilidades da Internet brasileira
> <caiu em eng.registro.br>
> Assunto: Re: [caiu] Indisponibilidade Lúmen
>
> Várias oscilações hoje com a Lumen na Região de Campinas e São Paulo. Mais
> alguém?
>
>   --
>
> *André Kiyohiko Osaki*
>
> On Tue, May 4, 2021 at 3:29 PM Leonardo Porto por (caiu) <
> caiu em eng.registro.br> wrote:
>
> > Aqui também, mesmo horário.
> >
> > Leonardo
> >
> > Em 04/05/2021 15:23, Edinilson J. Santos escreveu:
> > > Aqui sim, ontem, 03/05/2021 por volta das 19:00hs (antes disso,
> > > ficava indo e voltando).
> > >
> > > Edinilson
> > >
> > >
> > > Em 04/05/2021 08:52, noc em consuldata.com.br escreveu:
> > >> Bom dia. O link normalizou para vocês?
> > >>
> > >> Att,
> > >>
> > >>
> > >> Everton Araujo
> > >>
> > >> On 5/3/21 5:24 PM, Eduardo Duarte wrote:
> > >>> Aqui igual
> > >>>
> > >>> Jundiaí SP
> > >>>
> > >>> O link hoje parece pisca pisca de árvore de Natal
> > >>>
> > >>> Em seg., 3 de mai. de 2021 17:00, Alexandre Chaves Fonceca por
> > >>> (caiu) < caiu em eng.registro.br> escreveu:
> > >>>
> > >>>> essas sao as informações oficiais deles... em ordem cronologica
> > >>>> (de baixo para cima), e em horario GMT 0
> > >>>>
> > >>>> 2021-05-03 19:53:47 GMT - The IP NOC observed another reboot,
> > >>>> investigation and troubleshooting efforts remain ongoing.
> > >>>>
> > >>>> 2021-05-03 19:37:11 GMT - The IP NOC is reporting all affected
> > >>>> alarms remain clear and will continue to monitor and investigate
> > >>>> the fault for the next 4 hours. As such, the next notification
> > >>>> will be provided at
> > >>>> 23:45 GMT
> > >>>> or as additional information is obtained.
> > >>>>
> > >>>> 2021-05-03 18:49:21 GMT - The IP NOC reports that the Sao Paulo,
> > >>>> Brazil equipment has been restored to a full redundancy
> > >>>> condition. Traffic was migrated off the redundant uplink and
> > >>>> remains stable. The IP NOC will continue to monitor the services.
> > >>>> IP Engineering and the equipment vendor are still collaboratively
> > >>>> investigating the cause for the service impact.
> > >>>>
> > >>>> 2021-05-03 17:53:10 GMT - The IP NOC reports that the impacted
> > >>>> equipment in Sao Paulo, Brazil is still stable since the
> > >>>> redundant routing path was taken out of service. Services
> > >>>> continue to remain stable as well.
> > >>>> Cooperative investigations by multiple support groups along with
> > >>>> the equipment vendor remain underway.
> > >>>>
> > >>>> 2021-05-03 16:52:08 GMT - The IP NOC remotely migrated impacted
> > >>>> services to a redundant uplink, however some services have
> > >>>> continued to bounce in and out of service. The IP NOC,
> > >>>> Engineering teams, Field Operations and the equipment vendor
> > >>>> continue investigating the the cause for the service impact.
> > >>>>
> > >>>> 2021-05-03 15:55:43 GMT - Collaborative troubleshooting with the
> > >>>> equipment vendor, Field Operations and the IP NOC remain ongoing.
> > >>>>
> > >>>> 2021-05-03 15:07:56 GMT - Field Operations is onsite at the Sao
> > >>>> Paulo fault site. Collaborative troubleshooting with the
> > >>>> equipment vendor remains ongoing.
> > >>>>
> > >>>> 2021-05-03 14:11:40 GMT - The IP NOC reported a device has
> > >>>> restarted multiple times. Collaborative troubleshooting with the
> > >>>> equipment vendor remains ongoing. Field Operations are being
> > >>>> engaged as well.
> > >>>>
> > >>>> 2021-05-03 13:51:28 GMT - The IP NOC continues to work in
> > >>>> cooperation with the equipment vendor to troubleshoot the issue.
> > >>>> We understand how important these services are to our customers
> > >>>> and the issue has been escalated to the highest levels within
> > >>>> Lumen Service Assurance Leadership.
> > >>>>
> > >>>> 2021-05-03 13:48:56 GMT - The IP NOC continues to work in
> > >>>> cooperation with the equipment vendor to troubleshoot the issue.
> > >>>>
> > >>>> 2021-05-03 13:16:06 GMT - Cooperative investigations by multiple
> > >>>> support groups remain underway at this time.
> > >>>>
> > >>>> 2021-05-03 13:13:13 GMT - The IP NOC is working in cooperation
> > >>>> with Field Operations and the equipment vendor to troubleshoot
> > >>>> the issue.
> > >>>>
> > >>>> 2021-05-03 12:50:33 GMT - Cooperative investigations by multiple
> > >>>> support groups remain underway at this time.
> > >>>>
> > >>>> 2021-05-03 12:02:37 GMT - On May 03, 2021 at 11:46 GMT, Lumen
> > >>>> identifieda service impact in Sao Paulo, Brazil. As this network
> > >>>> fault is impacting multiple clients, the event has increased
> > >>>> visibility with Lumen leadership.
> > >>>> As such, client trouble tickets associated to this fault have
> > >>>> been automatically escalated to higher priority.
> > >>>> The IP Network Operation Center (NOC) is engaged and
> > >>>> investigating in order to isolate the cause. Please be advised
> > >>>> that updates for this event will be relayed at a minimum of
> > >>>> hourly unless otherwise noted. The information conveyed hereafter
> > >>>> is associated to live troubleshooting effort and as the discovery
> > >>>> process evolves through to service resolution, ticket closure, or
> > >>>> post incident review, details may evolve.
> > >>>>
> > >>>> 2021-05-03 10:27:08 GMT - Final Notification
> > >>>>
> > >>>> Start time: May 03, 2021 05:10 GMT Stop Time: May 03, 2021 05:43
> > >>>> GMT Second Start Time: May 03, 2021 06:07 GMT Second Stop Time:
> > >>>> May 03, 2021 07:23 GMT Root Cause: An equipment issue in Sao
> > >>>> Paulo, Brazil impacted client services.
> > >>>> Fix Action: The self-restarted, thus restoring services to a
> > >>>> stable condition.
> > >>>>
> > >>>> This service impact has concluded. Due to the nature of this
> > >>>> outage, it may be necessary to reset your services locally at
> > >>>> your equipment.  If after that action has been performed a
> > >>>> service issue prevails, please contact the Lumen Repair Center
> > >>>> for troubleshooting assistance.
> > >>>>
> > >>>> Customers with trouble tickets associated to this network event
> > >>>> do not need to open a separate Reason for Outage (RFO) ticket to
> > >>>> request formal documentation. Please allow 3-5 business days for
> > >>>> post incident analysisto conclude. Once final analysis has been
> > >>>> completed, all customers associated to this network event will
> > >>>> receive a formal RFO document via email from the Lumen RFO Team.
> > >>>> Additionally, a satisfaction survey link may be available.
> > >>>> We strive to provide thorough communications containing the
> > >>>> available information during a service disruption. Please let us
> > >>>> know if the updates you received during this event were
> > >>>> satisfactory.
> > >>>>
> > >>>> 2021-05-03 10:22:43 GMT - Additional support was engaged to
> > >>>> assist with investigations inconjunction with the equipment
> > >>>> vendor. The equipment vendor has advised that services have
> > >>>> remained stable since the device self-restarted for the second
> > >>>> time. Field Operations verified the equipment and the vendor has
> > >>>> advised that an on-going investigation into the reason for outage
> > >>>> will continue; however, as services have remained stable, a final
> > >>>> notification for the Event will be provided momentarily and the
> > >>>> reason for outage will be provided in the Formal RFO document
> > >>>> following the outage.
> > >>>>
> > >>>> 2021-05-03 09:28:18 GMT - Field Operations has been engaged to
> > >>>> dispatch to a site in Cotia, Brazil to perform additional
> > >>>> troubleshooting and to attempt a cold rest on the routing engine.
> > >>>> No ETA has been provided at this time.
> > >>>>
> > >>>> 2021-05-03 08:43:22 GMT - The equipment vendor continues
> > >>>> investigations into a suspected software issue at this time. The
> > >>>> IP NOC continues to monitor for service stability as
> > >>>> investigations progress.
> > >>>>
> > >>>> 2021-05-03 07:53:20 GMT - The equipment vendor continues
> > >>>> isolation efforts at this time. The IP NOC has advised that
> > >>>> services have self-restored again, but remain at risk of impact
> > >>>> due to the equipment issue. The IP NOC will continue monitoring
> > >>>> services for stability as isolation efforts continue.
> > >>>>
> > >>>> 2021-05-03 07:10:28 GMT - Investigations have determined that an
> > >>>> equipment issue is the root cause of both outages. The equipment
> > >>>> self-rebooted and restored; however, shortly thereafter the
> > >>>> equipment failed again. The equipment vendor has been engaged and
> > >>>> is investigating the cause of the equipment failure at this time.
> > >>>>
> > >>>> 2021-05-03 06:16:38 GMT - On May 03, 2021 at 06:07 GMT, Lumen
> > >>>> identifieda service impact in Sao Paulo, Brazil. As this network
> > >>>> fault is impacting multiple clients, the event has increased
> > >>>> visibility with Lumen leadership.
> > >>>> As such, client trouble tickets associated to this fault have
> > >>>> been automatically escalated to higher priority.
> > >>>>
> > >>>> The IP Network Operation Center (NOC) is engaged and
> > >>>> investigating in order to isolate the cause. Please be advised
> > >>>> that updates for this event will be relayed at a minimum of
> > >>>> hourly unless otherwise noted. The information conveyed hereafter
> > >>>> is associated to live troubleshooting effort and as the discovery
> > >>>> process evolves through to service resolution, ticket closure, or
> > >>>> post incident review, details may evolve.
> > >>>>
> > >>>> 2021-05-03 05:54:41 GMT - Final Notification
> > >>>>
> > >>>> Start time: May 03, 2021 05:10 GMT Stop Time: May 03, 2021 05:43
> > >>>> GMT
> > >>>>
> > >>>> Root Cause: An equipment issue in Sao Paulo, Brazil impacted
> > >>>> client services.
> > >>>> Fix Action: The self-restarted, thus restoring services to a
> > >>>> stable condition.
> > >>>>
> > >>>> This service impact has concluded. Due to the nature of this
> > >>>> outage, it may be necessary to reset your services locally at
> > >>>> your equipment.  If after that action has been performed a
> > >>>> service issue prevails, please contact the Lumen Repair Center
> > >>>> for troubleshooting assistance.
> > >>>>
> > >>>> Customers with trouble tickets associated to this network event
> > >>>> do not need to open a separate Reason for Outage (RFO) ticket to
> > >>>> request formal documentation. Please allow 3-5 business days for
> > >>>> post incident analysisto conclude. Once final analysis has been
> > >>>> completed, all customers associated to this network event will
> > >>>> receive a formal RFO document via email from the Lumen RFO Team.
> > >>>> Additionally, a satisfaction survey link may be available.
> > >>>> We strive to provide thorough communications containing the
> > >>>> available information during a service disruption. Please let us
> > >>>> know if the updates you received during this event were
> > >>>> satisfactory.
> > >>>>
> > >>>> 2021-05-03 05:27:56 GMT - On May 03, 2021 at 05:10 GMT, Lumen
> > >>>> identifieda service impact in Sao Paulo, Brazil. As this network
> > >>>> fault is impacting multiple clients, the event has increased
> > >>>> visibility with Lumen leadership.
> > >>>> As such, client trouble tickets associated to this fault have
> > >>>> been automatically escalated to higher priority.
> > >>>>
> > >>>> The IP Network Operation Center (NOC) is engaged and
> > >>>> investigating in order to isolate the cause. Please be advised
> > >>>> that updates for this event will be relayed at a minimum of
> > >>>> hourly unless otherwise noted. The information conveyed hereafter
> > >>>> is associated to live troubleshooting effort and as the discovery
> > >>>> process evolves through to service resolution, ticket closure, or
> > >>>> post incident review, details may evolve.
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> att,
> > >>>>
> > >>>> On 03/05/2021 16:51, Alexandre Chaves Fonceca por (caiu) wrote:
> > >>>>> sim, mesma coisa aqui..
> > >>>>>
> > >>>>> desde as 2 da madrugada.. chamado aberto e aguardando... pelas
> > >>>>> informações estao com problemas em equipamentos mesmo.. e nao
> > >>>>> sabem "ainda" o que ocorre e/ou pq..
> > >>>>>
> > >>>>> att,
> > >>>>> Specialist Linux Solutions
> > >>>>>
> > >>>>> facebook <https://www.facebook.com/specialistlinux/>instagram
> > >>>>> <https://www.instagram.com/specialistlinuxsolutions/>linkedin
> > >>>>> <https://www.linkedin.com/company/specialist-linux-solutions/>wh
> > >>>>> atsapp
> >
> > >>>>>
> > >>>>> <https://api.whatsapp.com/send?phone=5511976158584>
> > >>>>>
> > >>>>> Alexandre Chaves Fonceca
> > >>>>> Diretor,
> > >>>>> Specialist Linux Solutions
> > >>>>> E-mail:     alexandre em specialist.srv.br
> > >>>>> <mailto:alexandre em specialist.srv.br>
> > >>>>> Site:     www.specialist.srv.br <http://www.specialist.srv.br>
> > >>>>> Telefone:     (11) 3317-5042
> > >>>>>
> > >>>>> On 03/05/2021 16:16, Edinilson J. Santos wrote:
> > >>>>>> Está abrindo o chamado pelo Portal na URL:
> > >>>>>> https://latam.my.level3.com/portal
> > >>>>>> ?
> > >>>>>>
> > >>>>>> Pergunto pois, ultimamente, é praticamente a única coisa que
> > >>>>>> funciona direito nessa empresa.
> > >>>>>> Pelo menos envia emails regularmente informando a situação do
> > >>>>>> ticket.
> > >>>>>>
> > >>>>>> Edinilson
> > >>>>>>
> > >>>>>>
> > >>>>>> Em 03/05/2021 15:05, Leonardo Porto por (caiu) escreveu:
> > >>>>>>> Sofrendo aqui com mesmo problema, mas sem nenhuma informação
> > >>>>>>> adicional.
> > >>>>>>>
> > >>>>>>> Leonardo
> > >>>>>>>
> > >>>>>>> Em 03/05/2021 15:02, Luiz Fernando Mizael Meier escreveu:
> > >>>>>>>> Tarde!
> > >>>>>>>>
> > >>>>>>>> Com certeza alguém mais deve estar sofrendo com o tombo da
> > >>>>>>>> Lumen/CenturyLink/Level3/GlobalCrossing/Impsat.
> > >>>>>>>>
> > >>>>>>>> Circuito inoperante desde 8:50 e me informaram a pouco que a
> > >>>>>>>> única coisa que sabem é que foi em Curitiba.
> > >>>>>>>>
> > >>>>>>>> Alguém tem mais alguma informação?
> > >>>>>>>>
> > >>>>>>>> Luiz
> > >>>>>>>> _______________________________________________
> > >>>>>>>> caiu mailing list
> > >>>>>>>> caiu em eng.registro.br
> > >>>>>>>> https://eng.registro.br/mailman/listinfo/caiu
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
> > >>>>>>>>
> > >>>>>>>> https://eng.registro.br/mailman/options/caiu
> > >>>>>>> _______________________________________________
> > >>>>>>> caiu mailing list
> > >>>>>>> caiu em eng.registro.br
> > >>>>>>> https://eng.registro.br/mailman/listinfo/caiu
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
> > >>>>>>>
> > >>>>>>> https://eng.registro.br/mailman/options/caiu
> > >>>>>>>
> > >>>>>> _______________________________________________
> > >>>>>> caiu mailing list
> > >>>>>> caiu em eng.registro.br
> > >>>>>> https://eng.registro.br/mailman/listinfo/caiu
> > >>>>>>
> > >>>>>>
> > >>>>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
> > >>>>>>
> > >>>>>> https://eng.registro.br/mailman/options/caiu
> > >>>>> _______________________________________________
> > >>>>> caiu mailing list
> > >>>>> caiu em eng.registro.br
> > >>>>> https://eng.registro.br/mailman/listinfo/caiu
> > >>>>>
> > >>>>>
> > >>>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
> > >>>>>
> > >>>>> https://eng.registro.br/mailman/options/caiu
> > >>>> _______________________________________________
> > >>>> caiu mailing list
> > >>>> caiu em eng.registro.br
> > >>>> https://eng.registro.br/mailman/listinfo/caiu
> > >>>>
> > >>>>
> > >>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
> > >>>>
> > >>>> https://eng.registro.br/mailman/options/caiu
> > >>>>
> > >>> _______________________________________________
> > >>> caiu mailing list
> > >>> caiu em eng.registro.br
> > >>> https://eng.registro.br/mailman/listinfo/caiu
> > >>>
> > >>>
> > >>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
> > >>>
> > >>> https://eng.registro.br/mailman/options/caiu
> > >> _______________________________________________
> > >> caiu mailing list
> > >> caiu em eng.registro.br
> > >> https://eng.registro.br/mailman/listinfo/caiu
> > >>
> > >>
> > >> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
> > >>
> > >> https://eng.registro.br/mailman/options/caiu
> > >>
> > >
> > > _______________________________________________
> > > caiu mailing list
> > > caiu em eng.registro.br
> > > https://eng.registro.br/mailman/listinfo/caiu
> > >
> > >
> > > --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
> > >
> > > https://eng.registro.br/mailman/options/caiu
> > _______________________________________________
> > caiu mailing list
> > caiu em eng.registro.br
> > https://eng.registro.br/mailman/listinfo/caiu
> >
> >
> > --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
> >
> > https://eng.registro.br/mailman/options/caiu
> >
> _______________________________________________
> caiu mailing list
> caiu em eng.registro.br
> https://eng.registro.br/mailman/listinfo/caiu
>
>
> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>
> https://eng.registro.br/mailman/options/caiu
>
> _______________________________________________
> caiu mailing list
> caiu em eng.registro.br
> https://eng.registro.br/mailman/listinfo/caiu
>
>
> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>
> https://eng.registro.br/mailman/options/caiu
>