[caiu] Indisponibilidade Lúmen

Qua Maio 5 21:58:51 -03 2021

Boa noite

Lumen com várias falhas agora a noite

Att

Em qua., 5 de mai. de 2021 18:29, Eduardo Duarte <eduduarteneves em gmail.com>
escreveu:

> Aqui falha as 17:41hrs
>
> Em qua., 5 de mai. de 2021 17:21, Thiago Brandão Leiras <
> thiago.leiras em solutiontech.com.br> escreveu:
>
>> Percebi queda as 16:30 hs.
>>
>>
>> -----Mensagem original-----
>> De: caiu [mailto:caiu-bounces em eng.registro.br] Em nome de André Osaki
>> Enviada: quarta-feira, 5 de maio de 2021 17:00
>> Para: Lista das indisponibilidades da Internet brasileira
>> <caiu em eng.registro.br>
>> Assunto: Re: [caiu] Indisponibilidade Lúmen
>>
>> Várias oscilações hoje com a Lumen na Região de Campinas e São Paulo. Mais
>> alguém?
>>
>>   --
>>
>> *André Kiyohiko Osaki*
>>
>> On Tue, May 4, 2021 at 3:29 PM Leonardo Porto por (caiu) <
>> caiu em eng.registro.br> wrote:
>>
>> > Aqui também, mesmo horário.
>> >
>> > Leonardo
>> >
>> > Em 04/05/2021 15:23, Edinilson J. Santos escreveu:
>> > > Aqui sim, ontem, 03/05/2021 por volta das 19:00hs (antes disso,
>> > > ficava indo e voltando).
>> > >
>> > > Edinilson
>> > >
>> > >
>> > > Em 04/05/2021 08:52, noc em consuldata.com.br escreveu:
>> > >> Bom dia. O link normalizou para vocês?
>> > >>
>> > >> Att,
>> > >>
>> > >>
>> > >> Everton Araujo
>> > >>
>> > >> On 5/3/21 5:24 PM, Eduardo Duarte wrote:
>> > >>> Aqui igual
>> > >>>
>> > >>> Jundiaí SP
>> > >>>
>> > >>> O link hoje parece pisca pisca de árvore de Natal
>> > >>>
>> > >>> Em seg., 3 de mai. de 2021 17:00, Alexandre Chaves Fonceca por
>> > >>> (caiu) < caiu em eng.registro.br> escreveu:
>> > >>>
>> > >>>> essas sao as informações oficiais deles... em ordem cronologica
>> > >>>> (de baixo para cima), e em horario GMT 0
>> > >>>>
>> > >>>> 2021-05-03 19:53:47 GMT - The IP NOC observed another reboot,
>> > >>>> investigation and troubleshooting efforts remain ongoing.
>> > >>>>
>> > >>>> 2021-05-03 19:37:11 GMT - The IP NOC is reporting all affected
>> > >>>> alarms remain clear and will continue to monitor and investigate
>> > >>>> the fault for the next 4 hours. As such, the next notification
>> > >>>> will be provided at
>> > >>>> 23:45 GMT
>> > >>>> or as additional information is obtained.
>> > >>>>
>> > >>>> 2021-05-03 18:49:21 GMT - The IP NOC reports that the Sao Paulo,
>> > >>>> Brazil equipment has been restored to a full redundancy
>> > >>>> condition. Traffic was migrated off the redundant uplink and
>> > >>>> remains stable. The IP NOC will continue to monitor the services.
>> > >>>> IP Engineering and the equipment vendor are still collaboratively
>> > >>>> investigating the cause for the service impact.
>> > >>>>
>> > >>>> 2021-05-03 17:53:10 GMT - The IP NOC reports that the impacted
>> > >>>> equipment in Sao Paulo, Brazil is still stable since the
>> > >>>> redundant routing path was taken out of service. Services
>> > >>>> continue to remain stable as well.
>> > >>>> Cooperative investigations by multiple support groups along with
>> > >>>> the equipment vendor remain underway.
>> > >>>>
>> > >>>> 2021-05-03 16:52:08 GMT - The IP NOC remotely migrated impacted
>> > >>>> services to a redundant uplink, however some services have
>> > >>>> continued to bounce in and out of service. The IP NOC,
>> > >>>> Engineering teams, Field Operations and the equipment vendor
>> > >>>> continue investigating the the cause for the service impact.
>> > >>>>
>> > >>>> 2021-05-03 15:55:43 GMT - Collaborative troubleshooting with the
>> > >>>> equipment vendor, Field Operations and the IP NOC remain ongoing.
>> > >>>>
>> > >>>> 2021-05-03 15:07:56 GMT - Field Operations is onsite at the Sao
>> > >>>> Paulo fault site. Collaborative troubleshooting with the
>> > >>>> equipment vendor remains ongoing.
>> > >>>>
>> > >>>> 2021-05-03 14:11:40 GMT - The IP NOC reported a device has
>> > >>>> restarted multiple times. Collaborative troubleshooting with the
>> > >>>> equipment vendor remains ongoing. Field Operations are being
>> > >>>> engaged as well.
>> > >>>>
>> > >>>> 2021-05-03 13:51:28 GMT - The IP NOC continues to work in
>> > >>>> cooperation with the equipment vendor to troubleshoot the issue.
>> > >>>> We understand how important these services are to our customers
>> > >>>> and the issue has been escalated to the highest levels within
>> > >>>> Lumen Service Assurance Leadership.
>> > >>>>
>> > >>>> 2021-05-03 13:48:56 GMT - The IP NOC continues to work in
>> > >>>> cooperation with the equipment vendor to troubleshoot the issue.
>> > >>>>
>> > >>>> 2021-05-03 13:16:06 GMT - Cooperative investigations by multiple
>> > >>>> support groups remain underway at this time.
>> > >>>>
>> > >>>> 2021-05-03 13:13:13 GMT - The IP NOC is working in cooperation
>> > >>>> with Field Operations and the equipment vendor to troubleshoot
>> > >>>> the issue.
>> > >>>>
>> > >>>> 2021-05-03 12:50:33 GMT - Cooperative investigations by multiple
>> > >>>> support groups remain underway at this time.
>> > >>>>
>> > >>>> 2021-05-03 12:02:37 GMT - On May 03, 2021 at 11:46 GMT, Lumen
>> > >>>> identifieda service impact in Sao Paulo, Brazil. As this network
>> > >>>> fault is impacting multiple clients, the event has increased
>> > >>>> visibility with Lumen leadership.
>> > >>>> As such, client trouble tickets associated to this fault have
>> > >>>> been automatically escalated to higher priority.
>> > >>>> The IP Network Operation Center (NOC) is engaged and
>> > >>>> investigating in order to isolate the cause. Please be advised
>> > >>>> that updates for this event will be relayed at a minimum of
>> > >>>> hourly unless otherwise noted. The information conveyed hereafter
>> > >>>> is associated to live troubleshooting effort and as the discovery
>> > >>>> process evolves through to service resolution, ticket closure, or
>> > >>>> post incident review, details may evolve.
>> > >>>>
>> > >>>> 2021-05-03 10:27:08 GMT - Final Notification
>> > >>>>
>> > >>>> Start time: May 03, 2021 05:10 GMT Stop Time: May 03, 2021 05:43
>> > >>>> GMT Second Start Time: May 03, 2021 06:07 GMT Second Stop Time:
>> > >>>> May 03, 2021 07:23 GMT Root Cause: An equipment issue in Sao
>> > >>>> Paulo, Brazil impacted client services.
>> > >>>> Fix Action: The self-restarted, thus restoring services to a
>> > >>>> stable condition.
>> > >>>>
>> > >>>> This service impact has concluded. Due to the nature of this
>> > >>>> outage, it may be necessary to reset your services locally at
>> > >>>> your equipment.  If after that action has been performed a
>> > >>>> service issue prevails, please contact the Lumen Repair Center
>> > >>>> for troubleshooting assistance.
>> > >>>>
>> > >>>> Customers with trouble tickets associated to this network event
>> > >>>> do not need to open a separate Reason for Outage (RFO) ticket to
>> > >>>> request formal documentation. Please allow 3-5 business days for
>> > >>>> post incident analysisto conclude. Once final analysis has been
>> > >>>> completed, all customers associated to this network event will
>> > >>>> receive a formal RFO document via email from the Lumen RFO Team.
>> > >>>> Additionally, a satisfaction survey link may be available.
>> > >>>> We strive to provide thorough communications containing the
>> > >>>> available information during a service disruption. Please let us
>> > >>>> know if the updates you received during this event were
>> > >>>> satisfactory.
>> > >>>>
>> > >>>> 2021-05-03 10:22:43 GMT - Additional support was engaged to
>> > >>>> assist with investigations inconjunction with the equipment
>> > >>>> vendor. The equipment vendor has advised that services have
>> > >>>> remained stable since the device self-restarted for the second
>> > >>>> time. Field Operations verified the equipment and the vendor has
>> > >>>> advised that an on-going investigation into the reason for outage
>> > >>>> will continue; however, as services have remained stable, a final
>> > >>>> notification for the Event will be provided momentarily and the
>> > >>>> reason for outage will be provided in the Formal RFO document
>> > >>>> following the outage.
>> > >>>>
>> > >>>> 2021-05-03 09:28:18 GMT - Field Operations has been engaged to
>> > >>>> dispatch to a site in Cotia, Brazil to perform additional
>> > >>>> troubleshooting and to attempt a cold rest on the routing engine.
>> > >>>> No ETA has been provided at this time.
>> > >>>>
>> > >>>> 2021-05-03 08:43:22 GMT - The equipment vendor continues
>> > >>>> investigations into a suspected software issue at this time. The
>> > >>>> IP NOC continues to monitor for service stability as
>> > >>>> investigations progress.
>> > >>>>
>> > >>>> 2021-05-03 07:53:20 GMT - The equipment vendor continues
>> > >>>> isolation efforts at this time. The IP NOC has advised that
>> > >>>> services have self-restored again, but remain at risk of impact
>> > >>>> due to the equipment issue. The IP NOC will continue monitoring
>> > >>>> services for stability as isolation efforts continue.
>> > >>>>
>> > >>>> 2021-05-03 07:10:28 GMT - Investigations have determined that an
>> > >>>> equipment issue is the root cause of both outages. The equipment
>> > >>>> self-rebooted and restored; however, shortly thereafter the
>> > >>>> equipment failed again. The equipment vendor has been engaged and
>> > >>>> is investigating the cause of the equipment failure at this time.
>> > >>>>
>> > >>>> 2021-05-03 06:16:38 GMT - On May 03, 2021 at 06:07 GMT, Lumen
>> > >>>> identifieda service impact in Sao Paulo, Brazil. As this network
>> > >>>> fault is impacting multiple clients, the event has increased
>> > >>>> visibility with Lumen leadership.
>> > >>>> As such, client trouble tickets associated to this fault have
>> > >>>> been automatically escalated to higher priority.
>> > >>>>
>> > >>>> The IP Network Operation Center (NOC) is engaged and
>> > >>>> investigating in order to isolate the cause. Please be advised
>> > >>>> that updates for this event will be relayed at a minimum of
>> > >>>> hourly unless otherwise noted. The information conveyed hereafter
>> > >>>> is associated to live troubleshooting effort and as the discovery
>> > >>>> process evolves through to service resolution, ticket closure, or
>> > >>>> post incident review, details may evolve.
>> > >>>>
>> > >>>> 2021-05-03 05:54:41 GMT - Final Notification
>> > >>>>
>> > >>>> Start time: May 03, 2021 05:10 GMT Stop Time: May 03, 2021 05:43
>> > >>>> GMT
>> > >>>>
>> > >>>> Root Cause: An equipment issue in Sao Paulo, Brazil impacted
>> > >>>> client services.
>> > >>>> Fix Action: The self-restarted, thus restoring services to a
>> > >>>> stable condition.
>> > >>>>
>> > >>>> This service impact has concluded. Due to the nature of this
>> > >>>> outage, it may be necessary to reset your services locally at
>> > >>>> your equipment.  If after that action has been performed a
>> > >>>> service issue prevails, please contact the Lumen Repair Center
>> > >>>> for troubleshooting assistance.
>> > >>>>
>> > >>>> Customers with trouble tickets associated to this network event
>> > >>>> do not need to open a separate Reason for Outage (RFO) ticket to
>> > >>>> request formal documentation. Please allow 3-5 business days for
>> > >>>> post incident analysisto conclude. Once final analysis has been
>> > >>>> completed, all customers associated to this network event will
>> > >>>> receive a formal RFO document via email from the Lumen RFO Team.
>> > >>>> Additionally, a satisfaction survey link may be available.
>> > >>>> We strive to provide thorough communications containing the
>> > >>>> available information during a service disruption. Please let us
>> > >>>> know if the updates you received during this event were
>> > >>>> satisfactory.
>> > >>>>
>> > >>>> 2021-05-03 05:27:56 GMT - On May 03, 2021 at 05:10 GMT, Lumen
>> > >>>> identifieda service impact in Sao Paulo, Brazil. As this network
>> > >>>> fault is impacting multiple clients, the event has increased
>> > >>>> visibility with Lumen leadership.
>> > >>>> As such, client trouble tickets associated to this fault have
>> > >>>> been automatically escalated to higher priority.
>> > >>>>
>> > >>>> The IP Network Operation Center (NOC) is engaged and
>> > >>>> investigating in order to isolate the cause. Please be advised
>> > >>>> that updates for this event will be relayed at a minimum of
>> > >>>> hourly unless otherwise noted. The information conveyed hereafter
>> > >>>> is associated to live troubleshooting effort and as the discovery
>> > >>>> process evolves through to service resolution, ticket closure, or
>> > >>>> post incident review, details may evolve.
>> > >>>>
>> > >>>>
>> > >>>>
>> > >>>>
>> > >>>> att,
>> > >>>>
>> > >>>> On 03/05/2021 16:51, Alexandre Chaves Fonceca por (caiu) wrote:
>> > >>>>> sim, mesma coisa aqui..
>> > >>>>>
>> > >>>>> desde as 2 da madrugada.. chamado aberto e aguardando... pelas
>> > >>>>> informações estao com problemas em equipamentos mesmo.. e nao
>> > >>>>> sabem "ainda" o que ocorre e/ou pq..
>> > >>>>>
>> > >>>>> att,
>> > >>>>> Specialist Linux Solutions
>> > >>>>>
>> > >>>>> facebook <https://www.facebook.com/specialistlinux/>instagram
>> > >>>>> <https://www.instagram.com/specialistlinuxsolutions/>linkedin
>> > >>>>> <https://www.linkedin.com/company/specialist-linux-solutions/>wh
>> > >>>>> atsapp
>> >
>> > >>>>>
>> > >>>>> <https://api.whatsapp.com/send?phone=5511976158584>
>> > >>>>>
>> > >>>>> Alexandre Chaves Fonceca
>> > >>>>> Diretor,
>> > >>>>> Specialist Linux Solutions
>> > >>>>> E-mail:     alexandre em specialist.srv.br
>> > >>>>> <mailto:alexandre em specialist.srv.br>
>> > >>>>> Site:     www.specialist.srv.br <http://www.specialist.srv.br>
>> > >>>>> Telefone:     (11) 3317-5042
>> > >>>>>
>> > >>>>> On 03/05/2021 16:16, Edinilson J. Santos wrote:
>> > >>>>>> Está abrindo o chamado pelo Portal na URL:
>> > >>>>>> https://latam.my.level3.com/portal
>> > >>>>>> ?
>> > >>>>>>
>> > >>>>>> Pergunto pois, ultimamente, é praticamente a única coisa que
>> > >>>>>> funciona direito nessa empresa.
>> > >>>>>> Pelo menos envia emails regularmente informando a situação do
>> > >>>>>> ticket.
>> > >>>>>>
>> > >>>>>> Edinilson
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> Em 03/05/2021 15:05, Leonardo Porto por (caiu) escreveu:
>> > >>>>>>> Sofrendo aqui com mesmo problema, mas sem nenhuma informação
>> > >>>>>>> adicional.
>> > >>>>>>>
>> > >>>>>>> Leonardo
>> > >>>>>>>
>> > >>>>>>> Em 03/05/2021 15:02, Luiz Fernando Mizael Meier escreveu:
>> > >>>>>>>> Tarde!
>> > >>>>>>>>
>> > >>>>>>>> Com certeza alguém mais deve estar sofrendo com o tombo da
>> > >>>>>>>> Lumen/CenturyLink/Level3/GlobalCrossing/Impsat.
>> > >>>>>>>>
>> > >>>>>>>> Circuito inoperante desde 8:50 e me informaram a pouco que a
>> > >>>>>>>> única coisa que sabem é que foi em Curitiba.
>> > >>>>>>>>
>> > >>>>>>>> Alguém tem mais alguma informação?
>> > >>>>>>>>
>> > >>>>>>>> Luiz
>> > >>>>>>>> _______________________________________________
>> > >>>>>>>> caiu mailing list
>> > >>>>>>>> caiu em eng.registro.br
>> > >>>>>>>> https://eng.registro.br/mailman/listinfo/caiu
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>> > >>>>>>>>
>> > >>>>>>>> https://eng.registro.br/mailman/options/caiu
>> > >>>>>>> _______________________________________________
>> > >>>>>>> caiu mailing list
>> > >>>>>>> caiu em eng.registro.br
>> > >>>>>>> https://eng.registro.br/mailman/listinfo/caiu
>> > >>>>>>>
>> > >>>>>>>
>> > >>>>>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>> > >>>>>>>
>> > >>>>>>> https://eng.registro.br/mailman/options/caiu
>> > >>>>>>>
>> > >>>>>> _______________________________________________
>> > >>>>>> caiu mailing list
>> > >>>>>> caiu em eng.registro.br
>> > >>>>>> https://eng.registro.br/mailman/listinfo/caiu
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>> > >>>>>>
>> > >>>>>> https://eng.registro.br/mailman/options/caiu
>> > >>>>> _______________________________________________
>> > >>>>> caiu mailing list
>> > >>>>> caiu em eng.registro.br
>> > >>>>> https://eng.registro.br/mailman/listinfo/caiu
>> > >>>>>
>> > >>>>>
>> > >>>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>> > >>>>>
>> > >>>>> https://eng.registro.br/mailman/options/caiu
>> > >>>> _______________________________________________
>> > >>>> caiu mailing list
>> > >>>> caiu em eng.registro.br
>> > >>>> https://eng.registro.br/mailman/listinfo/caiu
>> > >>>>
>> > >>>>
>> > >>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>> > >>>>
>> > >>>> https://eng.registro.br/mailman/options/caiu
>> > >>>>
>> > >>> _______________________________________________
>> > >>> caiu mailing list
>> > >>> caiu em eng.registro.br
>> > >>> https://eng.registro.br/mailman/listinfo/caiu
>> > >>>
>> > >>>
>> > >>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>> > >>>
>> > >>> https://eng.registro.br/mailman/options/caiu
>> > >> _______________________________________________
>> > >> caiu mailing list
>> > >> caiu em eng.registro.br
>> > >> https://eng.registro.br/mailman/listinfo/caiu
>> > >>
>> > >>
>> > >> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>> > >>
>> > >> https://eng.registro.br/mailman/options/caiu
>> > >>
>> > >
>> > > _______________________________________________
>> > > caiu mailing list
>> > > caiu em eng.registro.br
>> > > https://eng.registro.br/mailman/listinfo/caiu
>> > >
>> > >
>> > > --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>> > >
>> > > https://eng.registro.br/mailman/options/caiu
>> > _______________________________________________
>> > caiu mailing list
>> > caiu em eng.registro.br
>> > https://eng.registro.br/mailman/listinfo/caiu
>> >
>> >
>> > --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>> >
>> > https://eng.registro.br/mailman/options/caiu
>> >
>> _______________________________________________
>> caiu mailing list
>> caiu em eng.registro.br
>> https://eng.registro.br/mailman/listinfo/caiu
>>
>>
>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>>
>> https://eng.registro.br/mailman/options/caiu
>>
>> _______________________________________________
>> caiu mailing list
>> caiu em eng.registro.br
>> https://eng.registro.br/mailman/listinfo/caiu
>>
>>
>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>>
>> https://eng.registro.br/mailman/options/caiu
>>
>