[caiu] Indisponibilidade Lúmen

Qua Maio 5 17:00:05 -03 2021

Várias oscilações hoje com a Lumen na Região de Campinas e São Paulo. Mais
alguém?

  --

*André Kiyohiko Osaki*

On Tue, May 4, 2021 at 3:29 PM Leonardo Porto por (caiu) <
caiu em eng.registro.br> wrote:

> Aqui também, mesmo horário.
>
> Leonardo
>
> Em 04/05/2021 15:23, Edinilson J. Santos escreveu:
> > Aqui sim, ontem, 03/05/2021 por volta das 19:00hs (antes disso, ficava
> > indo e voltando).
> >
> > Edinilson
> >
> >
> > Em 04/05/2021 08:52, noc em consuldata.com.br escreveu:
> >> Bom dia. O link normalizou para vocês?
> >>
> >> Att,
> >>
> >>
> >> Everton Araujo
> >>
> >> On 5/3/21 5:24 PM, Eduardo Duarte wrote:
> >>> Aqui igual
> >>>
> >>> Jundiaí SP
> >>>
> >>> O link hoje parece pisca pisca de árvore de Natal
> >>>
> >>> Em seg., 3 de mai. de 2021 17:00, Alexandre Chaves Fonceca por (caiu) <
> >>> caiu em eng.registro.br> escreveu:
> >>>
> >>>> essas sao as informações oficiais deles... em ordem cronologica (de
> >>>> baixo para cima), e em horario GMT 0
> >>>>
> >>>> 2021-05-03 19:53:47 GMT - The IP NOC observed another reboot,
> >>>> investigation and troubleshooting efforts remain ongoing.
> >>>>
> >>>> 2021-05-03 19:37:11 GMT - The IP NOC is reporting all affected alarms
> >>>> remain clear and will continue to monitor and investigate the fault
> >>>> for the
> >>>> next 4 hours. As such, the next notification will be provided at
> >>>> 23:45 GMT
> >>>> or as additional information is obtained.
> >>>>
> >>>> 2021-05-03 18:49:21 GMT - The IP NOC reports that the Sao Paulo,
> >>>> Brazil
> >>>> equipment has been restored to a full redundancy condition. Traffic
> >>>> was
> >>>> migrated off the redundant uplink and remains stable. The IP NOC will
> >>>> continue to monitor the services. IP Engineering and the equipment
> >>>> vendor
> >>>> are still collaboratively investigating the cause for the service
> >>>> impact.
> >>>>
> >>>> 2021-05-03 17:53:10 GMT - The IP NOC reports that the impacted
> >>>> equipment
> >>>> in Sao Paulo, Brazil is still stable since the redundant routing
> >>>> path was
> >>>> taken out of service. Services continue to remain stable as well.
> >>>> Cooperative investigations by multiple support groups along with the
> >>>> equipment vendor remain underway.
> >>>>
> >>>> 2021-05-03 16:52:08 GMT - The IP NOC remotely migrated impacted
> >>>> services
> >>>> to a redundant uplink, however some services have continued to
> >>>> bounce in
> >>>> and out of service. The IP NOC, Engineering teams, Field Operations
> >>>> and the
> >>>> equipment vendor continue investigating the the cause for the service
> >>>> impact.
> >>>>
> >>>> 2021-05-03 15:55:43 GMT - Collaborative troubleshooting with the
> >>>> equipment
> >>>> vendor, Field Operations and the IP NOC remain ongoing.
> >>>>
> >>>> 2021-05-03 15:07:56 GMT - Field Operations is onsite at the Sao Paulo
> >>>> fault site. Collaborative troubleshooting with the equipment vendor
> >>>> remains
> >>>> ongoing.
> >>>>
> >>>> 2021-05-03 14:11:40 GMT - The IP NOC reported a device has restarted
> >>>> multiple times. Collaborative troubleshooting with the equipment
> >>>> vendor
> >>>> remains ongoing. Field Operations are being engaged as well.
> >>>>
> >>>> 2021-05-03 13:51:28 GMT - The IP NOC continues to work in
> >>>> cooperation with
> >>>> the equipment vendor to troubleshoot the issue. We understand how
> >>>> important
> >>>> these services are to our customers and the issue has been
> >>>> escalated to the
> >>>> highest levels within Lumen Service Assurance Leadership.
> >>>>
> >>>> 2021-05-03 13:48:56 GMT - The IP NOC continues to work in
> >>>> cooperation with
> >>>> the equipment vendor to troubleshoot the issue.
> >>>>
> >>>> 2021-05-03 13:16:06 GMT - Cooperative investigations by multiple
> >>>> support
> >>>> groups remain underway at this time.
> >>>>
> >>>> 2021-05-03 13:13:13 GMT - The IP NOC is working in cooperation with
> >>>> Field
> >>>> Operations and the equipment vendor to troubleshoot the issue.
> >>>>
> >>>> 2021-05-03 12:50:33 GMT - Cooperative investigations by multiple
> >>>> support
> >>>> groups remain underway at this time.
> >>>>
> >>>> 2021-05-03 12:02:37 GMT - On May 03, 2021 at 11:46 GMT, Lumen
> >>>> identifieda
> >>>> service impact in Sao Paulo, Brazil. As this network fault is
> >>>> impacting
> >>>> multiple clients, the event has increased visibility with Lumen
> >>>> leadership.
> >>>> As such, client trouble tickets associated to this fault have been
> >>>> automatically escalated to higher priority.
> >>>> The IP Network Operation Center (NOC) is engaged and investigating in
> >>>> order to isolate the cause. Please be advised that updates for this
> >>>> event
> >>>> will be relayed at a minimum of hourly unless otherwise noted. The
> >>>> information conveyed hereafter is associated to live
> >>>> troubleshooting effort
> >>>> and as the discovery process evolves through to service resolution,
> >>>> ticket
> >>>> closure, or post incident review, details may evolve.
> >>>>
> >>>> 2021-05-03 10:27:08 GMT - Final Notification
> >>>>
> >>>> Start time: May 03, 2021 05:10 GMT
> >>>> Stop Time: May 03, 2021 05:43 GMT
> >>>> Second Start Time: May 03, 2021 06:07 GMT
> >>>> Second Stop Time: May 03, 2021 07:23 GMT
> >>>> Root Cause: An equipment issue in Sao Paulo, Brazil impacted client
> >>>> services.
> >>>> Fix Action: The self-restarted, thus restoring services to a stable
> >>>> condition.
> >>>>
> >>>> This service impact has concluded. Due to the nature of this
> >>>> outage, it
> >>>> may be necessary to reset your services locally at your equipment.  If
> >>>> after that action has been performed a service issue prevails, please
> >>>> contact the Lumen Repair Center for troubleshooting assistance.
> >>>>
> >>>> Customers with trouble tickets associated to this network event do not
> >>>> need to open a separate Reason for Outage (RFO) ticket to request
> >>>> formal
> >>>> documentation. Please allow 3-5 business days for post incident
> >>>> analysisto
> >>>> conclude. Once final analysis has been completed, all customers
> >>>> associated
> >>>> to this network event will receive a formal RFO document via email
> >>>> from the
> >>>> Lumen RFO Team. Additionally, a satisfaction survey link may be
> >>>> available.
> >>>> We strive to provide thorough communications containing the available
> >>>> information during a service disruption. Please let us know if the
> >>>> updates
> >>>> you received during this event were satisfactory.
> >>>>
> >>>> 2021-05-03 10:22:43 GMT - Additional support was engaged to assist
> >>>> with
> >>>> investigations inconjunction with the equipment vendor. The equipment
> >>>> vendor has advised that services have remained stable since the device
> >>>> self-restarted for the second time. Field Operations verified the
> >>>> equipment
> >>>> and the vendor has advised that an on-going investigation into the
> >>>> reason
> >>>> for outage will continue; however, as services have remained stable, a
> >>>> final notification for the Event will be provided momentarily and the
> >>>> reason for outage will be provided in the Formal RFO document
> >>>> following the
> >>>> outage.
> >>>>
> >>>> 2021-05-03 09:28:18 GMT - Field Operations has been engaged to
> >>>> dispatch to
> >>>> a site in Cotia, Brazil to perform additional troubleshooting and to
> >>>> attempt a cold rest on the routing engine. No ETA has been provided
> >>>> at this
> >>>> time.
> >>>>
> >>>> 2021-05-03 08:43:22 GMT - The equipment vendor continues
> >>>> investigations
> >>>> into a suspected software issue at this time. The IP NOC continues to
> >>>> monitor for service stability as investigations progress.
> >>>>
> >>>> 2021-05-03 07:53:20 GMT - The equipment vendor continues isolation
> >>>> efforts
> >>>> at this time. The IP NOC has advised that services have self-restored
> >>>> again, but remain at risk of impact due to the equipment issue. The
> >>>> IP NOC
> >>>> will continue monitoring services for stability as isolation efforts
> >>>> continue.
> >>>>
> >>>> 2021-05-03 07:10:28 GMT - Investigations have determined that an
> >>>> equipment
> >>>> issue is the root cause of both outages. The equipment
> >>>> self-rebooted and
> >>>> restored; however, shortly thereafter the equipment failed again. The
> >>>> equipment vendor has been engaged and is investigating the cause of
> >>>> the
> >>>> equipment failure at this time.
> >>>>
> >>>> 2021-05-03 06:16:38 GMT - On May 03, 2021 at 06:07 GMT, Lumen
> >>>> identifieda
> >>>> service impact in Sao Paulo, Brazil. As this network fault is
> >>>> impacting
> >>>> multiple clients, the event has increased visibility with Lumen
> >>>> leadership.
> >>>> As such, client trouble tickets associated to this fault have been
> >>>> automatically escalated to higher priority.
> >>>>
> >>>> The IP Network Operation Center (NOC) is engaged and investigating in
> >>>> order to isolate the cause. Please be advised that updates for this
> >>>> event
> >>>> will be relayed at a minimum of hourly unless otherwise noted. The
> >>>> information conveyed hereafter is associated to live
> >>>> troubleshooting effort
> >>>> and as the discovery process evolves through to service resolution,
> >>>> ticket
> >>>> closure, or post incident review, details may evolve.
> >>>>
> >>>> 2021-05-03 05:54:41 GMT - Final Notification
> >>>>
> >>>> Start time: May 03, 2021 05:10 GMT
> >>>> Stop Time: May 03, 2021 05:43 GMT
> >>>>
> >>>> Root Cause: An equipment issue in Sao Paulo, Brazil impacted client
> >>>> services.
> >>>> Fix Action: The self-restarted, thus restoring services to a stable
> >>>> condition.
> >>>>
> >>>> This service impact has concluded. Due to the nature of this
> >>>> outage, it
> >>>> may be necessary to reset your services locally at your equipment.  If
> >>>> after that action has been performed a service issue prevails, please
> >>>> contact the Lumen Repair Center for troubleshooting assistance.
> >>>>
> >>>> Customers with trouble tickets associated to this network event do not
> >>>> need to open a separate Reason for Outage (RFO) ticket to request
> >>>> formal
> >>>> documentation. Please allow 3-5 business days for post incident
> >>>> analysisto
> >>>> conclude. Once final analysis has been completed, all customers
> >>>> associated
> >>>> to this network event will receive a formal RFO document via email
> >>>> from the
> >>>> Lumen RFO Team. Additionally, a satisfaction survey link may be
> >>>> available.
> >>>> We strive to provide thorough communications containing the available
> >>>> information during a service disruption. Please let us know if the
> >>>> updates
> >>>> you received during this event were satisfactory.
> >>>>
> >>>> 2021-05-03 05:27:56 GMT - On May 03, 2021 at 05:10 GMT, Lumen
> >>>> identifieda
> >>>> service impact in Sao Paulo, Brazil. As this network fault is
> >>>> impacting
> >>>> multiple clients, the event has increased visibility with Lumen
> >>>> leadership.
> >>>> As such, client trouble tickets associated to this fault have been
> >>>> automatically escalated to higher priority.
> >>>>
> >>>> The IP Network Operation Center (NOC) is engaged and investigating in
> >>>> order to isolate the cause. Please be advised that updates for this
> >>>> event
> >>>> will be relayed at a minimum of hourly unless otherwise noted. The
> >>>> information conveyed hereafter is associated to live
> >>>> troubleshooting effort
> >>>> and as the discovery process evolves through to service resolution,
> >>>> ticket
> >>>> closure, or post incident review, details may evolve.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> att,
> >>>>
> >>>> On 03/05/2021 16:51, Alexandre Chaves Fonceca por (caiu) wrote:
> >>>>> sim, mesma coisa aqui..
> >>>>>
> >>>>> desde as 2 da madrugada.. chamado aberto e aguardando... pelas
> >>>>> informações estao com problemas em equipamentos mesmo.. e nao sabem
> >>>>> "ainda" o que ocorre e/ou pq..
> >>>>>
> >>>>> att,
> >>>>> Specialist Linux Solutions
> >>>>>
> >>>>> facebook <https://www.facebook.com/specialistlinux/>instagram
> >>>>> <https://www.instagram.com/specialistlinuxsolutions/>linkedin
> >>>>> <https://www.linkedin.com/company/specialist-linux-solutions/>whatsapp
>
> >>>>>
> >>>>> <https://api.whatsapp.com/send?phone=5511976158584>
> >>>>>
> >>>>> Alexandre Chaves Fonceca
> >>>>> Diretor,
> >>>>> Specialist Linux Solutions
> >>>>> E-mail:     alexandre em specialist.srv.br
> >>>>> <mailto:alexandre em specialist.srv.br>
> >>>>> Site:     www.specialist.srv.br <http://www.specialist.srv.br>
> >>>>> Telefone:     (11) 3317-5042
> >>>>>
> >>>>> On 03/05/2021 16:16, Edinilson J. Santos wrote:
> >>>>>> Está abrindo o chamado pelo Portal na URL:
> >>>>>> https://latam.my.level3.com/portal
> >>>>>> ?
> >>>>>>
> >>>>>> Pergunto pois, ultimamente, é praticamente a única coisa que
> >>>>>> funciona
> >>>>>> direito nessa empresa.
> >>>>>> Pelo menos envia emails regularmente informando a situação do
> >>>>>> ticket.
> >>>>>>
> >>>>>> Edinilson
> >>>>>>
> >>>>>>
> >>>>>> Em 03/05/2021 15:05, Leonardo Porto por (caiu) escreveu:
> >>>>>>> Sofrendo aqui com mesmo problema, mas sem nenhuma informação
> >>>>>>> adicional.
> >>>>>>>
> >>>>>>> Leonardo
> >>>>>>>
> >>>>>>> Em 03/05/2021 15:02, Luiz Fernando Mizael Meier escreveu:
> >>>>>>>> Tarde!
> >>>>>>>>
> >>>>>>>> Com certeza alguém mais deve estar sofrendo com o tombo da
> >>>>>>>> Lumen/CenturyLink/Level3/GlobalCrossing/Impsat.
> >>>>>>>>
> >>>>>>>> Circuito inoperante desde 8:50 e me informaram a pouco que a única
> >>>>>>>> coisa
> >>>>>>>> que sabem é que foi em Curitiba.
> >>>>>>>>
> >>>>>>>> Alguém tem mais alguma informação?
> >>>>>>>>
> >>>>>>>> Luiz
> >>>>>>>> _______________________________________________
> >>>>>>>> caiu mailing list
> >>>>>>>> caiu em eng.registro.br
> >>>>>>>> https://eng.registro.br/mailman/listinfo/caiu
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
> >>>>>>>>
> >>>>>>>> https://eng.registro.br/mailman/options/caiu
> >>>>>>> _______________________________________________
> >>>>>>> caiu mailing list
> >>>>>>> caiu em eng.registro.br
> >>>>>>> https://eng.registro.br/mailman/listinfo/caiu
> >>>>>>>
> >>>>>>>
> >>>>>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
> >>>>>>>
> >>>>>>> https://eng.registro.br/mailman/options/caiu
> >>>>>>>
> >>>>>> _______________________________________________
> >>>>>> caiu mailing list
> >>>>>> caiu em eng.registro.br
> >>>>>> https://eng.registro.br/mailman/listinfo/caiu
> >>>>>>
> >>>>>>
> >>>>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
> >>>>>>
> >>>>>> https://eng.registro.br/mailman/options/caiu
> >>>>> _______________________________________________
> >>>>> caiu mailing list
> >>>>> caiu em eng.registro.br
> >>>>> https://eng.registro.br/mailman/listinfo/caiu
> >>>>>
> >>>>>
> >>>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
> >>>>>
> >>>>> https://eng.registro.br/mailman/options/caiu
> >>>> _______________________________________________
> >>>> caiu mailing list
> >>>> caiu em eng.registro.br
> >>>> https://eng.registro.br/mailman/listinfo/caiu
> >>>>
> >>>>
> >>>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
> >>>>
> >>>> https://eng.registro.br/mailman/options/caiu
> >>>>
> >>> _______________________________________________
> >>> caiu mailing list
> >>> caiu em eng.registro.br
> >>> https://eng.registro.br/mailman/listinfo/caiu
> >>>
> >>>
> >>> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
> >>>
> >>> https://eng.registro.br/mailman/options/caiu
> >> _______________________________________________
> >> caiu mailing list
> >> caiu em eng.registro.br
> >> https://eng.registro.br/mailman/listinfo/caiu
> >>
> >>
> >> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
> >>
> >> https://eng.registro.br/mailman/options/caiu
> >>
> >
> > _______________________________________________
> > caiu mailing list
> > caiu em eng.registro.br
> > https://eng.registro.br/mailman/listinfo/caiu
> >
> >
> > --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
> >
> > https://eng.registro.br/mailman/options/caiu
> _______________________________________________
> caiu mailing list
> caiu em eng.registro.br
> https://eng.registro.br/mailman/listinfo/caiu
>
>
> --> PARA SAIR DA LISTA SIGA AS INSTRUÇÕES em:
>
> https://eng.registro.br/mailman/options/caiu
>