Major incident call

Person #24: Can you hear me? I think I'm hearing an echo. Hello? Person #50: Yes, Your Excellency, we can hear you. I think it's the bridge line. Everyone please mute when not— Person #33: Yeah, I hear you. It's fine. Person #29: We're all here. Person #24: Good. Yalla, let's go. Who's leading? Who's got the timeline? Person #50: So, Your Excellency, per the incident bridge protocol, I'll walk through the timeline. Incident declared at zero-seven-forty-two this morning. TAMM 4.0 production environment, mobile app and portal. Users reporting—initially login failures, then session drops for authenticated users. At peak we estimate... well, we're still calculating, but about sixty-three percent of active sessions were affected between zero-eight-hundred and zero-nine-fifteen. Person #24: Sixty-three percent. Ya Allah. That's—wait, session drops and login failures, same root cause? Person #50: That's what we're trying to confirm. The initial hypothesis was the ID provider, but the security team's early look showed the authentication service itself was healthy. The issue appears downstream. Person #49: No, the auth service was fine, I checked the logs. The token validation was passing. The problem was the session store. It was returning errors for about... forty seconds, then recovering, then failing again. Like a loop. Person #24: Session store. We didn't change anything there. Did we? Last night was a freeze. The freeze started— Person #33: Your Excellency, we had a change scheduled for last night. The AI assistant journey config, the new prompt templates for the Eid greetings. That was approved and went in at—what time—zero-one-thirty. Person #29: That's true, that was approved. I have the ticket. CHG-24817. But that was a config change to the assistant service, not the session store. Different pod, different namespace. Person #49: But it shares the same Redis cluster. If that config push triggered a failover, or if there was a memory spike— Person #50: Hold on. Let me just—the change management record shows the change was approved. But we also have an unapproved change that went in around zero-two-fifteen. A vendor pushed a hotfix to the API gateway. That's not in the change calendar. Person #24: What? Who? Which vendor? Person #50: The API gateway vendor, Your Excellency. Their support team, apparently responding to a ticket we didn't open. They pushed a WAF rule update. No change record, no communication to us. Person #29: Yaani, they just—they logged into the gateway and made changes? Who has those credentials? Person #50: That's the question we're working. Per our access policy, only the platform team and myself should have production gateway access. The vendor has a break-glass account, but it's supposed to be sealed and only used with a two-person sign-off. Person #44: The break-glass account was used. At zero-two-fourteen. I saw the audit log entry. They authenticated from a vendor IP, loaded the new rule set, and the gateway restarted. That restart is what dropped the session store connections. Person #24: So the vendor caused this. We didn't cause this. We're in a freeze, ya jamaa, and the vendor pushed a change in the middle of the night. Person #50: That's the current assessment, Your Excellency. We're still validating, but the timeline fits. Gateway restart at zero-two-fourteen, session store errors starting zero-two-forty, incident alerts at zero-three-oh-five. Person #33: But wait—if the gateway restarted, why would it take twenty-five minutes for the session store to start failing? Shouldn't that be immediate? Person #49: Because the gateway doesn't connect to the session store directly. It's the backend services that hold the session. When the gateway restarted, it dropped all the connection pools, and the backends tried to reconnect. But the session store had its own issues—it was already under memory pressure. We had a leak. I flagged it last week. Person #37: The memory leak—is that related to the new journey flows? The ones that went live after GITEX? Person #49: No, it's the session cleanup job. It's been running slow for two weeks. We kept pushing it down the backlog. And now with the reconnect storm— Person #24: Wait, wait. You're telling me we have a memory leak in production that's been known for two weeks and nobody escalated it? Person #49: It was in the backlog, Your Excellency. It was scheduled for the next release. We didn't think it was urgent. It was slow, not broken. Person #37: Where was the escalation? Did anyone flag this to the change advisory board? Person #50: I'll take that. It was on the weekly operations review, but it wasn't escalated to CAB. It wasn't marked as a production risk. That's a process gap on our side. Person #24: Okay. Okay. Let's focus. What's the status now? Is it stable? Person #50: As of ten-forty, we rolled back the vendor's WAF rule set. The gateway is back to the previous configuration. Session store errors have dropped to near zero. We're monitoring. Person #33: So it's basically fixed? The rollback solved it? Person #49: The immediate issue is fixed. But we still have the memory leak. And the reconnect storm—the backends have been reconnecting all morning, so they're not fully stable. I'd say we're at ninety percent, not a hundred. Person #24: Ninety percent is not a hundred. When do we get to a hundred? Person #49: If we restart the session store cleanly, we can clear the memory. That would take about five minutes. But it would drop all active sessions. Everyone would have to log in again. Person #33: That's bad. It's Friday. People are using the app for the weekend. And we have the Eid greeting push notifications going out at noon. If people open the app and have to log in again— Person #24: Eid greetings. Right. That's today at noon. Who's pushing that? Person #33: The CX team. They scheduled it through the engagement service. It's automated, goes out at twelve sharp. But if sessions are dropping— Person #29: The push notifications don't require an active session. They go through the notification service, which is separate. Users will get the notification. When they tap it, they'll need to authenticate. If we restart the session store after the push goes out, it's fine. Person #33: But if we restart before noon, then users who got the notification at noon will open the app and see the login screen. That's a terrible experience. The CX team will have my head. Person #24: The user experience matters, but stability matters more. What does the monitoring look like right now? Are we trending stable? Person #50: Yes, Your Excellency. Error rates are down to baseline. Latency is... slightly elevated but within SLO. The memory leak is still there, but it's not accelerating. We could hold for a few hours if we needed to. Person #24: So we hold until after the push. Then we do a controlled restart. Inshallah, that's the plan. Person #49: I'd rather do the restart sooner. The leak is a time bomb. If we get another spike— Person #24: What would cause another spike? The push notifications go out at noon. If a hundred thousand people open the app at once— Person #49: That's exactly the spike I'm talking about. The notification goes out, people open the app, they create new sessions, the session store has to write all those sessions, and with the leak, it'll run out of memory faster. We could see another incident at twelve-thirty. Person #24: So we either restart now and accept the login screen, or we restart after the push and risk a spike that causes another incident. Person #37: Is there a middle option? Can we scale up the session store? Add more memory before the push? Person #49: We can add nodes. But the leak is in the cleanup job, so adding nodes just postpones the problem. It might be enough to get us through the afternoon. Person #24: How long does it take to add nodes? Person #50: We have the automation. We can scale out in about fifteen minutes. But we'd need approval from the cloud operations team—we have a quota, and the current allocation is at seventy percent. Person #37: I can talk to the cloud team. We have a standing agreement for capacity during incidents. I'll get that started. Person #24: Do it. And then after the push, we do the restart. Ya jamaa, let's make sure we're communicating this properly. Do we have a status page update? Person #50: We posted an initial status at zero-eight-fifteen. We updated it at zero-nine-thirty when we rolled back the gateway. We should post another update now that we're monitoring stable. Person #24: What does it say? Person #50: Some users may have experienced login issues. The issue has been resolved. We are monitoring." That's the latest. Person #24: That's not enough. People are going to ask what happened. The press might pick it up. We need to be more transparent. Person #50: I can draft an update that says we experienced a technical issue with our authentication systems, which was identified and resolved. We apologize for any inconvenience. We're monitoring to ensure stability. Person #24: Fine. But add that we're investigating the root cause and will provide a full report. We need to show we're on top of it. Person #40: Your Excellency, before we continue—I want to flag that the vendor needs to be held accountable. This is the second time they've made an unapproved change. Last time it was a config tweak on the document management system. We had a call about it. We agreed they'd only touch production with our written approval. Person #24: Is that true? This is the second time? Person #40: Yes, Your Excellency. In July. They changed a document retention policy without approval. It caused a compliance issue. We had to restore from backup. We brought it to the vendor review board, but I don't think any formal action was taken. Person #37: I remember that. We sent them a warning letter. But we didn't revoke their break-glass access. That was a mistake. Person #50: I'll take responsibility for that. The access review was scheduled for Q3. It wasn't completed. The break-glass account should have been rotated after the July incident. Person #24: I want that access revoked today. And I want a formal incident report sent to the vendor's regional office. Not their support team. Their management. Person #40: I'll draft it. I'll send it to you for approval before it goes out. Person #24: Good. And I want to know—why did they make this change? What were they trying to fix? Person #50: Their support team said they detected a spike in requests to the gateway. They thought it was an attack. They pushed a WAF rule to block the traffic. But the spike was actually from our own load testing—the QA team was running a stress test on the sandbox environment, but the traffic was routed through the production gateway due to a misconfiguration. Person #24: Wait. Our QA team was load testing, and their traffic went through production? Person #29: That's what I want to understand. The sandbox environment should be isolated. How did traffic from the sandbox route through the production gateway? Person #50: We're investigating. The initial finding is that the load testing tool was configured with the production gateway URL instead of the sandbox URL. It was a configuration error in the test script. Person #33: Whose test script? Who ran it? Person #50: The QA team. They were testing the Eid greeting push—the load that the notification service would generate. They set up the test this morning, and they used the production URL by mistake. Person #24: So the vendor saw traffic, thought it was an attack, and pushed a WAF rule without telling us. And our QA team accidentally sent traffic to production. This is a comedy of errors. Person #29: But the vendor still violated the change freeze. Even if they saw suspicious traffic, the protocol is to alert us, not to make changes. They have our on-call number. Person #40: Exactly. We have a clear escalation path. They didn't use it. They bypassed the process and made a change. That's the core issue. Person #44: To be fair, the WAF rule they pushed was actually effective. It blocked the load testing traffic. If they hadn't pushed it, the gateway might have been overwhelmed. Person #49: No, it wouldn't have been overwhelmed. The load test was generating maybe two thousand requests per second. The gateway can handle ten times that. The vendor overreacted to a non-threat. Person #44: I'm just saying, the rule they pushed wasn't malicious. It was a misjudgment, not a deliberate act of sabotage. Person #40: I don't care if it was malicious or not. They violated the freeze. They violated our trust. They need to face consequences. Person #24: Okay, enough. We'll deal with the vendor later. Right now, I need to know: what's the plan for the next two hours? We have the push at noon. We need to make sure the system is stable. What are the immediate actions? Person #50: First, we're scaling out the session store. That will give us headroom for the push. Second, we're monitoring the error rates and latency. Third, we're preparing for the controlled restart after the push. Person #24: Who's doing what? Assign owners. Person #50: I'll coordinate the scale-out with the cloud operations team. Ya3ni—let me just confirm, Faisal will handle the infrastructure, and I'll monitor the metrics. The restart after the push will be led by the engineering team. I'll make sure the approvals are in place. Person #29: I'll coordinate with the QA team to make sure they don't run any more tests against production. And I'll update the incident timeline. Person #33: I'll handle communications with the CX team. They're going to want to know if the push is affected. Person #24: Good. And I want a full postmortem by Monday. No exceptions. The vendor incident, the QA misconfiguration, the memory leak that wasn't escalated—I want all of it in the report. Person #49: Can I just say, the memory leak wasn't a surprise. I flagged it in the standup last week. But we had a sprint commitment, and the product team didn't want to prioritize it. Person #33: That's not fair. You flagged it as a minor issue. You said it was a cleanup job that was slow. You didn't say it would cause an outage. Person #49: I said it was a memory leak in the session store. I said it would become critical if we didn't fix it. I even put it in the risk register. Person #33: The risk register is a graveyard. Nobody reads it. Person #24: Enough. The postmortem will cover this. We'll figure out the process failures then. Right now, I need everyone focused on the next two hours. Person #50: Your Excellency, I have a request. We need to prepare a slide for the Executive Council. The Chairman's office asked for a brief update on the incident before the noon push. Nothing detailed, just a status summary. Person #24: A slide. By noon? We're in the middle of an incident. Person #50: Yes, Your Excellency. The Chairman's office said it's for a briefing with the Executive Council. They want to know what happened, what we're doing, and when it will be resolved. Person #24: Fine. Who can write it? I don't have time to draft a slide right now. Person #33: I can draft it. I'll keep it simple: what happened, what we fixed, what we're monitoring. I can have it ready in twenty minutes. Person #24: Do it. Send it to me for review before it goes out. And keep it honest. Don't overclaim. We're not fully stable yet. Person #33: Got it. I'll draft it now. I'll keep it to three bullet points. Person #24: Yalla. Let's stay on this call. I want updates every fifteen minutes. Who's next? Person #29: I've confirmed with the QA team. They've stopped the load test. They're reviewing their test scripts to make sure the sandbox URL is correct. They'll provide a written confirmation by end of day. Person #24: Good. What about the vendor? Have we contacted them? Person #40: I'm drafting the email now. I'll send it after we've revoked their access. I don't want to give them a head start on a response before we've contained the situation. Person #50: Access revocation is in progress. I'm coordinating with the IAM team. It should be complete within the hour. Person #24: Make sure it's done before the push. We don't want them making any more changes during the peak. Person #37: The scale-out is underway. I've spoken with the cloud team. They're adding three nodes to the session store. ETA, fifteen minutes. Person #24: Good. And the restart? When do we do it? Person #50: We'll wait for the push to complete. That's twelve-oh-five, twelve-ten. Then we'll do a rolling restart of the session store nodes. We can do it one node at a time to minimize impact. Users might see a brief interruption, but it should be less than a minute. Person #49: A rolling restart won't fully clear the memory. We need to restart all nodes to clear the leak. A rolling restart will just shift the load. Person #50: That's true. But a full restart means all sessions drop. We're trying to minimize user impact. Person #49: The leak is in the cleanup job. It's not in the session data itself. We can restart the cleanup job without dropping sessions. That would clear the memory. Person #24: Wait, is that true? We can fix the leak without dropping sessions? Person #49: Yes. The cleanup job is a separate process. We can restart just that process. It'll clear the memory that's been allocated to the job. The session data is stored in a different part of the memory space. Person #24: Why didn't you say that earlier? That's the solution. Restart the cleanup job, clear the memory, no user impact. Person #49: I was about to suggest it, but we were discussing the vendor and the push. The restart of the cleanup job is a simple operation. It takes about two minutes. Person #24: Do it. Now. And let's monitor the memory usage. If it comes back, we know the leak is worse than we thought. Person #49: I'll do it now. I'll report back in five minutes. Person #24: Good. So we're doing a—wait, what do we call this? We're doing a cleanup job restart, not a full restart. That's the plan. Person #50: Yes, Your Excellency. That's the plan. We'll restart the cleanup job, monitor memory, and then proceed with the push at noon as scheduled. Person #33: And if the memory leak comes back? We'll have to do a full restart anyway. Person #24: Then we'll cross that bridge. But for now, we have a plan. Let's execute. Person #37: I want to add something. After this incident, we need to review our vendor access policies across all platforms. Not just TAMM. All our common platforms. ADERP, ServiceNow, the document management system. If vendors can push changes without approval, we have a systemic risk. Person #24: Agreed. Add it to the postmortem. And I want a vendor access review completed by the end of next week. No extensions. Person #40: I'll lead that. We'll review all break-glass accounts, all vendor support access, and all change management records from the past quarter. Person #50: I'll support with the access logs. We have the audit trail for all production environments. Person #24: Good. Anything else? Any other issues we need to address before the push? Person #50: One more thing. The status page. We need to update it to say we're investigating the root cause and will provide a full report. The current message says the issue is resolved, which is not entirely accurate. Person #24: Agreed. Update the status page. Accuracy first. We do not want the Chairman seeing "resolved" when it's not. Khalas. Person #29: Your Excellency, just to confirm—the noon push. We have the team standing by. If the cleanup job holds, we go as scheduled. I'll send the go/no-go at 11:45. Person #24: Perfect. And let's make sure we have someone from the vendor on the line during the push. Not just on standby. Actually on the call. I want them to see what happens live. Person #40: I'll make sure. They've been cooperative today, but I want their change lead to explain in writing why this was pushed without approval. I want it in the postmortem. Person #37: And I want it signed. Not just an email. A formal acknowledgment of the breach. This goes to their account manager and their regional director. Person #50: I'll draft the language and send it to you both for review before we send anything officially. Person #24: Good. Anything else on the push itself? Person #49, are you seeing any improvement yet? Person #49: I'm looking at the memory graph now. It's stabilizing. The cleanup job is holding. I'll still do a full report in ten minutes, but early signs are good, Your Excellency. Person #24: Mashallah. Okay. So we have a plan, we have owners, we have a timeline. Let's not let this happen again. This is the second vendor incident this quarter. Person #33: And it won't be the last unless we automate the change approval. I'm just saying—if we had a CI/CD gate that blocked unapproved pushes, this never happens. We can build it in two weeks. Person #24: You're not wrong. But let's get through today first. Add it to the postmortem as a recommendation. We'll discuss next week. Person #33: Fair enough, boss. I'll put it in the doc. Person #44: Just one thing. In the postmortem, we should include the fact that the vendor's change was not just unapproved—it was also not tested against the staging environment. We have evidence of that in the logs. Person #24: Noted. Thank you for that. Okay, I think we're done here. Anything else before we wrap? Person #37: One last thing. The Chairman's office has requested a one-slide summary of this incident for the Executive Council meeting on Sunday. Can someone own that? Person #24: I'll take it. I'll send it to you for review by end of day. Yalla, everyone. Good work today. Let's hold the line. We'll reconvene at 11:45 for the go/no-go. Shukran.