welcome everyone thank you so much for joining us today um super excited to to have you all here um for those of you that don't know me I'm Chris one of the co-founders um and CPO here at incident IO and this is the second in our four-part series on redefining Incident Management So today we're going to be talking about the incident way which is like broadly our philosophy on Incident Management um operational excellence and sort of broken down into 10 easy to consume hopefully principles on sort of how we see the world of Incident Management
done really well um and this sort of comes from a mixture of our own experience we have been in many in incident um but also from the experience of our our customers who like represent some of the most like recognized brands in in the world um so I guess like on to running order for the day um so today we have a panel discussion we're going to be joined by uh Stephen um one of my co-founders and the CEO here at incident IO um and we'll be joined by uh mark vanika uh from Checker trade
who is one of our customers um we'll get to intros shortly um first just a tiny bit of housekeeping so the event today is going to last um approximately 45 minutes um a reminder that we can't see or hear uh the audience um but please do use the the comments in the side and the Q&A to ask any questions as you go through um so we'll do around like 30 minutes of on the panel discussion and then we will open up and sort of take some of those questions and and ask them of Mark and
Steven um as well um usually we have like way more questions than we can get to on these events um which is totally fine and that shouldn't stop you from submitting um but if we don't get to your question we will try and follow up in the sort of post event uh followup that we uh that we do here so I guess like without further Ado let's um bring uh Stephen and Mark on stage uh as if by Magic um great to have you here um I guess before we dive into panel discussion let's do
a quick round of intros um maybe Mark do you want to kick us off with a brief int yourself yes sure so I'm I'm Mark fiker I am an engineering manager at czecher trade where I am responsible for our platform and our inser management overall um for the people outside the UK you might not know us as well but if you're in the UK everybody in the UK knows the Checker trade jingle and it's ingrained into our brains but Checker trade is a company that enables homeowners to connect with trades and get jobs done in
and around the house get things fixed Etc so that's that's the company nutshell it is definitely a very very catchy jingle can attest to that um Stephen um I imagine most people are aware of you but like uh for those that aren't you want to give us a brief intro to yourself yeah absolutely so I'm Stephen I started this company with Chris and Pete about three I don't know three years and change ago at this point um so Chris is based just outside of London but I'm in New York uh so a lot of our
customers came from the US and I moved out here uh early last year to s be closer to them so uh I did put it in the comments but this I'm in an office right now with no air conditioning uh so we'll see how this uh see how this goes in 30 degrees celsius weather sounds sounds fun uh I'm also in an office with no air conditioning and it is unseasonably like well maybe seasonably hot here in the UK at the moment um let's let's uh Soldier on through right let's jump into jump into panel
um so we're talking about the incident way as I said um broadly we we sort of cover these in in pairs as we go through things um first up I think these two principles are sort of like for me like the foundation of like a healthy incident culture um so they address sort of in my mind like two of the most important questions that any organization should be able to answer which is like what even is an incident at our organization and and who are they for now we're we're kind of relatively opinionated in the
incident way around like what these things are but I think there's still a reasonable sort of like amount of like Freedom around the edges of those things to interpret them and sort of apply them in your own context um I guess with that in mind and to kick us off Mark like how do you folks at Checker trade think about like what incidents are and who should get involved yeah so for us we very much see the in are for everyone as well and over the last year we've done a lot to sort of enable
other areas of the business to get involved in that if we go back to sort of the traditional way of how we managed incidents and how we handled incidents it was very much an engineering only sort of club for lack of a better term where it was just engineering getting or a couple of Engineers getting together solving a small problem here and there and the why the business would not necessarily see any of the impact around that uh and you know as you said it is very reactive one of the things that we strive to
get to is yep it's all urgent yes we all need to fix it but we also want to be proactive about it and more engaged with our stakeholders whether that's within other areas of the business or even our actual customers whether it's trades or homeowners who are having issues with various parts of the systems um so yeah that's kind of how we see it at the moment nice and I guess like Stephen coming to you like I you you were obviously have a quite a unique vantage point on like these things being at incident I
and we have probably quite a quite a sort of far end of the spectrum way of dealing with incidents but like yeah how how do you think about these two things of like the what and the who yeah I think um so for background like me and Chris and Pete started this company coming from a place called mon in the UK which is a regulated Bank uh and as a result like banks are really complicated and there is lots of stuff going wrong in very like little and large ways uh and as a result like
I think that informed sort of why or the approach that we took here and like I think in general like stuff generally is going stuff is generally going wrong quite a lot more frequently at lower severity than it is like at at very high severity but a lot of the time people might not call these things incidents you might call them bugs you might call them like a blip of like 500 errors but you wouldn't necessarily declare them as an incident I think there's a different approach because I think if you only declare incidents when
the worst happens you sort of end up with a couple of bad things like one people are scared to declare incidents because they don't you know it's like oh this must mean that something terrible is happening and do I want to be the person that declares the terrible thing that can be intimidating uh and then I think the second thing which is more important is like people are poly practiced at the process of running incidents because it might be like once a month once every couple of weeks style thing uh and we see it pretty
differently like we think of it as like urgent reactive work that takes you away from whatever it is you are doing at that given time this can look like a outage this can look like a bug this can look like a particular error that a customer is facing it can also look like the power going out at your office uh there are many different like things that go wrong uh and like these are fundamentally like ways that these are these are problems that people need to solve together uh and they need to communicate internally but
also sometimes to the rest of the company about them and we think of those as incidents as well um so that's sort of like what are incidents in terms of who to involve in them I think there's sort of two schools of thought uh so one is like incidents are a technical problem kind of a nerdy Sr focused problem where we talk about you know P95 response times and uh like a lot of like very deep technical stuff that you would find in data dog I think that is part of an incident but it like
doesn't encapsulate the much broader context in which an incident happens like we think of incidents as not a technical problem but as mostly a human communication and coordination problem uh and like kind of tell telling you a story from monzo it's like let's we had some pretty bad outages at monzo and that meant that because of let's say some issue with like etcd which is a piece of technology we used that we weren't able to process card payments like it's sort of you know academically interesting that it started in etcd but in reality we have
hundreds of thousands of customers that can't buy groceries which means they're coming to customer support it also means that if we have not managed to process a certain amount of payments then we may have disclosures we may need to make to the regulator and sort of all of these things don't sound like technical problems to me they sound like customer support problems executive decisions uh compliance and all of this is fundamentally like a human communication and coordination issue and as a result like it just feels so it felt so obvious I think to me Chris
and Pete at least that like this isn't an SRE problem this is a problem for everyone to help solve but then a lot of the tools that we ended up picking up up uh like we always described it as like there's just loads of Json and like logs and all this nerdy stuff whereas that's very intimidating if you're a customer support leader a CEO you kind of just want to get out of the technical land and into the kind of human communication and coordination land so yeah I uh I think we are at a maybe
a different spot to some of the industry but i' from what I've seen with the customers we work with we generally tend to Trend more in this direction over time so I'd say it's potentially a limitation of the tooling that folks are using at the moment that makes it sort of not easy uh not necessarily easy to go along this um this trend nice yes I feel this is going to be a theme of today's uh panel but we are going to have to move on um I would love to dig in and ask like
a bajillion questions here um in the interest of covering covering all the various things we want to chat around um let's let's talk a little bit about about these two so um I sort of you know again like principles aligned to how or organizations as a whole kind of engaged with things going wrong I think um uh the first one being like who who's responsible for like actually raising the alarm um and like fundamentally see this as like everyone is the is the sort of answer to that and there's a few reasons for that I
think um which is broadly like we want to create an environment and and I think the healthiest organizations have this environment of shared responsibility where it is not engineering who have to you know build the thing ship the thing keep an eye on the thing look after the thing like there are many many more eyes that exist out there whether it's your entire customer base your customer support as the sort of proxy for that um execs who maybe log in and try to use something maybe they log in to check a trade to book a
trades person like well this is rubbish it doesn't work like for some reason like those are sort of things that should should be the responsibility of everyone I think it hwks back a little bit to like I think the Toyota way where they had to like everyone should be able to pull the big red rip cord to sort of stop the production line kind of thing and the benefits of this are obviously like it's lot a lot less likely for things to go on noticed um and then the sort of side benefit of that is
like your response times are going to be better right the sooner you know about things the sooner you can jump on them um and then I think like the second one is sort of like a little bit of like a follow on there which is like you know how how should you manage incidents and like for us it is default to public versus default to private which is maybe a little bit of a departure from how historically things were done which was like something bad's happen let's keep it behind closed doors um and I think
the benefits of this for me like are it like de escalates incidents makes them a lot less scary if you can see them happening frequently it's a great way to catalyze learning where bunch of different departments or people in Cross teams can start of see what's going on what's going wrong how it's dealt with which sort of builds that operational M muscle and it feeds into that first one around like the reporting side of things like the more you see incidents publicly happening the more kind of normal it feels to declare those incidents um Stephen
you I guess you touched on this a little bit but like like how how did this work at at monzo these things um like what was your sort of viewpoint from there where we were working in that that regulated environment yeah so when I was at monzo this guy called Chris Evans uh ended up building some internal tooling uh which later got open sourced and I think sort of I came to that I I came to monzo sort of seeing this tooling so I'd never been I'd never seen the before times uh of us using
this and I think there was a few uh a few really interesting differences like one it was embedded into slack so if you agreed with my previous statement that like incidents are a human communication and coordination problem then people tend to use their human communication and coordination tools to solve this which is slack or teams or Zoom you know choose your choose your weapon of choice but it was like where we were doing the thing but badly uh that then meant that if it met us where we were solving the problem but it made it
really easy to declare incidents and I think just in technology like the story of great technology has generally been about reducing friction and making things easier and you get all these like spillover benefits that tend to happen after the back of that it's being easier to declare incidents meant that we got a lot more incidents around the company uh and it meant that people were better at running incidents so it's like the flip of what we I said uh happened there and it was I can imagine some people in the audience are thinking like this
sounds crazy don't you just get incidents declared by people around the company that have very little context and cause a lot of noise I think honestly our experience was like not really uh if I was going to stack up like human declared incidents versus alert declared incidents from Daya dog and Page Gutt and places like that alerts were probably 100 to a THX as noisy or had like a false positive rate compared to the humans so as a result there was this like some of the things that you might be worried about about letting more
people raise the alarm like just didn't happen in practice to us and I think this kind of hitting on the notion of transparency like we had this incident Channel which was a channel where probably like three quarters of the organization had joined it and I think that this is just like a stream of information coming through and you could just understand what was going on in the organization uh monzo itself had a value of default to transparency so there is maybe some like monzo isms that that stuck around here that might make it easier but
in general like it accelerated I think a lot of learning about how to run incidents well because you just got to see it and this isn't me saying like all incidents should be managed in the open so for example if you have a security incident where you have an internal attacker uh you don't want to declare that you're you know going to a sort going off to that person that's something you want to deal with privately but I I stick with I guess the monzo ism of like default transparency on this stuff like it generally
tends to have a lot of benefits uh like spillover effects into the organization uh and as a result it kind of makes you better at running incidents makes you better understanding who else is having incidents and how to respond well there just all this stuff that is uh not the thing you might not the thing you might think of when you're just declaring them but it ends up being sort of more powerful a bit later on yeah yeah love it love it Mark I guess coming to you like um Checker trade like do you have
folks everywhere declaring incidents all the time we've opened it up for sure to to allow everybody to raise incidents um what we did do is in certain areas especially with some of our more agents that are on the phone going via team leads to avoid an absolute torrent of potential for instance coming in so I have a little bit of an element of of span of control there but it seems to be working really well for us um I do agree with what Stephen said as well incense should be open they are very open uh
except for security related ones for very obvious reasons um but other than that yeah we we 100% want to make sure that everybody has the the capability and and willingness to raise an issue even if it turns out that it's not actually an issue in the end of the day it's more important we are aware of something that has been noted and has been spotted and who knows there might well be something underlying that could potentially lead to a wider in later on right so at least in this way we have something to look and
investigate rather than an alert from dayto dog or from Google op Suite or something that might indicate nothing so yeah yeah makes a ton of sense it's sort of like the cost of declaring and turning something down is is a lot less than the cost of like someone not reporting and you missing something um and that that point of actually filtering into team leads as sort of like a way to sort of reduce the raw number of like IC sort of like at the at the sort of full end of the spectrum something we did
at monzo as well I think that's honestly like a great great thing which is it can become very difficult to train like there is like a pragmatic lens to this everyone can raise the alarm which is like if you're in an or of 5,000 people and everyone's just like oh I've seen something and it's like well no it's like there are there are good like pragmatic ways to kind of filter those things up and apply a little bit of quality control around around it but um yeah makes a ton of sense makes a ton of
sense right I'm going to keep us moving on Pace because uh that is my job um onto onto communication which these two two things here are about um we're not going to have time to talk about the second I will give a quick sort of summary in a sec but like the first one I think is very important here which is like we often find ourselves talking about how like the best incidents are founded on like great communication um and I think if we talk about like trying to engage the whole organization if you try
and pull people away and deal with incidents somewhere where the organization is not you're sort of immediately facing this kind of uphill battle to get you know people over into your platform or your dashboard somewhere which is something that we've kind of really really lent into with incident iio and it sort of I think it harks back to for me like you know when when an incident happens like people are looking for like the most efficient way to be able to solve that thing they want to literally just get their job done and if you
look at like some years ago where everyone was working in an office that everyone would jump into like a meeting room a war room right and they would get around a table because just face Toof face communication was like the fastest way to get get sh done basically and you'd write on a whiteboard because it was there um and I think the sort of like fast forwarding to these days like it's almost like a zero probability that you will have every single person in the same place and so you default to like these async communications
whether it's slack teams something else there and your whiteboard gets replaced with like a slack channel that sort of thing um I guess like Mark for you like how how looking at like Checker trade like how were things before incident iio for you and then like you presumably do now Orient around slack as your thing for instant response and like how how is that actually working out for you yeah so kind of what I alluded to earlier that um it would be a small group getting together and it would have been in slack they would
create a small Channel somewhere and there would be no centralized way of announcing what what was going on so it almost felt like it was a little bit hidden to some extent um so with that regard then now having that capability which I know is still in slack but having that more public visible having pings having announcements all those of things has really opened that up a lot better to enable us to have better communication also to the point where we've actually been able to draft in other areas of the business to help us with
certain elements of that communication so it doesn't have to be just the person running the incident we got other people able to help now to help Drive communication outwards whether it's to areas of the business or to a customers which is a massive Improvement of where we were 12 to 18 months ago and has made everybody life a bit easier if something is really on fire we all know that it's a stressful situation to to be in and get involved in so having that flexibility to drag these people in and get those things off your
back makes your life a lot easier yeah I I completely completely agree I think I think that sort of when you start to see incidents where like the power of bringing in Customer Support to to deal with things where you're dealing with the rest of the incident it is just like it's like legit gamechanging I think makes things like far far easier to be all gravitating to to that point and like I don't know we we often talk about like you know the kind of philosophy around like why why slack and and it's like generally
generally like want to put literally everything to run your process at your fingertips and it gives us all of those levers to be able to do that really really well um and so the only time you should have to go and leave the you know leave the slack channel is to go and actually fix the problem which is like the kind of we'll get to shortly which is like you know who who's responsible for fixing the problem you know humans automation that kind of thing um I I said I'd sort of very briefly touch on
the second one I think um it is probably an entire an entire event in and of itself talking about like customer Communications and things um but like broadly our our Viewpoint of this in the context of like the incident way is like I think very few organizations historically have lent into being very public about their incidents or communicating well during them um I think I first my first real Encounter of this done to the to quite an extreme was monzo where status page usage was like incredibly frequent um it changed over the time that I
was at monzo and you know early days we post everything all the time and then realized that when you have 8 million customers that actually becomes like a negative for customers and you have to balance those two things out but um yes I'm going to I'm going to skip over all of the nuance and detail and just say we believe it is good to communicate whether your customers are internal external one to many one to few whatever it is you're doing it is generally a thing that that feels a bit unintuitive but actually uh very
positive for folks so um I think the the top one here is sort of probably one of the you know topic dour I guess uh automation AI could be an entire session again in an of itself um like our view here is is relatively straightforward I think but it is a little different I think to what we're seeing with some other people in the market which is you know Automation and AI has a very very firm and very useful place in dealing with incidents um but not as a sort of replacement for humans but like
a way to level up and make humans more efficient and more capable during these incidence um and I think the either line we don't want to cross and we're not comfortable doing and and would sort of hold relatively firm on certainly for now is like that automation should not be taking actions on your behalf and like fundamentally if you look at the reason behind that I think it's quite sound which is if you are having an incident your system is not working as you expect it and I use system in the broadest sense of like
you know your Technical Systems like your infrastructure and your software and whatever else is going on there but also like the human component that makes that up right for for some reason something is not working as expected and when I look at that kind of situation it's like what do I have more confidence in being able to sort of adapt to those conditions is it a human that has you know been around for 30 years uh seeing the world and doing things or an AI that's been trained on a big data set um and I
think that's like I think it's a hard argument to sort of go against I guess like Mark how how how strongly do you agree with that like how how far would you feel comfortable like automating things in incidents I think there's elements of the incident management process that are worth looking at automating but I would very strongly agree that s the fixing component definitely not like you said if if there is an error somewhere yeah you can train a model to to look for that specific error but that error could be 10 different causes and
10 different results and I don't feel comfortable personally to to allow some sort of automations and then resolve an issue and potentially actually put more fire as a result of it right and and you ends up in a worst situation uh for us as a company a lot of our reputation hangs on having our systems working properly and our trades and our consumers being able to do what they need to do with it and we pride ourselves on that reputation and if that gets damaged that's that's a very hard thing to come back from when
you run that risk if you let everything sort of automated fix for you you run a big risk of that so in an instant absolutely I will take things like the AI components and stuff to help us drive an instant to conclusion but it will not let it fix anything for us yeah I think the point you make around like automating the process is is interesting um and sort of like a a good thread to pull on which is like the the way I kind of roughly think about this is like what is the cost
what is the greatest cost of that automation going wrong and so you know if you have some automation which is like I'm going to send an email to you know the leadership team when some certain things happen and then someone accidentally you know sets the wrong conditions on an incident and it sends an email I'm like okay it's not the worst and if we assume that you know 95% of the time people do the right things it's a net positive right the downside there is not particularly particularly big I think um as you say it
is the sort of nuance of like when something goes wrong in a technical system and there could be a hundred different reasons for it and you want to do a thorough amount of De buing first probably not worth the cost of switching it off and switching it on again automatically um even if that is the thing that the human ends up doing it's like there is some good process to follow to get there um Stephen I guess like uh We've obviously thought like long and hard about this um do you think like as as like
incident iio we will end up like trusting trusting automation more over time yeah I think um I liked the way that Mark was talking about it which is like help with the pro you know help me with the process don't I don't need your help with the problem um I think so for context I started a fraud detection company before this that you sort of like I don't know whatever machine learning was before the new wave of AI and I think the places where that was generally useful was sort of like within bounded context so
I know what it can and cannot do and like where there's sort of predictability which is I know what this thing is probably going to do given the situation and I think they're often best used where like there's some level of acceptable error in the system so like fraud detection is like you're not going to spot every bit of fraud but if you can spot most of bit you're in a good place uh and then you also have all the generative stuff which is like this is an image you can't really say this is a
99 out of 100 image or a 94 out of 100 that there's sort of no wrong or right answer you know being very reductive I don't think any of that stuff is true in incident response which is you often have very like complex and subtle issues that evade humans uh for a long time debugging this stuff and then I think the cost of FSE positives is very very high I think if you get one thing wrong people will turn off your automated incident resolution thing forever uh and they will never turn it back on and
I think we see that honestly with some software in the market today already especially around things like alert grouping and stuff like that it's like if it gets it wrong once you don't trust it um I think the sort of I'm skeptical of like like endtoend Incident Management if that's what we're calling this which is like I find the issue I suggest to fix send a PR merge it monitor it deploy it like I don't think that's going to work um we have had literally zero pull from customers to do anything like hey please go
in SSH onto this server and restart the database and fix things for me that is not the problem that we are being asked to solve we are being asked to solve like the process problem um I think where it has been helpful and where we will trust it more over time is around anything to do with like human communication and coordination so for example summarizing incidents like this has been a wildly popular feature across our customers to the extent to which like the vast vast majority of all summaries written for incidents are AI generated or
assisted at this point um finding follow-ups so if you close an incident we'll scan through everything and find all the things that you said that was like oh we should really fix this thing or we should uh you know add more service like we will find that stuff and create it as follow-ups like that I think is the powerful stuff because it's again we're not taking action on your behalf we're just saving you time and mental energy and as a result it's uh I think we're not going to suffer from the like zero falce positive
stuff that other people will uh and like the sort of one one sentence we say a lot internally around AI is like exoskeletons not robots it's like we're sort of trying to build the uh you know the mech suit that you can wear to Lev yourself up but we're not trying to sort of automate or replace folks entirely um but yeah so I think we will trust it but only within specific situations yeah I agree I think um I think the next Evolution will be sort of more automated evidence Gathering that again is there to
help a human make a decision so you know maybe it's not SSH into a box and go and take some actions but maybe it's like go and go and like tail some logs and Pull It in or do you know you know what I mean like start to get those reps in or go and look at the changes that have gone out last like PR that that happened and tell me what might be uh you know might be a contributor to this thing um again we are dangerously short on time so the next one I'm
going to skip over only other other in fact I will say one thing which is like I think it is really really important and I will would love to do a whole session on this like learning actions post incidents thing I think the the way I would summarize this one incredibly incredibly succinctly is like both learning and actions are really really important things to do actively after an incident and the reason we have this in the incident way is because I think a lot of people go hunting for actions and miss out on the learning
and my viewpoint is if you go hunting for learning you will definitely get two actions as a result of that and so like they are not the same same goals and going for one definitely is the the better one than the other anyway uh it pains me to skip over it because I would love to chat more about it but we will come back there'll be more more of these events where we'll get into it um right last two um I sort of broadly cover like how we model incidents um as like an industry this
is not us as incident iio um and how we draw conclusions from those things so the first of these is like really addressing the there is like a I don't know I've seen like a sthe of comments around like oh it's very hard to have severities on incidents because like you know what even is a major incident or you know is it major minor is it critical like you can't fit this in a box because we're messy humans or like you know how how What stages this incident in is it are we investigating are we
fixing are we monitoring and like these are like just like the messy complexities of life but the Viewpoint that we have here is like just because they are wrong and reductive and do not perfectly represent the world they are actively very helpful so being able to shape an organization and have a common language to talk about roughly where you are in an incident or how bad an incident is those sorts of things I think are objectively helpful um I sort of the thing that comes to mind here is the sort of famous George box the
the stat statistician um which was like you know all models are wrong but some of them are useful which is like generally generally what we're saying with this thing um and I think like the second Point here is like the kind of the follow on from the first which is if we assume that we have this kind of model of an incident which by definition reduces down the complexity into this sort of like neat box then there is sort of no world where we feel it is sensible or pragmatic that you should draw conclusions from
the data that that tells you about because it's literally skipped over all of the complexity and instead that data is an incredibly useful artifact for sort of shining a spotlight steering you in the direction of where you might want to go hunting a little deeper here um so that's that's broadly what what these are about um I guess Mark coming to you like how how do you folks make use of like incident data to understand things a little bit more like check a trade yeah so we we do quite like to look at the data
and help Drive sort of decisions not only from just an instant point of view but actually creating some form of model for us internally to have a better understanding of where some of the severities are because like you said it is difficult to pick oh this is a P0 this is a P1 so we we create a framework of some reference points with regards to impact that then help make that decision for people um and again it's all very flexible right there's multiple people working on inant people look at the data that they see they
might reference previous instance and in similar areas and oh that was a p P3 or P2 so let's make this a similar one so we have all these data points that we can look at at any given time in the system and I think across engineering particular we find quite a lot of value in there um now that the business is more involved I can only see that become more valuable over time because it will enable people to have better reference points and a better understanding if if somebody raises incident because they see an issue
and they raise it as a P1 or a P0 constantly that triggers certain things of automation that goes to leadership we come want to avoid that if it's not really that so those data models have become quite important for us to then help make those decisions I think yeah that's so sounds like wildly pragmatic Mark um which is you know something that something that Engineers often sort of get a little bit hung up on is like it's it's probably going to be all right if we can measure up against like previous incidents and figure our
way along those kind of things it's like actively helpful to get there um I guess like stepen um similarish kind of question to you like uh we obviously use like our own product like as as like CEO like what what data do you care about within within incidents yeah I think of it as like sort of risks and tax so the first one is like risks in the business and the way to understand how we use it is like we declare yes product security incidents there but we also declare uh go to market related incidents
this might be something around like our Revenue operation stack CRM Salesforce course all that side of the world that any issues there get declared as incidents but also customer support so for example if we have too many customer support tickets in the queue and we're going to we may breach SLA then that will be declared as an incident and I think of it as a way of tapping into risk in the business because hey if it turns out that we are more frequently at you know falling over in this part go to market or this
part of customer support is having issues then that helps me understand like okay like where do we need to where do we need to add extra resources and then I think the second thing that is uh is important there is like reactive Works reactive work is ultimately a tax that you pay on proactive work and you can choose to pay down that tax if you want to with some more proactive work uh but it helps me understand like what teams what parts of the company are are doing that and like people might think about this
as like technical debt poorly written code but I think about it as like an organizational level View of like okay do we actually need to go and do a bunch of work in rops do we need to go and do a bunch of work in infrastructure to make like base level resiliency like improvements um I think like that stuff's been really useful I honestly we're too small for me to even look at things like MTX and all of that stuff I care more about like where is it happening and how much time is it how
much time is being spent versus anything that's a bit more aggregate than that yeah yeah yeah we are we have that luxury of being being small enough that we can go and do all of that stuff um but I do generally think that like this is the point of the data right which is like Shine the spotlight on the area that you can go and focus down um so I've I've even found this with with some of our incidents where we've had very very busy spells and ended up using like our product to steer me
towards a team that is generating more incidents than any other and be like great well they've just shipped a thing and they're in very much like reactive mode because there's a lot of like you know we're in like beta with someone with shipping product and you know you're kind of in that break fix cycle and say okay well that's where it is that's totally explainable and I can sort of see the context there and if I was looking at like an mttr graph that probably would have obscured all of that nuance and uh yeah so
it's sort of yeah it's interesting interesting um right that that sort of concludes the instant way that is 10 10 principles well maybe eight because we skipped over two um I think we probably have about about seven uh minutes um so let's we got a few questions which is which is awesome um let me bring the first one up um which is broadly like practically practically speaking like how do you see incidents for everyone playing out in real incidents um Mark maybe coming to you first if you if you if you have a good answer
here yeah I we we we see with when we have instance and like said because it's open people can join the Channel people get dragged into the channel but with the tooling obviously you have the opportunity to become either a participant or an observer and we noticed that a lot of people will stay sort of in that Observer mode so just to keep an eye on what's going being up in the loop of what's Happening uh we don't see everybody joining on huddles that are happening for example but that's okay they they are there they
get the information that they need they get the updates that they need and the feedback that we're getting from across the business is that they they're finding value in it and there is an element of if an instant is big enough it's all hands- on deck and everybody jumps in a helps out and souls and we got to make sure we all work together to then everybody be on board but even for smaller ones it it just helps for people to have that understanding and I think it also then leads to people feeling more confidence
they see the process they understand what's going on and then if there is a case where they do need to raise it into themselves or get involved themselves they have now seen a bit more and almost takes a bit of weight off their shoulders because they already are a little bit more confident about actioning it themselves yeah yeah I think I think that's that's totally true and I think um in terms of like encouraging encouraging this like everyone everyone getting involved in real incidents I think something that I've seen I've seen people be very good
at bringing people into the live incident and a lot less good at bringing them into the post incident flow which is broadly like you know you have customer support people who maybe are on the hook for dealing with all the customer coms and they are a very fundamental very fundamental part of like how that incident was run and then you get to the PO incident FL sort of part of the phase and suddenly it's like the engineers are like well this is now an engineering thing um so we're not really interested in like looking And
discussing the process and we've actually done we did this at monzo where we did separate posst incident processes for like the process and the technical problem and two separate places just because they were so sort of Divergent in in how they would sort of uh run through um anything to add Stephen or should you want me to move on move on said it perfectly well you've got an absolute doozy of a question coming up so um brace yourself um I've read it so I will summarize for you which is John John Paris of Skyscanner um
so it isort of when it comes to like shared understanding of like the states of what incidents you are in um basically John's saying like as they've grown in in maturity they found that it's harder to align on the sort of the life cycle stages of the incident so are they in I presume this is like are they in a fixing state or an investigating state or a monitoring State um that's becoming harder whereas like severity is sort of converging by the sounds of it on a shared understanding of like this is very clearly a
s 1 to a s four sort of thing um if you've seen anyone else seeing seeing this problem what your thoughts on on this one uh first off John uh hello uh good to hear from you I think it is a I think it's a great question and um I think so so like some from personal experience like especially where the sorts of incidents where this will generally become more of a problem are the ones where you actually have like different sort of streams or types of in multiple things are being solved at the same
time essentially so there are some things which are fixed but there are other things that need to happen and won't be done for a while so a great example is um usually like some kind of incident related to payments where you may have stopped the system from being wrong and sending out the payments immediately but there is also maybe payments that went to the wrong account for wrong amounts or things like that which is like things that will be remediated um and ideally like you don't want that to have to take a very long time
to do um so I think uh I think that this is something we haven't really allowed people to model that well in our product yet um I do recognize it like as a real thing um we are working on essentially like multiple sort of like streams of incidence at the same time um so I think it will give us a little bit more Fidelity to be able to break apart the fact that you might be in multiple stages at the same time however uh I have no answers for you directly John on how to solve
this problem I think it is just like fundamentally incidents are messy and you can be at different stages at different times for different problems in the incident uh and as a result um I think it's uh I think it's honestly just a tricky one but we will sort of do what we can to help uh the tools better match reality I guess yeah I think that's solid I think my my sort of I mean Chris yeah exactly you are come and come and give a better answer than I did I think I think my my
answer on this is often like um I think it's very easy people to get hung up on applying the label in the in the incident of like is this fixing is it monitoring is it whatever and I think my my usual answer to teams who get hung up on on this or severities or whatever else is like what do you need to do differently like does moving from one state to another actually cause some action to happen is it sending emails is it doing things if it if it's not it's academic and it's like pick
one get on with get on with your your incident solving if it has some implications and there's some actions being taken then uh look at those and decide which ones of those you think you should be taking at that time um but it's a great question I haven't actually seen it sort of playing out sort of explicitly but John you know where we are and I'm very happy to come and chat to you about it if if that's helpful um nice well I think we are dangerously close to time um so I think with that
um Stephen mark thank you so much for joining um I think it's been a lot of fun uh appreciate you appreciate taking the time and answering all all my questions um and with that I think we are uh we're going to call it at that point um so thank you so much everyone else for joining along here um hope hope you found this useful um if you do want to go and read up more about what we've been discussing so the incident way um there will shortly be a QR code replacing my face um or
if you want to head to the website you can head to incident ioth way or slash thee incident way we have sort of covered all of our bases on URLs there um and with that thanks so much uh we will see you next time