Hello um I'm Pete welcome I'm co-founder in CTO here at instant I really appreciate everyone who's joined us this evening today depending on where in the world you are um super excited to have you all here this is the first ever um webinar or kind of live event in our four-part Series so redefining instant management um super excited to be getting into this and it's you know intended to cover a lot of the topics That we talk about all the time like internally to the company and with our customers and we thought it was about
time to talk about some a bit more openly as well so super excited to be to be getting into that today um I think you know the rationale for doing this really is that building an instant management product means that we as a company have a really front row seat to the way that you know the companies in the world that are kind of leading the Charge are handling incidents and the way that that is changing and part of our role as we see it is to help customers accelerate that and I think you know
that's fantastic for the company that's fantastic for the customers we speak to but for loads of people outside of that sphere you know you don't get as much access to that so we wanted to reflect on the hundreds of thousands of incidents that we've helped customers solve and share I guess some Of the learnings and some of the trends that we're seeing in the industry let's see how it goes it's a bit of an experiment I'm super super excited to get into it um before we I guess like dive in uh to topics we'd love
to cover just a little bit of housekeeping so um in a minute I'm going to invite Lisa up and I'll we'll talk more about that in just a second but to set the tone the event's going to be about 45 minutes um I'm sure if Lisa and I got into it we Could talk for far far longer but I'm not going to subject you to that um we can't see or hear the audience which is very like unnerving but we can see the chat function um and we can see Q&A so feel free to chat
as you go there's a couple of other folks from the team here they'd be more than happy to riff with you guys um and we're going to make some time at the end for questions so please put things in the Q&A um we'll keep an eye on that and and Lisa and I would Love to answer questions at the end um however um first I would love to cover some some topics that are sort of near and dear to our heart and for that I'd like to invite Lisa up to the stage um I think
you know I'll let Lisa hi Lisa um Talk a bit more about herself in a second but you know to set the scene Lisa was one of the first people that um I kind of asked to come and you know uh join instant IO and take a crazy leap um with that Point barely a product um sort Of no investment no nothing so she's had a really front row seat to every customer we've put on every incident we've ever had she's caused her Fair few um as have I and I guess like as folks that
have tackled hundreds of incidents in our careers and seen customers run essentially like hundreds of thousands of these things I think you know I really really wanted to get Lisa's voice in on this one um and I know firsthand that she has lots of very strong Opinions and I'm looking forward to hearing them um Lisa I guess like before we jump jump in might be helpful for everyone just to share a little bit of context on kind of what you do now and kind of your role in the team because I think that sets the
scene quite nicely for some of the topics we're going to talk about um so hand over to you yeah um hi everybody uh very excited to to be here this is my first webinar so um we'll see how that goes for everyone um So yeah I've spent most of my time um in like the response product um and working focusing on how people respond to incidents how can we build a product that is delightful and useful in that kind of moment of high stress um and I think that's that's something that I've always really enjoyed
because the stakes are high in that you're interacting you know the product that we build someone is interacting with at a moment where like things are not going to plan and They're not feeling great um and that means that you've got a chance to really Delight them and make their day if you get it right also absolutely a chance to make them hate you if you get it wrong um but I kind of enjoy that kind of slightly more like high stakes part of the product um and also thinking about what I would have wanted
because these are all the tools I basically get to spend my day building things that I have wanted for my whole career um and that's Really exciting and whether that's stuff to help with the really big high severity 20 people in a room incidents or whether it's stuff to help with like just the little bugs and sort of day-to-day noise that that happens in a in a team I think it's really interesting thinking about how do you build something that works for both those groups of people for bigger companies for smaller companies and for those
people who are in that like really High stress moment um so yeah that's me and I'm kind of excited to get into it nice likewise um and I think sort of topically I thought we would start with home turf for you uh and talk a bit about response so I think you you you mentioned there that you know incidents are high stakes they're high risk they're high stress and I think you know the reason that I wanted to cover this one first is that I think that is traditionally how everyone sees Incidents in historically right
it's like the old ways their major issues that typically happen infrequently they're complex they're hard to navigate and I think what we've seen is a real Trend in the other direction there um which is incidents being something anything that takes you away from unplanned work with a degree of urgency um whether big or small I guess you have driven a large amount of the I guess like culture change that we've had at Incident IO similar one that we've seen in other companies where it's like let's try and treat as many things as possible as incidents
I guess like what why do that is a question that I would be asking if I didn't already the answer um so yeah i' like would love any thoughts you've got on that yeah so I think I can't tell whether I always believed this or whether I've like eaten my own Kool-Aid a little bit but um at the beginning of of the company fairly Obviously we had this new product and we wanted to be using it as much as we could so we were like we're going to dog food it and we're going to dog
food it for everything and I think we kind of thought at the time that we'd probably push the boat out a bit too far there and that wasn't really how our customers should should really use our product in the real world and then sort of doing it we discovered that actually it just felt really good not just from the point of View of all of the the dog fooding and the the stuff we were learning about our own product but also as a way of codifying the way that you do reactive work and particularly in
the early days a lot of our sort of selling points and the way that we were building relationship with customers was all about providing really really great support and building those really good relationships with customers and actually using incident response as a Framework doing all the same things that you do for a you know critical incident but sort of in microcosm works really really well for those for those smaller incidents and really helps you to build trust so that's communicating really clearly internally and externally making sure you're showing your working so that everyone else in
the team can learn from it so that you're spreading the knowledge of how you actually debug this stuff thinking about being really clear With both the team but also like bluntly yourself about where you are in the phase of an incident and do you understand what the problem is and are you trying to fix it or are you monitoring it and all of that hygiene just made us better at responding to those smaller like unplanned bits of work and I think then what we found is we started talking to customers about it and they were
kind of like oh yeah actually I this this kind of works for Me too and now that we were sort of you know two three years on from that I can now sort of understand the the reasons for that which is it's about applying structure to reactive work and if you're in a normal environment where you don't have that structure it ends up being happens quietly in the corner it's often very invisible to the team to management and the result of that is that you end up with sort of this invisible workload that's dragging your
team and you can't See it and you can't talk about it and no one really knows who's doing it and why and for who and I think that Incident Management and good Incident Management tooling is a really good way of like pulling that stuff out in the open and then you can interrogate it and be like oh why is this particular integration cause so many incidents like do we need to do some investment there you can start to prioritize and you're just seeing so much more and then being Much more structured in the way that
you you fix and communicate about this stuff yeah 100% I think um one of the things that in in my role uh sorry I had a bit of lag there Lisa can you hear me okay yeah I think I think you're all good now okay brilliant sorry I just had a moment of cut out the irony of having an incident during a live event about incidents is not lost on me um but may we can we can use this um I'm sure the social team will have a field day um but Yeah I think one
thing that I've noticed as well that I've particularly appreciated him in my role I guess as CTO is like why do I want more of these small incidents I think for all the reasons that you've said for sure I think one of the things that has happened before as well is like kind of like the small incidents that then become big incidents like the Snowballs and being able to like catch those when they're tiny and like have everything Already there when it's like a tiny fire that suddenly erupts into like a flaming Inferno and if
you already got a fire extinguisher on site like that's incredibly useful and I think that I found very interesting as well is kind of you know we've had a number of ones where it's like oh it's probably nothing but let's just declare an instant anyway and then it's like oh my God this is very not nothing and also help please and and that's where I think it's been You know super interesting seeing that seeing that philosophy sort of lead to a better outcome even for major incidents not just for smaller ones yeah the other the
other thing I just thought of is um it's also about learning your tooling so where I've been in the past you don't do your incident process until something's really wrong and that means you're both having to fix some of the most difficult issues and learn all your tooling at the same time whereas if you use your Tooling every day then it's just sort of like your tooling times 10 which is much easier where you're kind of using all of it versus being like I don't I don't know what we call incidents I don't know what
severities are I don't know who what a lead role is like all of you don't want to be discovering all of that stuff while your website's down yeah 100% I think like what one thing that maybe just like came into my head while you were talking about all of That is kind of you know familiarity with tool totally agree that's really important and I think you know one of the reasons that a lot of people wouldn't declare more incidents or you wouldn't take the philosophy that you've just gone with is essentially like your processes are
just a bit rubbish um and so it's like really painful to do that I'm kind of I'm gonna I'm gonna swerve the bias of someone that literally runs an instant management Product Company But like if we if we talk about for example like um something that we hear all the time from customers right is uh you know the postmortem process is really really painful right it's like we treat incidents so seriously that if we did that for 10 things a day we would never do any other work I guess sort of like maybe for the
benefit of folks that are maybe in that position like how how do you in the team view this like because like I I know for a fact that You don't follow a super comprehensive detailed postmortem on absolutely every single instant we have that would be insane so like how how do you balance that how do you figure out like I guess how to have a response to that CTO who's like I don't want to pay the cost on every incident so I think it's about having an adapting an Adaptive process so with most things with
most processes like if you think about how you run a project Some projects are going to take six months and you're going to be really really structured and detailed and planned them and other projects are going to take four days and essentially someone's going to put their headphones on write up a really quick plan and come up with something sensible and we're used to the idea that we have different muscles for delivering product projects and I think incidents are just the same thing where you want to have different Lay different amounts of essentially bureaucracy and
paperwork and also so empower the people responding to the incident to push back if they don't feel like it's adding value like we've all written a postmortem where we would sat for four hours staring at notion being like why am I doing this with my life I'm doing it to tick a box because my manager told me to and as soon as you're there you you've totally lost and you've lost all of the engagement from Responders and that's the point where people will stop declaring incidents because they don't want to do the paperwork whereas if
you say hey do you think a postm would be useful here and someone's like nah then you know maybe you challenge them if you disagree but maybe you just say okay cool well let's let's move on with our day and I think it's just about making sure you have those different muscles and you can use them in the right way equally if you end Up in a culture where you never write any postmortem even when everything is on fire you know that's that's probably also a problem that you've got to talk about yeah I think
the the ex both extremes are bad right and I think the the key is like having a process that just like ability and is adaptive to to whatever type of incident you're running I think like one thing you mentioned there is like the cost of writing the Postmortem or you know um the effort involved I think like I'd love to move this on to um thank you for slides uh automation so like this is an area where I guess like the idea of you know lots of manual efforts and lots of pain sort of just
in particularly in 2024 just starts to scream uh llms and Ai and all sorts of other exciting buzzword I think applied in the context of incidence is it's an interesting topic and I think we should talk about it bit because it's Something we've seen coming from customers it's something we've worked on internally ourselves um kind of I guess in the old world if incidents are manual rely on people following some flowchart in a Google doc and the new world is one where we're kind of trying to figure out how to use the chat gpts of
the world how to use like Ai and pattern matching like you know between those two like the second one definitely sounds better but it's not free and it doesn't come Without cost and it doesn't come without tradeoffs I think talking about some of those would probably be interesting or I think so um if not people in the chat can tell us to move on um I think like there's a school of thought that says AI is just going to replace everything um what would you say to people who say that it's like why why even
build this products in you know 10 years time like five years time AI is just going to do it all like why are there not loads of Ai ai only instant management products already on the market I guess would be a question that I think is interesting yeah so I think um the the way that I've seen AI be successful and it's possible that that is because of the limitations of our current llms and as llms get get better this won't be true anymore but like my current view is that AI is really really good
at augmenting humans and making humans faster more reliable and overall like Much better incident responders and I think you know we've obviously got some AI features in our product that we've built and the stuff that we found that really resonated with people was like a little bit kind of magic here and there to just Grease the wheels and keep things moving and I think that is also what automation has always done so if we think about Automation and AI as being not so dissimilar this is like the same way That you have a little Alias
or you have a little script that you run to lock the master Branch that's that's automation you know we've been able to do that for for years and years maybe at some point you just tell AI to lock the master branch in your phone and it does it but it's still kind of the same thing where you as the human are in the driving seat and the reason that I think that is I don't think we're going to get to a point where we have ai that is good Enough and trustworthy enough so if we
think about like hallucinations for example do you want to give AI the kinds of power that you give your engineers on your team around you know rot credentials to your database because if that goes wrong you're going to have real egg on your face and you're going to be going back to your customers saying oh so the AI B dropped the database and we're really sorry and it's going to take us eight hours to restore From the back you like blame the intern isn't it it's sorry our AI just you know deleted all your data
oh bad but but I think it's also like the risk there the risk reward just isn't right whereas the risk reward of getting an AI that gobbles up all of the information from all of your incident channels from all of your docs and like tells you hey the last time somebody had this problem they found they ran this command and it did this and you know providing all that Context in a really useful way like that feels like you know I'm a superpower I i' I'm now like a a superum who can do all of
this stuff 10 times faster but there's still a human in the loop who is checking who's understanding who's maybe prompting the AI to say oh okay I kind of think we need to look at the database can you can you pull up the logs and I think that's the kind of you know Iron Man version of this that I see which is the the thing that's changing is you're No longer typing you're probably talking you're probably talking to something that has a lot of context about your organization specifically that you've trained on the data that
you have and the other humans that are in in your organization and so you suddenly have the collective knowledge of everyone in your team at your fingertips rather than having to to learn that and on board onto a team so I think that's that's the amazing future that I see but it's not I'm just going to give the keys to my database the AI yeah 100% I I think sort of you know to sort of show behind the curtain a little bit um I don't have this is where I would love a lovely slide with
some nice analytics and graphs of our AI usage but it's like one of the things that we've seen in just in terms of behavior change in what our customers did before they use instant ey but also using instant I so sort of forget the forget our product for a Second like is that the number one use cases that people are using this stuff for is is the like it's the the the Grump work the stuff that's just like I need to go through a thousand slack messages and try and pull out some useful pieces of
information it's not actually the like I want you to be really smart and restart my kubernetes cluster or anything like that it's the you know can you help me with the thing I have to do in every incident 10 times That's the things that people actually reach first if you if you look at what they actually do versus what people say they want actually you know exactly what you've just described I think you know literally have the grph graphs to prove it um so that's been that's been super super interesting because that was that was
not my expectation coming in right it like my expectation was that people would want you know that that chat GPT experience that like you know magic give It to an AI and everything just just works and um very much not the case so yeah principles of exoskeletons not robots going very strong instant IO and uh yeah working really well um let's move on uh to communication so this is something where I mean we were literally talking about our status page the other day right um for for an issue we were having with a third party
and it's it's something that I think every every organization has to wrestle with and Struggle with which is like how much should you be communicating with customers during an incident and I think there is zeelot at both ends of the spectrum I think if I try and draw a distinction between old and new I think you know historically and certainly when I first became an engineer you know you would never talk to your customers about instance right they they're something to be hidden there're something to be ashamed of they're you know maybe you Email some
people on closed channels you have like an angry account manager somewhere um and in general talking about instance publicly is is is a sign of weakness it's something to avoid um whereas like the new world you're seeing tons and tons of people being way more open with like their incidents and their outages and following up after and I think you know just personally like monzo was really good at this um the culture there was exceptional like you Can read some absolutely fascinating posts about how we managed to like up or mess up sorry a load
of Cassandra clusters and like there's these incredible writeups right and you know one of our customers is Etsy who were doing that long before anyone else was doing that and they had amazing writeups and they got rightly applauded for that I think we're seeing more and more people both head in that direction and want to head in that direction you know No one's forcing them something thing is making them want to do that and I think you know that's that's maybe a good first first kicking off point which is like why why why does everyone
suddenly care about being more transparent being more open talking about their incidents um yeah any thoughts on that Lisa um yeah so I think if you if you put yourself in the shoes of a team where something has gone horribly wrong uh so let's say your your whole platform is Down and internally you're like we know it's going to be down for an hour you know we we to restart we've got to restore from a backup we know it's going to take about an hour yeah I think that like you've got two options there right
you can either tell people or not tell people and historically people largely voted for not telling people um and there there are pros and cons of that right the the pros are that the people who aren't trying to use your product Don't know and they probably never find out maybe that's that's kind of uh Saving Face maybe it saves you SLA credits whatever it is um the cons of that approach are there are some number of people who are very much aware that your stuff is down and they feel like you are cheating because you're
not telling them what's going on and they don't know what to do so they don't know whether you're going to be up in five minutes or an hour um and they will Probably take to social media and they will scream about you and I think probably that is the single sort of biggest change here is that people's behavior when those things go wrong has changed from sort of accepting that software goes down and that's just the world we live in to actually a world where most software works most of the time and the expectation is
that software is available 247 like the days of you know weekend maintenance windows Are gone and so if something's down you go on to Twitter and you moan about it and you say hey why is this thing down what's happening and when you're in that situation if you get ahead of it and you communicate then you can totally change the conversation so I think like Etsy as you said and also like Linea I think we're really good at this Linea recently had an issue where they ended up to do a roll back and they sort
of took the lots of their platform down for a couple Hours and they communicated really clearly during the incident they sent write ups to everybody who was affected it was all very transparent and the result of that was both as a customer I knew what to do and what would and wouldn't work and how to sort of plan my day and I also trusted that they would tell me if something similar happened again I trusted that they were fixing it like I knew somebody knew and was trying to fix it and I also thought they
were Good Engineers because it's become a proxy for really good engineering culture because if you are good engineers then probably the postmortem isn't going to make you look that stupid because it was probably a few different weird things that happened all on the same day might might make you look really stupid yeah I say you um but like if you can communicate about it well and you can talk about the various things that you've done both in Terms of how you fixed the issue but also what you've done kind of looking forwards I think that
you can get to a point where you're really building trust with your customers particularly if you have technical customers who are looking and suddenly they're empathizing with you they're not judging you they're not angry they're like oh man I would hate to be that I would hate to have woken up to that today I'm really really glad that my my app's fine and I'm sending You like hug UPS hug UPS or whatever it is and that just can't happen if you've not got that communication yeah yeah I think it's a real kind of there's like
a a sort of curve of how much gratitude you get or how much trust you build and it's sort of you know there's a there's a point where sort of you start with no news is good news and you end up being like well no news is highly suspect because you know everyone has issues like there just no way that you've had Nothing go wrong and so now I start to go like well if you're not going to tell me do I need to check do I need to poke and that's where you get a
kind of you know if you have a CLE if you've got a fiveyear history of no incidents I just I just flat out don't believe you right um and I think anyone that's worked in a company where maybe they've swept an incident under the rug knows that that is likely what's happening and so yeah I think I think the trust building Exercise is a really really strong strong point and I think it's kind of you know for an organization particularly like you said a technical one or if you have a technical product like one of
the easiest ways to build trust is just to be open and kind of honest about what's going wrong and sort of you know you won't always get I think one thing to call out is like you won't always get the reception that you necessarily hope for there and sort of You know I've I've been militantly like everything should be you know we should publish we should talk about status Pages um sorry talk about incident on our stasis page and I think one of the things that maybe uh curbed my enthusiasm for an open uh instant
world was like my experience in larger Enterprises where particularly when there's like a lot of money on the line or a really really fragile relationship on the line or there's a competitor That's nipping at your heels who will use every opportunity to kind of you know slate you in the eyes of the customer like actually there are quite material and often very commercial trade-offs that you're making there and a well-meaning engineer like me in that position potentially puts the company in a really difficult one um I think like you know are there any other considerations
that like you would flag I know like we've talked about this Right like we have the conversation of you know is this status page worthy and generally our bias is obviously yes but there are circumstances where you also have to layer in a bit of like how will this be interpreted right will people understand that this isn't actually a big deal we're trying to be open or will they proxy well every other incident on the status page is only there when it's really really bad so this must be really really bad and actually you kind
of Suffer from being a lead the charge and be a first mover are there any other things that you you've heard come up with customers where sort of they've been nervous to do this or nervous to embrace this or anything you know folks that are maybe listening should should think about before going back and being like you know pitchforks for open incidents with their leadership team um yeah so I think one thing that can be really challenging is when you don't Have a lot of clarity so when you understand exactly what the problem is you
roughly know how long it's going to take you to fix it you roughly know what the impact is you're in a really good place to update your status page with something that is useful and not going to like induce panic but there are some incidents where at least for the first part you can be in a place where actually you don't have very many facts you have like some alerts firing it's Possible that no one's noticed it's possible that everybody's noticed it could be anything in between and you don't really have anything useful to say
other than like I think maybe the database is a bit sad and that's that's a really difficult message because then you're portraying a a sense of like you know everybody should be worried but we don't really know how worried you're you can't really tell your customers what they should do because you don't know What they should do um and so I think that we should uh we need to think carefully about making sure we wait until we have enough Clarity that it's like a meaningful message or putting something that's very deliberately not meaningful of just
saying like we think something's happening and we don't know and we're really trying to figure it out as quickly as we can and we'll be back with you in 10 minutes with more information um and then I think the Other thing that we've seen is competitors really go after each other on this stuff and depending on the space you're in I think particularly if you've got non-technical buyers it's a it's a bit of an easier sell where you're like look there's loads of red circles on their status page they're unreliable you should use us look
at our status page put two pictures next to each other like we know that this gets put into sales Decks that get presented to people and I Think that's that's the most compelling thing to be worried about like that's the thing that I get worried about um and I think it's vicious as well right it's like it's not a it's not a hypothetical like we literally seen it and you know um seen seen folks use it against each other and it's it's one of the things that we struggle with right it's like do we how
much do we want to encourage this from people and does that make us you know are we actually helping Or are we hurting in a world which isn't quite ready for that um yeah and I think I think it it has to be about your competitors and what they do and you know you want to be kind of you basically you want to be a bit ahead but if you are in a different post code then it could easily come back to bite you um and you've got to be careful about the understanding your customer
base and the way that your customer basis likely perceive it and then also what are the Things you can do to to communicate and soften that and change their perspective on it nice yeah totally agree with that um I'm going to move this on if that's okay because I want to talk about um learning we talk we talked briefly about sort of sharing postmortem often behind that there's a lot of interesting debugging and diagnosis that goes on um but that's that's more on the sort of technical side of things I think it is also a
lot of learning that happens Inside organizations just around like your process how you resolve incidents like how you how you see them just like the pp perception internally in organizations and how you know are they a learning opportunity are they something to be avoided I think you know to do the kind of old world new world comparison I think you know in the old world certainly like my initial experience was like incidents are generally something to be avoided They're kind of you know whether on your status page or internally evidence of weakness maybe like one
team having many incidents everyone kind of looks at the corner of their eye and is like man are those guys okay like you know they don't seem to know what they're doing and I I think that's a culture that probably is all too familiar for a lot of people um which leads to some of the behaviors we've just talked about I think one Trend we've seen with particularly our Customers which makes sense they're buying us specifically because they care a lot about incidents but you know whether you buy us or not like companies increasingly are
moving towards a world where incidents are seen as kind of a valuable way to learn both about kind of where your failures might be and how to mitigate them but also more about your organization like you know there's nothing there's nothing that will be more telling about how your organization Really works than like you know you're losing a million dollars a minute at 3:00 a. and you very rapidly find out you know who the key people are who the decision makers are you know where your blockers are where your friction points are it's an incred
incredibly efficient way in many ways of like flushing out a lot of organizational structure um if not the most painless one I think kind of what's yeah how would how would you reflect on this leas like I I guess sort Of it's I've said sort of two things that feel like they're maybe in opposition to each other which is like you know incident as evidence of weakness is bad incidence as a way to learn is good but incidents are still a way to find your weaknesses like how do you how do you think about that
like you know how should we be using them and how do we kind of I guess Foster more of an incident or a good thing culpture versus incident or like something to be frowned Upon so I think the first thing is if you can't see the incidents that doesn't mean they're not there so kind of going back to the point that we started with like if you don't know they're there that doesn't mean your organization is fine it just means you don't know about them um and once you know about them you can do stuff
about them so the way that I would I would sort of reframe that is I think incidents are a really good way of finding weaknesses in both your Organization and your systems whether that is like some kind of technical issue that you've got or maybe it's oh we only have two people in the company you know how this thing works and one of them was on holiday and the other one was in a pub and so we just about made it work but actually it was and that's totally hypothetical right Lisa like there's never actually
been a situation that practically happened definitely never wrote code while on the phone to Somebody in a pub I say I have literally had Lisa dial in from a pub not just once on multiple occasions for instance inant so it's a it's a real scenario uh both resolved excellently might I add but anyway back over to you um so yeah I think that they're they're good ways of seeing that stuff but you only see them if you're looking for it so it's really easy to take an incident like the the monzo Cassandra outage I think
is a really good one um which is like Chris Evans who's our um our CPO ended up writing uh a a really good blog post about it while he was in mono and essentially they made a mistake with the configuration of this tool that they were using and that resulted in some very bad things and they learned a lot about the tool and you could easily stop there and be like okay I now know how this tool works and I didn't before and the the the new way I think that we're that we're sort of
talking about here is Like cool but what about all the other things that you can learn so we're not just looking at the specific thing that went wrong now we can start to look at and they do in this in this really great postmortem like well how do we get into this position well that was because we actually don't have very many people on staff who do know about this tool very well I wonder if that's a thing we can change and you start to sort of look at the broader impact on like particularly When
you're thinking about people and process and how did we communicate with customers oh we should probably have some templates of how we do that because we were writing it on the Fly and that feel good and when you start reflecting on all of those things you suddenly get your sort of like risk reward or or your the the scales turn a bit from how bad the incident was versus how much benefit you're getting and so I think it's the we had a line really early on in our Positioning that was called like don't count your
incidents make your incidents count um which I stand by as a as a positioning line where it's like you're you're going to have incidents so what you want to do I think I think Pete's internet has finally I'm back I'm back it's all good I'm alive um sorry about that team yeah like if you if you're gonna have incidents anyway you should at least make sure that you get the best value out of them and step one is Knowing that they're happening and then step two is like really interrogating them and thinking about what you
can learn from them without you know taking it too far and writing postmortem for no reasons about tiny bugs that nobody cares about so you know that there's there's a middle ground there but I think it's like the way that I would always think about it is probably you're not doing enough on your most critical incidents and probably you're doing too Much on your smallest incidents and if you kind of shift the effort you'll probably learn more from really thinking about those those really big moments and also the big near misses um versus like putting
that effort into your sort of weekly postmortem that maybe no one is reading and maybe aren't that valuable yeah a document is a really nice uh sort of artifact of evidence but is not necessarily sort of proof that you have extracted the value right it's Like the conversation is often the most useful thing let alone like the the write up um I'm GNA move this on to our our last Point um because I guess we've talked a lot about incident as like an isolated thing of like you know they happen we learn from them we
use stasis Pages like all of these behaviors that we're seeing sort of in Trends over time in in the industry I think one of the areas that is almost like of the most uh calcified is around sort of Monitoring your Incident Management program or like monitoring the performance of your of your team when it comes to incidents and I think generally you know if you look back at how most companies do this like it's MTT something right it's like that's that's the metric some some teams use mttr for like response meantime to detect meantime to
resolve like there's there's usually somewhere in a deck or a dock a metric Um that everyone is desperately trying to Monitor and there's definitely a place for those right and we should we should talk about that because I'm not saying like MTX is like a bad thing um but I think one of the things I've noticed is like although some customers do care about this they're like this is not the first metric that everyone asked for right this was not when we launched insights everyone wasn't clamoring on the door for MTX metrics like that Wasn't
what they wanted and I thought that was very interesting um and I guess the the question then is like you know okay whether it's not MTX or mttr like what is it like what are the insights that we should be looking at what are the ways we should be evaluating our instant response in aggregate um and I think that's very interesting so like I guess I guess like you run a team instant you care about this what do you look at right because you know I expect Every Tech Le to be thinking about the performance
of their team when it comes to incidents like how do you do that like you know when I say is everything good what's the graph you showing me what's the like analysis you'd give me um so I I live in there are there are two dashboards I look at one is um the pager load dashboard and that is basically about are we burning people out are people being paged at 3 o'clock in the morning repeatedly on multiple Evenings in a row um and that is about the the well-being of the team so it's not really
about the system or the impa because uh Pages happen for all sorts of reasons some of them are customer impacting some of them aren't um but having a having a metric of like how much is my team getting disrupted I think is really really important and then the second one is about workload so workload is uh our sort of we use slack data to basically try and figure out how Long you are spending on this reactive work and that's the number I've always wanted and it genuinely makes me excited and I'm also upset that I
can't get it in other places like i' I'd love for sort of to get it about other kinds of work that my team does um but being able to just look at a graph and quantify the feeling of life oh it feels like it's been a bit rough for the last two weeks and you see a graph that does that and you're like oh yes validation isn't it It's like I'm not just making it up there's a graph that says it's really bad but but then also you can cut the data and that's the magic
bit right where you're just like oh okay we've got we've had this massive Spike what's it caused by oh it's this integration or this customer or this deal you know whatever it is that's causing that you can now see it whereas before you might get as far as like this doesn't feel good but you certainly could never get To like what was actually the problem but why yeah and I also find that my gut's just not not as good as I'd like it to be like when I when when the RO when the actual data
is sort of put up against my gut feeling I'm sometimes right but I'm not right often enough to not want the data yeah yeah absolutely it's sort of like you know you go it's definitely definitely this it can't Poss or it can't possibly be that and it's like I think it might be that yeah Totally agree with that um I just had a quick look at Q&A out the Cor I uh and we've got a bunch of good questions so I'm going to suggest we move on to those if that's or rightly sir um but
thanks so much I'm sort of aware that I pitched this to you as a conversation about interesting instant management topics and what I've basically done is Grill you and Q&A you for about 40 minutes um but I imagine lots of other folks out there um at various stages of instant Management Journey d uh probably really enjoyed that so thank you so much um would love for you to hang around for Q&A if that's all right um because I think it' be great to hear your thoughts on some of these um so just taking a look
at the chat um let's maybe talk about Kay's question um I think that's that's an interesting one so we were talking about status Pages earlier and talking about like what you publish or postmortem and what you write About and I think uh if we can get K's question up on screen like K's question was like what's your approach to balancing sort of transparency versus proprietary information and it's a really good one because you know sometimes you can be very open about like what went wrong and other times you kind of like it's very difficult or
it's very exposing to do that um and also you know you talked about technical audiences and non-technical audiences Sort of that that also makes a difference I guess sort of like what would be your rubric for like the degree of transparency to go with let's maybe start there I think the the thing I'd start from is anything that's going to change a customer's Behavior they need to know so that's like the the easy bit so anything that's like how long are we going to be down what works what doesn't work from a customer's perspective that
has to go out otherwise like your coms Just aren't really doing their job I think that you then get into like a murky bit of like how much information do you give about the why as opposed to the like what is wrong um and I think that is a lot about knowing your audience so for us we would often give a lot more information about the why than at a payment provider company like I was at before where actually your customers don't care at all they probably don't understand um and you're not adding Value so
I think it's it's got to be tailored to your audience and understanding like who your customers are who are the people that are on your status page um and and also thinking about you know what are your red lines about what you're not going to expose to people because you probably have some and let's not pretend that those aren't there um and so it's kind of you know you've got to try and try and walk that walk that line which is obviously going To be a judgment call um but I think the the thing to
focus on that I would always use to drive it is like what will change my customer's behavior is the first thing that you've got to say and then what will build trust you know is it actually if you give them a little bit of more information are they going to feel much more comfortable or are they going to feel much more stressed yeah I think like one of the one of the things that I've definitely noticed when It comes to that side of things is like often a bit of paralysis because you're not sure what
to post means you don't post anything and actually that's actively worse and it's kind of it's always better to say we know something is broken we don't yet know what we're digging into it or like you know a number of our customers have been affected by an incident you just give like high level overview and say like we have emailed all affected customers for Example even things like that it's like how would they know if they are affected how would they know if this matters to them and if you can give people that confidence it
doesn't matter that you're not providing super Gran details at that point there's time for a postmortem write up later where you can think really hard about this but usually you're doing this when something is on fire and so stressing about like how much to share and how much not to share In an incident I I wouldn't worry too much and in the postmortem you've got plenty of time to kind of figure out you know what what degree of information is actually helpful here um I think we've got time for maybe one more um from Lucas
I thought this be an interesting one which is like how do you know um if an incident is big or small if not every incident has a postmortem which I think is an interesting question so it's kind of if you don't have if you don't really Dig into it how do you know how bad it really was I think you there's probably a number of of of answers to this I think sort of from from my two sense I think a postmortem is sort of not the way that you should be in my view sort
of like digging into and communicating how big the blast radius was is is useful as a way to sort of double click on that impact and maybe it's worse than you thought and you find that out through doing the postmortem that's Potentially an outcome so maybe it's worth it but in general things like blast radius affected customers that should all be happening in the context of of the incident investigation itself rather than you know needing a document to elicit that information like that the document is a way to like topological like document what happened a
document is a way to sort of collate all the impact together versus you know being the only way that you Should be able to sort push people to dig into into the incident itself if that makes sense like it can be a useful forcing function for an investigation but in theory that shouldn't be necessary there are other ways to kind of push for that investigation and the postmortem becomes sort of the cherry on top right it's the is the evidence of all of your hard work and all of the interesting things that you found um
it's maybe just a reframing of of how I Would look at it I would expect my team to investigate every bug um the postmortem as a forcing function s shouldn't ideally be needed um but yeah nice um that just takes us straight to time uh and somehow we managed to lose Lisa at least on my screen so she's done uh she's done her 45 minutes um and uh and she's off um thank you so much Lisa in your absence really really appreciate it um timing could not have been more perfect and the answers Similarly excellent
as always um and you know unsurprising there's a reason that I asked you to come be employee number two after all um thanks so much to everyone for listening Sor we didn't get to all the questions um I think you know hopefully you found this interesting engaging it's the first it's a series of experiments we're kind of just trying to open the doors a little bit to some of the things that we've learned some of the things we've seen um if you are Interested in learning more about this stuff um about the product about instant
management we have tons of free resources um not trying to you know sales pitch you um we're going to put a QR code up on the screen now and uh that'll sort of take you to a bunch of other useful materials and references otherwise um sort of you know have a have a lovely rest of your week and thanks for thanks for tuning in we really appreciate it