DevOps Best Practices
How to Have Tough Conversations with Cloud Providers, Vendors, and Everyone Else You Are Paying
by James Burns
Error rates skyrocket. A critical service slows. SLAs breach by the dozen.
When things break — and one of your vendors is clearly at fault — even the most seasoned engineers can lose their cool.
“What the heck is going on! Why didn’t we know about this? Fix it! Fix it now!”
In situations like this, it’s easy to get upset. But, of course, this won’t help anyone.
You turn the screws, You tear down the bridge, Flimsy as it is, It's business like – Cake, You Turn the Screws
Effective vendor relations are ultimately about the timely resolution of issues. Yet, this is only made possible through strong relationships, data-driven communications, and empathy on both sides.
When you build your business on other companies’ services (cloud storage, APIs, etc.), you get to focus on your unique value, but you also become dependent on other people — not on your team — quickly fixing issues that are breaking your business. In order to do that, they need to trust you, and you need to provide them with accurate information.
Vendors, Vendors Everywhere
The rate of business growth today, particularly in startups, is driven largely by effective use of vendors. A great idea even with great developers can only go so far. There are many other things necessary to make customers aware of a product, to deliver that product, and to understand how that product is used. You might use a cloud platform provider for compute or serverless, managed databases, or APIs for payments, telecommunications, chat, or shipping. Even though these vendors are not part of the core value proposition for a company, they will be critical to the customer experience and the company brand.
Over the past five years I’ve acted as the primary technical contact for many vendors including cloud, observability, alerting, and auditing companies. I’d like to share some key lessons that can make everyone happier in the relationship (or at least understand what they’re not happy about.)
1. Start With Data
Whether your point of contact with a vendor is email support, a web form, chat, or paging a TAM, having data in a consistent format that’s been shown to be effective is the fastest way to get something fixed. Time series charts with annotations often provide significantly faster escalation and resolution.
Many people seem to believe that if the service is down for them, it’s down for everyone and the vendor must clearly know. This is rarely the case. You may be the only company that has business impact or that has the observability tooling to see the issue (or both).
Thus, every initiation of vendor contact will be more successful if it includes:
- A screenshot / link / graphical example of the behavior to be discussed
- A textual explanation of how that information is being interpreted
- What that means to you as a customer, either realized impact or projected short-term risk for impact to your business
- A specific question or requested action
An Example Vendor Communication
2. Be a Partner
When something is seriously wrong, there is a lot of stress, and it’s often the case that you’re feeling more pain than your vendor. Still, it is generally much more effective to delay a desire to “turn the screws” until well after the incident. Treating the contacts at the vendor as partners (rather than outsiders) who are working together with you to solve a shared issue has worked magic for me. Time and time again. It allows you to easily ask questions like the following and get real answers:
- What other information can I share with you to help make the problem clearer?
- Are you seeing the same thing?
- How are you currently prioritizing this?
- How can we help?
- Is there anything that isn’t as impacted?
Vendors have to contextualize what you see into their systems which often look quite different from your view. Every opportunity you have to encourage their belief of and empathy with your view of their systems is a way to accelerate resolution.
3. Admit Error
Sometimes it really isn’t the vendor’s systems. The faster you can let them know that you’ve found the issue and that it wasn’t them, the less resources they waste and the more they view you as trustworthy — even if you were wrong this time. If you don’t update a vendor in a timely fashion, they can waste many hours trying to find something that doesn’t exist.
Fast forward to the next time you ask for help and request they prioritize resources for you: Your vendor will hesitate. They will likely place someone else’s needs above your own. Don’t let that happen.
A Need for Observability
When you look to interact with any vendor, especially in a real time scenario, you must think through how you’ll know how your usage of that vendor offering is actually working. You also need to understand how to see that interaction in the context of your key business transactions. Without these two pieces of information, you can’t take a proper data-driven approach, nor can you be a good partner. You will also find yourself not wanting to admit error or even being able to, since you’ll be unable to find causes for issues.
There is a common theme to all three lessons: without observability that provides consistent, trustworthy, actionable information, you cannot have effective vendor relations.