FactSet increases developer efficiency, improves workflows, and reduces MTTR with Lightstep
Headquarters: Norwalk, CT
Industry Segment: Financial Technology
Limited visibility into dependencies
Manual process for finding and resolving errors
Incomplete understanding of system health
Complete system visibility
Reduced MTTR from days to hours
Real-time understanding of system health
FactSet makes it easy for financial professionals to convert raw data into powerful insight, reporting, and analysis. The company gives its more than 125,000 users the edge to outperform with informed insights, workflow solutions across the portfolio lifecycle, and industry-leading support from dedicated specialists.
Their complex and highly-trafficked system includes hundreds of services, thousands of instances, and multiple languages that ultimately send Lightstep nearly five million spans/minute.
Lightstep provides FactSet with full visibility into their deep system, enabling their developers to resolve incidents faster, understand dependencies, and ensure better customer experience.
Resolving Complex Issues in Hours Instead of Days
“I used Lightstep to track down a hard-to-find issue a number of layers below our entry point into a massive data store,” said Ed Thompson, a software engineer.
Thompson noticed one of their API calls was performing poorly. He was able to reproduce it one time, but the issue was intermittent, and only occurred once a day. A Lightstep trace was able to show this one request and track it through several layers of infrastructure that Thompson’s team wasn’t responsible for.
“We were able to rule out issues in our backend service and track this issue down to the most fundamental layer — totally obscured from our team previously,” said Thompson.
To resolve an intermittent issue in the lower level infrastructure, Thompson would have had to find the infrequent issue manually with logging, discover the team responsible and, notify them, explain the issue, and then wait for a response.
“It took me three to four hours to resolve this issue. Without Lightstep, this would have easily taken one or two days,” said Thompson.
Full System Visibility — Including Third-Party Dependencies
“Lightstep makes dependencies obvious, very quickly,” said Thompson.
“Just by looking at a trace I can see ultimately if there is, say, a sql dependency that is not apparent by executing these codes. And I can clearly see the topography of the dependency chain across the system.”
Full-system visibility with Lightstep means that FactSet can understand how performance issues from databases, third-party dependencies, and managed services propagate up into the application layers and impact the customer experience.
“There was an alert that went off for a service with a pool of hosts behind a load balancer. With Correlations, we could immediately see which backend host was the one that was having an issue,” said Logan Capaldo, a principal software engineer.
“One of the biggest advantages Lightstep offers is that all of sudden you see all the dependencies — even those you weren’t originally tracking or focused on,” said Bhu Boopathy, a software engineer.
"I used Lightstep to track down a hard-to-find issue a number of layers below our entry point into a massive data store."
Reduced MTTR from days
"We were able to rule out issues in our backend service and track this issue down to the most fundamental layer — totally obscured from our team previously."
Surfacing Once-Hidden, Logless Errors
FactSet has also found Lightstep helpful for understanding issues for services that are not instrumented or logging errors.
“I’ve used Lightstep to fix issues that are otherwise very difficult to identify, let alone resolve,” said Capaldo.
One example of this is when there was a request that didn’t get to the host and was unable to log anything. Capaldo and his team were still able to see the issue, because the client side was instrumented in Lightstep.
“You actually see the error spike in Lightstep, even though you couldn’t see the errors in the service itself,” said Capaldo. “The errors weren’t being logged where people were used to looking at, but Lightstep was able to help us resolve the issue.”
“It takes so little effort to get a lot of value out of Lightstep.”
Improved MTTI (Mean Time to Innocence)
FactSet has alerts set up on their services based on performance and availability metrics. When one of these goes off, a ticket is filed internally that may or may not fire a page depending on how critical the service is considered.
Through Lightstep’s Snapshot feature, every time a ticket is filed, the exact real-time state of the system is captured via a latency histogram and related dependencies are available to share.
“With Snapshots, we can very quickly say ‘this is not my fault’ and return to other work,” said Capaldo. “It’s saved us a lot of time — and stress.”
“Lightstep doesn’t just reduce MTTR, it reduces mean time to innocence. It has been tremendously helpful,” said Boopathy.
FactSet is beginning to take advantage of a new Lightstep feature, Service Health for Deployments, which automatically shows what impacted a service’s performance during and after a deployment, and surface why it happened.
“With Service Health for Deployments, we can quickly identify a specific regression and share it with the team responsible,” said Capaldo. “It’s already improving how we deploy to production.”
Empowered to Resolve Issues Before They Affect Customers
The FactSet team is now automatically and proactively made aware of certain issues they need to investigate with the help of Lightstep.
“There are many occasions where the automated monitoring in Lightstep notified us of an issue and we were able to resolve the issue before client impact,” said Thompson.
“Lightstep has helped change our behavior to look at services before there are performance issues,” said Boopathy. “Lightstep makes it easy, in part because it’s so inexpensive in time and effort to proactively address outliers and possible issues.”
In addition to automation, Lightstep empowers FactSet with tools to answer challenging questions about service and system health, even at the unique customer level.
“When Lightstep says things like ‘unlimited cardinality,’ they actually mean it,” explained Capaldo. “It really surprised me — there are no limits, just knobs you can tune to get as specific as you need.”
“This is really effective when combined with scale,” said Capaldo. “We’re sending 1000s of spans that are overly unique and Lightstep makes smart choices about what, and what not, to pay attention to.”
“It’s amazing what Lightstep can do to help deliver a great customer experience.”
World-Class Technical Support
In addition to providing full visibility into FactSet’s distributed system, Lightstep has been an exceptional technical partner with developers across the company.
“We use Lightstep as a benchmark for what a good vendor is,” said Boopathy.
“The interactions have all been incredible. Sometimes, we get into technical discussions with other vendors and it’s an uphill battle to get the point across. Not the case with Lightstep. Everyone we deal with is highly technical and very responsive.”
“Whether it’s a bug, a feature request, or a pull request on GitHub, the Lightstep team is available to help,” said Capaldo.