Distributed Security Alerting
There are lots of tools that are really good at scanning for vulnerabilities and misconfigurations. Getting the scanner results to the right people and ensuring those people have the knowledge and resources to fix the problem is vital to the usability of these tools. This talk will dive into the open source tools and techniques Spotify has created to help our developers understand and learn from security scans.
While there are hundreds of tools available to scan for misconfigurations and vulnerabilities they are usually made for security professionals. However they tools become most effective when helping normal developers catch problems with their setup. As a security engineer it becomes easy to spend all of your time redirecting these alerts in order to fix the last mile problem. Even when it is not manual many of the alerting frameworks that exist are not great for security products. During this talk we will dive into distributed security alerting. We will start by discussing ownership of security in products at Spotify. We will examine alerting and hardening to understand when we would use alerting. We will look into different types of alerts that the security team generates and how ideally we would want teams to react to them. Then we will put ourselves in the shoes of a developer getting the alert and think of all the things we would want to act in the way the security team would like. After that we dive into Comet. Comet is a open source tool created by the Spotify team to help us deal with alerts that we want to send out to developers. We will go through the features that enable teams to use it easily like alert batching and automatic escalation. We also will dive into how the security team develops it. We will examine the architecture and the code to understand why the modularization makes it so easy to write new alerting sources. Lastly we will dive into the metrics that Comet collects. We will understand the types of metrics we built for ourselves as scanner owners. These allow us to look at how effective our alerts are. We will also look at the metrics we built for product managers. These metrics allow them to understand how their work is being scanned. It also allows them to compare themselves to other parts of the organization and gives security numbers as hard evidence when discussing the security of products in other parts of the organization.