Associate Operations Engineer

At Blizzard Entertainment, Associate Operations Engineers in our Global Network Operations Center (GNOC) use systems and network expertise combined with software engineering to provide proactive monitoring and incident response of services across our world-wide operations. The role is comprised of talented engineers that are focused on rapid detection, triage, and mitigation of player-impacting issues and continuous improvement of service operations processes through monitoring, signal:noise, correlation, and automation.

Blizzard's games and platforms reach a global audience of passionate gamers. The GNOC is the nexus point between our games, platforms, and all of the teams dedicated to creating and supporting them. Responsible for player and employee-facing services, Associate Operations Engineers are the vanguard of service reliability, helping ensure that our creators can create and players can play.

As an Associate Operations Engineer at Blizzard, you may find yourself...

• Employing your systems/network/cloud knowledge as a first responder to triage problems and improve accuracy of detection
• Managing engagement, documentation, and stakeholder communications during critical service outages
• Driving player and user impacting issues to resolution
• Processing routine request for access, ACL, network, and systems maintenance
• Adjusting thresholds and duration of alert triggers to reduce false positive alerts
• Developing automation to reduce toil and increase velocity
• Building positive and collaborative relationships across the company
• Helping define and instrument Service-Level Objectives to ensure epic player experiences
• Adopting Containers and Kubernetes for new and existing services
• Applying everything-as-code methodologies across configuration, infrastructure, orchestration, and elsewhere Being part of an on-call rotation to ensure coverage across our 24/7 teams

You may succeed in this role if you...

• Love to solve novel and exciting problems
• Dislike solving the same problems over-and-over- so you automate or eliminate them
• Are inspired to make everyone's job easier by improving workflows
• Are comfortable digging through metrics, logs, and whatever else is available to triage and fix an incident at any time
• Strive to be better, smarter, and faster tomorrow than you are today
• Enjoy trying new technologies to improve what we're doing today
• Are okay using older technologies that may not be perfect, but are good enough and low maintenance
• Naturally spread the philosophies and practices of DevOps to others
• Like to collaborate with others to solve problems, share knowledge, and provide feedback
• Can self-assess the needs of a system or team, and make a case to prioritize that work
• Relish working with software, network, cloud, and systems engineers to solve problems across all tiers of the stack
• Help your peers succeed as much as you can

Types of projects you may work on...

• Evaluation and tuning of correlation patterns to improve alert clustering
• Integrating metadata with event correlation platform to improve enrichment / dependency mapping
• Develop scripts to automate information gathering / context for specific alert conditions
• Create microservices to enable self-service functionality (including ChatOps / Bots)
• Develop automations for auto-remediation of known-error incidents.
• Working closely with service owners to highlight opportunities to reduce interrupt work caused by incidents
• Integrating monitoring and logging with systems to improve observability and enable Service-Level Objectives

Areas of Expertise for an Operations Engineer at Blizzard

Operations Engineers at Blizzard are expected to have deep knowledge in the technologies used across the enterprise. Here's a non-exhaustive list of technologies Operations Engineers may be exposed to:

• Service Operations Framework/Workflows (ITIL, Incident/Problem/Event/Change Management, DMAIC)
• Monitoring (Prometheus, Kibana, Grafana, Elasticsearch, LogicMonitor, Datadog, Cloudwatch, GCP Operations)
• Alert Correlation Platforms (BigPanda, Moogsoft, ScienceLogic, Tivoli/Netcool)
• Linux (bash, debugging, tuning)
• Networking (triaging, packet loss, routing, ACLs)
• Programming (Python, JavaScript, Go, C#, Groovy, Shell, PowerShell)
• Automation Platforms (StackStorm, Rundeck, SALT)
• Cloud Services and Architecture (AWS, GCP, OpenStack)
• Identity Management and Directory Services (Active Directory, OpenLDAP, Okta)
• Logging (Splunk, SysLog, ELK Stack, Linux Journal, grep)
• Source Control (GitHub Enterprise, Perforce, SVN)
• CI/CD (Jenkins, Argo, Spinnaker)
• Service-Level Objectives (SLI, SLO, SLA, Error Budget, Burn Rate)
• Distributed Systems (architectures, micro-services, high-availability, elections, discovery)
• Configuration Management (Puppet, Hiera, Terraform, Terragrunt, Ansible, Consul)
• Container Computing (Docker, Kubernetes, Service Mesh)
• Proxies and Load Balancing (Nginx, HAProxy, Envoy)