Site Reliability Engineering Lead
About Graphcore
How often do you get the chance to build a technology that transforms the future of humanity?
Graphcore products have set the standard in made-for-AI compute hardware and software, gaining global attention and industry acclaim. Now we are developing the next generation of artificial intelligence compute with systems that will allow AI researchers to develop more advanced models, help scientists unlock exciting new discoveries, and power companies around the world as they put AI at the heart of their business.
Graphcore recently joined SoftBank Group – bringing large and ongoing investment from one of the world’s leading backers of innovative AI companies.
Job Summary
We are seeking an experienced Site Reliability Engineering leader to build and lead a new SRE organization responsible for the production operation of a rapidly scaling AI supercomputing platform. The environment combines highly customized compute, high-performance networking, storage and supporting infrastructure, and will grow through multiple phases of deployment.
This is a rare opportunity to establish the reliability function for a new platform from the ground up. The platform and its operational model are being developed in parallel and will ultimately support a 24x7x365 production service with stringent availability requirements.
You will take the SRE organization from initial formation through production launch, stabilization and scale. This includes hiring and developing the team, defining the operating model, establishing production readiness and incident-management practices, and ensuring reliability and operability are engineered into the platform from the outset.
SRE is responsible for the operational capability required to run the platform reliably in production, while partnering with engineering teams that remain accountable for the reliability and operability of the systems they build.
This is not a purely managerial position. During the development and early production phases, the SRE Manager will be expected to work directly with engineering teams, develop a deep understanding of the platform, and participate in troubleshooting and incident response.
Over time, success will increasingly mean building the people, processes, automation, tooling, and operational discipline that allow the organization to operate effectively without depending on you for day-to-day escalation.
Responsibilities and Duties
- Build the SRE Organization:
- Build and develop the team from its initial formation through full 24x7x365 production operations, including defining roles, interviewing and hiring team members, establishing career expectations, and developing future technical leaders.
- Mentor engineers and team leads, develop successors, and build an organization capable of operating effectively without depending on any single individual.
- Work with leadership to forecast staffing requirements as the platform grows from initial deployment through full production scale.
- Establish the Production Operating Model:
- Define the operating model for an SRE organization, including staffing and coverage model, escalation paths, on-call responsibilities, incident management, handoffs, production access, change management, and operational readiness requirements.
- Establish clear operational interfaces with Datacenter Operations, engineering teams, vendors, and other service owners.
- Establish and continuously improve production readiness standards, runbooks, operational procedures, failure-mode documentation, escalation processes, and incident response practices with a strong emphasis on automation and engineering over manual operational work.
- Develop training, cross-training, simulation, and production incident-response exercises to ensure the team can operate independently and confidently.
- Engineer Reliability Into the Platform:
- Embed with platform engineering teams during development to gain deep technical knowledge of the system and ensure reliability, serviceability, observability, and operational requirements are incorporated into the platform before production.
- Lead the development of SLOs, operational health indicators, alerting standards, incident severity definitions, and reliability reporting appropriate for a large-scale production infrastructure service.
- Build a culture in which recurring operational problems are engineered out through automation, improved observability, better platform design, and elimination of unnecessary toil.
- Ensure the SRE organization can rapidly diagnose and mitigate issues across compute, networking, storage, and supporting infrastructure.
- Develop strong technical depth within the team while maintaining access to specialist expertise in critical areas such as high-performance networking, storage, observability, and platform scheduling.
- Lead Production Operations:
- Lead or participate in major production incidents as necessary, particularly during platform development, launch, and early production.
- Establish a blameless post-incident review process focused on identifying systemic improvements and ensuring corrective actions are completed.
- Serve as the senior operational authority for the SRE organization and represent production reliability concerns in engineering and leadership discussions.
Required Skills and Experience
- Significant experience leading or building an SRE, Production Engineering, Infrastructure Reliability, or comparable function supporting large-scale, highly available production infrastructure.
- Experience taking a new or rapidly evolving platform through production readiness, launch, stabilization, and ongoing operation.
- Experience building and operating sustainable 24x7x365 production support or on-call organizations.
- Strong understanding of modern Site Reliability Engineering principles, including SLOs, incident management, observability, automation, toil reduction, capacity management, and production readiness.
- Strong incident leadership experience, including managing high-severity, multi-team production incidents under significant time pressure.
- A strong systems engineering background with a working understanding of Linux, networking, storage, distributed systems, automation, and production infrastructure.
- Ability to operate effectively at both the leadership and hands-on technical levels, moving comfortably between organizational design, architecture discussions, troubleshooting, and incident response.
- Demonstrated ability to lead, hire, mentor, develop, and retain a growing team of strong technical engineers at various levels of capability
- Strong judgment regarding when to solve an immediate operational problem and when to invest in eliminating the underlying failure mode.
- Strong written and verbal communication skills, particularly during incidents and when communicating technical risk to senior leadership.
- Ability to manage conflicting priorities under a pressured environment
Desired but Not Required
Candidates are not expected to have experience in all the areas below. Experience in several would be particularly valuable:
- High-performance networking technologies such as InfiniBand, RDMA, RoCE, or large-scale Ethernet fabrics.
- Large-scale parallel or distributed storage systems.
- Workload schedulers and orchestration platforms such as Kubernetes, Slurm, or comparable systems.
- Production observability and service-level monitoring for complex distributed infrastructure.
- Custom or early-generation hardware, firmware, or environments where hardware and software are developed concurrently.
- Operational relationships and escalation processes involving datacenter operations and infrastructure/hardware/networking vendors.
- Rapid capacity expansion, datacenter migration, or transitions between temporary and permanent production environments.
What Success Looks Like
- The SRE organization is staffed, trained and capable of supporting the platform through production launch and subsequent growth.
- Production readiness, observability, incident management and operational engineering practices are embedded into how the platform is developed and operated.
- The team has the technical depth, automation and operating discipline to resolve most production issues independently, with leadership escalation reserved for genuinely exceptional events.
In addition to a competitive salary, Graphcore offers flexible working and a comprehensive benefits package designed to support your health, wellbeing and financial future. Our benefits include medical, dental and vision coverage, Flexible Spending Accounts (FSAs), Health Savings Accounts (HSAs), disability and life insurance, a 401(k) retirement plan, commuter benefits, wellness services and an Employee Assistance Programme (EAP). We welcome people of different backgrounds and experiences; we're committed to building an inclusive work environment that makes Graphcore a great home for everyone. We offer an equal opportunity process and understand that there are visible and invisible differences in all of us. We can provide a flexible approach to interview and encourage you to chat to us if you require any reasonable adjustments.
About Graphcore
Graphcore develops a microprocessor designed for AI and machine learning applications.
More jobs at Graphcore
- Staff AI Performance Engineer
- Principal AI Performance Engineer
- Senior AI Performance Engineer
- Supply Quality Engineer
- Director - Post-Silicon Validation (Functional)
- Staff Robotics Engineer
- Staff Hardware Engineer
- Senior Systems Engineer
- Principal Security Architect
- Principal SLT Test Engineer