Technical Lead/ Associate Technical Lead – Observability Engineer

About the job

Technical Lead/ Associate Technical Lead – Observability Engineer

Sysco LABS is the captive innovation arm of Sysco Corporation (NYSE: SYY), the world’s largest foodservice company. Sysco is a Fortune 500 company and the global leader in selling, marketing, and distributing food products to restaurants, healthcare, and educational facilities, lodging establishments and other customers who prepare meals away from home. Its family of products also includes equipment and supplies for the foodservice and hospitality industries. With more than 72,000 colleagues, the company operates 334 distribution facilities worldwide and serves approximately 725,000 customer locations. For fiscal year 2023 that ended July 1, 2023, the company generated sales of more than $76 billion.

Operating with the agility and tenacity of a tech–startup, powered by the expertise of the industry leader, Sysco LABS is perfectly poised to transform one of the world’s largest industries.

Sysco LABS’s engineering teams based out of Colombo, Sri Lanka and Austin and Houston, TX, innovate across the entire food service journey – from the enterprise grade technology that enables Sysco’s business, to the technology that revolutionizes the way that Sysco connects with restaurants and the technology that shapes the way those restaurants connect with customers.

Sysco LABS technology is present in the sourcing of food products, merchandising, storage and warehouse operations, order placement and pricing algorithms, the delivery of food and supplies to Sysco’s global network, the in-restaurant dining experience of the end-customer and much more.

As a Lead Observability Engineer, you will be responsible for designing, implementing, and maintaining observability solutions for complex systems and applications across Sysco. You will work closely with development, operations, monitoring and ITSM teams to ensure that the systems and applications are monitored, logged, and traced effectively, allowing for efficient troubleshooting, debugging, and performance analysis. You will also be responsible for defining observability standards and best practices, driving the adoption of observability technologies, and continuously improving the observability posture of the organization. You will be a key stakeholder in Sysco Major Incident Management process supporting technology teams to troubleshoot and resolve complex production issues faster using Datadog.

The Role:

• Define, document, and enforce observability standards and best practices across the organization for the three primary pillars of observability (Logs, Traces and Metrics) and research and develop new solutions for other pillars of observability (like RUM, Synthetic Monitoring, Network monitoring and profiling)
• Collaborate with development and SRE teams to identify and address areas of improvement in the observability stack while ensuring that observability is integrated into the software development lifecycle.
• Design and develop standard dashboards for critical metrics for various Sysco applications and services using the observability data.
• Continuously monitor and analyze system and application performance data, identify trends and anomalies, and make recommendations for improvement.
• Monitor and maintain Sysco’s observability tool stack (Datadog and Servicenow ITOM), ensuring they are up to date, healthy, secure, and compliant.
• Responsible for Datadog platform usage and capacity planning to ensure technology teams can consume all the platform features effectively.
• Stay up to date with the latest trends and advancements in observability technologies and best practices, evaluate the viability of such in Sysco’s context and provide thought leadership.
• Collaborate with technology teams to troubleshoot and resolve complex production issues related to system performance, and reliability through observability tools and actively engage in Sysco’s Major Incident Management process for incident response and resolution.
• Provide training, mentorship, and guidance to other team members on observability concepts, tools, and practices.
• Enable integrations and build custom integrations where necessary to onboard new data sources and metrics to Datadog platform.
• Drive process optimization via automation to ensure L1 monitoring team can monitor the applications effectively and efficiently.

The Profile:

• Proven experience as an Observability Engineer or similar role in Software Engineering or SRE with a strong understanding of observability concepts, methodologies, and tools.
• Expert level experience in monitoring and logging technologies, both open source and closed source. Previous working experience in Datadog or Dynatrace will be added advantage.
• Strong programming and scripting skills (e.g., Python, Go, JavaScript) for automation and customization of observability solutions.
• Deep understanding of system and application architectures, distributed systems, microservices, and cloud computing.
• Experience with DevOps practices and tools (e.g., Git, Jenkins, Docker, Kubernetes) for continuous integration, deployment, and delivery.
• Excellent analytical, problem-solving, and troubleshooting skills.
• Strong communication and collaboration skills to work effectively with cross-functional teams.
• Ability to work in a fast-paced, dynamic environment and adapt to changing requirements and priorities.
• A working knowledge in Network is needed. Fundamental knowledge of TCP/IP stack, application protocols (DHCP/DNS/HTTPs) and networking concepts (HSRP/NAT/VPN/VLANs/802.1x/Wireless/Clustering/High Availability/Load Balancing)
• Strong in troubleshooting incidents in production environment

Why you should join Sysco LABS:

• An attractive base compensation and benefits package which is comfortably above market rates.
• The opportunity to work with high caliber individuals across the organization.
• Technical training and soft skills development programmes.
• Fast career growth, recognition and senior leadership opportunities for good performers
• A flexible, diverse and entrepreneurial work environment and a fun work culture that celebrates success regularly.