Sr. DevOps Engineer, Product Development Engineering

Comcast Philadelphia, PA

About the Job

Business Unit:

As a member of the LIVE (Linear Infrastructure Video Engineering) team within the VISE (Video Infrastructure and Security Engineering) organization, the primary task is to support operational engineering, data collection, analysis, monitoring and metrics configurations, application deployments, as well documentation and procedure development for Comcast Video Services and Video Platforms product lines.

In your role, you will bridge the gap between development and operations. Tasks will include but not limited to, implementing monitoring, metrics, and alerting configurations to quantify scope of reported linear video issues using tools and technology such as Grafana Kibana, InfluxDB, Prometheus, and Kubernetes. Create new metrics and identify monitoring deliverables to improve site reliability. Evaluate new code releases for basic reliability and provide systems integration support to our access network and divisional partners for deploying remote phy markets, and provide guidance to QA, Development, and Divisional/Regional stakeholder teams.

You'll be working with some of the best and brightest minds in Comcast while we operate and improve the video delivery infrastructure that will power the next generation of television. Success in this role is characterized by a stable and robust platform, with fewer customer-facing issues, defects and outages.

Responsibilities:

  • Function as a key member of the team responsible for the day-to-day system administration and operation of Comcast's video delivery systems with concentration on metrics data collection, analysis, reporting, tools and solutions development (dashboard development), systems integration, and operational support.
  • Participate in the design review of new technical features and architecture changes to the Linear Video delivery systems.
  • Perform Linux system administration, installation, configuration, and troubleshooting on hundreds of bare metal servers.
  • Participate in planning, testing and execution of system and application level upgrades.
  • Perform troubleshooting of Linear Video issues and problem resolution activities with other support teams (e.g. network support, divisional support, and vendors).
  • Produce Root Cause Analysis that focuses on recommendations and lessons learned and avoid problem recurrence.
  • Become proficient with Kubernetes command line interface for running commands against Kubernetes clusters when troubleshooting.
  • Assemble training materials for divisional support to enable divisional and customer care support capabilities.
  • Maintain productive relationships with peer organizations & and vendors.
  • Participate in 24/7 On-Call rotation.
  • Ability to travel required.

Here are some of the specific platforms and technologies we work with:

  • Linear Video
    • Audio/Video Compression MPEG
    • Multiplexing CAP1000
    • Edge QAM APEX1000, NSG
    • QAM to IP and IP to QAM Conversion
    • IP Video CIF Dash
    • Harmonic Prostream
    • SEMs
  • Linear Video DAI (Digital Ad Insertion)
    • Spotlight back office
  • CDN
    • Traffic Monitor
    • Traffic Router
    • Traffic Operations
    • Traffic Stats
    • ATS (Apache Traffic Server)
  • Cloud and Network Infrastructure
    • Cisco & Juniper Routers
    • F5 load balancers
    • VMWare, OpenStack
  • Logging/Monitoring/Host Management
    • ELK
    • Grafana
    • Prometheus
    • Puppet
    • Git

Skills & Requirements

  • 10 years Linear Video experience
  • 5 years UNIX/Linux administration
  • 3 years in-depth experience with IP networking (TCP/IP packet analysis with tcpdump and Wireshark, DNS, SNMP etc.)
  • Experience with open source tools and technologies such as Prometheus, Alert Manager, Elastic Search, Kibana, and Grafana
  • Experience with monitoring probes (Sentry, Sencore, Ineoquest)
  • Creation of shell scripts to automate tasks and perform basic data analysis
  • Experience with Kubernetes/Docker and Remote PHY is a plus




Comcast is an Equal Employment Opportunity/Affirmative Action/Drug-free workplace employer