Senior Infrastructure Engineer - Logging

Get Referred

Job Description

Why American Express?


There’s a difference between having a job and making a difference.


American Express has been making a difference in people’s lives for over 160 years, backing them in moments big and small, granting access, tools, and resources to take on their biggest challenges and reap the greatest rewards.


We’ve also made a difference in the lives of our people, providing a culture of learning and collaboration, and helping them with what they need to succeed and thrive. We have their backs as they grow their skills, conquer new challenges, or even take time to spend with their family or community. And when they’re ready to take on a new career path, we’re right there with them, giving them the guidance and momentum into the best future they envision.


Because we believe that the best way to back our customers is to back our people.


The powerful backing of American Express.

Don’t make a difference without it.

Don’t live life without it.


If you want to invest your time and energy into creating innovations that make a difference for a global IT services organization, then join the Enterprise Monitoring, Tooling and Engineering team at American Express.  Be a part of the team responsible for introducing and supporting technology that improves the availability, performance and efficiency of American Express’ IT operations.


EMTE seeks a Logging Engineer with the ideas, knowledge, and strengths to enhance how American Express Technology uses log monitoring and machine data to gain greater Operational Intelligence.  This Senior Infrastructure Engineer will align all engineering designs with American Express’ architectural enterprise standards and promote the adoption of best practices offered through the Enterprise Logging Center of Excellence. Success for this individual’s performance and outcomes will be measured, in part, on the engineer’s ability to lead and collaborate with team members, technology partners and other stakeholders to create innovative solutions that achieve personal goals and those set by organizational leaders and the team.

Job Description

As a Logging Infrastructure Engineer you will:

  • Maintain and enhance multiple Logging aggregation tools/environments (e.g., Splunk, Elastic, ELK, Red Hat Linux)

  • Assist various teams with data onboarding into Splunk and/or Elastic

  • Assist users with log search queries, dashboards, and applications for use by Operations, Development, and Management personnel

  • Mentor Splunk/Elastic users and administrators
  • Work closely, at a deep technical-level, with engineering teams to ensure solution designs are consistent with American Express Technology’s architectural vision, platform/product roadmaps, enterprise standards, guidelines and principles

  • Ensure compliance with security standards and assist in audit preparations.

  • Develop, document and implement enterprise standards and procedures
  • Monitor environment and computing resources for reporting and capacity planning.

  • Maintain systems documentation
  • Assist with the administration/support of other EMTE platforms as necessary

  • Participate in 24x7 on-call support rotations for monitoring and automation tools during business hours, nights and weekends

  • Function as an active member of an agile DevOps team, consistently contributing to the team and its Agile practices (tools, common components, and documentation)

  • Adopt DevOps methods and roles in support of monitoring and automation tools/services

  • Assist in troubleshooting various system, network, and application issues using log data

  • Follow Incident/Problem/Change Management, SOX and PCI processes
  • Perform all activities in a timely manner, as required, to contribute toward Enterprise-level compliance of internal/external processes, standards and regulatory controls.

  • Perform other duties as assigned. 


  • Knowledge of Splunk and/or Elastic administration and maintenance.

  • Knowledge of Splunk and/or Elastic cluster construction and administration.

  • Knowledge of Splunk and/or Elastic application development and optimization.

  • Ability to write scripts in one or more languages. (shell, Perl, Python, Ruby, etc…)

  • Familiarity with Red Hat Enterprise Linux 6 and 7.

  • Experience creating and supporting highly available enterprise production environments.

  • Fundamental knowledge of TCP/IP networking, subnetting and routing concepts, and distributed computing concepts;

  • Ability to self-direct personal activities to achieve goals and meet commitments

  • Self-motivated leader with strong interpersonal skills and ability to work in cross-functional and inter-organizational teams.

  • Ability to persuade and influence others without direct control.

  • Able to manage multiple projects tasks and those of supporting team members needed to meet multiple demands in a dynamic, fast paced environment.

  • Strong analytical, logical reasoning

  • Strong troubleshooting skills and experience working within a heterogeneous environment. 

  • Ability to solve problems quickly and independently

  • Ability to automate processes

  • Strong written and verbal communication skills, with the ability to influence cross-functional teams, business and/or vendor partners, and technology leaders

  • Able to develop/make presentations, facilitate discussions and provide technical demonstrations in 1:1, small group and large group settings.

Required Background
  • 1-5 years’ experience with systems analysis/programming, incorporating design methodology, infrastructure support and/or network administration

  • 1 or more years’ experience in Splunk and/or Elastic production administration

  • Experience working in a team/workgroup setting

  • Bachelor’s Degree in computer science, computer engineering preferred, or experience in related field required

Desirable Skills and Experience
  • Working knowledge of Linux administration: configuration and tuning, networking, logging, storage, and installation and integration of third-party software.

  • Programming background in any applicable language.

  • Familiarity of SOX, PCI DSS and other regulatory standards helpful

  • Ability to design, and present training to varying levels of users.

  • Experience with automation tools such as Ansible or Puppet

  • Experience with Kafka
  • Experience with other Operational Intelligence and monitoring tools such as: Dynatrace, AppDynamics, ICINGA, Prometheus, Graphite/Grafana, InfluxDB, etc.

  • Prior experience in DevOps or DevOps-like environment (Practices that emphasize the collaboration and communication of both software developers and operations engineers)

    • Working knowledge of Application Development workflow and Agile Methods

    • Experience working with Scrum or Kanban-related tools and concepts (e.g., Jira, Rally, Epics, Stories, estimating story points, etc.)

Professional and Leadership Qualities for Success
  • Must be a highly motivated, energetic self-starter who excels in fast-paced, dynamic, team environments and committed to getting results

  • Strong technical acumen, passionate about learning and trying new technology

  • Ability to self-direct personal activities to achieve goals and meet commitments

  • Strong analytical, logical reasoning

  • Ability to solve problems quickly and independently

  • Excellent organizational/time management skills, able to manage multiple tasks

  • Strong interpersonal skills, strong written/verbal communications skills (i.e., presentations, documentation, emails, reports, etc.)

  • Innovates through experimentation, failing fast, and continuous improvement

  • Seeks and offers constructive feedback, willing to learn from mistakes

  • Provides astonishing customer service, while exhibiting an attitude of excellence and absolute integrity in everything 


Employment eligibility to work with American Express in the U.S. is required as the company will not pursue visa sponsorship for these positions. 


ReqID: 19016666
Schedule (Full-Time/Part-Time): Full-time
Date Posted: Oct 25, 2019, 3:41:43 PM