Lead Data Engineer Job at The Phoenix Group, New York, NY

  • The Phoenix Group
  • New York, NY

Job Description

Role Overview

The Lead Data Engineer will own the design, development, and maintenance of robust data architecture and pipelines leveraging cloud-based technologies, supporting strategic analytics, AI/ML initiatives, and data governance initiatives across the organization’s enterprise-scale environment. Collaborating with cross-functional teams, this individual will ensure high-quality, secure, and performant data solutions that enable decision-making and operational excellence.

Key Responsibilities

  • Design, build, test, deploy, and support scalable data pipelines utilizing orchestration tools (Airflow, Astronomer), data transformation frameworks (dbt), cloud data platforms (Snowflake), SQL, and Python.
  • Develop and optimize cloud-based data architecture, including database schemas, data warehouses, security roles, workload isolation, cost management, and performance tuning within Snowflake.
  • Implement medallion data architecture principles—raw/bronze, silver, and gold layers—for structured and semi-structured data management.
  • Lead migration and modernization of legacy workflows to cloud orchestration platforms, establishing reusable patterns for dependency management, retries, backfills, alerting, recovery, and ensuring idempotency.
  • Troubleshoot and maintain the full runtime environment of orchestration platforms, including schedulers, workers, executors, metadata stores, queues, and autoscaling components.
  • Collaborate with cloud infrastructure teams to manage AWS services such as S3, IAM, EKS, ECS, Fargate, EC2, Lambda, Secrets Manager, KMS, CloudWatch, and networking components.
  • Build secure, resilient integrations across APIs, SFTP, databases, cloud storage, vendor APIs, flat files, Excel files, and event-driven systems.
  • Develop ingestion and processing pipelines for structured, semi-structured, and unstructured data, including OCR/image-based document extraction.
  • Embed data quality testing, monitoring, lineage, reconciliation, and observability into data workflows to ensure reliability and compliance.
  • Partner with Data Governance, Security, and Business teams to enforce data ownership, classification, access policies, retention, and auditing standards.
  • Build and maintain CI/CD pipelines for orchestration workflows, transformations, and infrastructure code, ensuring automated testing, environment promotion, deployment, and rollback capabilities.
  • Support production environments through on-call and incident management processes, conducting root cause analysis, stakeholder communication, and corrective actions.
  • Establish reusable frameworks, engineering standards, and best practices to improve platform reliability, consistency, and delivery velocity.
  • Lead technical workstreams, mentoring engineering team members through design reviews, code reviews, documentation, and troubleshooting.

Core Qualifications & Requirements

  • Minimum of 5+ years in data engineering or software engineering roles with progressive responsibilities.
  • Proven experience with cloud-based data platforms, including Snowflake, AWS cloud infrastructure, and data orchestration tools such as Airflow/Astronomer.
  • Strong expertise in designing and implementing data architectures, including dimensional and medallion models.
  • Hands-on experience migrating and maintaining production workflows in orchestration platforms.
  • Deep understanding of cloud compute environments, including Kubernetes, containers, schedulers, workers, and autoscaling architectures.
  • Proficiency in AWS services such as S3, IAM, EKS/ECS, Lambda, Secrets Manager, KMS, and CloudWatch for security, storage, and monitoring.
  • Skilled in integrating complex data sources and systems, including APIs, SFTP, relational and NoSQL databases, cloud storage, and file-based workflows.
  • Experience processing semi-structured and unstructured data, including OCR, document extraction, and image processing.
  • Strong background in implementing data quality controls, automated testing, data lineage, and observability tools.
  • Knowledge of data governance principles, access controls, data classification, retention policies, and audit requirements.
  • Experience with CI/CD pipelines, Git workflows, and automated deployment practices for data solutions and infrastructure.
  • Excellent troubleshooting skills across cloud infrastructure, data pipelines, application code, and security permissions.
  • Ability to lead technical initiatives and communicate effectively with both technical teams and business stakeholders.

Job Tags

Full time

Similar Jobs

Motorcycle Safety Academy

Motorcycle Mechanic Job at Motorcycle Safety Academy

 ...Motorcycle MechanicWe are seeking a skilled and dependable Motorcycle Mechanic to maintain and repair our fleet of training motorcycles. The ideal candidate will have strong technical skills, attention to detail, and a commitment to ensuring all motorcycles are safe,... 

Dr. Squatch

Strategic Operations Associate Director Job at Dr. Squatch

 ...! About the Role: Dr. Squatch is looking for a Strategic Operations Associate Director who will be a key leader within the Operations...  ...is responsible for all candidate presentations to our hiring managers. To protect the interests of all parties, Dr. Squatch will... 

Intrepid Digital

Search Engine Optimization Manager Job at Intrepid Digital

 ...Coach and coordinate: Guide analysts/specialists; collaborate with...  ...the curve: Navigate the AI-era search landscape (AI Overviews, SERP...  ...industry landscape, including search engine algorithms, competitors, and...  ...~ Unlimited PTO ~ Above-market remuneration ~ World-class client... 

SRI Tech

Azure DevOps Engineer Job at SRI Tech

 ...98033Employment Type: Full TimeIndustry: Computer SoftwareClient: WiproContact: Nidhi TotaCompany: SRI Tech SolutionsRole : Azure DevOps EngineerLocation : Redmond WAEmployment Type : Fulltime JD Strong Experience as Azure Cloud Software Engineer who would be responsible... 

Medical Delivery Services

Delivery Driver Knoxville, TN - 3 Days a Week Job at Medical Delivery Services

 ...Job Description Job Description Medical Delivery Driver (Independent Contractor 1099) Medical Delivery Services is contracting Independent Drivers (1099) for medical delivery routes using your own vehicle. No delivery experience required. Fast onboarding and steady...