Job Description
US Work Authorization Requirement:
Candidates must be legally authorized to work in the United States without employer sponsorship. This includes, but is not limited to, U.S. Citizens, Permanent Residents, and other individuals with valid U.S. work authorization.
Job Description:
Responsibilities:
Design and implement scalable data pipelines using Azure Databricks and Azure Data Factory.
Develop and maintain robust ETL/ELT workflows using Apache Spark and PySpark DataFrames.
Build and optimize data pipelines for efficient data ingestion, transformation, and processing of large datasets.
Utilize data governance tools to manage data access, security, compliance, and data lifecycle.
Implement data quality frameworks and maintain data lineage across enterprise data platforms.
Design and support modern data architecture using Lakehouse and distributed data processing principles.
Develop high-performance Spark and SQL-based data transformations.
Supp
Apply Master Data Management (MDM) concepts to ensure data consistency, integrity, and standardization.
Work within the Azure Databricks ecosystem to deliver scalable, reliable, and enterprise-grade data engineering solutions.
Required Skills:
Azure Databricks
Apache Spark / PySpark
Azure Data Factory
SQL
Data Pipelines and ETL/ELT
Data Governance and Data Lineage
Lakehouse Architecture
