Description
Design, develop, and maintain real-time data streaming pipelines using Spark Streaming or Azure Functions. We are seeking a highly skilled Senior Data Engineer or Data Architect with extensive experience in Synapse and Azure data services. The ideal candidate will have a deep understanding of Spark and Lakehouse architectures, as well as a proven track record of implementing and managing data solutions using Synapse services and other Azure tools.
Design, develop, and maintain real-time data streaming pipelines using Spark Streaming or Azure Functions. We are seeking a highly skilled Senior Data Engineer or Data Architect with extensive experience in Synapse and Azure data services. The ideal candidate will have a deep understanding of Spark and Lakehouse architectures, as well as a proven track record of implementing and managing data solutions using Synapse services and other Azure tools.
Key Responsibilities
Qualifications
Requirements:
· Proven experience in real-time data streaming using Spark Streaming or Azure Functions.
· Strong proficiency in Python for data processing and automation tasks.
· Hands-on experience with Kafka for data ingestion and message queuing.
· Familiarity with Redis for caching and fast data retrieval.
· Knowledge of data lake architectures and best practices for data storage and retrieval.
· Solid understanding of software engineering principles, including design patterns, testing, and documentation.
· Strong problem-solving skills and ability to work in a fast-paced, collaborative environment.
Preferred Qualifications:
· Experience with cloud platforms such as Azure
· Knowledge of other programming languages and frameworks related to data engineering.
· Familiarity with data modeling, ETL processes, and data lake concepts.
