Data Engineer II
We are seeking a Data Solutions Engineer to join our Big Data Product team
- Developing and validating Big data products which runs on the large Hadoop cluster.
- Developing and testing ETL process
- Building big data and batch/real-time analytical solutions that leverage emerging technologies.
- Perform data migration and conversion activities on different applications and platforms.
- Design, develop and test data ingestion pipelines, perform end to end automation of ETL process and migration of various datasets.
- Perform data profiling, discovery, analysis, suitability and coverage of data, and identify the various data types, formats, and data quality issues which exist within a given data source.
- Develop transformation logic, interfaces and reports as need to meet project requirements.
- Participate in discussion for technical architecture, data modeling, ETL standards, Migration activities, collaborate with Product Managers and Architects to establish the physical application framework (e.g. libraries, modules, execution environments)
- Provide technical guidance to the other team members and contribute to the technical design and development for Data migration and Data Quality framework.
- Tuning performance optimization of data pipelines
- Develop unit and integrated automated test suites to validate end to end data pipeline flow, data transformation rules, and data integrity.
- Develop tools to measure the data quality and visualize the anomaly pattern in source and processed data.
- Bachelor’ s degree in Computer Science or equivalent education/training
- 4- 5 years of Software development and testing experience.
- 3+ years of Working experience on tools like Hive, Spark, HBase, Sqoop, Impala, Kafka, Flume, Oozie, MapReduce, etc.
- 3+ years of programming experience in Scala, Java or Python
- Experience with development and automated testing in a CI/CD environment. Knowledge of GIT/Jenkins and pipeline automation is must.
- Experience with developing and testing real-time data-processing and Analytics Application System.
- Strong knowledge in SQL development on Database and/or BI/DW
- Strong knowledge in shell scripting
- Experience in Web Services -
- API development and testing.
- A solid understanding of common software development practices and tools.