Search Jobs

Data Engineer II

California, CA 00000

Posted: 09/15/2020 Industry: Back-End Development / Programming, Data Development / Management Job Number: 10484

Job Description


We are seeking a Data Solutions Engineer to join our Big Data Product team

Requirements
  • Developing and validating Big data products which runs on the large Hadoop cluster.
  • Developing and testing ETL process
  • Building big data and batch/real-time analytical solutions that leverage emerging technologies.
  • Perform data migration and conversion activities on different applications and platforms.
  • Design, develop and test data ingestion pipelines, perform end to end automation of ETL process and migration of various datasets.
  • Perform data profiling, discovery, analysis, suitability and coverage of data, and identify the various data types, formats, and data quality issues which exist within a given data source.
  • Develop transformation logic, interfaces and reports as need to meet project requirements.
  • Participate in discussion for technical architecture, data modeling, ETL standards, Migration activities, collaborate with Product Managers and Architects to establish the physical application framework (e.g. libraries, modules, execution environments)
  • Provide technical guidance to the other team members and contribute to the technical design and development for Data migration and Data Quality framework.
  • Tuning performance optimization of data pipelines
  • Develop unit and integrated automated test suites to validate end to end data pipeline flow, data transformation rules, and data integrity.
  • Develop tools to measure the data quality and visualize the anomaly pattern in source and processed data.


Preferred Requirements
  • Bachelor’ s degree in Computer Science or equivalent education/training
  • 4- 5 years of Software development and testing experience.
  • 3+ years of Working experience on tools like Hive, Spark, HBase, Sqoop, Impala, Kafka, Flume, Oozie, MapReduce, etc.
  • 3+ years of programming experience in Scala, Java or Python
  • Experience with development and automated testing in a CI/CD environment. Knowledge of GIT/Jenkins and pipeline automation is must.
  • Experience with developing and testing real-time data-processing and Analytics Application System.
  • Strong knowledge in SQL development on Database and/or BI/DW
  • Strong knowledge in shell scripting
  • Experience in Web Services -
  • API development and testing.
  • A solid understanding of common software development practices and tools.

Job Requirements

Chrystal

Meet Your Recruiter

Daniel Clifford

Apply Online
Apply with LinkedIn Apply with Facebook Apply with Twitter
Chat With A Recruiter

Send an email reminder to:

Share This Job:

Related Jobs:

Login to save this search and get notified of similar positions.