Knowesis, Inc.

Senior Big Data Engineer – Cloudera Migration

Viewed 0 times

Job Description

Knowesis is seeking an experienced Cloudera Engineer to support a contract with the Department of Defense (DoD) Office of Inspector General (OIG) in all aspects of design, development, and delivery of data solutions.

This position requires U.S. Citizenship (applicants without proof of U.S. citizenship will not be considered due to the position’s security clearance requirement).

Duties and responsibilities include, but are not limited to:

  • Ensure Cloudera installation and configuration is at optimal specifications (CDP, CDSW, Hive, Spark, NiFi).
  • Perform critical data migrations from CDH to CDP.
  • Design and implement big data pipelines and automated data flows using Python/R and NiFi.
  • Assist and provide expertise as it pertains to automating the entire project lifecycle.
  • Perform incremental updates and upgrades to the Cloudera environment.
  • Assist with new use cases (i.e., analytics/ML, data science, data ingest and processing), Infrastructure (including new cluster deployments, cluster migration, expansion, major upgrades, COOP/DR, and security).
  • Assist in testing, governance, data quality, training, and documentation efforts.
  • Move data, and use YARN to allocate resources and schedule jobs.
  • Manage job workflows with Oozie and Hue.
  • Implement comprehensive security policies across the Hadoop cluster using Ranger.
  • Configure and manage Cloudera Data Science Workbench using Cloudera Manager.
  • Troubleshoot potential issues with Kerberos, TLS/SSL, Models, and Experiments, as well as other workload issues that data scientists might encounter once the application is running.
  • Communicate with users, other technical teams, and senior management to collect requirements, describe data modeling decisions and data engineering strategy.

 Qualifications/Skills

  • Understand the Hadoop and Spark architecture and the future of distributed processing frameworks.
  • Experience with HDFS and MapReduce for all file-related operations.
  • Experience implementing security by installing and configuring an authentication and authorization system such as Kerberos for all services in the cluster.
  • Knowledge of how to add, remove, and rebalance nodes in a cluster using cluster management tools.
  • Experience configuring Cloudera Manager to perform administrator operations. 
  • Experience running MapReduce and Spark applications in a Hadoop cluster.
  • An understanding of all the Hadoop daemons along with their roles and responsibilities in the cluster.
  • Solid understanding of Cloudera Data Science Workbench architecture.
  • Knowledge of HW/SW/Network/Security requirements for CDSW installation & upgrades.
  • Experience with deployment of CDSW on Cloudera Data Platform (CDP).
  • Knowledge of common tasks and guidelines related to cluster management on which CDSW may be hosted.
  • Knowledge of how to use TLS/SSL to enforce secure encrypted connections, using HTTPS and WSS, to the Cloudera Data Science Workbench web application.
  • Linux system administration experience.
  • Proficiency with a modern programming language, including R, Java, and/or Python.

Education: Bachelor of Science or higher in Computer Science, Engineering, Mathematics, or related technical discipline, with six (6) years minimum experience.

Certifications: Security +

Preferred Certifications: Cloudera Certified Professional (CCP) or Cloudera Certified Associate (CCA)

*Due to the COVID-19 Pandemic, Knowesis, Inc. requires all employees to be fully vaccinated as a condition of employment. An employee is considered fully vaccinated two weeks after receiving the second dose of a two-dose COVID-19 vaccine or one week after receiving a single-dose COVID-19 vaccine. All newly hired employees will be required to provide proof of vaccination. 

 

Equal Opportunity Employer/Protected Veterans/Individuals with Disabilities

The contractor will not discharge or in any other manner discriminate against employees or applicants because they have inquired about, discussed, or disclosed their own pay or the pay of another employee or applicant. However, employees who have access to the compensation information of other employees or applicants as a part of their essential job functions cannot disclose the pay of other employees or applicants to individuals who do not otherwise have access to compensation information, unless the disclosure is (a) in response to a formal complaint or charge, (b) in furtherance of an investigation, proceeding, hearing, or action, including an investigation conducted by the employer, or (c) consistent with the contractor’s legal duty to furnish information. 41 CFR 60-1.35(c)

Job Summary

wave-1-bottom
Potomac MD Location
Full Time Job Type
Knowesis, Inc.

Similar Jobs

The largest community on the web to find and list jobs that aren't restricted by commutes or a specific location.