Big Data Engineer Job Descriptions, Average Salary, Interview Questions

As a Big Data Engineer at ABC Company, you’ll be responsible for data analysis solutions to automate the understanding and protection of the world’s largest digital enterprises. Your data science skills will be translated into opportunities for customers through the use of algorithmic, statistical, machine learning, and visualization techniques. You will demonstrate initiative and creativity by proposing ways to address problems often with large or incomplete data sets and validate findings using an experimental and iterative approach.

Your job

Your job will be to develop software to store and analyze massive amounts of customer enterprise data to reduce threat surface, increase cyber resilience, and facilitate digital transformations of some of the world’s largest enterprises. Specifically, you will:

• Leverage cloud-based data stores and write software using exciting, modern approaches to manage information on massive cyber environments.
• Collaborate with members of the engineering team to increase their big data engineering IQ and understand the needs of our customers.
• Design, develop, and implement novel detectors and classifiers of critical systems and activity through machine learning and other modern approaches.

What you should have accomplished

We’ll assess your skills and ensure your impact will scale with your ability. What we’re looking for to start:

• Bachelor’s degree or higher in mathematics, statistics, engineering, or computer science
• 3+ years of experience with modern data science and machine learning approaches, big data analytics, and software development skills
• Knowledge and ability to explain both the code and the underlying statistical or analysis approach
• Has used by modern data science algorithms/models
• Analytical and problem-solving skills
• Communication skills, critical thinking, and strategic thinking skills above and beyond the technical knowledge and implementation experience

We’re looking for an experienced Data Engineer to help with work centering around the most critical applications that manage client data and data privacy platforms. The Data Engineer will focus on designing, developing, and supporting all of our data solutions, and this role specifically focused around the data part of our systems. This person will work closely with business leads to design and build innovative solutions.

What You’ll Do

• Build, review, and improve new and existing data pipelines, improve data structure and availability
• Participate in design, development, and implementation of robust, high volume data-oriented solutions with big data technologies (Python, Spark, Hadoop, etc.)
• Works independently to determine methods and procedures on new or special assignment to find solutions to existing problems
• Responsible for creating re-useable processes that help implement each solution
• Having broad expertise or unique knowledge, uses skills to contribute to development of company objectives and principles and to achieve goals in creative and effective ways
• Works with stakeholders to understand the business needs and gather requirements to develop appropriate solutions. Prepares business and technical documentation

About you

• Ability to juggle multiple projects simultaneously and manage time efficiently
• Able to do your best work in a team setting and autonomously
• Owns a problem to the end
• Proud to share in team’s success
• Well-developed interpersonal skills
• Wants to grow a career with a great company.

What you’ll bring

• Bachelor’s degree in computer science, engineering, mathematics, related technical discipline or equivalent experience is required
• 3+ years of experience as a Data Engineer or in a similar role
• Experience with big data technologies is required (Python, Hadoop, Spark, etc.)
• Knowledge of data management fundamentals and data storage principles
• Experience with data modeling, data warehousing, and building ETL pipelines
• Proficiency in Python and Spark required
• Experience with SQL and relational database development is required
• Extensive experience working in a Unix environment is a must
• Excellent analytic skills, deadline focused, detail oriented, well organized, and self-motivate
• Experience writing Unix Shell Scripts is required

Preferred qualifications

• Experience with Maven and Git
• Proven success in communicating with users, other technical teams, and senior management to collect requirements, describe data modeling decisions and data engineering strategy
• Knowledge of software engineering best practices across the development lifecycle, including agile methodologies, coding standards, code reviews, source management, build processes, testing, and operations

As a Big Data Engineer, you will be responsible for engaging in the design, development, and maintenance of the big data platform and solutions. This includes analytical solutions that provide visibility and decision support using big data technologies.

The role involves developing data integration solutions and working in a cross functional team focused on building Products and Solutions. Working with Product owners, system administrators, Data Engineers, data scientists, and data architects, you will ensure that solutions built meet the business demands.

Responsibilities

• Develop ELT processes from various data repositories and APIs across the enterprise, ensuring data quality and process efficiency
• Develop data processing scripts using Spark
• Work in a hybrid Data environment with cloud technologies and enterprise Datawarehouse
• Identify, investigate, and solve data quality issues and make sure the data is secured and reliable
• Develop relational and NoSQL data models to help conform data to meet users’ needs
• Address performance and scalability issues in a large-scale data lake environment

Minimum qualifications

• 5+ years of experience working with Big data solutions, including databases (SQL/No-SQL) and building ETL and Data pipelines using cloud services
• Fluency in SQL with other programming languages, such as Python
• Experienced in design, implementation, maintenance of custom ETL, Data pipelines
• Experienced in analyzing data to identify deliverables, gaps and inconsistencies

Preferred qualifications

• 5 years of experience with big data/Hadoop distribution and ecosystem tools, such as Hive, Spark, Kafka and AWS Glue, Lambda, EMR
• Experienced in design and implementation of batch and real time data pipelines
• Experienced in modeling and writing complex queries[H(2]
• Experienced in Data warehouse concepts and Dimensional Modeling
• Experience with Data quality monitoring, performance tuning and End to End process optimization
• Bachelor’s/Master’s degree in computer science, information technology, or a related field or equivalent experience
• Knowledge with Linux system administration, Linux scripting, and basic network skills
• Experience with coding against and developing REST API’s