Big Data Software / API Engineer

IBM   •  

San Francisco, CA

Industry: IT Consulting/Services


5 - 7 years

Posted 425 days ago

This job is no longer available.

You will be responsible for helping build an enterprise platform that is performant for an enterprise-wide data lake and cognitive services on the cloud.

You will work closely with otherdevelopers and internal IBM teams using the platform.

Success in this role requires the ability to collaborate successfully with cross-functional teams. Given the varied types of people you will work with, both technical and not, the ability to communicate effectively in person, in writing, over the phone, on a whiteboard, or on a video conference is essential.


Successful candidates should come from a strong data engineering background and must have significant experience with the back-end, APIs, cloud and big datatechnologies as well as traditional data warehouses.

* The ideal candidate will have significant experience with much of the following or equivalent:

*Experienced with Java development (must have at least 5 years of experience)

*Experience with developing and deploying RESTful/RPC-based services (must have at least 2 years of experience)

*Experienceworking with relational databases and solid knowledge of SQL (must have at least 3 years of experience)

*Experience with using search tools (e.g. ElasticSearch, Solr) (highly desirable)

*Experienceworking withBig Data tools in the Hadoop ecosystem (highly desirable)

*Experience with Enterprise authentication and authorization tools (highly desirable)

* 7+ years of big data software/API or backend software experience in reputed companies

* Strategic thinker with proven ability to build and manage scalable big data cloud systems using open source tools and big data

technologies such as Hadoop, HBase, Spark, etc.

*Experience in cloud-based Agile DevOps experience with PaaS & IaaS

*Experience with SQL, NoSQL, relational database design and methods for efficiently retrieving data

*Experience evangelizing solutions internally and train users on how to best optimize usage of platform

*Experience navigating internal processes for data, support, and hardware resources

*Experiencecollaborating with highly technical people and comfortable doing so; team leadership is a plus

* Ability to communicate effectively in person, in writing, over the phone, on a whiteboard or on a videoconference is essential

Specific Technical Skills and Knowledge:


* ElasticSearch

* Data knowledge, query languages, SQL, BigSQL, NoSQL

* Test-driven development methodology, JUnit, JMeter

*Hadoop, Oozie, Python

* Spark, Scala, Node.js, Flink and otherbig data analytics technologies

* Docker, containers, K8s

* Authentication, JWT, OAuth2

*Big data ingestion, processing, querying and tools

* API, middleware, and software development

* Cloud development and administration

* DevOps and Continuous Integration

*Agile Software Development, teamwork, and collaboration

* Technical leadership

Required Professional and Technical Expertise

* 5+ years of related work experience: in software engineering, architecture and design-designing and delivering large scale distributed software systems, preferably in large scale global business, service oriented architecture (SOA) principles and Web services standards and best practices

* Hands-on experience as a software architect designing and delivering large scale distributed software systems preferably in a large scale global business

* Capability to architect highly scalable distributed systems using open source tools and big datatechnologies such as Hadoop, HBase, Spark, Storm, etc.

*Experience in, cloud-based agile, DevOps with PaaS & IaaS, using Big Data batch and streaming tools SQL, NoSQL, relational database design, and methods for efficiently retrieving data

* Gather and process raw data at scale (including writing scripts, web scraping, calling APIs, write SQL queries, etc.)

* Programming in Java, Scala, Python, SQL

*Bachelor's Degree

Preferred Professional and Technical Expertise

* Communicate vision to diverse technical and non-technical audiences.

* Doctorate level degree


EO Statement

IBM is committed to creating a diverse environment and is proud to be an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, disability, age, or veteran status. IBM is also committed to compliance with all fair employment practices regarding citizenship and immigration status.


$110K - $120K