
PySpark Backend Engineer - AI Security R&D Israel
- ישראל
- משרה קבועה
- משרה מלאה
- Design and implement backend logic using PySpark within Databricks, focusing on data processing using Spark Structured Streaming.
- Develop and maintain microservices and backend components in Python and C#, deployed in Kubernetes environments.
- Collaborate with product managers and backend teams to translate business requirements into Spark-based logic and services.
- Optimize Spark jobs for performance, reliability, and cost-efficiency in a cloud-native environment (Azure).
- Own features end-to-end: from refining requirements with product managers, through design and implementation, to deployment and support.
- 5+ years of experience in backend development, with a strong focus on data-intensive applications.
- Hands-on experience with PySpark and Spark Structured Streaming in production environments - must
- Solid understanding of Spark internals, job optimization, and streaming semantics.
- Proficiency in Python. Experience with C# is an advantage.
- Experience building and deploying microservices in Docker/Kubernetes environments. Familiarity with cloud platforms (Azure/AWS/GCP)
- Experience working in Databricks
- Experience in cybersecurity or real-time data processing domains - advantage.