Data/AI Engineer

Overview:  We are looking for a Senior Data Engineer with deep expertise in Lakehouse architecture, real-time data streaming, cloud data infrastructure, and microservices development on Azure Kubernetes Service (AKS). You will play a central role in designing and delivering next-generation data pipelines, BI solutions, AI/ML platforms, streaming APIs, and scalable microservices that power Guidepoint's research and analytics products.  This is a high-impact, hands-on engineering role. You will work closely with data architects, data scientists, analysts, frontend engineers, QA, and DevOps teams to translate complex business requirements into scalable, reliable, and observable data systems.  This is a Hybrid role from our Pune office. What You'll Do: Data Engineering & Lakehouse  Design, build, and maintain ETL pipelines, data ingestion workflows, and table schemas on Azure Databricks to support BI, analytics, and AI/ML use cases  Architect and optimize the Lakehouse using Delta Lake on Databricks, ensuring reliability, performance, and cost efficiency  Build and support data pipelines from business applications such as Salesforce, NetSuite, and other enterprise systems  Develop and maintain Knowledge Graph models, entity relationship structures, and NLP-based insight pipelines  Maintain data governance, data privacy standards, and compliance best practices throughout the data lifecycle  Perform root cause analysis on data and processes to identify opportunities for improvement  Collaborate with data architects, scientists, and business consumers to populate and optimize the data warehouse for reporting and analytics  Microservices & AKS Development  Develop and support scalable web APIs and microservices using Python and Azure Platform Services  Build new applications, services, and platforms; optimize existing solutions and refactor legacy components using modern, scalable architectures  Design, implement, and deploy microservices on Azure Kubernetes Service (AKS) using Docker, Kubernetes, Helm, and Azure DevOps YAML pipelines  Perform end-to-end deployments including infrastructure setup, configuration, and monitoring on AKS  Decompose portions of legacy applications into modern microservices architecture  Design and manage JSON payloads and payload contexts for inter-service communication  Engage in database schema design and management, including updating tables and rows for large datasets  Collaborate with cross-functional teams — Full-Stack, QA, DevOps, and Product — in agile SDLC processes  Real-Time Streaming & SSE  Design and implement robust SSE (Server-Sent Events) endpoints using Python frameworks (FastAPI, Flask, Django) for real-time event delivery to web and mobile clients  Build and maintain asynchronous backend services using asyncio, aiohttp, or similar libraries for non-blocking, high-concurrency streaming  Architect streaming data pipelines integrating SSE with upstream message brokers — Kafka, Redis Pub/Sub, RabbitMQ  Optimize connection lifecycle management: reconnection logic, heartbeat signals, event ID tracking, and graceful shutdowns  Collaborate with frontend teams to define and evolve SSE event schemas and API contracts  Implement observability across streaming services: distributed tracing, structured logging, and metrics using Prometheus, Datadog, or OpenTelemetry  Engineering Excellence  Write comprehensive unit, integration, and load tests for all data, streaming, and microservices components  Write and maintain robust CI/CD pipelines using Azure DevOps YAML pipelines  Participate in architecture reviews, code reviews, and on-call rotations  Maintain thorough technical documentation and mentor junior engineers on best practices in data engineering, Lakehouse architecture, streaming systems, and microservices    What You Have: Required  Bachelor's degree in Computer Science, Engineering, or a related field from an accredited university  7+ years of professional data engineering and/or backend software engineering experience  Advanced SQL expertise across relational and NoSQL databases (SQL Server, Neo4j, Elasticsearch, Cosmos DB)  Strong hands-on experience building and optimizing data pipelines on Azure Databricks  In-depth knowledge of Delta Lake, Data Warehousing, and Lakehouse architecture  Highly proficient in Spark, Python, and SQL  Proven experience designing and deploying microservices on AKS using Docker, Kubernetes, and Helm  Hands-on experience with Azure DevOps YAML pipelines for CI/CD automation  Experience with SSE or real-time streaming — event stream formatting, retry logic, connection management  Strong grasp of async Python: asyncio, async/await, event loops  Experience with message brokers: Kafka, Redis Streams, RabbitMQ, or similar  Proven track record of processing and extracting value from large, complex, and disconnected datasets  Excellent stakeholder management and communication skills across global, cross-functional teams  Proven leadership skills with a strategic mindset and passion for driving innovation  Nice to Have  Experience with Fivetran for data integration  Familiarity with BI tools such as Power BI  Experience building and deploying ML and feature engineering pipelines using MLflow  Knowledge of Knowledge Graph development (e.g., Neo4j) and NLP-based analytics  Familiarity with cloud-based AI/ML services and Generative AI tools  Experience working in a compliance-based environment (building and deploying compliant software throughout the SDLC)  Familiarity with API gateway configuration for streaming (NGINX, Kong, Azure API Gateway)    What We Offer:  Competitive compensation  Employee medical coverage  Central office location  Entrepreneurial environment, autonomy, and fast decisions  Casual work environment  About Guidepoint:  Guidepoint is a leading research enablement platform designed to advance understanding and empower our clients’ decision-making process. Powered by innovative technology, real-time data, and hard-to-source expertise, we help our clients to turn answers into action.  Backed by a network of nearly 1.75 million experts and Guidepoint’s 1,600 employees worldwide, we inform leading organizations’ research by delivering on-demand intelligence and research on request. With Guidepoint, companies and investors can better navigate the abundance of information available today, making it both more useful and more powerful.  At Guidepoint, our success relies on the diversity of our employees, advisors, and client base, which allows us to create connections that offer a wealth of perspectives. We are committed to upholding policies that contribute to an equitable and welcoming environment for our community, regardless of background, identity, or experience.    #LI-AD2 #LI-HYBRID     Offices: Pune, Maharashtra, India (Pune);