NIQ Hiring Drive: NIQ is currently hiring for the job of Data Engineer (Python with Polars) in Chennai in Chennai and Pune. This position offers the opportunity to be a part of large-scale systems for data, which will allow enhanced product analytics and insights from global data. The ideal candidate will have the necessary expertise in Python data programming as well as high-performance data processing as well as the most modern cloud-based data structure.
About the Role
The job involves conceiving developing, constructing, and sustaining large-scale data processing workflows to support NIQ’s Digital Shelf. The focus of the job is on data pipelines with high performance for images and automating data workflows. This requires hands-on experience with Python, Polars, and PostgreSQL and a dedication to excellence in engineering as well as reliability and scalability.
This position offers an amazing chance to use the latest data technologies and contribute to the process of performance optimization through efficient data pipelines.
Key Responsibilities
The Data Engineer who works in this job will perform many different tasks related to workflow management, design, and optimization. The main responsibilities are:
- Maintaining large-scale data pipelines by using Python along with Polars.
- Implementing end-to-end data processing systems that are focused on speed, accuracy, and reliability.
- Designing modular and testable software with clarity with scalability in mind.
- Conducting unit and integration tests to verify data integrity and the consistency of workflow.
- Management of the data model, query and optimization by using PostgreSQL (PSQL).
- Making and maintaining CI/CD pipelines by using GitHub Actions, Jenkins, or GitLab CI to ensure seamless integration, and automatic deployment.
- Utilizing Kubernetes and containers-based deployment environments to improve scaling data applications.
- Integrating embedded databases, such as DuckDB to make data analytics more efficient.
- Utilizing computer vision and image processing tools to increase the ability to match data from products.
- Working with teams across departments to improve data infrastructure as well as internally-developed analytics tools.
- Monitoring the performance, debugging, and enhancing existing systems of data for improved efficiency and maintenance.
Required Skills and Qualifications
To be successful in this position Candidates must satisfy the following professional and technical prerequisites:
- Expertise in the writing of large-scale data pipelines utilizing Python as well as Polars.
- An understanding of the best practices for data engineering that include ETL creation as well as performance optimization.
- Expertise in PostgreSQL to optimize queries and schema creation.
- Experience working in cloud environments like AWS, Azure, or Google Cloud Platform (GCP).
- Knowledge of Kubernetes is essential to orchestrate containers and improve performance of deployment.
- Experience with CI/CD tools such as GitHub Actions, GitLab CI or Jenkins to automate test and deployment.
- Experience with unit testing frameworks like Python for ensuring code reliability.
- The exposure to DuckDB or similar databases to perform data processing in a light weight.
- A basic understanding of the image processing or computer vision technology used in data workflows.
- A strong problem-solving capability and a methodical approach to troubleshooting and improving performance.
Job Summary
- Location: Data Engineer (Python with Polars)
- Company: NIQ (NielsenIQ)
- Location: Chennai / Pune
- Job Type: Full-Time, In-Office
- Working Days: 5 Days per Week
Application Process
Candidates who are interested should be familiar of Python-based engineering for data, ETL systems, and pipeline development that focuses on analytics. A strong communication skill and a keen eye for particulars are essential to successful performance in this position. For more information, applicants are invited to go to NIQ’s official NIQ career page. They can send current resumes that highlight relevant project experiences, technical skills and professional accomplishments.