Your new company

The Python Data Engineer is a software development role within the Data Team of the client's IT Department. This position is part of a project-based engagement, focused on building web scrapers and data ingestion pipelines for the client's Tokyo Japan Power Trading Team.

Your new role

• Collaborate with BAs to clarify requirements in a fast-paced environment.
• Develop scalable Python and PySpark scrapers integrated with existing Databricks frameworks.
• Ingest structured and unstructured data from websites (HTML, PDF, Excel, APIs, CSV, etc.).
• Build and test scrapers in Databricks (DBX), following GM design patterns and using shared libraries.
• Use ADF for orchestration and integrate with GM frameworks for logging, error handling, and translation.
• Write clean, reusable Python code for data processing, automation, and transformation.
• Monitor, debug, and maintain data pipelines to ensure reliability and fast issue resolution.
• Review technical specifications and work closely with FO Analysts for validation and clarification.
• Ensure scrapers support Japanese websites with power market data and adhere to HTML/API nuances.

What you'll need to succeed

• Strong expertise in Python programming and SQL.
• Hands-on experience with web scraping and industry best practices.
• Proficient in web scraping using libraries such as BeautifulSoup, Camelot, lxml, re(regex) etc.
• Familiar with Python libraries like deep_translator for language translation (nice to have).
• Knowledge of modern cloud-based data architectures, including Data Lakehouse on Databricks.
• Experience with Databricks and Azure is highly desirable.
• Good understanding of Big Data frameworks like Spark and file formats like Parquet.

• Source code management e.g. Azure DevOps, Git
• Agile delivery methodologies such as SCRUM or Kanban
• Knowledge and work management tools (e.g., JIRA, Confluence)
• Certified in Data Engineering, Azure or Python

What you need to do now

If the above role is a good fit for you, or if you know anyone who might be good for the role, please get in touch with an updated CV and I'll be in touch!

EA Reg Number: R1660446

EA License Number: 07C3924

Company Registration No: 200609504D