Organisations in Australia are looking more and more for data engineers and data scientists to help them make sense of their data and improve how they make business or run their businesses through the smart use of its existing data.
Hays Jobs Report, which covers skills in demand for the months of January to June 2020, had data engineers and data scientists in the Top Five sought-after skills for digital technology. Although the report was developed before the novel coronavirus outbreak, it shows the trend of businesses seeking data professionals in Australia.
Data scientist and data analyst are also listed on LinkedIn’s annual Emerging Jobs Report 2020 for Australia, which lists the top 15 emerging jobs in a specific region or country.
Computerworld Australia spoke to three data specialists to learn more about their paths to succeeding in their desired careers.
Getting started in a data engineering career
Deepika Wadhwani, a data engineer at the Australian Securities Exchange (ASX), got her first job with Accenture as soon as she completed her bachelor degree in computer engineering in Mumbai, India. She started as a software engineer and always worked around data.
“I learnt quite a few new technologies and worked with some really interesting people on very interesting projects. I can say where I am today, is more or less where I wanted to be when I started out—working with data,” Wadhwani explained to Computerworld Australia.
In her role today, Wadhwani acts upon the data needs of ASX’s big data platform. On the data science side, her focus is to ensure all products, dashboards, and data deliverables follow the same framework to make the data as insightful as possible.
Peter Crooks, a lead data engineer at REA Group, always had his eyes set on data. He told Computerworld Australia that in his first full-time role, after completing his studies, he worked as a software engineer, building IT service management software for a small business.
“I was working on developing an integrated reporting and visualisation tool into that software, and working with clients building out custom data sets based on their requirements, so I had a data focus from early on,” Crooks said.
Since then, he realised his interest was really in working with data, which he started pursuing first on personal projects before deciding to dedicate his career to data as well, which has been his focus now for several years.
Today, Crooks’s work includes taking large data sets from many different sources and from these building an accurate and up to date view of the property market. He explained that working with data requires specific skills and the best way to develop those is hands on experience.
Vikash Kumar, a data engineer at Cbus Super, always worked with data having started as a data warehouse support analyst. He later moved to an ETL (extract, transform, load) developer role with Tata Consultancy Services (TCS), which gave him insight of all touchpoints and how a data solution build is a team work from requirements to go-live.
Through this role, he moved to Australia to work at TCS customer Latitude Financial Services. “It provided me a great opportunity to perform detailed data analysis, presenting insights from existing data to business, consolidating data from source systems such as mainframes and data warehouses to get the final view and finally come up with business rules, in agreement with business, to be implemented as a data and BI solution,” he said.
In 2016 he joined Medibank as a data engineer, moving on to his current role with Cbus Super in January 2020.
Achieving the desired role and remaining relevant
Keeping up to date with new technologies and industry trends always makes the base of what any IT professional needs to be aware of, but each profession has specific things to look out for.
Wadhwani pointed out that it is important to be respectful of corporate behaviour and the vast expanse of functional knowledge that you get from systems interacting together. “There is nothing more honest than genuinely respecting your work, people you work with, and their time and ideas. This respect gives me the ‘hard working’, ‘keep trying to achieve more’ attitude that I really believe is very important to be successful.”
Understanding the context of the data you are working with—considerations when working with property data differ greatly if you are working with, say, meteorological data than with e-commerce behavioral data— is also crucial, Crooks said, alongside knowing what framework or method is suitable for the project you’re working on.
Kumar believes that to be successful as a data engineer a professional must be able to envision a solution from end to end. Other main skills to have include a good grasp on traditional and new data concepts; understanding the types of stakeholders a data engineer works with and their perspective; having some knowledge of relevant languages (such as Python and R), SQL, Apache Spark, GitHub and one of the BI tools; and getting a data-relevant cloud certification.
“My various roles in the data world helped me grow my skills in every area which are critical to succeed, be it core analytical skill or soft skills such as stakeholder management,” Kumar told Computerworld Australia. “New data solution builds and implementations enabled me to do a lot of research on tools and techniques which over the years have built a solid foundation for me as a data engineer and a mindset of building relevant and useful products.
Skills employers look for when hiring a data engineer
According to Australian jobs marketplace Seek, the common salary of a data engineer working in New South Wales is $130,000 a year.
Robert Beckley, regional director of Hays Information Technology in Australia, said that Hays noticed an increase in demand for certain IT roles during the COVID-19 lockdown. This has mostly shown in the form of new contractors in a range of infrastructure roles from service desk to engineers.
To get her skills, Wadhwani chose to acquire a BA degree—a personal preference, she said. She also said that when she is helping in the hiring process she looks for people who are curious and who have a strong love for data and working on data problems and everything surrounding data.
Crooks studied computer science but has worked with professionals who have minimal formal qualifications but gained on-the-job skills as well as professionals who have a PhD. As long as you acquire the skills you can go far, he said.
Kumar completed a graduate degree in computer applications and also holds a masters degree in computer applications. Although he is not familiar with what is taught in a diploma in Australia, he believes a diploma course would provide an opportunity for people to acquire the sought-after knowledge and skills.
He said if he was to hire a data engineer, he would look for a team player with experience and knowledge in SQL, Python and/or R, any ETL tool, big data technologies, CI/CD, and cloud platform and with analytical and problem solving skills
Wadhwani said that if you are passionate about data and set yourself to learn something new, however small, every day “I don’t think that even the sky is the limit in the times we are today”.
Crook suggested that if you’re not currently working in data engineering but think you might be interested, take the time to investigate and build out some personal projects. ”There are so many interesting publicly available data sets to work with. Find some data that you’re interested in and build a project around it,” Crooks said. “This will build up your skills, and allow you to see if data engineering is something you might want to pursue as a career.”