Menu
Stats NZ faces a further challenge: shoring up its Integrated Data Infrastructure

Stats NZ faces a further challenge: shoring up its Integrated Data Infrastructure

With work patching the 2018 census still under way, Stats NZ's data platform needs significant investment

Liz MacPherson (Stats NZ)

Liz MacPherson (Stats NZ)

Statistics NZ's Integrated Data Infrastructure (IDI) is the envy of the world, but is facing some serious challenges as user demand soars.

Demand for the IDI has grown so much that its current infrastructure is struggling to cope.

That infrstructure is also key to helping Stats NZ patch together a full set of results from the deeply flawed 2018 census. One in seven kiwis failed to fully complete the census which was taken online for the first time.

Stats NZ told Parliament it needs to ensure it has a stable IDI platform because demand to load new datasets cannot be fully met, creating a backlog of requests.

"The IDI is a specialist tool designed for statisticians and researchers and currently requires a degree of specialist knowledge and skills for access and insights," Stats NZ said. "There are opportunities to support ease of access for the targeted user groups."

As a prototype, the IDI was experimental and was not fully featured, Stats NZ said. There is, however, scope to increase its processing capabilities to meet the current and future demand.

Stats NZ said it needs to scale up the IDI from an "evolved prototype" to a data warehouse model that better reflects customer demands and on a larger scale.

"Customers have signalled an interest in increasing the frequency and flexibility of IDI data uploads, whilst also accommodating an increase in datasets (number of and data volume)," Stats NZ said. "A revised data linking methodology within the IDI may support this objective."

User groups would also like Stats NZ to make it easier to work with IDI data and to gather insights through the development of more accessible projections and interfaces.

Demand for IDI access is soaring.
Demand for IDI access is soaring.

Such improvements would help Stats NZ meet its vision of “unleashing the power of data to change lives” by increasing the amount and type of insights from a wider range of researchers.

However, the investment required is above Stats NZ current operating baseline budget.

The agency has worked to make data more discoverable by developing a training dataset made from synthetic data, able to be used outside the data lab to help new users learn how to use the IDI. 

Stats NZ will also publish metadata to increase the information about the datasets and their variables available to the public and researchers to improve their knowledge of the data available.

Additionally, Stats NZ has sought to make data more accessible by supporting user collaboration and working to find ways that help communities share insights, experience and resources with each other.



Follow Us

Join the newsletter!

Or

Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.

Tags Stats NZCensus 2018IDIintegrated data infrastructure

Events

Why experience is the new battleground for partners

Join us for an exclusive webinar, in association with Hewlett Packard Enterprise and Technology Services Industry Association (TSIA) and learn about the latest industry insights and how technology services continue to evolve to deliver differentiated value, and how partners can be successful in 2021 and beyond.

Featured

Slideshows

The Kiwi channel gathers for the 2020 Reseller News Women in ICT Awards

The Kiwi channel gathers for the 2020 Reseller News Women in ICT Awards

Hundreds of leaders from the New Zealand IT industry gathered at the Hilton in Auckland on 17 November to celebrate the finest female talent in the Kiwi channel and recognise the winners of the Reseller News Women in ICT Awards (WIICTA) 2020.

The Kiwi channel gathers for the 2020 Reseller News Women in ICT Awards
Leading female front runners honoured at the 2020 Reseller News Women in ICT Awards

Leading female front runners honoured at the 2020 Reseller News Women in ICT Awards

The leading female front runners of the New Zealand ICT industry joined together for the annual Reseller News Women in ICT Awards event at the Hilton in Auckland, during which hundreds of guests celebrated 13 outstanding individuals who won awards, chosen from more than 50 finalists representing over 30 organisations.

Leading female front runners honoured at the 2020 Reseller News Women in ICT Awards
Channel gathers to celebrate the Reseller News Innovation Awards 2020 winners

Channel gathers to celebrate the Reseller News Innovation Awards 2020 winners

More than 500 channel leaders gathered in Auckland on 21 October at the ​Reseller News Innovation Awards ​2020 to celebrate the achievements of the New Zealand technology industry's top partners, start-ups, vendors, distributors and individuals.

Channel gathers to celebrate the Reseller News Innovation Awards 2020 winners
Show Comments