Stats NZ is taking a new tack with a Cloudera-based Data Hub
Statistics New Zealand is seeking members of the Cloudera partner ecosystem to implement and support a new Cloudera-based data management platform.
The selected partner or partners will support the platform, called the Stats NZ Datahub, and augment the agency's data engineering team with the implementation of business solutions, a new tender stated.
"Stats NZ is seeking support to achieve the strategic ambition of providing a sustainable and future proofed data-processing environment by implementing a Cloudera based Datahub," tender documents read.
"The first deliverable solution, by 30 Apr 2019, will be ingest, validation and preparation of data to load the Stats NZ Integrated Data Infrastructure (IDI)."
Once this initial phase has been completed, Stats NZ said it will investigate other projects which would benefit from the use of the Datahub including the potential to re-platform the IDI from SQLServer to the Datahub.
"We are cognizant that we will require support of the successful partner(s) as well as ability to reach back to the software manufacturer (Cloudera)," the tender stated.
Cloudera's software is the most popular commercial distribution of the open source Hadoop Big Data platform.
Stats NZ said the physical infrastructure for the Datahub will be provided and managed by Revera under its existing All of Government infrastructure-as-a-service (IaaS) contract - licenses have already been bought for the hub with Gold support.
The Datahub production cluster, which has been reviewed and confirmed by Revera, will be three master nodes, five worker nodes, and admin node and an edge node, while the Production cluster will have a data capacity of 160TB.
There will also be a disaster recovery cluster in a remote Revera managed data centre.
Potential suppliers can bid for professional services to Implement, certify and accredit the Stats NZ Cloudera Datahub and to support the Cloudera Software and/or provide additional, experienced resource to support to the Stats NZ data engineering team.
Stats NZ describes itself as a "non-traditional statistical agency" seeking to push the boundaries and "lead the way for New Zealand’s data future".
"Our vision is ambitious – to move from an organisation that provides statistics on specific topics, to an organisation that now unleashes data to change lives," the tender document stated. "This vision underpins everything that Stats NZ does, and everything we will do in the future."
Cloudera opened a local office in New Zealand in 2016.
