Corporations are racing against time to create archives that allow retrieval of emails in response to increasingly common civil suit discovery motions in the US.
Email is becoming a standard part of civil discovery. Opposing counsel is well aware that email archives are rich veins to mine, and last year unstructured data was specifically added to the discovery list in Rule 26 of the U.S. Federal Rules of Civil Procedure. And the threat isn't confined to Global 2000 corporations. Any organisation can face a civil suit, and any civil suit can include the requirement to produce relevant email. CIOs in organisations that are as yet unaware of this legal trend should alert their CEOs to the danger.
For most enterprises today, the challenge is to find the requested email at all. In most organisations the archive is little more than a pile of forgotten tapes originally intended as backup in case of a computer crash. How many IT organisations today can even find the tapes from two years ago?
However, say the experts at Wikibon.org, even as enterprises scramble to find misplaced tapes and create a real archive that will allow them to produce sets of historic emails if required, they should think beyond this purely defensive requirement to active use of the archived data.
Email contains valuable data
"Most unstructured data has very little metadata associated with it," said Wikibon.org cofounder David Floyer at a public Peer Incite meeting held by the organisation last week. "Email is an exception. Email headers contain a great deal of useful information about who is talking to whom and how organisations really work.
"There is a huge requirement for automated data classification for semantic search," he said. "These are essential for organisations to protect themselves."
For example, he says, the human resources department may need to search the archive for early evidence of emotional or sexual harassment, sales may want to search for indications of contract abuse, and purchasing can search for evidence of employee theft. Executives then can intervene early to avoid problems before they reach the stage of civil or criminal action.
Martin Tuip, manager of business development at email archiving house Mimosa Systems and a meeting participant, suggested that analysis could go far beyond the initial concern over legal issues to supply insight into how the organisation actually works, who influences whom and what the unofficial lines of communications are.
This could provide insights, for example, into why some projects succeed while others fail. He envisions organisations creating a single repository of unstructured data that can then be mined by business applications as needed for information.
This will require a new kind of archive, said consultant and Wikibon.org Chief Content Officer Peter Burris. "People think of archives as inactive. We are proposing that in the future, archives will be active systems. Various kinds of applications will run on top of that archive," he says.
Those applications will evolve quickly, Floyer suggests. He envisions a two-tiered architecture, with a long-lived infrastructure consisting of the archive itself along with its associated hardware and data capture and semantic classification software. The business applications will run on top of that as a separate layer to provide the flexibility to support their rapid evolution.
This, Martin predicts, will result in a split in the archive administrator's function, which Burris compared to the split in the data management function between the data and database managers. In this case, archive administrators will focus on the underlying archive, while a new function — which Wikibon.org cofounder David Vellante called "records management" and said would be a business-facing rather than an IT technical position — would evolve to manage the business applications layer.
This will become a complex new area of activity, with important legal requirements, fast-evolving business needs and potential for fast expansion beyond email to other forms of unstructured data ranging from word processing documents and spreadsheets to web pages, Floyer says. And with that likely expansion will come an explosion of metadata associated with those documents, driven in part by a need to satisfy expending discovery requirements in civil and criminal actions and in part by such concerns as documenting and improving user experiences on corporate web sites.
This, Floyer says, will create a major new opportunity for outsourcers. "Many organisations will not want to deal with all this complexity internally," he said. "They will be looking for service providers who can guarantee their methods will meet legal requirements and support business users.
"Email is the lifeblood of most organisations today," Floyer said. "Archiving the institutional information contained in it and other forms of unstructured data will provide huge potential value beyond what it will save in potential court-mandated fines for failure to produce mandated documents in a timely manner. As IT organisations race to create their initial archives, they should keep the vision of the ultimate active archive in mind and build consciously toward that goal."