If the digital oil field is producing anything, it’s data. An active exploration survey can create hundreds of terabytes of data daily in formats ranging from paper well logs to raw sensor output. Using these data requires specialized knowledge, training, software, and, most importantly, a community of individuals who understand both the processing of raw information and interpreting the products that result.

These hurdles are formidable, and many organizations fail to overcome them. Research firm and global market intelligence company IDC estimates that approximately 90% of the big data collected by enterprise organizations go unanalyzed. The majority of data collected are destined for life as “dark data” – data that are never found again, processed, or analyzed effectively after being initially gathered.

Rethinking dark data

The dark data challenges are many. Companies collect and store vast amounts of data for compliance reasons or to capture market intelligence. These data are not always placed in relational databases but instead are stored as unstructured file types on collaborative share sites, spreadsheets, or file repositories.

Managing these dark data directly, in the variety of formats and sheer size in which they are stored, is an increasingly unfeasible proposition. Often, data remain isolated in both technological and organizational silos. These data are too large to easily transfer and are restricted to proprietary formats. An effective search is impossible across a range of formats and storage locations, making it difficult for members of an organization to see what data assets are commonly held.

Solving the dark data problem requires new ways of thinking. Extraction professionals can no longer expect to store, move, and share whole files and datasets using the specific software used to create them. The industry needs to focus not just on source data but on information about these data.

Metadata – abstracted information that describes source data assets – present an emerging opportunity to summarize and search through dark data holdings quickly and efficiently without moving the source data themselves. Metadata models are hardly standardized, however, and metadata presented in conventional formats are far from being effective decision tools.

The industry needs innovative approaches to visualizing metadata and making them actionable. This method has been used in dashboards and web portals but needs to be extended to a better, lighter, and more shareable digital medium to attack and solve the dark data problem at scale. This type of visual metadata platform is being built with ClipCard.

Getting the big picture on big data

Meta-visualization via ClipCard can provide users with a summary of source data. These abstracts can be shared securely within an organization and at hierarchical levels of detail without transferring large source data files across overburdened networks. The total body of these metadata summaries fuels a new kind of enterprise search engine that can put an organization’s entire body of data at the fingertips of any user instantly.

How users interact with information can change when they start building meta-visualization catalogs. Data can now be represented as abstract summaries of whole entities that can be located geographically and by time. Information can be selected by place name, through a map interface, or by topic-specific tags to make sure the user is drilling down to the information needed.

Meta-visualizations create a framework for applying metadata standards and elevating topic-specific terminology within visual summaries. This, in turn, optimizes the search and summarization of data across an organization. Data assets and even domain experts from different parts of the enterprise can be easily tagged with descriptors from dictionaries or company glossaries, making data classification easier for users. Relationships can be mapped between objects, and data can be organized in collections based on project or subject manner. Additional information can be added through an easy visual interface to make these summaries even more useful. These include descriptive text, topic-specific tags, impact metrics, and links to other data and data owners. All of this can be done without altering backend databases or the source data themselves because it is accomplished through the new layer of visual metadata summary.

Delivering this experience on mobile platforms as well as on the desktop is crucial. E&P work does not end at 5 p.m., nor is it usually conducted in an office. Metadata visualization must be available remotely and on any device users have on hand.

There’s an app for that

Thanks to innovative productivity software, users expect on-demand collaboration and visualization tools to work anywhere and at any time. Users are increasingly comfortable with touch-screen technologies and web applications (apps) for discovering and analyzing information. Just like today’s consumers, extraction professionals are more and more accustomed to swiping touch-pad screens or checking smartphones to search for and find specialized targets through massive unstructured datasets. Consumers do this when they search for restaurants, products, travel destinations, and people. Professionals deserve the same quality of experience when managing project assets, viewing market projections, and collaborating with global teams.

This app-driven archetype for searching, finding, and acting on unstructured information has only just barely begun to be a part of enterprise software and the data scientist-dominated world of dark data and analysis. Visualizing metadata makes it possible to deliver a radically useful experience to nontechnical users on any device without moving or altering source data.

Virtual reality

With rich, visual summaries of data assets, the digital oil field will rely not only on where data are physically stored but on virtual central repositories of searchable metadata. Sharing meta-visualizations through secure cloud services keeps the original data on company servers but allows users to view results, share them with other team members, and act upon the information instantly.

The result is an intuitive, powerful mobile application and interface that empowers users wherever they are to create and share information from all parts of the digital oil field. Survey well data statistics can be sent to the home office, and test flows can be monitored remotely. Current climate and environmental information used to determine operations can be sent from the home office to the exploration team with a simple swipe. Archived data and drawings from older surveys can be sent directly to field engineers to complement new survey results as they happen.

Digital oil field 2.0

The digital oil field is ready for a transformation in its approach to managing and organizing the vast amounts of data that are produced daily. Users in all sectors of the oil and gas industry will benefit from putting dark data to practical use. But companies won’t break their dark data dependencies through formal computer science research or a legacy software stack that focuses on source data directly. Taking a fresh approach to meta-data by rendering them in visual, searchable, actionable form is the only effective approach to bringing dark data to light.