We've entered a new age based in technology and huge volumes of data. All this data will help to shape our future world.

What this means for us is we can make smarter decision and act faster. But it hinges on our ability to read and make sense of it all.

The problem this creates is, with so much data how do we really make sense of it all?

Let's look at the example of Google Person Finder.

In 2010 one of the largest earthquakes on record hit Haiti. 316,000 dead and over 3 million affected by this tragedy. Added to the horrors, thousands of people went missing.

In 2011 the world watched another major disaster. An earthquake hit just off the coast of Japan. It was so powerful it moved Japan's main island (Honshu) 2.4m east. The follow-on tsunami killed thousands, with thousands going missing.

Previously there's been no single reference point to find missing people in crisis like these. At the time of these disasters, data flooded into networks and databases of different aid agencies. This data came from response teams, individuals and other aid agencies.

There was so much data that it jammed databases. As more data arrived it became too difficult to filter it all and find missing people. What should have taken hours took days or weeks to do. Another problem was there was no collaboration of data across different agencies.

At this point Google stepped in and did what government couldn't. With the software and technology they had at their disposal they created a single search point to find missing persons. They called it 'Google Person Finder'.

This meant people could jump onto Google Person Finder to search for missing loved ones. This was a better outcome for those seeking information. It also took the strain off emergency response teams trying to process all this information.

Now this wouldn't have been possible without two crucial parts to the equation.

  1. A huge amount of data
  2. The technology to interpret all the data

A Lifetime of Books Created Everyday

People who can read and interpret data the best and fastest will be so-called 'Wizards' of the future.

You'll be shocked at how much data we create on a daily basis. Someone has to make sense of it all. Humans can't do it on their own. Computers lack the human creativeness of data analysis. So entrepreneurial companies realise they must put the two together.

Just how much data am I talking about? IBM (NYSE: IBM) estimates we create about 2.5 Exabytes of data every single day.

That's the equivalent of about 625 Million DVD's worth of new data, per day. Another way to look at it is that it's more data than every book that's ever been written.

Cisco Systems (NASDAQ: CSCO) estimate by 2016 global Internet Protocol (IP) traffic will be over 110 Exabytes per month. Added to this Global Mobile data traffic will be over 11 Exabytes of traffic per month.

All this data comes from devices and sensors. Your phone, GPS, weather stations, CCTV, things you post online, new websites, etc. Data comes from everywhere.

The collective terms for all this data is very technical...it's called 'Big Data'.

But the Government Might Kick Down My Door!

I recently asked a group of friends the question, 'Are you worried that you give away too much information?' The overwhelming answer was, 'Yes.'

Our basic human nature means we want to keep personal information private and to ourselves. It's rooted in a feeling of mistrust. Mistrust of governments and major corporations. And I can understand that.

However, there does seem to be an Orwellian belief that as we switch the GPS function on our smartphone, Julia Gillard and her spooks at ASIO will know exactly which cheek you just scratched.

There are also theories about the humble online search. Look up words such as 'terrorism' or 'al-Qaeda' and the NSA, CIA and FBI will kick in your front door. Next thing you know you're being strung upside down at an 'undisclosed location'.

Let's think about this. What if it was harder for governments and organisations to know about us if we gave them more data? What if there was so much Big Data that they couldn't tell the difference between a man or a mouse?

I know it sounds a bit daft, but there's method to my madness. What if we could overload the systems of organisations by simply creating too much data for them?

This serves a double purpose. Blast a system with too much data and they overload. It becomes a thick fog of Big Data. In short, they don't have the technology to make sense of it all.

It could mean we have the information available to allow us to interact with our digital environments more efficiently. Yet also hide from those that shouldn't be able to see what we're up to.

Alvin Toffler, a renowned writer and futurist, termed the idea (before the internet even existed) of Big Data in his book Future Shock. He described it simply as 'information overload'.

In addition, it's private industry, not governments, which have technology and software to process Big Data and make sense of it. These companies are your typical Silicon Valley start-ups that have built their business around Big Data.

To make sense and draw out legitimate, meaningful information from enormous data sets can make a company. When governments don't have the capabilities to make sense of their data, they turn to those who can.

This Company Does What No Others Have Done Before

One example of this is a company called Palantir Technologies. They are a software provider. And their software helps organisations like the CIA and FBI interpret Big Data.

Palantir does more than just help government agencies. They also work with financial, scientific and humanitarian organisations to help make better decisions. Helping them all to answer questions that are difficult to see in the fog of data.

Palantir is a known term for fans of J.R.R Tolkien. To explain, Palantir are 'seeing-stones' from the Lord of The Rings. And that's what Palantir believe their technology is. It's the 'seeing-stones' of Big Data.

A big part of the work Palantir does is rooted in their mission. Their mission to make sense of Big Data whilst maintaining civil liberties. Palantir describe it as,

'a core component of (our) mission is protecting our fundamental rights to privacy and civil liberties. Since its inception, Palantir has invested its intellectual and financial capital in engineering technology that can be used to solve the world's hardest problems while simultaneously protecting individual liberty.'

In this sense what does protecting civil liberties mean? Well that's the trick part of Palantir's technology. They can tag and screen data from its source. This means they can enable or hide data based on different authority levels.

A good example would be a Medical Researcher with a huge database of DNA information. They would overlay Palantir's software over the database to find connections, links and patterns.

If the Police wanted to use the database to link crimes to particular DNA matches they could...but they could not use that information against a linked person.

The data gets a tag to say it's been obtained by the Medical Institution, not the Police. So the linked person's identity remains undisclosed.

Palantir software is world leading. It carefully balances the use of data to make smarter decisions with the protection of privacy and civil liberties. It's important for this to occur. As there must be trust that our data is used for purposes of good and not evil.

Another example is the 2012 London Olympics. London City police used an app and location services to create 'heat maps' of crowds around London. This helped monitor crowd gatherings and control traffic flow.

They did this by sending out alerts and updates to the app users. It created smoother flow of foot traffic, avoided crushes at events and overcrowding at Tube stations.

Sadly, this week the bombing in Boston provides us with another example. It's likely this will be the largest investigation ever from the use of 'crowd sourced' data. The collection of images and data from the internet is on a scale never seen before.

Google launched Person Finder again this week for the Boston crisis. And you can safely say Palantir's software is running at full steam. No doubt the FBI are using it help track down the people that committed these terrible crimes.

We will continue to create more data whether we like it or not. As we connect our digital lives with our day-to-day living, more data will flow from our activities to databases around the world.

We're inadvertently creating an information overload. And that's not a bad thing.

Google and Palantir are just two examples of companies that work to ensure we use information properly. In a way that helps us. Now that has to be a good thing in times of crisis.

Regards,

Sam Volkering
for The Daily Reckoning Australia