Big Data and the Bright and Messy Future
Mark Davis
JUL 15, 2015 15:34 PM
A+ A A-
Big Data and the Bright and Messy Future
by Mark Davis, IEEE Computer Society Member, and Distinguished Engineer, Dell
It began with a simple problem: how do I figure out who is accessing my web site and what they are viewing? I should take a step back, actually, because it began even before that with the grand challenge of trying to crawl and index an exponentially growing number of websites in the early 21st Century. But that challenge was unique to Google and a few other players, while analyzing web site access was more ubiquitous among the burgeoning social media sites that were coming of age around the same time.
The solution to both of these data quandaries came in the form of new ways of storing and processing massive amounts of data that broke away from traditional databases and focused on distributed data processing techniques. These techniques also didn’t rely on supercomputers that were only affordable by nation states, but instead could be built from often unreliable commodity computers and disk drives. And today these technologies have already become a critical part of low-friction startups that can be built to solve new types of problems. More established enterprises and organizations are also investing heavily in this next wave of pervasive data analytics and expect that investment to continue to grow over the next decade.
The possibilities of big data are tantalizing and palpable. Businesses can start to analyze their buildings and correlate AC systems with weather patterns, reducing costs and enhancing planning for heat islands and other effects. Wearable health monitors can provide longitudinal data that feeds into medical planning and epidemiological estimates. Cities can detect road wear patterns and proactively improve responses, improving the lives of people and reducing vehicle wear. Web sites and mobile apps can better contextualize the information that helps you work and be entertained.
Perhaps even more intriguing is the emerging possibility of advancing intelligent systems design and training using big data. Recent improvements in algorithms like simulated neural networks have been combined with big data sets to show human-like or better performance on a number of tasks where advances had been stalled for a number of years and incremental improvements were only being made using extremely complicated software architectures. Automatic image understanding is one such area, but large-scale language translation has also shown advances. In each case, big data collections were a prerequisite, but cheap hardware and big data software infrastructure were important components to the results.
We are all familiar with the narrative theme of technology gone awry, from Mary Shelley’s Frankenstein to Terminator. Almost all literature and video storytelling relies on conflict of some sort, and this one is known as Man vs. Technology, though perhaps we need to update that to People vs. Technology in a more modern world. Big data carries risks, too. We already routinely see data breaches due to hackers exploiting our connected world to gather large-scale personal information. Here we see an area where security, privacy and even physical safety collide with big data. If we can’t secure data and safeguard privacy, everything from transit systems to our physical safety are at risk. For instance, stalkers and thieves are already mining social media for clues about our locations and habits. And inverting this, big data technologies are also being used to detect breaches to find bad guys.
One of the most interesting risks also carries with it the potential for enhanced reward. Don’t they always? That is, some economists see economic productivity largely stabilizing if not stagnating.  Industrial revolutions driven by steam engines, electrification, telephony, and even connected computing led to radical reshaping of our economy in the past and leaps in the productivity of workers, but there is no clear candidate for those kinds of changes in the near future. Big data feeding into more intelligent systems may be the driver for the next economic wave, though revolutions are always messier than anyone expected.
[%= name %]
[%= createDate %]
[%= comment %]
Share this:
Please login to enter a comment:

Computing Now Blogs
Business Intelligence
by Keith Peterson
Cloud Computing
A Cloud Blog: by Irena Bojanova
The Clear Cloud: by STC Cloud Computing
Computing Careers: by Lori Cameron
Display Technologies
Enterprise Solutions
Enterprise Thinking: by Josh Greenbaum
Healthcare Technologies
The Doctor Is In: Dr. Keith W. Vrbicky
Heterogeneous Systems
Hot Topics
NealNotes: by Neal Leavitt
Industry Trends
Internet Of Things
Sensing IoT: by Irena Bojanova