Is More Data Always Better?

google think magazine data overload obesity information ideas processing analysisThere has been a discovery in the online marketing and data/statistics world in the last few years. We have had more websites, products and tools created online than we can possibly keep track of. The terms to describe this deluge of activity we have been hearing the most are “data overload” and “information overload” from both companies and consumers. This Google Magazine uses the term Data Obesity to describe this phenomenon.

They ask the question, why is more data always better?

I think the idea of “more data us better” is common from people who lived before the Internet was prevalent. We had to work hard to find data. Researching something meant going to a library and looking in a card catalog (or maybe something called Gopher) and then finding your way around the Dewey decimal system to find that book. And then sometimes they didn’t even have the book because it was checked out or possibly it was just filed wrong because nobody understood the Dewey decimal system.

On a related note recently we got invited to my cousin’s wedding in Santa Fe New Mexico. My dad promptly went to the library and checked out 3 books on Santa Fe and New Mexico. I cringed. He asked how to find out the flights to book something without a travel agent. I realized I have been traveling since 2000 this way and he stopped traveling about that time so he never has. I introduced him to Travelocity, it was mind blowing and a bit of data overload compared with the OAG book he used to use in the 80’s.

The point here is that finding data was really difficult. People had control over its distribution because it was in print. When it became more freely accessible due to Google and other companies efforts we assumed this would be good, because people could remember where to find it and use it whenever we wanted. We never thought it would get this big so fast. Now travel sites are overwhelming, they have too many choices and there are too many of them trying to get you to opt into something you don’t want while being over charged for bringing a suitcase on a flight. This is just one example of how data has gone exponential so quickly.

Others of us have come to a data overload conclusion when they have 200 emails in several in-boxes, 1000+ rss reader posts from feeds waiting, several work projects, 500+ Facebook wall posts in their feed and hundreds of tweets that have gone un-read. This is among a climate where you have to follow-up with projects 5-10 times to get things done, post blogs/tweets/FB status updates daily to keep on people’s radar, empty the DVR so it doesn’t get overloaded and auto delete something you really wanted, listen to the radio on the way to work just in case something big happens and still find time to scoop the litter box before it gets full and the cats poop on the floor.

And the real purpose in all those tweets/FB posts and feeds is that you business changes yearly and if you don’t know about the latest trend and some real insights about it before your boss asks about it, you won’t have a job for all that long. (in digital marketing)

Having data overload be a “good” problem to have from some people’s perspective (as in that it is growth oriented). The democratization of publishing combined with tracking methodology and databases have all contributed to this problem, giving everyone a voice, a potential following of readers, a data trail to analyze and method to say something important online 24/7/365.  And then we have an even bigger problem of processing what is being said, figuring out if it is important or not and sharing/processing/saving it in some way if it is. Acting on that data is way down the line and many of us don’t even get there.

And this isn’t even the big problem with data overload. Where will we store it all? Why do tweets disappear from search so quickly? Because there are millions of them and the failwhale is full. According to the ThinkQuarterly UK, there are 800 Exabytes of data/information created every two days. It took humans from the beginning of civilization until 2003 to create the first 800 Exabytes, and we’re on a roll now.

Where does all this seemingly random data go? How will we know what it says without having to go into a database table and read specific field information? Where are the software tools to manage all this and still give humans the ability to customize the out put in ways that match the behavior or business purposes that we really need? Does any of this stuff ever get deleted?

These are all huge questions we have to answer as more people publish, share, create, track and do business online. We also have to weigh the possibilities of sharing data openly and locking it behind walls as well as how will people comprehensively find what they need when they want to as well as gauge the validity/accuracy of the information presented?

I’m betting on paid services for personal and business data management/archiving & Analysis tools. We will pay for good analysis, good data access & processing and good reliability/backups when we feel the pain of missing good insight, losing good data and just too much happening. Both personally and professionally. But unless you know how to work with SAP, SPSS, SQL, Oracle or a bunch of other systems data management is largely out of your control at this point. They are the librarians of our digital data and they need to find a workable way to Dewey decimal system it back into order and allow us to use it as humans need to.

Advertisements

Spring Weather – April, May, June

Today is the first day of spring and it has been over 40 degrees and sunny all day in Chicago. This has been a minor miracle considering that from November 1, 2007 through February 29, 2008 this winter has been awful. We have had snow every week during that time and sometimes snow fell 2 or 3 times a week. We didn’t know what to do with all of it, how to clear it all or how to keep the ice from piling up all over everything. We got through it somehow and it is safe to go to the grocery store again and carry things up my back stairs that are outside now that they are finally not covered in ice. (a typical 3 floor walkup building in Chicagoland)

I just wanted to say happy spring and welcome to warmer weather in Chicago. I will not miss this winter, nor wish this much snow on anyone. I hope that we don’t get such an extreme summer also since heat is just as difficult in some ways. Although, this global warming thing is getting weirder by the year. Anyway, go get some spring clothes out of storage and take a walk outside and enjoy the spring weather.

Google Earth adds a weather layer to maps

This is interesting. Google has partnered with weather.com and Naval Research Labs Meteorology labs to provide weather information on Google Earth maps. (maybe on gas pumps too?) I think this is very logical and makes perfect sense. I am suprised though with all the $ Google has that they didn’t launch some sattelites and provide weather on their own. This new way of doing business with partnerships is not the traditional way for Google. They like to own the technology for themselves and not outsource. So, I am curious if these other sources will be reliable enough and timley enough for google’s demands. It can’t be an easy job providing anything for Google, they are pretty perfect all the time and expect you to be too.

I wonder what Tom Skilling thinks of the new weather on Google earth?

Chicago Marathon Halted Due to Warm Hot Weather

This is odd. Yes it is hot today in Chicago, but I never have seen the LaSalle Bank Chicago Marathon get canceled or stopped due to weather concerns. I guess they figure that no one was anticipating this kind of hot humid weather when they trained for it over the past 6 months. Currently my thermometer shows 86 degrees farenheight, but it was supposed to get up to 90 degrees today too. They did get a winner and finisher in the marathon before it was halted at the 3 hour mark half way through. Kenyan Patrick Ivuti won the marathon by one 500th of a second with the last final stretch being neck and neck sprint the whole way. There were 36,000 runners participating in the Chicago Marathon today and I am sure none of them were thrilled with the hot humid weather that we got during the race.  One runner was reported to have died, a man who was 35 and from Michigan.  It is unseasonably warm for this time of year. I have not had an October I can remember where I have had to run my air conditioning like this before. I am not sure if it is global warming? or something else happening, but it is really hot.

Fall Colors – What I like about Autumn

I ahve been feeling that fall is almost here, and this weekend I think it hit. Here are some of my favorite things about fall:

1.  Walking through crunchy leaves on the sidewalk. Both the color and the sound are great.

2. The trees turning great colors and the falling leaves as you walk or drive by.

3. Pumpkins on everyone’s front step. Some carved around halloween and then the squirrels cart them off in pirces to be eaten.

4. The decorations on all the houses and all the candy in the office, although too much of both is not a good thing.

5. The temperature out. It’s perfect with a light jacket in Chicago, but not too cold yet.

6. The harvest moon and the indian summer sunset.

7. The beginning of a new Fall TV season and all the Oscar Hopeful Movies that are thinking movies rather than gratuitously stupid entertainment.

8. Leaving the windows open so the cat can enjoy the sounds of the outside for a few weeks before it gets too cold.

9. Getting my pre-thanksgiving tradition of getting all my shopping done, and avoiding the cold and the crowds of the Holidays.

10. Little kids in costumes at Halloween. Big Kids in costumes aren’t as cute.

Have a great Fall.