Is More Data Always Better?

google think magazine data overload obesity information ideas processing analysisThere has been a discovery in the online marketing and data/statistics world in the last few years. We have had more websites, products and tools created online than we can possibly keep track of. The terms to describe this deluge of activity we have been hearing the most are “data overload” and “information overload” from both companies and consumers. This Google Magazine uses the term Data Obesity to describe this phenomenon.

They ask the question, why is more data always better?

I think the idea of “more data us better” is common from people who lived before the Internet was prevalent. We had to work hard to find data. Researching something meant going to a library and looking in a card catalog (or maybe something called Gopher) and then finding your way around the Dewey decimal system to find that book. And then sometimes they didn’t even have the book because it was checked out or possibly it was just filed wrong because nobody understood the Dewey decimal system.

On a related note recently we got invited to my cousin’s wedding in Santa Fe New Mexico. My dad promptly went to the library and checked out 3 books on Santa Fe and New Mexico. I cringed. He asked how to find out the flights to book something without a travel agent. I realized I have been traveling since 2000 this way and he stopped traveling about that time so he never has. I introduced him to Travelocity, it was mind blowing and a bit of data overload compared with the OAG book he used to use in the 80’s.

The point here is that finding data was really difficult. People had control over its distribution because it was in print. When it became more freely accessible due to Google and other companies efforts we assumed this would be good, because people could remember where to find it and use it whenever we wanted. We never thought it would get this big so fast. Now travel sites are overwhelming, they have too many choices and there are too many of them trying to get you to opt into something you don’t want while being over charged for bringing a suitcase on a flight. This is just one example of how data has gone exponential so quickly.

Others of us have come to a data overload conclusion when they have 200 emails in several in-boxes, 1000+ rss reader posts from feeds waiting, several work projects, 500+ Facebook wall posts in their feed and hundreds of tweets that have gone un-read. This is among a climate where you have to follow-up with projects 5-10 times to get things done, post blogs/tweets/FB status updates daily to keep on people’s radar, empty the DVR so it doesn’t get overloaded and auto delete something you really wanted, listen to the radio on the way to work just in case something big happens and still find time to scoop the litter box before it gets full and the cats poop on the floor.

And the real purpose in all those tweets/FB posts and feeds is that you business changes yearly and if you don’t know about the latest trend and some real insights about it before your boss asks about it, you won’t have a job for all that long. (in digital marketing)

Having data overload be a “good” problem to have from some people’s perspective (as in that it is growth oriented). The democratization of publishing combined with tracking methodology and databases have all contributed to this problem, giving everyone a voice, a potential following of readers, a data trail to analyze and method to say something important online 24/7/365.  And then we have an even bigger problem of processing what is being said, figuring out if it is important or not and sharing/processing/saving it in some way if it is. Acting on that data is way down the line and many of us don’t even get there.

And this isn’t even the big problem with data overload. Where will we store it all? Why do tweets disappear from search so quickly? Because there are millions of them and the failwhale is full. According to the ThinkQuarterly UK, there are 800 Exabytes of data/information created every two days. It took humans from the beginning of civilization until 2003 to create the first 800 Exabytes, and we’re on a roll now.

Where does all this seemingly random data go? How will we know what it says without having to go into a database table and read specific field information? Where are the software tools to manage all this and still give humans the ability to customize the out put in ways that match the behavior or business purposes that we really need? Does any of this stuff ever get deleted?

These are all huge questions we have to answer as more people publish, share, create, track and do business online. We also have to weigh the possibilities of sharing data openly and locking it behind walls as well as how will people comprehensively find what they need when they want to as well as gauge the validity/accuracy of the information presented?

I’m betting on paid services for personal and business data management/archiving & Analysis tools. We will pay for good analysis, good data access & processing and good reliability/backups when we feel the pain of missing good insight, losing good data and just too much happening. Both personally and professionally. But unless you know how to work with SAP, SPSS, SQL, Oracle or a bunch of other systems data management is largely out of your control at this point. They are the librarians of our digital data and they need to find a workable way to Dewey decimal system it back into order and allow us to use it as humans need to.

Advertisements

Predictions for the next 10 years

2020 predictions vision of the home video media center family roomBack in 1999 I went to a conference at the Field Museum in Chicago called The Next 20 Years (sponsored by ZDNet, I still have the button that says Believe in Technology).

Now that we’re rolling over the odometer to 2010 I can honestly say that none of the predictions about string theory have come true.

It was an interesting idea though, to think about what is possible now and in the future and speculate in ways that may inspire people to do more, make things better and improve life.

I have been thinking a lot about this decade ending in the last few weeks and aside from an obvious comment about how blindingly fast it went by, I’m skipping the recap and these are some thoughts for the next ten to twenty years.

Disclaimer: These are just my ideas as one person, who analyzes things for a living, and I don’t have a lot of data to prove any of it. Take it with a grain of salt or as entertainment only.

1. Photo Recognition will be big. And I am not talking about face recognition software. But with smartphones we mostly have decent cameras at our disposal that are connected to the internet 24/7. I have been thinking I’d like to be able to redeem the coke-points my husband collects by snapping a picture of the cap rather than entering the number on a form online (boring and slow). This is the exact stuff that QR code readers are used for that work for UPS tracking and a whole bunch of other applications. Expect them to be used as the new coupons, contests, offline-online gaming and a whole bunch of other stuff. Then maybe by the time all that is common place facial recognition of images will be working online.

2. You will probably work in an industry that does not exist yet. Continuing education is a must. I say this because my life is an example. I work in Online Marketing and data tracking for ad agencies and this didn’t exist as a job or a technology available to most companies in 1999. I have to make sure I spend time learning on the job and off the job each year because things change a lot. This does not make having a family easy and we have no idea if we will do that as a result, but it means that you have to be curious about new stuff and be willing to investigate it and you may end up the local expert when you’re the only one with that knowledge. And learn a lot of math.

3. Taxes will go up. All this BS about lowering taxes to stimulate business and rich people spending will go away since we can’t fund the programs required, can’t borrow any more as a government and we would still have the lowest taxes for those rich people to pay when compared to other developed economies. Interest rates and inflation may follow, and of course oil prices crunching a lot of people out of the middle class. Someone will finally do the math proving that investment in hiring new people at a company and creating jobs is inversely related to lowering taxes on the rich and everyone else.

4. There will be a whole new batch of media mavens that we listen to and we will like them because they are curators not experts. No one person will be able to create enough content or be syndicated to as many channels, mediums and messages as would be possible in this fragmented media world. The people you will look to for advice are blogging now, looking at thousands of sources of information, knowing how to process it, evaluate what is good-bad-meaningless and just filter down to the good stuff. We need people like this because the big media push to produce new stuff 24/7/365 is too much for one person to go through and we all still have jobs/families/houses to attend to. And not everyone wants to spend every day plugged into a screen reading constantly. We just want those wow, aha moments. Eventually maybe this 1000 cable channels, commercials every 10 minutes, 100 blog posts a day, constant content model will streamline due to lack of popularity of most of it (no ROI) but as there is more digital space available someone will put something on it, with no guarantee of quality because people seem to randomly stumble upon things still and listen/watch/interact with amusement/laziness/procrastination of their day job. 

5. Expect more digital sensors everywhere. And this could mean in our clothing, in our fridges, on the roads, in our homes. There is a lot of bandwidth for transmitting data and ways are improving for processing data and analyzing it (without human intervention, or programming needed). I foresee more real-time data on traffic and alternate routes in my car guided by my voice requests (like Knight Rider’s Kit?). I foresee clothing measuring weight and texting me that I shouldn’t eat any more calories today. I foresee my fridge telling me the milk has gone bad again and there is a cracked egg leaking all over it. We may spend all day responding to automated messages. These may be an upgrade fee kind of thing but I think at some point the regular cost will include it because the data will be so valuable and targetable for marketers. The recent privacy discussions prove that people are becoming more aware of ad tracking as well as digital capabilities and the younger generations don’t want to go back to a time without it. But we do need better security options for this to work or an opt in policy for managing what companies know and how we want to get/share/target this info.

6. We’re going to get a whole lot more competition from China, South America and Africa for jobs. Companies are going there for operations now and not just to supply their own regions with goods and services. All the Bill & Melinda Gates (plus Oprah, Warren Buffet & That Facebook guy too) funding health/education programs in Africa will create a continent of healthy people who have jobs that used to be here related to their natural resources and possibly other areas as well. China will continue to be a leader in growth and the US needs to define itself. I always wonder why there is such an emphasis on making sure all the other countries have the help they need to solve their problems by these foundations and not the ones with people starving/not getting educated or employed in the USA. Also Immigration, population growth and birth rates in the US will all drop by 2020. (based on what I saw from the census in 2010)

7. The market will continue to be tumultuous. Up, down, sideways. It isn’t connected to real people or the economy as we know it anymore. We’re not sure how to gauge it or if it will make any positive growth in 10 years. With higher interest rates in 2012-2013 CDs may be the hot investment again.

That is it for now, but I may have more ideas later. One thing is for sure, let’s get out there and party like it’s 1999.

rolling over the odometer 1999 2000 2010 100000 miles

How to create a webtrends custom report

webtrends menu
the webtrends menu in my old company login, new one isn’t as nice.

First off I am going toclarify that I am working in WebTrends self hosted software version 8 and that I have administrator access to do this. Without those factors you won’t neccissarily have the same experience setting up custom reports.

I have already had the developers add tags to the site to track custom events (SDC tags, DCS MultiTrack events) and I don’t work with the actuall tagging on the website so I will not address those concerns here.

I’m starting from the point where the tags are live on the site and I need to set up this custom report so we can start gathering data. If you need retroactive data because the tags were placed on the site months ago and you didn’t create the report, it is only possible if you create the report and re-process all the data from scratch for that time period and this is usually done in a new profile. This takes a long time (sometimes weeks if you have years of data) and the people maintaining the servers and data for your webtrends setup usually hate you if you ask. So, let’s stay out of that area and go build a report that will collect data in a table from this point forward.

One of the most frustrating thing about WebTrends is that nothing is connected and it is a bunch of database tables. Nobody in Marketing thinks that way or interacts with a system like this so it is completley foregin. (‘m getting the feeling that the developers feel the same way too) You have to go into several tables in the database behind the webtrends software and create the structure for the report. It sucks but that is the way it works.

For every custom report you need at least three things. A Measure, A Dimension and A Report. (you may also want a filter but that isn’t required).

1. Testing – Some people (webtrends actually advises this) like to create a copy of the profile to test the reports (or any changes to the profile) before they add or change anything for real. This reduces the probablility that you will have some change corrupt or crash your data. If you depend on this data and have a lot of custom reports and data already set in the profile you’re working with, start with a copy profile first and don’t endanger the main data set. Work in that sandbox until you find the right combination of settings, write them down and then go back to the main profile when you’re sure the report exports the right data in the right format. There is a one day waiting period to collect data to test in most cases unless you’re running on-demand (of which most of my advice may not apply to) or you have updates and processing every few hours. Also only select the most recent month’s data in the copy profile, you don’t need all the data to run a test.

2. A Dimension is what most people would call a metric name. It is the data being collected from those custom tags. Just because the tags are in the WebTrends format and live on the site doesn’t mean that the WebTrends software knows that they are there or what to do with that data. You need to create a dimension to name this data and create a table to collect it in. Go to Dimensions in the admin menu. (some systems show custom report menus under reports & profiles others in Admin, check both) Select new on the upper right corner. Name it. Give it a column name for reports. Use the navigation to go through the process steps to finish creating this measure. You will need the actual tag names for this. In my case it is WT.it. Not terribly descriptive but hopefully functional. Do not select to activate across all profiles because you also need to go into the profiles (the main one and the copy) you are working with and enable this dimension there so it knows which profile it is working with. You don’t want to affect everyone elses profiles with this extra data, just yours. They need a drop down within the dimension setup to allow you to select this all at once but of course that doesn’t exist to confuse us more.

3. A Measure is another function you need to set up manually (in a database table?) and tell WebTrends that along with the Dimension you just created a table for, should you count or sum this data? In a system like DART, this is so simple it is a drop down choice on an export menu, here you have to configure the backend system of the millennium falcon to get it set. Silly but true. Go to Measures on the custom reports menu. Select new on the upper right corner. Name it. Give it a column name for reports. Use the navigation to go through the process steps to finish creating this measure. You will need the actual tag names for this. You must also go into the profiles (the main one and the copy) you are working with and enable this measure there so it is activated. Ditto about not selecting all profiles on the measure setup so that this works. For my purposes I had the WT.ti (page title tag) and 4 parameters that were possible when that tag was used. (ShareThis: Facebook, Twitter, Email & MySpace) and they all got created as additional measures also. This will give us more granularity in the data showing us not only which pages had share this activity but which site it was shared to.

4. The Custom Report finally! – After about 4 hours of setup for the setup you can finally go do what you originally set out to do. Go to the reports link under custom reports.

The menu for setting up the report asks for the info about the dimensions and measures you just set up as well as how you want the data compiled. I always allow for sorting ability but you don’t have to if you want to save space and if you just export it all anyway it may not matter.

The setup for the report makes more sense having seen everything we’ve already talked through. It is also worth noting that if your template for webtrends software doesn’t have a left nav bar location for the custom reports you have to find the template being used now for your back-end-interface then you have to go into that version of template and add custom reports to that list. I found mine under report configuration, report designer in the report and profiles section of the menu. See screen shot at left. Click the top line of the left nav in the template to deselect the chapters already there and add a new report and list it in a new chapter for custom reports.

You also need to enable the report in the profile itself. Each version of WebTrends has a slightly different menu, but in the edit-report menu of the profile there is a reports tab that lists all the custom reports set up in the system and you select the ones you want to enable.

After this your data in the test copy profile should work, then the trick is repeating it all over again in the main profile and remembering all the steps.

Why is Black Friday such a big deal?

I’ve been thinking about the holiday shopping season and why Black Friday and the entire Thankgsiving weekend is so crazy for shoppers. Why do all these people go crazy trying to get the best deals? Why do they spend so much time out in the malls that weekend buying things? Why is it specifically just this weekend that is crazy?

Some things have occurred to me this year since it is the first one I have had living with my husband in a house vs a condo/townhome setup. One thing has struck me for sure this year. People are scheduled. Everyone has events to attend, multiple family members to see, possibly neighbors or coworkers also and way more work around the house and at their day job than anyone can really handle.

What does this really mean?

People don’t have a lot of time to prepare for Christmas even though they feel the weight of the expectations from kids, family members and themselves after seeing the onslaught of advertising and promotional decorating that happens around town every year.

With way too much going on you usually end up with all your time too busy and scheduled to start shopping until the Thanksgiving event is checked off the list. Then you luckily have a three day weekend with the family that people tend to sacrifice on doing all this work in prep for the bigger deal holiday down the road.

This may be the one weekend  that moms can get out of the house to shop without the kids in tow, since their husbands/relatives are likely off from work also and can watch the kids at home. suprising the kids is impossible if they’re in the store with you so this weekend is important for the element of surprise.

Stores have long made a big deal out of this being the “busiest shopping day of the year” because it is the “first official day of the Christmas Season” even though Holiday decorations usually pop up the day after Halloween. The inside lingo of calling the day “Black Friday” became public about 6-7 years ago as companies admitted that stores may be in the red financially (operating at a loss) for the year until this day because so much of their business is done in the Christmas Holiday Season.

Since then people and marketers have been even more obsessed with having the biggest sales to attract the most buyers and people want the best deals. Entire websites have sprung up (and mobile apps) to track all the prices from competing stores and give you the inside scoop on getting the lowest prices for those items your kids and relatives want.

Some people shop online but how many of us hate it when something is pictured on a site and looks different/color/size/texture than you thought and has to be mailed back and it is twice the hassle of getting it at the store? It is also equally frustrating to see something online and then go to the store to find it isn’t even stocked and you wasted a trip.

So are these special sale prices on Black Friday really all that great? From what I have seen most of the really low prices on high demand items are very limited (4-6 per store, hence the waiting in line to get in first) and most people don’t get them. The other aspect of the deals that you hear less about is that these items are never the high end good quality items on sale. We have been looking at LED TVs this year and none of the ones I’ve seen on sale have been the 240 hz models. Only the 60hz and 120 hz models are on sale. It seems like a way to make a big fuss about people going to your store to get a good deal and limiting the sales to lower quality items and in limited quantity.

In addition to that, the Christmas holiday seems to have little or no religious meaning for most Americans at this point. It is a family oriented economic event. I agree with the ideas of getting together with family and spending time with those who really mean a lot to you in your life. I also believe that everyone has a few things they can’t afford or won’t quite go buy even though they need them and it is family and friends that should help them out this time of year and that is where the gifts come in. But this Santa brings everyone everything they ever wanted thing has to stop. As does the over-buying that the TV/Radio ads tend to promote.

I don’t know if it is worth going through all the craziness of shopping on Black Friday or Thanksgiving Weekend unless you are like us, both employed and scheduled for something on every day of all the weekends from Thanksgiving until the end of the year. Then it is your only chance to get things bought and decorated before family starts arriving expecting your house to look perfect. I am just trying to figure out when I will have time to clean it.

If you’re into Black Friday Sales you probably have: Kids, Inlaws, Someone staying with you over the holidays or a million christmas lights on your house and lawn. You probably live near a large metro area or in a city that has big box stores.

If you’re not you may be a: College Student, Single Guy or Girl or just not have kids. You probably don’t get into the decorating thing either. Someone else in the family probably hosts all the events and you get more sleep per night than they do per week.

A side note: It has also been reported that the Wednesday before Thanksgiving has become the second most crowded night in bars in the year. This used to be a big binge night for College students becasue they didn’t have class the next day and would be back home with all their old friends to meet up and hang out. It is interesting that it has flowed into the single people at older ages category and possibly others also. Maybe they drink because they know what is coming?

Five reasons to Tweet

how to grow your twitter presence graph chart table ideas marketing advertisingThere are millions of useless tweets out there on Twitter.com. Here are some suggestions for things I’d like to see tweeted more:

1. Tweet to say thanks. Not enough people do this in real life, business or online.

2. Tweet something you found useful or helpful.

3. Tweet your reactions to something surprising. Good or bad reactions to products are actually very helpful to the product dev process for companies.

4. Tweet ideas you think may help others. Especially ones not related to your core business. I believe that ideas from outside companies can revolutionize how they think by bringing in fresh info.

5. Tweet back comments to questions. This informal way to survey has developed into a valuable real time tool for people to find answers to real questions.

Things I’d like to see less of:

1. Breakfast, lunch and dinner photos.

2. Celeb following and retweeting

3. Drunk Tweeting

4. Flame tweet wars

5. Lame product pitches disguised as articles, white papers and special product sales deals without full disclosure.

Got any ideas to add?

Last Day At Job Checklist for Work

Today is my last day at the company I work for. I have a new job starting next Tuesday that took 6 months to find. I’m excited to be moving into a cool new job but there are loose ends to tie up here first. This made me think about the process that we use to exit jobs that aren’t because of a layoff or a company shutting down (these take much longer). These are the things I’ve been doing that I realized weren’t things I had thought about before:

1. Distribute the knowledge – I’ve spent all week training people on what it is that I mysteriously do to create the reports I have been doing over the last 4.5 years. Also training a new person this week on how to do analysis from scratch has been a large chunk of the time spent.

2. Clean out personal stuff – I took home all my personal stuff yesterday and then I have today to look over things again in case I missed anything.

3. Distribute valuables – in a controlled economy like the office, good monitors, good chairs and the nick knacks that amuse us are all in limited supply. Distribute these items to those who have been gracious to you over the years and could genuinely use them.

4. Summary Goodbye email – say something witty and genuine to the office or department and leave your contact info for them if they need it.

5. Tie up all paperwork – COBRA insurance paperwork comes in the mail but they may need a paper copy of  your resignation letter with a signature to officially process things. Check when your health insurance ends, mine goes through the end of the month if I don’t opt for COBRA although I probably will until I’m sure that the new insurance will be set up and active. 401K’s can stay put, this one doesn’t charge to stay in the plan so I’m keeping it where it is at the moment.

6. Take any last pictures if you want to remember it. I have a view of Chicago from the 31st floor so I took a lot of pictures.

7. Go out drinking one last time with the gang. And take pictures.

Did I miss anything?

I’m sad that I’m loosing my view of the beautiful yet grizzly city. I’m sad I’m going to miss some great people. I am also dissapointed that I am loosing my great insurance.

I’m happy I’m moving to a manager role. I’m happy to jump into new projects. I’m happy to delve into consumer marketing after 10 years with recruitment marketing and media. I’m happy to never do data entry again or see another media matrix. I’m happy to learn more about webtrends and do more SEM work again. I’m happy to cut my commute time in half, gain my NPR back in the mornings and never ride a diesel fume filled METRA train again.

As you can see the positives out weigh the negatives. So, I’ll be blogging from a much lower altitude in OPRF as of next week.