Is More Data Always Better?

google think magazine data overload obesity information ideas processing analysisThere has been a discovery in the online marketing and data/statistics world in the last few years. We have had more websites, products and tools created online than we can possibly keep track of. The terms to describe this deluge of activity we have been hearing the most are “data overload” and “information overload” from both companies and consumers. This Google Magazine uses the term Data Obesity to describe this phenomenon.

They ask the question, why is more data always better?

I think the idea of “more data us better” is common from people who lived before the Internet was prevalent. We had to work hard to find data. Researching something meant going to a library and looking in a card catalog (or maybe something called Gopher) and then finding your way around the Dewey decimal system to find that book. And then sometimes they didn’t even have the book because it was checked out or possibly it was just filed wrong because nobody understood the Dewey decimal system.

On a related note recently we got invited to my cousin’s wedding in Santa Fe New Mexico. My dad promptly went to the library and checked out 3 books on Santa Fe and New Mexico. I cringed. He asked how to find out the flights to book something without a travel agent. I realized I have been traveling since 2000 this way and he stopped traveling about that time so he never has. I introduced him to Travelocity, it was mind blowing and a bit of data overload compared with the OAG book he used to use in the 80’s.

The point here is that finding data was really difficult. People had control over its distribution because it was in print. When it became more freely accessible due to Google and other companies efforts we assumed this would be good, because people could remember where to find it and use it whenever we wanted. We never thought it would get this big so fast. Now travel sites are overwhelming, they have too many choices and there are too many of them trying to get you to opt into something you don’t want while being over charged for bringing a suitcase on a flight. This is just one example of how data has gone exponential so quickly.

Others of us have come to a data overload conclusion when they have 200 emails in several in-boxes, 1000+ rss reader posts from feeds waiting, several work projects, 500+ Facebook wall posts in their feed and hundreds of tweets that have gone un-read. This is among a climate where you have to follow-up with projects 5-10 times to get things done, post blogs/tweets/FB status updates daily to keep on people’s radar, empty the DVR so it doesn’t get overloaded and auto delete something you really wanted, listen to the radio on the way to work just in case something big happens and still find time to scoop the litter box before it gets full and the cats poop on the floor.

And the real purpose in all those tweets/FB posts and feeds is that you business changes yearly and if you don’t know about the latest trend and some real insights about it before your boss asks about it, you won’t have a job for all that long. (in digital marketing)

Having data overload be a “good” problem to have from some people’s perspective (as in that it is growth oriented). The democratization of publishing combined with tracking methodology and databases have all contributed to this problem, giving everyone a voice, a potential following of readers, a data trail to analyze and method to say something important online 24/7/365.  And then we have an even bigger problem of processing what is being said, figuring out if it is important or not and sharing/processing/saving it in some way if it is. Acting on that data is way down the line and many of us don’t even get there.

And this isn’t even the big problem with data overload. Where will we store it all? Why do tweets disappear from search so quickly? Because there are millions of them and the failwhale is full. According to the ThinkQuarterly UK, there are 800 Exabytes of data/information created every two days. It took humans from the beginning of civilization until 2003 to create the first 800 Exabytes, and we’re on a roll now.

Where does all this seemingly random data go? How will we know what it says without having to go into a database table and read specific field information? Where are the software tools to manage all this and still give humans the ability to customize the out put in ways that match the behavior or business purposes that we really need? Does any of this stuff ever get deleted?

These are all huge questions we have to answer as more people publish, share, create, track and do business online. We also have to weigh the possibilities of sharing data openly and locking it behind walls as well as how will people comprehensively find what they need when they want to as well as gauge the validity/accuracy of the information presented?

I’m betting on paid services for personal and business data management/archiving & Analysis tools. We will pay for good analysis, good data access & processing and good reliability/backups when we feel the pain of missing good insight, losing good data and just too much happening. Both personally and professionally. But unless you know how to work with SAP, SPSS, SQL, Oracle or a bunch of other systems data management is largely out of your control at this point. They are the librarians of our digital data and they need to find a workable way to Dewey decimal system it back into order and allow us to use it as humans need to.

How to create a webtrends custom report

webtrends menu
the webtrends menu in my old company login, new one isn’t as nice.

First off I am going toclarify that I am working in WebTrends self hosted software version 8 and that I have administrator access to do this. Without those factors you won’t neccissarily have the same experience setting up custom reports.

I have already had the developers add tags to the site to track custom events (SDC tags, DCS MultiTrack events) and I don’t work with the actuall tagging on the website so I will not address those concerns here.

I’m starting from the point where the tags are live on the site and I need to set up this custom report so we can start gathering data. If you need retroactive data because the tags were placed on the site months ago and you didn’t create the report, it is only possible if you create the report and re-process all the data from scratch for that time period and this is usually done in a new profile. This takes a long time (sometimes weeks if you have years of data) and the people maintaining the servers and data for your webtrends setup usually hate you if you ask. So, let’s stay out of that area and go build a report that will collect data in a table from this point forward.

One of the most frustrating thing about WebTrends is that nothing is connected and it is a bunch of database tables. Nobody in Marketing thinks that way or interacts with a system like this so it is completley foregin. (‘m getting the feeling that the developers feel the same way too) You have to go into several tables in the database behind the webtrends software and create the structure for the report. It sucks but that is the way it works.

For every custom report you need at least three things. A Measure, A Dimension and A Report. (you may also want a filter but that isn’t required).

1. Testing – Some people (webtrends actually advises this) like to create a copy of the profile to test the reports (or any changes to the profile) before they add or change anything for real. This reduces the probablility that you will have some change corrupt or crash your data. If you depend on this data and have a lot of custom reports and data already set in the profile you’re working with, start with a copy profile first and don’t endanger the main data set. Work in that sandbox until you find the right combination of settings, write them down and then go back to the main profile when you’re sure the report exports the right data in the right format. There is a one day waiting period to collect data to test in most cases unless you’re running on-demand (of which most of my advice may not apply to) or you have updates and processing every few hours. Also only select the most recent month’s data in the copy profile, you don’t need all the data to run a test.

2. A Dimension is what most people would call a metric name. It is the data being collected from those custom tags. Just because the tags are in the WebTrends format and live on the site doesn’t mean that the WebTrends software knows that they are there or what to do with that data. You need to create a dimension to name this data and create a table to collect it in. Go to Dimensions in the admin menu. (some systems show custom report menus under reports & profiles others in Admin, check both) Select new on the upper right corner. Name it. Give it a column name for reports. Use the navigation to go through the process steps to finish creating this measure. You will need the actual tag names for this. In my case it is WT.it. Not terribly descriptive but hopefully functional. Do not select to activate across all profiles because you also need to go into the profiles (the main one and the copy) you are working with and enable this dimension there so it knows which profile it is working with. You don’t want to affect everyone elses profiles with this extra data, just yours. They need a drop down within the dimension setup to allow you to select this all at once but of course that doesn’t exist to confuse us more.

3. A Measure is another function you need to set up manually (in a database table?) and tell WebTrends that along with the Dimension you just created a table for, should you count or sum this data? In a system like DART, this is so simple it is a drop down choice on an export menu, here you have to configure the backend system of the millennium falcon to get it set. Silly but true. Go to Measures on the custom reports menu. Select new on the upper right corner. Name it. Give it a column name for reports. Use the navigation to go through the process steps to finish creating this measure. You will need the actual tag names for this. You must also go into the profiles (the main one and the copy) you are working with and enable this measure there so it is activated. Ditto about not selecting all profiles on the measure setup so that this works. For my purposes I had the WT.ti (page title tag) and 4 parameters that were possible when that tag was used. (ShareThis: Facebook, Twitter, Email & MySpace) and they all got created as additional measures also. This will give us more granularity in the data showing us not only which pages had share this activity but which site it was shared to.

4. The Custom Report finally! – After about 4 hours of setup for the setup you can finally go do what you originally set out to do. Go to the reports link under custom reports.

The menu for setting up the report asks for the info about the dimensions and measures you just set up as well as how you want the data compiled. I always allow for sorting ability but you don’t have to if you want to save space and if you just export it all anyway it may not matter.

The setup for the report makes more sense having seen everything we’ve already talked through. It is also worth noting that if your template for webtrends software doesn’t have a left nav bar location for the custom reports you have to find the template being used now for your back-end-interface then you have to go into that version of template and add custom reports to that list. I found mine under report configuration, report designer in the report and profiles section of the menu. See screen shot at left. Click the top line of the left nav in the template to deselect the chapters already there and add a new report and list it in a new chapter for custom reports.

You also need to enable the report in the profile itself. Each version of WebTrends has a slightly different menu, but in the edit-report menu of the profile there is a reports tab that lists all the custom reports set up in the system and you select the ones you want to enable.

After this your data in the test copy profile should work, then the trick is repeating it all over again in the main profile and remembering all the steps.

Five reasons to Tweet

how to grow your twitter presence graph chart table ideas marketing advertisingThere are millions of useless tweets out there on Twitter.com. Here are some suggestions for things I’d like to see tweeted more:

1. Tweet to say thanks. Not enough people do this in real life, business or online.

2. Tweet something you found useful or helpful.

3. Tweet your reactions to something surprising. Good or bad reactions to products are actually very helpful to the product dev process for companies.

4. Tweet ideas you think may help others. Especially ones not related to your core business. I believe that ideas from outside companies can revolutionize how they think by bringing in fresh info.

5. Tweet back comments to questions. This informal way to survey has developed into a valuable real time tool for people to find answers to real questions.

Things I’d like to see less of:

1. Breakfast, lunch and dinner photos.

2. Celeb following and retweeting

3. Drunk Tweeting

4. Flame tweet wars

5. Lame product pitches disguised as articles, white papers and special product sales deals without full disclosure.

Got any ideas to add?

WebTrends Email Stats Reports How To Setup

I love that WebTrends is a good solid web analytics reporting solution, but I really find the setup process for just about anything with this system to be very confusing and lengthy. I’m sure there is a reason for this (could be data integrity processes or cost savings) but I really just need a step by step list when I need to get something done quickly and someone to tell me where these oddball parts of the process exist. Therefore I’m writing a list to explain this process so I have it written down and other people can find this info too.

(technical note I’m using webtrends software 8.1, not the webtrends hosted solution)

Today the task at hand is setting up automated reports of webtrends data to be sent monthly by email. The duration of the data collected and the frequency of the reporting schedule are both flexible, it can be daily, weekly, monthly, quarterly or yearly.

The first step is to go to the administration menu from your login. There go to Report Designer and Templates. You can select one of their templates, I needed to create a new one.

Then name your template. Go to next, select the content by adding (and naming) a new chapter, then adding content to that chapter from the add report link on the menu above. Select the “built in report” list from the drop down to get the standard metrics available in webtrends. Check the boxes of the metrics you want included, I would say 4-8 per report is enough before you have too much data for someone to really use. You can make changes to the layout, although I was not looking for that level of detail now.

Click next at the bottom of the page. Then you have some configuration settings, like for wrapping text lines on long urls (ok) and how many rows of data in the reports (20-50 max for readability, top 5 is good).

Click next again and give profiles access to this report. I noticed mine are already given universal access and grey-ed out so nothing much to do with this screen. Then click save.

Next you then go back to the profiles list (admin menu and web analysis and reports & profiles) and edit the profile you want to get this report to add the report to the profile. This is one of those steps I think is redundant and should be automated or brought into the setup process before this because its confusing. You wave over the profile in the list (don’t click it) and get a menu with “actions” and edit is one of them.

From there go to reports in the top menu and on the drop down go to report templates. Click the box by your report to select it, ignore the second checkbox that is labeled default because it will change the default reporting style in the profile to this new report, and that isn’t the intention here.

Then go back to the admin menu a third time and to the scheduler menu (bottom) and then schedule jobs and click the button for a new job. This is the email setup part. Under job type select scheduled report and follow through the pieces of menu from left to right as you fill out each section. First select your profile you want reported on, next give the report a name and assign it to a user (yourself). (note this is also how to disable the emails with the check box below, no idea why this is hidden here). Report type: general report. Output type can me a database, pdf, excel/csv or pdf. I chose pdf because it looks professional and we don’t have to install Microsoft office/word on the server in order to export it. Its the only option that does not require that except the database. Number of data rows to report is up to you, I usually do top 20.

Next add the report destinations, this is where you need the email info. Add your email as the from, add theirs as the to address. Also, cc yourself on these reports so you get them too. Add the SMTP server address (if you don’t have the SMTP address it will hold up all of your other scheduled jobs, so don’t set this up without it.) So, the software knows where to connect to send it from. (contact IT about this if you don’t have it) You can also FTP it if you like your data that way, or save to a folder on the server. (not as exec friendly though) 

Under templates, complete view is ok. Under reports, here you select the reports you want to include. These are a duplicate of the ones you selected above, maybe redundant but this is literally the process we took on the phone with the WebTrends helpdesk people. Report type: standard again, date range: its up to you. Scheduling is next on the menu, you can’t run it on the 1st of the month because data may not have compiled yet in all time zones so the 2nd of the month is the first you can run a monthly report with the most recent previous month’s data. Ditto lag time for dailies, weeklies etc. Run once or run weekly/monthly/daily, as you choose.

The host binding section he literally told me to ignore. So I have no idea what that means. Then you get a summary page at the end and click save.

You just wait now and see if everything gets delivered correctly. It is good that the report is only generated once per month on the date you specify as a job that processes, so it can run data in the past (vs only from the point you created the report, forward like custom reports do because they create their own database table) and it won’t clog up your processing queue with a lot of memory/processing because it’s just once.

I wish there were short concise directions for setting up webtrends email reports like this on the web already but I realize that nothing is easy or self explanatory with database systems or webtrends. It’s just part of the territory until next generation tools come around, and no I don’t mean Google Analytics (which is almost as confusing now to beginners). Someday this has to get simpler in process so more people can use it.

Bounce Rates on Google Analytics

google analytics bounce rate pages exits ratesI was just discussing what Bounce Rates were in Google Analytics and thought this could be a potentially confusing term and would be helpful to blog about. I also work with WebTrends .

We have a client that has a site with us that had a high bounce rate and a high exit rate. (50% for some pages) Anything above 20% would be something worth looking into in my opinion, but the differences change depending on the site, product, sales process and design so everyone has their owne level of normal as a benchmark and you try and improve from there.

They wondered if this Bounce Rate was an issue, as many clients would.

The thing is, it may not be an issue to have a high Bounce Rate because if people land on a product description page and then click to buy (or in our case, apply) is this really bad?

Well the qualifier for a Bounce Rate is that they viewed that one page and left. This does not include someone clicking on a link on the page to buy/apply. That would be an exit. They would not have viewed any other page on the site or interacted (clicked) anything else either. This bounce would be from hitting the back button or clicking the x button on the browser.

Exits from the site are considered people who have viewed more than one page and finished their visit. They may click to apply/buy or they may x-out of the window or they may reload the home page. (just a few of many examples) One tricky thing is when someone gets a site that launches a new window for a page you click on. That is typically an exit and new site visit. 

So, is this good or bad? For this client I think it is ok, because they are very stringent about who they are looking to hire and when people see the extensive requirements I am pretty sure most people would realize whether they had a shot at the job or not very quickly and either click forwards in the path to apply (on an Applicant Tracking System application site, (don’t ask, too many sites linked with too many processes)) or back out. It is very straight forward and very few other options are on the page.

How do you reduce bounce rates?

I never hear people talk about strategies to get more qualified traffic to these pages, I just hear about providing more info on the page to help them convert. That is a great strategy and if you can link exact search terms to the appropriate products/jobs with a page designed for one clear desired action then you are doing well. If you can suggest other related alternatives on the same page, maybe on the right sidebar, you are doing even better. If you have an email sign-up that says, not what you are looking for? Sign up here and we’ll email you when new ones come up. Great. But if you have a lot of traffic bouncing even then, you may want to look at the source. What words are your pages optimized for and why do those keywords not match what you’re providing or asking people to do? Maybe search is also not the right medium to find people based on the Google Insights search volume for that term and you are getting similar searches/clicks but not for what you offer. Maybe reel in the search efforts and go for more qualified means of finding these specific people like email, targeted display ads (by content/interest, behavior or location) or offline communication. (gasp!)

Remember Google Analytics (or any analytics package) is not just about a bunch of numbers and bunk. If you can’t figure out what the human behavior is behind the numbers or what the actual user/customer wants they don’t mean much of anything except that your site is up and running.

Why FriendFeed is no big deal

I am so lost on why so many people are gaga about friendfeed. Of all the web apps that have come out this one has the least technical savy and is not at all new in function. Is it useful? Yes, but has it been around somewhere else since 2002? YES!

I started using Livejournal in 2002 and one of the things that hooked me about it was that all my friend’s LJ blog feeds were listed in reverse cronological order on a friends page. This was great. I didn’t have to go to each blog individually to read them and I could scroll and scan through a day’s worth of posts in a few minutes. Later they added the ability to add other RSS feeds into LJ for reading as a friend. I added everything I was watching at that time. I fed everything from fickr feeds to blogs like Scoble into the mix. They even allowed you to categorize your friend feeds into different groups for more organized reading commenting was also possible direct from the feed although if it was not an LJ blog the comment would not be submitted back to the original site, just on LJ.

All this was great but almost too great. I stopped using RSS all the time in about 2006 because there were too many feeds to read and too much information to process. I couldn’t keep up. It was taking hours to get through everything even just scaning and skimming. I pretty much abandoned the feed page there even though I still post occasionally.

So, basically I am not excited about FriendFeed since it is just another incarnation of something I already have and there were limits to how much information and how many feeds and posts I could reasonably process a day. I have actually gone backwards in tech terms now and I use link buttons in my browser for the blogs or sites I read daily and I know that when that row across the top of the browser is full, one has to go if I add one. It’s is kind of a limit to how many I can monitor and keeps it under control. I also use the feedbliz email subscriptions a lot too. It creates a lot of extra email, but it is easy to scan and delete if it isn’t pertinent info and if there isn’t good info for a month I unsubscribe.

Anyway, maybe some day soon people will come to their senses and find a way to balance being aware and alert of what is happening on the net but not wasting an entire day following blog posts and feeds. It is a huge hit to productivity in America because the discovery process is buzzy and addictive when you find something huge and you always think it will be there and then your day is lost.

New Google YouTube Ads

I just tried to install the YouTube Google ad format on my other blog. it was interesting, and it isn’t really suited that well for my blog. I got it to work and you can see it here. Chicago Restaurants

Here is what I would like from Google and YouTube to make it better:

1. Better targeting of videos. Maybe choose specific videos that have relevance to your content. Their filters aren’t working for me. I even gave them Chicago keywords as a guide and they still are serving random stuff that isn’t related to Chicago in any way. Maybe use those keywords to do a tag search and just serve those that have those tags? Or let me pick them individually.

2. The size is not right for my blog. It’s too big for anywhere pretty much, so I opted for above the posts because it will blow out the side bars if I put it there. What might work is to update the wordpress theme to alternate posts with video units on the main page. Or I may have to put it inside the posts only, and hope those posts have some relevance to the videos. This won’t work until we get better targeting.

Anyway, it’s an interesting way to syndicate their content out and get this huge affiliate network going generating eyeball views for their content and keeping YT top of mind in consumer’s eyes when so many other video sites are popping up every day.