Archive for the 'Answers Spotlight' Category

What do star ratings mean?

Friday, August 25th, 2006

ratings.jpg

I, like many other Google Answers researchers, pay close attention to the ratings that the paying customers give for our answers. But what do those stars really mean?

I recently spent a few days in Paris, staying at a two-star hotel. Sure, it was on a noisy street – but that’s because it was in a supremely convenient location. Sure, the room was pokey – but it was clean and cheap. And the breakfast, although not included in the price, was generous, delicious, reasonably priced, and was served in a bright and cheery room with sparkling cutlery and crockery.

Maybe the hotel was only two-star rated because the room didn’t have a trouser press or satellite TV. But I wasn’t in Paris to press my trousers or watch TV.

The point is: everyone has a different set of standards by which they rate something, so we should not expect every customer to have a rating scale that matches our own.

The point was driven home by a recent question about car-sales websites which was answered by politicalguru-ga. The customer rated it “only” three stars, but gave a tip.

Some researchers take three stars to mean that the client is not satisfied with the answer given, but I see it differently. I reckon it usually works like this:

  • 1 star = very poor
  • 2 stars = poor
  • 3 stars = acceptable
  • 4 stars = good
  • 5 stars = very good

The Google Answers Help and Tips page says something slightly different (though I doubt that many customers have studied it carefully):

To rate an answer, click on the “Rate Answer” button in the answer field. You can rate an answer from 1 star (poor) to 5 stars (great) … Google Answers suggest that you rate an answer based on its content (how helpful was the information), clarity (how clearly was the information communicated), and tone (how friendly and polite was the writing).

A slightly different rating scale is offered by Google Video. When you hover your mouse over the rating stars you see the following guidelines:

  • 1 = poor
  • 2 = below average
  • 3 = average
  • 4 = above average
  • 5 = excellent

None of this really matters: the point is that ratings are always subjective and can certainly be inconsistent. If you bear that in mind, they can still provide useful feedback to the researcher.

ftPLEEEEZE

Sunday, August 6th, 2006

I usually write about things that I know something about, but I fear that I’ve already covered everything in my meager repertoire. 

Anyway, today is a bit different. I’m writing on a topic about which I know next to nothing…ftp.

In researching an open question at Google Answers, about world timezone data, I came across what looks to be a very informative site on the topic of timezone databases:  Sources for Time Zone and Daylight Saving Time Data

 

Everything I need to answer the aforementioned question is here, no doubt.  Problem is, I can barely make heads or tails of it. 

There’s a mysterious world out there of the ftp-heads; folks who prefer storing and trnasfering information using an ancient method known as file transfer protocol, rather than the more recent hyper-text transfer protocol, or good old http, that powers the internet. 

Why use ftp?  I haven’t a clue, except that most of those involved seem to programmers, so perhaps there’s an advantage in that world that I can’t quite make out. 

But I suspect it has more to do with ftp’s appeal as the sort of street slang of the internet.  FTP-ers have a secret language, known only to insiders, and it rarely seems to get translated into http-happy language. 

Take the timezone link that I mentioned above.  The site starts off with:

The tz database…The public-domain time zone database contains code and data that represent the history of local time for many representative locations around the globe.

BINGO!  That’s exactly what I want.  But where the @#$%^&*! is it?

The site is full of links…to pages on time zone boundaries, daylight-saving rules, something called the GNU C Library, and a site on FTP distribution.

But the actual timezone database is, how shall I put this, elusive.  At least to me, it is.  Not even a specialized ftp search engine was any help. 

And if you want to paly around with a file or two, just try unzipping things with file extensions like rmp, gz-tar, sqx.   Poor old WinZip just throws up its arms and chokes.

 

There are plenty of other folks who have figured out how to access timezone files and put them to use, such as:

TWiki.org Service: Date and Time Gateway – Timezone Selector

Current Time in 1000 Places

etc., etc.

But nowhere does anyone seem to provide the sort of Timezone 101 instructions that a non-programmer like myself needs in order to get going.

By this point, you’ve probably recognized my little write-up here for what it really is – 50% whining, and 50% pleading.  But just in case, let me be explicit….

For the thousands of Web-Owl readers out there…a little help?

Thanks.

pafalafaga David Sarokin

Latin translations

Monday, July 31st, 2006

dictionary.jpg

A popular genre of questions at Google Answers is translations from English to Latin.

Sometimes the questions are driven by personal curiosity, sometimes the translations are related to schoolwork, and some are motivational messages. Sometimes someone just wants a pompous-sounding byline for a website, and a surprising number are messages of love.

Since the Google Answers service started, around 200 such questions have been answered, at prices ranging from $2 to $60.

The following translations were for tattoos – better get those ones right!

Nearly as critical was this one, to be engraved onto an Ipod as a gift:

Here's just a tiny sample of the rest:

There are many more.

Researchers providing these translations included hlabadie-ga, guillermo-ga, pinkfreud-ga, tutuzdad-ga, juggler-ga, livioflores-ga, but mostly alanna-ga who also provided two further links of interest: Latin phrases used in English, and Latin proverbs and locutions.

I've don't speak Latin myself, but back when I was at school my friend Neil used to quote ridiculous Latin phrases. One of his favourites was Quanti canicule ille in finestre? which translates as How much is that doggie in the window? and can be found with other similar nonsense at Latin Phrases and Expressions.

(photo by Sanja Gjenero)

Answers are not just research

Thursday, July 27th, 2006

Although we call ourselves Researchers, and Google calls us that too, what we do can often go beyond that.

The service is called Google Answers, yet what the customers ask for often goes beyond an answer to a question. Here are some of the things we have done recently:

These aren't things you can “just Google for”. It's a whole different job from research-oriented questions like Omega 3 requirements in pregnancy or Incidence and prevalence of asthma.

QINANs: The Untouchables

Sunday, July 23rd, 2006

At any given moment, there are hundreds of unlocked, unanswered questions at Google Answers awaiting a researcher's attention, including, typically, about 25 questions priced at $200, and even more in the $100-200 range.  The unanswered questions have a combined list price of, easily, $10,000 or more.

At the same time, there are dozens of researchers scanning what I call the QINAN list — Questions In Need of An Answer.   Each researcher is hungry for their next lock, and hoping that, this month at least, they'll earn something approaching minimum wage.

So why aren't the researchers locking the QINAN's, and socking away those thousands?

This has been one the major surprises to me about Google Answers…the sheer number of questions that, while doubtless posted in good faith, are virtually unanswerable.

I've picked out one or two in several price categories to give you a bit of a feel for what I mean.

 

Q:non linear optimisation $200

Huh?

This one is asking a highly technical question involving the solution to a mathematics problem in C++ programming.  It's the type of thing you either know, or not.  There's no searching for a solution for this one.

 

Q: Names of engineers in Palm Springs  $200.00

This asker wants the names of ALL designers and engineers and CADD operators and a few others who work for any civil engineering firm in several California cities.

Ha, ha.

Actually, it's hard to articulate WHY this question is unanswerable, but it is.  First off, we can't promise ALL of anything, as there's always the probablility that we've missed one or two.  But the main problem is that firm-by-firm employee directories just don't exist, and no amount of skillful research on our parts can bring them into existence.

 

Q: Investment accounts that allow debit card access? Sub accounts?   $150.00

This one's interesting, because it probably is answerable with the right amount of clarification and effort.  But I would have to charge the asker $150 just to read through and make sense out of the convoluted question about specialized features of investment accounts.  Then I'd have to start calling around for information about such accounts (which is generally a lost cause, because the firms don't want to give out a lot of information to me, until I give them the details about my finances, etc, and they feel they have me on the hook as a potential customer). 

Then, I'd have to start a long round of Clarifications with the asker (which may never be seen, given the problems with the notification system) asking if such-and-such a bank sounds like the right one…and so on, and so on.  It really would take thousands of dollars worth of effort, and at the end of it all, there's a good likelihood that there simply isn't a firm out there that offers the type of specialized accounts that are needed. 

And the asker has made clear that an answer of 'it can't be done' just ain't gonna cut it.

 

Q: Retail Sales United States and Western Europe   $120.00

Wants to know every major retailer in the US selling SuperMax razors.  And their sales!  And their profit margins! 

And then do the same thing for all of Western Europe.

I'm out of breath, just thinking about it.

 

Q: USDA Schedule C position?  $100.00

This one broke my heart a bit, because I have some pretty deep background knowledge on this topic, and I would have like to have helped here.  But the back and forth only made it clear that the asker didn't want what this researcher had to offer, and seemed to want some sort of magic that, I suspect, no one can provide. 

 

Q: I am looking for scientific research papers that state GHB is safe   $100.00

GHB is a dangerous and illegal drug, and this asker wanted proof that it's 100% safe. Sorry….

 

and lastly there's:

 

Q: attorney lied about healthcare How do I get legal redress.  $5.00

At least it isn't all caps!  But the question is unclear — it's amazing how many like this are posted — and the process needed to better understand what's being asked here hardly seems worth the effort, particularly for a legal mess that we would be unlikely to help untangle. 

 

These are by no means the worst of the bunch, or even particularly representative of the unanswerable questions at Google Answers.  Just a look at a not atypical bunch of QINANs from yours truly,

pafalafaga David Sarokin

Google Answers Researchers are People Too

Friday, July 21st, 2006

by Knowledge_Seeker 

Reflecting a bit on the numbers in my previous post and in the other posts about the number, locations, and answering rate of Google Answers Researchers, I’ve decided I should try to dispel what may be a common misperception.

When you read Google’s description of More than 500 carefully screened Researchers…” and then learn that we hail from hundreds of locations in dozens of different countries, you may get a sense that we are 500 separate entities working in isolation – that from our kitchens, offices, upstairs bedrooms, libraries and cafes, all we focus on is grabbing the next question.

But no, Google Answers is not a melee of hundreds of researchers all scrapping for questions and trying to grab locks and post quick answers before other researchers can get them. We are a relatively small group of dedicated Researchers who work steadily (and sometimes collaboratively) through the list of unanswered questions providing timely and (most importantly) accurate substantiated answers.

Yes there are hundreds of Researchers, but the core group of active Researchers has changed little over the years. Many of the names on my list of those who have recently answered questions are Researchers who have been actively answering since the service began in 2002. Sure, some have fallen off the bottom of the list and a few have come out of hibernation and have joined us along the way, but all-in-all we are the same people.

Despite our physical remoteness, Google Answers Researchers frequently work together. We have at least one forum and one Google Group where we can meet. The forum I belong to (provided by one of the researchers) has 75 members – again shaking down to 25 or 30 active participants. Here we discuss and critique answers, debate ethical questions, get advice on dealing with difficult clients, and ask for assistance on answers.

At any given time we might be arguing the ethics of providing a customer with information on how to commit suicide, finding a few more references for a colleague who feels his answer is a bit short, asking if someone in Germany could please make a quick local phone call to confirm a fact, or alerting a Researcher that a customer is requesting her. Mindful that the customer is more important than the almighty dollar, we willingly hand off full or partial answers when our lives take us unexpectedly from our work.

Google Answers Researchers are perfectionists. For many of us, a 3-star (and for some, even a 4-star) Answer is unacceptable. A rejection is devastating. When an answer gets rejected or rated one- or two-stars, the Researcher often brings it to the attention of the group. Together we dissect it and try to determine if the problem was the customer, the Researcher, a misunderstanding, or simply a rating mistake. If it was the Researcher, we say so. Yes, we’re friends, but when it comes to our work, the customer comes first.

Google Answers Researchers go far beyond just answering questions. We may be independent contractors, but most of us care as much about Google Answers as the Google employees do. We put a great deal of thought and effort into improving our own service to the customers and improving Google Answers overall.

In our forum we maintain a large resource center with links to helpful sites, databases, and tools. In addition to what Google supplies to us, we have put together our own training guidelines, Researcher FAQ’s, and Beginning Researcher guide. We have brainstormed and submitted to the Google Answers Editors an extensive list of suggestions for improvements and changes to the service. We have taken part in an “Owlet” program set up by the editors to apprentice new Researchers.

Google Answers Researchers serve as front-line watch dogs for the Google Answers Editors, alerting them to inappropriate questions, comments, and answers. We also monitor the media and the web for references to Google Answers, correcting misconceptions and errors as they arise. We’ve corrected our Wikipedia entry in several different languages, have thanked reporters and researchers for delivering balanced articles, have debated with critics and skeptics, and have carefully watched as competitors rise and inevitably fall.

In short, while we may be “More than 500 carefully screened Researchers”, Google Answers Researchers are committed to Google Answers as a whole, not just to our own paychecks, star ratings, or personal fame on the website. We are wholly dedicated to keeping Google Answers the highest quality answer service on the net.

Thanks for listening,

-K~

The top Google Answers researchers

Thursday, July 13th, 2006

Which Google Answers Researcher has answered the most questions?

That would have to be GAR powerhouse bobbie7-ga, with 3192 questions answered.

Next in line seems to be the veritable pinkfreud-ga, with 2190 questions answered, followed closely by juggler-ga on 2045.

Other high rankers are tutuzdad-ga on 1704, pafalafa-ga on 1532, scriptor-ga on 1464 and  justaskscott-ga on 1107.

Have I missed anyone who has answered over a thousand questions?

I’ve answered a whopping 187.

How many researchers are there?

Tuesday, July 11th, 2006

How many Google Answers researchers are there? If you go to the Google Answers Home Page it says that “More than 500 carefully screened Researchers are ready to answer your question” (you don’t see that message when you are logged in).

How many of those researchers have answered questions? Knowledge_seeker-ga counted 57 who had answered one or more questions within the last 60 days.

Researchers come and go. I have answered quite a few questions recently, yet I’ve been very inactive at times. Twice I have received a December check from Google for a mere $1.50, for answering just one $2 question. It seems that on December 31st the accountants like to square the books.

Anyway, back in 2004 Benjamin Edelman of Harvard University studied Earnings and Ratings at Google Answers. He checked all the answers posted from the start of the service in April 2002 until November 2003, and found 534 distinct question answerers.

There is a lot of other fascinating information in Edelman’s study. For example, did you know that tipping correlates with number of links provided in the answer? Edelman also found that earnings increased with researcher experience – he estimated that every extra question a researcher answers raises their earnings by about two cents per hour.

Coming back to those 534 researchers: what else do we know about them? We know they come from many parts of the globe – aceresearcher-ga once assembled a partial list and found researchers from Argentina, Australia, Austria, Barbados, Belgium, Brazil, Canada, Chile, China, Colombia, Cyprus, Czech Republic, England, Germany, Hungary, India, Ireland, Israel, Italy, Japan, Lebanon, Malaysia, Netherlands, New Zealand, Norway, Philippines, Poland, Romania, Russia, Scotland, Singapore, South Africa, Spain, Switzerland, Tanzania, Turkey and the United States.

Google Answers Researcher Activity

Tuesday, July 11th, 2006

Because it’s a slow day here and has been raining since morning, I figured I’d do a bit of counting myself. I did a quick hand count of how many of the “600+” researchers have been actively answering questions for the past 60 days.

Answer: 57

Of those, 54 were active in the last 30 days.

Of those, during the last 30 days:

1 Answered 75 questions

1 Answered 50 questions

1 Answered 44 questions

10 Answered 20-30 questions

13 Answered 10-19 questions

The rest answered fewer than 10 questions apiece.

These numbers are very much in keeping with a similar count I did about 2 years ago. I no longer have the exact numbers, but I remember the total number of active researchers being just around 60, with a small subset of those picking up a high percentage of the workload.

I leave it to commenters to decide what this all means …

-K~

The life of a Google Answers question

Monday, July 10th, 2006

What happens to a question after it is posted to Google Answers? Here’s a rough idea.

Most of this is wild extrapoliation from my own experience; the rest is based on quick approximate counts from publicly-accessible pages at Google Answers.

  • One person posts the question
  • A few percent of questions are removed by the Google Answers Editors because they contravene the Terms of Service for Google Answers – for example, because they include identifiable personal information
  • A few percent of questions are accidentally posted more than once. Researchers usually post a comment to alert other researchers to the duplicate question
  • Over 600 researchers have been approved by Google to answer the question
  • Around a hundred researchers will read the question title, of whom…
  • Perhaps twenty researchers will read the question body.
  • Half a dozen researchers will lock the question to conduct some preliminary research, of whom
  • Two will research the question in depth, of whom
  • One will ask for clarification, after which
  • There will be an exchange of clarification details of between zero and a dozen or more messages (averaging around three).
  • Finally, for about 50% of clarified questions,
  • One researcher will post an answer.

This will result in:

  • Between zero and a dozen or more request for clarification and clarifications, averaging two or three, plus
  • Between zero and over a thousand comments, mostly from non-researchers.

The question will then:

  • Be rated in about one-third of cases: usually five stars, sometimes four stars, occasionally three stars, rarely two stars and occasionally one star
  • Be tipped in about 25% of cases, usually from $1 to $10 but can be up to $100
  • Be rejected by the customer, or withdrawn by the researcher, in under 5% of cases

Of the unanswered questions:

  • A small number will be closed early, but
  • The vast majority will expire normally.

Of the answered questions:

  • Under five percent will be referenced and linked to from a future question

Does anyone have any other metrics of interest to suggest?