There’s a new Question Answering system (it’s something different from a search engine, as I wrote in the past) available online for testing: True Knowledge (it just needs a quick registration.)
From my point of view, the best way to test a system of this kind is by posing real questions to it, questions that, in the past, we have already searched the internet for an answer using standard search engines or other sources (especially Wikipedia.) In fact, testing questions invented on the spot is not very useful (if not useless), because we may tend to follow the examples provided (too easy) or ask weird questions which no one would actually ask in a normal situation.
For this purpose, I’ve been collecting a list of about fifty questions (which I update from time to time.) I know it’s not a very long list, but it’s carefully created, balanced and representative of both easy and difficult tasks to be carried out in this sector, from the point of view of an insider.
Even though I’m well aware of the huge complexity that must be faced in order to implement effective systems of this kind, I have to say that here the results are disappointing: only 3 out of the fifty questions obtain a correct answer (while a fourth question is not fully satisfying), with a percentage of 7%. As most questions are very simple (for example, “Who won the Nobel prize for Chemistry in 1999”), I was actually expecting something better (for the above mentioned question Google already provides the correct answer in the first link of the results.) It’ s true that this is just a beta version, however even small variations of the suggested questions seem to invalidate the process, and this makes me doubtful on the soundness of the approach, and on the applicability in real situations.
When the system replies, it seems like magic indeed, but this happens so rarely that the magic disappears and what remains is the distinct sensation of a nice experiments but actually a useless tool. The effort is remarkable, anyway and I will surely test the system again in a few months, but at this stage of development I have to say that unfortunately, the tool is not able to save us time, yet (and as for the future, we will wait and see.)