Hi Mike! Thank you for starting such an interesting conversation!
Agreed it is a hellishly difficult task, and probably THE MOST difficult task of the CitSciMedBlitz challenge! What you're doing is the very essence of textbook learning! When you read text, your brain identifies the concepts in the text, organizes how the different concepts are related in the text, and then integrates it into your existing framework on the subject. You can think of learning as the addition of new experiences, ideas, etc. into a network of information you've gathered based on your experiences. Now imagine just how much knowledge is in the biomedical literature if we think of every article as a knowledge dump of scientific experience! I think we could solve a lot of questions more quickly if there was a better way of harnessing that knowledge!
Text mining tools can help us figure out what genes, diseases, and proteins are in an abstract (to a certain extent), but figuring out how these different things are related? Not easy at all! In fact, there was so much doubt as to whether or not this could be done by non-specialists, that we actually had to have an experiment to test it before we spent the effort developing Mark2Cure (hence the paid amazon mechanical turkers). Agreed that mentioning this isn't a great motivational tool
Given how difficult it is to read an abstract, we chose to allow deeper relationship options other than the binary 'relates to', 'has no relationship to' because the most important information is in the specific relationship between the concepts and it would be a waste of users' efforts if they read and understood the text, but weren't able to submit more than 'relates to' vs 'doesn't relate to'