Accessibility-Aware Reranking and Relevance Feedback demo

The Accessibility-Aware Reranking and Relevance Feedback demo demonstrates the accessibility-aware search engine functionalities developed as part of the CUbRIK project. The results of an image search engine are reranked (reordered) according to their accessibility to the user, who is considered to have a number of vision impairments, hereby cataract and protanopia (red-green color blindness). Moreover, the user can provide feedback to the system, by selecting alternative rankings of the results, which he/she considers as more suitable to his/her impairments. This feedback is used to update and fine-tune the user impairment profile which is maintained by the system.

The demo can be accessed at the following address:

Four user accounts have been created as examples of users having different combinations of vision impairments. The credentials for the four users and their impairments are the following:

  1. username: user000, password: 0123 - User with protanopia.
  2. username: user001, password: 1234 - User with cataract.
  3. username: user002, password: 2345 - User with both protanopia and cataract.
  4. username: user003, password: 3456 - New user, who is presented with vision tests, in order to determine the amount of disability in the supported vision impairments.

The demo search engine is for images related to fashion, so the user can search for e.g. shoes, skirts, jeans, hats etc.

cubrik image 1

Below follow more details about the functionalities of the demo.

New users, such user003 above, are first presented with a set of vision tests, which measure the amount of disability in the supported impairments, by e.g. using Ishihara color blindness tests and measuring the contrast sensitivity. The results of these tests consist the user impairment profile, which holds information about the impairments of the users and is used in order to rerank the search results.

cubrik image 2

The reranking is accomplished by first extracting accessibility scores from the images, i.e. numerical values assessing how accessible an image is to people having the supported impairments. Many alternative rankings are calculated, corresponding to different trade-offs of the various impairments. The one that is closer to the profile of the user is selected to be shown on the screen.

The results are presented in six panels (columns). The first panel (Original results) is the original ranking of the results, considering only their relevance to the query. The second panel (Accessibility results) is the ranking of the results, by considering only their accessibility to the user. The third panel (User vision simulation) simulates how the results of the second panel are viewed by the user, with the specific impairment profile. The fourth panel (Info) shows the accessibility scores of the images of the second panel, in a scale from 0 (not accessible) to 1 (accessible). The fifth panel (Reranked results) is the final ranking of the results, where both their relevance to the query and their accessibility to the user is considered.

The last panel (Feedback), presents alternative rankings graphically, in a Pareto diagram. The rankings are presented as points, so that those which are more suitable for people having protanopia are closer to the protanopia axis, and respectively for cataract. The ranking which is currently presented in the previous columns is represented by a black point in the diagram. The initially selected ranking is the one which is closest to the user impairment profile, which is depicted as a blue point. The user can select one of the alternative rankings by clicking on the respective point on the diagram, or by using the sliders below. After a ranking is selected, the results in the other panels are reordered accordingly. If the user considers one of the alternative rankings to be more suited to his/her impairments, he/she can send it as feedback to the system, in order to fine-tune his/her stored impairment profile. This effect is apparent in subsequent search sessions, where the position of the user profile in the Pareto diagram is updated.

Relevant publications:

  1. I.Kalamaras, A. Drosou, & D. Tzovaras, “Multi-objective optimization for multimodal visualization”. IEEE Transactions on Multimedia, accepted with minor corrections.