Make a website accessible is part of my daily job, and I enjoy that.
The W3C Web Accessibility Initiative has a checklist to help you make sure your sites are accessible. The checklist is divided up into three priority levels. Priority 1 being the most serious problems and Priority 3 being the least serious. From the checklist:
Each checkpoint has a priority level assigned by the Working Group based on the checkpoint's impact on accessibility.
[Priority 1] A Web content developer must satisfy this checkpoint. Otherwise, one or more groups will find it impossible to access information in the document. Satisfying this checkpoint is a basic requirement for some groups to be able to use Web documents. [Priority 2] A Web content developer should satisfy this checkpoint. Otherwise, one or more groups will find it difficult to access information in the document. Satisfying this checkpoint will remove significant barriers to accessing Web documents. [Priority 3] A Web content developer may address this checkpoint. Otherwise, one or more groups will find it somewhat difficult to access information in the document. Satisfying this checkpoint will improve access to Web documents.
Some checkpoints specify a priority level that may change under certain (indicated) conditions.
Another great way to at least understand the challenges involved is to run your site through a screen reader. JAWS is one of the popular screen readers and they have a free demo that you can download. It has a short time limit (40 minutes) but at least it gives you an idea of what the visually impaired have to work with.
Other screen readers you can try with a downloadable demo:
Window-Eyes
MAGic screen magnification software
Read also this article on Screen-Reader Accessibility
And also Accessify with an almost complete list of resources