FILE | B | NIST Technical Support Services for Accessible Voting Systems-Usability Performance - Attachment1

FBO DAILY ISSUE OF AUGUST 22, 2008 FBO #2461
DOCUMENT

B -- NIST Technical Support Services for Accessible Voting Systems-Usability Performance - Attachment1

Notice Date: 8/20/2008
Notice Type: Attachment1
NAICS: 511210 — Software Publishers
Contracting Office: Department of Commerce, National Institute of Standards and Technology (NIST), Acquisition Management Division, 100 Bureau Drive, Building 301, Room B129, Mail Stop 1640, Gaithersburg, Maryland, 20899-1640
ZIP Code: 20899-1640
Solicitation Number: NB894040-8-05964
Archive Date: 8/27/2008
Point of Contact: Mario A. Checchia, Phone: 301-975-8407, Mary Alice Powe,, Phone: 301-975-8567
E-Mail Address: mario.checchia@nist.gov, maryalice.powe@nist.gov
Small Business Set-Aside: Total Small Business
Description: B – NIST Technical Support Services for Accessible Voting Systems-Usability Performance, July 30, 2008 This is a solicitation for commercial services prepared in accordance with the format in FAR Parts 12 and 13, as supplemented with additional information included in this notice. This announcement constitutes the only solicitation; quotes are being requested and a separate written solicitation will not be issued. Solicitation Number NB894040-8-05964 is issued as a request for quotation. This solicitation is set aside exclusively for small business concerns. The North American Industry Classification Code (NAICS) is 511210. The NIST has a requirement for technical support services for accessible voting systems-usability performance. I. Background Information -Enacted by Congress in October 2002, the Help America Vote Act (HAVA) legislation gives the Commerce Department's National Institute of Standards and Technology (NIST) a key role in helping realize nationwide improvements in voting systems. NIST's Information Technology Laboratory (ITL) is coordinating the agency's HAVA efforts through its expertise in areas such as computer security, laboratory accreditation procedures, and usability. The HAVA created an Election Assistance Commission (EAC) to oversee voting standards work. Reporting to the EAC is the Technical Guidelines Development Committee (TGDC), which makes recommendations on voluntary standards and guidelines related to voting equipment. The Human Factors and Privacy (HFP) Subcommittee of the TGDC focused on recommendations for the development of voluntary voting system standards and guidance in the areas of human factors, usability, accessibility, and privacy. NIST is providing technical and administrative support to the TGDC. The EAC adopted a new set of Voluntary Voting System Guidelines (VVSG) in December 2005. The EAC asked the TGDC and NIST to develop a more comprehensive and re-organized version of the VVSG. A complete draft of this updated version was delivered to the EAC in the summer of 2007. NIST will address changes to VVSG as requested by the EAC after the public comment period. NIST will also develop test methods for the VVSG and address technical issues as they arise during discussions with the TGDC and the EAC. The TGDC has directed NIST to develop usability performance-based requirements in addition to design requirements under the guidance of the HFP Subcommittee. Some of these usability performance requirements for the voter interface will take the form of benchmarks based on the results of research, which have produced usability metrics, a test protocol, a standard test ballot, and a controlled set of test participants. Initial work at NIST has shown that this test protocol, the NIST Voting Performance Protocol (VPP) is a valid usability test and is repeatable. The NIST VPP is intended to be a repeatable, controlled experiment. The goal is not primarily to provide a “realistic” measure of voting behavior, but rather to isolate and measure the relative usability effect of various voting systems – and this requires that all other potential variables be held as constant as possible.) Finally, it is important to note that the goal of the benchmarks is to discriminate among voting systems to identify those systems with poor usability. The NIST VPP and the derived benchmarks are intended for use by voting system tests laboratories to certify systems as meeting the requirements in the VVSG. Other pass/fail performance tests for usability and accessibility in the form of tightly controlled usability tests are also under development. The benchmark tests are intended to be applied to all voting systems, including accessible systems. Preliminary investigations of the accessible systems have uncovered additional issues specific to voters with disabilities which point out the need to look more carefully at these systems and how they are tested. For additional background information, see: 1.http://vote.nist.gov 2.http://vote.nist.gov/PublicHearingsandMeetings.html, especially the March and May 2007 Plenary meetings for the Draft Voluntary Voting System Guidelines and the HFP presentations. 3."NIST Special Publication 500-256, Improving the Usability and Accessibility of Voting Systems and Products". http://vote.nist.gov/Final%20Human%20Factors%20Report%20%205-04.pdf 4.Whitney Quesenbery et al, "Defining a Summative Usability Test for Voting Systems", Usability Professions Association, September 2004. http://www.upassoc.org/upa_projects/voting_and_usability/documents/voting_summative_test.pdf 5.NIST preliminary report on “User-Based Conformance Test for the Usability of Voting Equipment”, http://vote.nist.gov/032906User-BasedConfTesting3-10-06.doc 6.NIST preliminary report on “Human Performance Metrics for Voting Equipment”, http://vote.nist.gov/032906performance-metrics3-10-06.doc II. Objective of this Procurement -For the most part, voting systems examined thus far were NOT designated as accessible voting systems. Accessible voting systems must meet both the usability requirements and the accessibility requirements of the VVSG. HAVA mandates one such system per polling place. However, it is very important that the accessible systems for voters with disabilities also meet the benchmarks and to understand what other usability performance issues occur with these systems. NIST has some preliminary data about performance, but there are many gaps. The overall objective of this procurement is to obtain Contractor assistance to gather additional data on the usability performance of accessible systems using the NIST VPP and other methods and to determine what changes, to ensure that the VVSG has testable usability performance requirements for accessible voting systems. In coordination with the NIST COTR, the Contractor will accomplish the following: •Identify usability performance tests for accessible voting systems for the Voluntary Voting System Guidelines (VVSG). •Design the protocols for usability performance tests using the VPP where possible. •Pilot test and collect data with tests participants with a range of different disabilities •Analyze the usability performance data to determine the potential of the performance tests for VVSG conformance tests. •Identify future research to further enhance the usability performance requirements for accessible voting systems. In order to conduct this research, the Contractor must have the following the qualifications: extensive experience testing systems for accessibility, performing usability testing with test participants who have disabilities, knowledge of human factors, usability, accessibility, and statistical analysis, some knowledge and experience with usability and accessibility testing of voting systems (or similar kiosk-based systems). III. Scope of Work - The Contractor shall perform usability testing research to support the development of performance requirements for accessible voting systems. The Contractor shall provide testing sites and participants for the usability testing and associated data collection. The Contractor shall provide detailed documentation of the tests, data collected, and statistical analysis. IV. Specific Tasks- The Contractor shall provide all labor, supervision, materials, equipment, and facilities (unless otherwise stated herein) to complete the following tasks: 1.Identify potential usability performance tests for accessible voting for the Voluntary Voting System Guidelines (VVSG). 2.Design, pilot test, and analyze usability performance tests on accessible voting systems identified in Task 1 to determine their potential for VVSG conformance tests. 3.Identify issues and future research to further enhance the usability performance tests for accessible voting systems. These tasks shall be implemented as follows: Task 1: Identify potential usability performance tests for accessible voting systems for the Voluntary Voting System Guidelines (VVSG). The Contractor shall: (a) Identify gaps in performance testing of accessible voting systems and potential usability tests to address those gaps. (b) Develop a plan for which tests to develop further based on the findings in (a) and discussions with the NIST COTR.(c) Work with NIST COTR to determine Human Subject Studies or Paperwork Reduction Act Requirements and provide descriptions to the NIST COTR to submit for approval by NIST, if necessary. Task 2: Design, pilot test, and analyze usability performance tests on accessible voting systems identified in Task 1 to determine their potential for VVSG conformance tests. The Contractor shall: (a) Select the usability performance tests, design the test protocols, and choose the test participant demographics (including considerations of test participants who are blind, have low vision, have mobility and dexterity disabilities, and have intellectual disabilities). The demographics shall be discussed and agreed upon with the NIST COTR. (b) Continue to work with NIST COTR to adjust Human Subject Studies or Paperwork Reduction Act Requirements and provide descriptions to the NIST COTR to submit for approval by NIST, if necessary. (c) Conduct pilot tests of selected test protocols. (d) Analyze all data and report all findings. (e) Provide findings in detail with NIST COTR. Task 3: Identify issues and future research to further enhance the usability performance tests for accessible voting systems. The Contractor shall: (a) Further analyze all data collected to determine additional demographics and testing variations that would strengthen the tests or address gaps. (b) Identify future research areas based on the findings in (a) (c) Provide all findings with NIST COTR and other NIST staff as requested by the NIST COTR. (d) Attend NIST voting project meetings as requested by the NIST COTR. V. Reporting Requirements and Deliverables - The Contractor shall provide to the Government the following deliverables and reports in accordance with the following schedule. The NIST COTR will confirm the schedule and deadlines. Deliverables/Due Dates: 1.Report on gaps and testing from Task 1. Due two (2) months after contract award date. 2.Report on draft test protocols, demographics and goals of each test from Task 2. Due four (4) months after contract award date. 3.Report on results from the pilot testing. Due July 31, 2009. 4.Report on issues and future research plans from Task 3. August 31, 2009. 5. Final report summarizing all work. Due September 30, 2009. VI. Period of Performance and Location Work is to be Performed Period of Performance is the effective date of award through September 30, 2009. Work shall be completed at the contractor’s facility. VII. Inspection and Acceptance As duties and responsibilities are completed, the Contractor shall request review and inspection by the NIST COTR. Tasks completed in an unacceptable manner may require corrective action by the Contractor. Within 5 business days of receipt of reports, the NIST COTR shall provide the Contractor of any required revisions or deliverable acceptance. VIII. Government Furnished Property and Responsibilities The Government will provide to the Contractor through the NIST COTR: 1.All voting equipment needed for the planning and usability testing in Task 1 and Task 2 2.All details of NIST VPP, metrics, test materials, test data, benchmark data, and statistical analyses from previous investigations. 3.Access to the NIST campus for meetings. IX. Exhibits and Attachments None. X. Invoices The Contractor may submit equal monthly invoices based for the work accepted and completed on the tasks. XI. Quality Assurance Surveillance Plan The purpose of this plan is to provide a quality assurance surveillance for the work to be conducted under this contract. The plan provides a basis for the NIST COTR to evaluate the quality of the Contractor's performance. The oversight outlined in this plan will help to ensure that the topics and assessment are maintained at the required levels through the Period of Performance. Further, this plan provides the NIST COTR with a proactive way to avoid unacceptable or deficient performance. Performance Standards: 1.Quality Level: By monitoring the Contractor, the NIST COTR will determine whether the Performance Standards set forth in the Purchase Order have been attained. Performance Standards for all tasks are specified in the Performance Requirements Summary of the Performance Work Statement. 2.Frequency: During the performance of this Purchase Order, the NIST COTR takes periodic measurements (i.e., conduct surveillance), as specified, and will analyze whether the frequency of measurement is appropriate for the work being performed. Adjustments may only be made by a modification to the Contract. 3.Management Responsiveness: The NIST COTR will determine whether the Contractor has managed the Contract effectively and efficiently with successful completion of the topics and assessment as specified in the Performance Standards set forth in the Performance Requirements Summary. XII. Performance Requirements Summary Task 1: Identify potential usability performance tests for accessible voting systems for the Voluntary Voting System Guidelines (VVSG). Desired Output: Report describing detailed analysis of gaps and types of usability performance tests that could address the gaps. Meetings attended as required. Phone and email discussions held as needed. Performance Standard: Report delivered by Contractor is comprehensive, that is, (1) covers all types of disabilities, and (2) describes usability tests that are performance based. The Contractor responds promptly to phone calls and emails and is in attendance at meetings. Report and other written communications are written clearly, in grammatically correct English and delivered on time. Monitoring Method: Reviewed by COTR. Task 2: Design, pilot test, and analyze usability performance tests on accessible voting systems identified in Task 1 to determine their potential for VVSG conformance tests. Desired Output: Report describing draft test protocols, demographics, and goals of each test. Report on results from the pilot testing. Meetings attended as required. Phone and email discussions held as needed. Performance Standard: Reports delivered by Contractor are comprehensive, that is, (1) cover all decision points, all testing details and data collected, and (2) contain conclusions based on the analyses and/or data. The Contractor responds promptly to phone calls and emails and is in attendance at meetings. Report and other written communications are written clearly, in grammatically correct English and delivered on time. Monitoring Method: Reports reviewed by NIST COTR Task 3: Identify issues and future research to further enhance the usability performance tests for accessible voting systems. Desired Output: Report discussing issues that suggest needs for additional research discovered by Contractor while performing the usability testing experiments in Task 2 and 2. Meetings attended as required. Phone and email discussions held as needed. Performance Standard: Report contains list of at least three (3) additional research areas with each derived from an analysis of the data collected in Tasks 1 and 2. Report and other written communications are written clearly, in grammatically correct English and delivered on time. -Monitoring Method: Reports reviewed by NIST COTR. If further information is required about the facilities, please refer to the NIST website, www.nist.gov. Deliverables: Monthly reports: The monthly reports shall include the details of the effort, including hours spent in each activity. This report will include but not be limited to reports on any of the related activities pertaining to this contract detailing accomplishments during the specified time period. COTR: The Contracting Officer’s Representative (COR) is : to be determined, National Institute of Standards and Technology. Place of Performance: Contractor Site. Price Quote – The contractor shall submit the best potential fixed monthly price to provide the services of the PWS. The resulting purchase order shall be negotiated on a fixed price basis. The award shall be made on a best value basis. The best value evaluation factors while relatively equal are listed in order of importance, include the following: 1) Approach - The likelihood of effectively meeting the requirements; 2) Past Performance - The relevance and quality of prior performance; and 3) Price - The amount, realism and consistency of the evaluated price. The evaluation shall be based on use of the adjectival scheme as follows: Outstanding (O) - Conspicuously striking in eminence; Excellent (E) - Very good of its kind; eminently good, superior or first class; Good (G) - Beneficial and worthwhile; sound and valid; Adequate (A) - Reasonably sufficient and suitable; Marginal (M) - Minimally suitable at the lower limit; Unsatisfactory (U) - Not acceptable, not adequate to carry out. The solicitation document and incorporated clauses are those in effect through Federal Acquisition Circular Number 2005-23. The provisions and clauses may be downloaded at http://www.acqnet.gov.far. The following provisions and clauses shall apply to this solicitation. FAR 52.212-1, Instructions to Offerors – Commercial, applies to this acquisition and there is no addenda to this provision; FAR 52,212-2, Evaluation - Commercial Items. The Government shall award a purchase order resulting from this solicitation to the responsible offeror whose offer conforming to the solicitation will be most advantageous to the Government, price and other factors considered. The following factors shall be used to evaluate offers: (i) approach, (ii) past performance, (iii) price; FAR 52.212-3, Offeror Representations and Certifications - Commercial Items, applies to this acquisition and there is no addenda to this provision; FAR 52.212-4, Contract Terms and Conditions - Commercial Items, applies to this acquisition and there is no addenda to this provision; FAR 52.212-5, Contract Terms and Conditions Required to Implement Statutes or Executive Orders - Commercial Items, applies to this acquisition; FAR 52.204-9, Personal Identity Verification of Contractor Personnel; FAR 52.227-14, Rights in Data – General; 52.217-9 Option to Extend the Term of the Contract and other appropriate Department of Commerce clauses(See Attachment 1 to this solicitation for some additional clauses that may apply.) All interested contractors must be actively registered in the Central Contractor Registration (CCR) at the CCR website www.ccr.gov and must provide a completed copy of the Offeror Representations and Certifications. Interested contractors are invited to review the above Performance Work Statement “NIST Technical Support Services for Accessible Voting Systems-Usability Performance, July 30, 2008 ” and provide an electronic response to be received no later than August 27, 2008 addressed to Mario.checchia@nist.gov. It is the responsibility of the contractor to confirm NIST's receipt of quote. Contractors must submit a complete response. Responses that are not substantially complete will be returned. Offers of partial performance shall be rejected. A electronic version in MS Word or Adobe Acrobat and the electronic version shall be in the form of an email with attachments. A one page letter shall transmit the quotation signed by an individual authorized to commit the organization. Each response shall be on 8 1/2-inch x 11-inch paper, in a commercially standard font, not smaller than a size 12 font. Each page in the response shall be separately numbered. A page of paper printed on both sides is considered two pages. The quote shall include the letter of transmittal and shall be limited to twenty(20) single sided pages. Submit quotes arranged in three sections as follows: Section I - Approach, which includes identification and qualifications of relevant personnel and technical management approach; Section II - Past Performance for relevant experience which describes at least three (3) relevant projects performed in the last three years and quality of performance providing contact individuals names and telephone numbers; Section III - Price, firm fixed price for the required technical consulting services. Each firm fixed price shall include sufficient data to allow the Government to determine the amount, realism and consistency of the fixed price. Award shall be made to the responsible contractor that offers the best value to the Government. The Government shall not be liable for any costs incurred in response to this request. The Government reserves the right to make a single award, multiple awards, or no award as a result of this RFQ. It is the responsibility of the contractor to confirm NIST's receipt of quote.
Web Link: FedBizOpps Complete View
(https://www.fbo.gov/?s=opportunity&mode=form&id=38324b48fd4f3baa5067c08f891d68b2&tab=core&_cview=1)
Document(s): Attachment1; File Name: Attachment1-other clauses (attachment1-clauses.doc); Link: https://www.fbo.gov//utils/view?id=dad8b6fccd05826da0828a965930cd31; Bytes: 36.00 Kb; Note: If links are broken, refer to Point of Contact above or contact the FBO Help Desk at 877-472-3779.
Record: SN01645622-W 20080822/080820221006-38324b48fd4f3baa5067c08f891d68b2 (fbodaily.com)
Source: FedBizOpps Link to This Notice
(may not be valid after Archive Date)

| FSG Index | This Issue's Index | Today's FBO Daily Index Page |

Loren Data's SAM Daily™

B -- NIST Technical Support Services for Accessible Voting Systems-Usability Performance - Attachment1