Mission report: 24 may to 4 june, 2004: east asia learning achievement study: ssa/thla/2003/0000d984-0

MISSION REPORT: 5th to 23rd February, 2006: PAKE/2006/00000032
Purpose
The purpose of this report is to provide a brief summary of my mission to Lahore
during the period 5th to 23rd February, 2006 as part of the Punjab Examinations
Commission project.
My main duties during the mission were:
 to continue working towards the establishment of the PEC;  to advise on operational matters such as setting up a data entry facility for building the candidate registration database, arrangements for printing examination papers, arrangements for capture of markers’ sheets following the examinations, recruitment of IT staff for the Commission, recruitment of an Acting CEO, and appointment of the Governing Council;  briefing the IT staff on the computer systems needed to support the  assist subject specialist groups in the analysis and interpretation of pilot study Establishment of PEC
Several meetings were held with the Secretary (Education) and the Special Secretary
(Schools) to brief them on progress with establishing the PEC and, in particular, the
appointment of an Acting CEO and the Governing Council.
It was decided that the Secretary (Education) would move urgently to appoint an
Acting CEO on the basis that the appointee would not receive any preference over
other candidates should he/she apply for the permanent post. It was also agreed that I
would share the CTA function with Ted Redden. A copy of the broad plan for
provision of CTA support is given at Attachment 1. It should be understood that both
Ted and I stand ready to return to Lahore at short notice if the need arises and we can
re-arrange our commitments to other UNICEF projects.
It was also agreed that, in the light of the urgency attending many aspects of the
establishment of the PEC and the implementation of its operational plans, the
appointment of the Governing Council would be made after the completion of the
grade 5 examinations this year.
It was also noted that Jamil Najam had taken up his appointment as the Acting
Operational Manager.
Data Entry Facility
The immediate need is for a facility that would have sufficient throughput capacity to
process 1.2 million candidate registration/admission forms during a four week period
in March, 2006. An ideal facility located in the Children’s’ Library was identified by
the Special Secretary (Schools). The facility is comprised of 60 Windows desktop
machines linked via a Windows LAN to a Windows fileserver, all located within one
large room. This facility would have sufficient capacity to handle the candidate
registration data entry workload in March, 2006 if it were to be operated three shifts a
day, seven days a week. The Director of the Children’s’ Library agreed to make the
facility available to PEC for the month of March for candidate registration data entry
and, if needed, for the month of May for data entry of markers’ sheets. On this
understanding, plans were made to utilize that facility at least for the March data entry
workload.
However, at a meeting of the Board of the Children’s Library on the 20th February,
2006 a formal request for use of the facility was rejected; it is understood that the
Board was concerned about the security implications of a 24 hour-a-day operation.
Other options that were explored included the use of the PITE computer facility and
the Punjab EMIS data entry room. However, the combined capacity of those facilities
could handle only about 30% of the projected workload. Moreover, the operation
would be spread over two locations which would pose considerable logistical
difficulties.
Subsequently, Jamil Najam suggested that the Punjab Government’s IT Department
would have equipment available for that purpose and believes that by 24th February,
2006 he should be able to secure agreement to that proposal and, provided that data
entry operators and shift supervisors are also available, data entry operations could
commence on 1st March, 2006.
It will be necessary to monitor very closely progress with this proposal.
Data Entry Operators
In order to accommodate the projected data entry workload it will be necessary to
employ three shifts each of 60 operators – a total of 180 operators – prepared to work
seven-days-a-week for four weeks in March. PMIU’s experience is that there are
many experienced data entry operators available in the market for short term projects.
However, there are obvious logistical difficulties involved in UNICEF employing 180
date entry operators each on a separate SSA. A better arrangement would be for
UNICEF to negotiate an institutional contract with a local IT firm. I have spoken with
Mr. Nazim, a representative of Future Soft the firm that carried out data entry for the
pilot study; he assured me that Future Soft has substantial experience in entering
education data and has the capacity to field quickly the required number of operators.
Presumably there are other firms with a similar capacity. As a matter of urgency I
recommend that quotations be sought from three firms asking for the supply of 180
experienced data entry operators to work on a three shift, seven-days-a-week basis
during March, and a decision taken to contract the firm offering the most economic
and effective arrangement.
Shift Supervisors
All operations at the data entry centre must be carried out under the control of a
PEC/DPI officer acting as shift supervisor, who will be responsible for all aspects of
shift operation. The whole process will be under the control of the Acting Operations
Manager. Jamil Najam has undertaken to identify three appropriate persons who are
currently in DPI and have the necessary skills and experience.
Printing of Examination Papers
The printing, bundling and distribution of examinations papers are huge tasks that
must be accomplished within a compressed time frame. In total, the projected print
volume amounts to 35 million A4 pages that must be bundled into approximately
1000 packs, one for each examination centre.
The Special Secretary (Schools) and Jamil Najam have investigated the option of the
Government Printing Press carrying out the printing and bundling of papers. The
Government Press, however, is not set up as a secure printing operation and Jamil
Najam, in particular, has very strong reservations about the effectiveness of any
security provisions that could be put in place for the duration of the print run. Indeed,
he has expressed the view that leakage would occur within two hours of the Press
receiving the papers.
Additionally, the Government Press indicated that the unit cost of printing would be
0.75 rupees per page. This is much more costly than last year’s.
The private presses that carried out the work last year could again be contracted. This
has attractions because they carried out the work last year without any leakage.
A third option is to engage a firm that would install equipment such as the RISO RZ
370 mini press or Ricoh JP 3000 digital duplicators in the Commission’s office,
supply trained operators, paper and ink and offer a unit cost in the order of 0.40
rupees per page. This option has the decided merit of allowing the Commission to
control the security of the printing operation and of gaining experience in operating an
in-house press. If the Commission took up that option and found it effective and
economic it could negotiate to purchase the equipment at a favourable price after the
completion of the print run.
Adoption of this option would depend on the Commission having its own secure
premises in which to install and operate the equipment.
I believe that this is by far the most attractive option and recommend that Jamil
Najam seek firm quotations from at least three firms willing to enter into such an
arrangement and that a decision be taken quickly to engage the firm offering the best
package.
Entry of Candidates’ Responses
The examinations are conducted at examination centres and supervised by district
invigilators. At the completion of each examination the invigilators batch the
completed papers and send them under conditions of strict security to the cluster
centre linked to that examination centre.
At the cluster centre the papers are inspected by markers and every candidate’s score
on each question is transcribed to markers’ sheet. On each marker’s sheet is a table
with up to 45 columns (the number of columns varies from paper to paper) and up to
50 rows (one for each candidate). One sheet constitutes one batch of data. The
markers’ sheets are then collected and sent under strict security to the data entry
centre in Lahore.
At the data entry centre in Lahore the data on the markers’ sheets must be entered into
the Examinations database for subsequent analysis. This data capture can be carried
out using data entry operators or it can be performed using flat-bed OCR scanners.

Now that the examination papers have been finalised the total volume of data to be
captured can be calculated; this amounts to 350 million keystrokes. To accommodate
this workload within a four week period would require an operation employing 100
data entry operators working three shifts-a-day, seven-days-week.
The alternative approach of using OCR flat-bed scanners such as the Kodak i810
would enable the work to be completed within one week. The Kodak i810 scans black
and white images at 45 pages per minute (ppm). The workload is 7.2 million written
scripts recorded on marker’s sheets (one A4 page) with 50 scripts a sheet. This
represents 144,000 pages to be scanned. Assuming 24 hour operation, the i810 could
scan 64,800 pages a day, hence, the whole operation could be accomplished in two
and a half days.
The labour cost of entering 350 million keystrokes is approximately $60,000 (US) at
the market rate of one rupee per 100 keystrokes. The capital cost of the i810 is of the
order of $10,000 (US). Hence, the OCR scanner option is far less costly than manual
data entry, even after allowing for the capital cost of the scanner. The equipment
would also provide a capacity for scanning next year’s candidate registration forms,
and in 2007 the combined candidature for grades 5 and 8 would approximate two
million.
This approach, however, may be objected to on the grounds of accuracy as compared
with manual data entry. Such an objection would be compelling if it were based on
high speed scanning of hand written alphabetic characters using low-end, desk top
scanners that are engineered as a consumer durable item. However, the approach
considered here relies on a high-end, flat-bed scanner that is engineered to deliver
high levels of accuracy and high throughput rates. This kind of scanner is routinely
used to scan financial documents, passports, custom declaration forms and passenger
movement cards in many countries.
In this regard, it should be noted that the markers will write only numerals on their
sheets, not alphabetic characters. This is important because there are far fewer degrees
of freedom involved in scanning numbers than alphabetic characters and, hence,
higher levels of accuracy are achieved with the former than the latter. In contrast, data
entry operators make many more errors when keying long lists of numbers than when
entering alphabetic text such as name and address fields. This is because they
anticipate the “next” character from the context of the material they are entering,
which they cannot do with a meaningless stream of numbers.
Having regard to these arguments, I strongly recommend that the Commission
acquire an appropriate high speed, flat bed scanner and scan in the markers’ sheets. If
this recommendation is accepted it will have to actioned quickly as the IT Section will
have to thoroughly trial the equipment when implementing the software needed to
validate the markers’ sheets data.
Recruitment of IT Staff
More than 200 applications were received for the two IT posts, ten of whom were
called for interview. It was recommended that Zulfiqar Ali be offered the IT
Manager/senior systems analyst post and Kashif Saeed the senior programmer post.
Both are very experienced IT professionals with sound records of achievement and
solid hands-on knowledge of the software technologies that the Commission will
employ. It is crucial that they take up their posts as soon as possible. I have briefed
them on the software that needs to be constructed, tested and implemented over the
next three months, and will maintain close contact with them during that time.
Pilot Study
The pilot study has been very successful, both in terms of developing final papers and
building capacity in the subject specialist groups. Additionally, the pilot study has
brought into clearer focus many of the problems that have been evident with the
present curriculum documents; these matters are discussed in detail in Ted Redden’s
mission report and are reflected in the planned CTA support for later in 2006.
Software Specifications
Work on specifying the software needed for processing candidate registration,
markers’ sheets, and analysis and reporting of examination results commenced during
my mission in November, 2005 and continued in Australia after my return. During my
last mission, after analysing the pilot study data it became clear that we should report
candidate performance using performance bands based on the distribution of question
difficulty estimates as well as on distribution of candidate ability estimates. This has
meant that I will need to specify additional reports at candidate, school, tehsil, district
and province levels. I have commenced work on these report specifications and will
forward them directly to the IT Section within the next two weeks.
Acknowledgements
Finally, I should like to acknowledge with gratitude the many forms of assistance
rendered by UNICEF, DPI(Elementary) and PMIU. In particular, I should like to
thank Maurice Robson, Deepak Baijrachary, Tariq Saeed, Ted Redden and Jamil
Najam for their professionalism and commitment, and UNICEF for its hospitality.

Dr Ken Vine
27th November, 2006

Source: http://pec.edu.pk/system/files/Ken_Vine_MISSION_REPORT_FEB_2006.pdf