Fermilab
| | | | | | | Beams-doc-925
Version: 1 Document type: Note |
Notes on Transferring MI-Notes to the Beams Division Document Database
Bruce C. Brown
Fermi National Accelerator Laboratory*
P.O. Box 500
Batavia, Illinois 60510
12/01/2003
*Operated by the Universities Research Association under
contract with the U.S.
Department of Energy.
Abstract
Using the Beams Division
Document Database (DocDB) for storage, the MAIN
INJECTOR NOTES Web Page has been re-implemented. Many inadequately
stored documents have been located in various formats and stored in
DocDB. In addition to web page you are now viewing, this Beams-doc-925
will store various files which describe/document this effort to assist
anyone who seeks to further improve the quality of MI-Notes
online.
Introduction
The Beams Division has had a variety of notes systems which
document various aspects of the complex and its operations. These
systems have evolved in various ways since their introduction in the
era when all documents were circulated as paper copies. In the fall of
2002, it was recognized that the work of the Main Injector Group was
being impeded by the fact that their web system for documents was
difficult to maintain and available new documents had not been added
for a couple of years. A document database system which had evolved by
use in several experimental groups was implemented to replace the
MI-Notes web page system. This system was then made available to
various groups and now is open for use across the Beams Division.
The Main Injector Group has maintained a formal note system since
1989. In the summer of 1998 and again one summer since then, summer
help has scanned old documents and converted them to a machine
readable form. The access to them has been provided via a web page
(MI
Notes). The system has not been successfully maintained for
documents created since 1999. In parallel with implementation of the
Beams Division Document Database, MI-Notes documents were transferred
to the new system. This document will describe the efforts needed to
achieve the present new implementation and store documents and
spreadsheets which were created to record activities and document
status of the effort.
Document Transfer
Winter 2002 through Spring 2003
Document numbers were reserved for MI-Notes by Bruce Brown and Ruth
Becker (Main Injector Department Administrative Person), starting in
November 2002. A spreadsheet was implemented to provide topics for
Ruth to use and it has been maintained for various uses since.
MI_Notes_Transfer_Audit_Nov2003.xls is the current form of that
spreadsheet and it is part of Beams-doc-925. Beams doc numbers were
reserved for all notes on the existing MI-Notes pages plus some
additional notes in Ruth Becker's lists by Jan 14, 2003. The process
of transferring the documents then started. Many problems were
discovered and processes begun to address them.
- The initial efforts to place the documents on the web involved
conversion to html. In addition to problems with content, links had
been added to the end of the documents to return the reader to the
MI-Notes page or the Main Injector Department page. These had been
hardcoded to a machine name which caused them to fail since that
machine had been replaced.
- The process required to get the desired content, including
embedded images, was complex. The software suggested for the process
was not available to Ruth Becker so initial transfer of .html documents
was carried out by Bruce Brown. Ruth transfered some .pdf files.
- Examination of the .html files demonstrated that many of the documents
contained figures which had not been included in the .html conversion process.
It was decided that if a more complete document existed in the available
paper files, that copy would be re-scanned to create a suitable new version.
- In addition, the process of file conversion employed optical
character recognition (OCR) which was adequate for understanding some
document but failed dramatically for some (an especially dramatic failure is
MI-Note-0017.html). Re-scanning was also selected to solve this
problem.
Two lists were used for MI-Notes in addition to the web page. On a
printed version of a computer file, Note numbers were assigned upon
request by initially writing them onto the paper copy, then updating
the file. The list of MI-Notes was also maintained through the years
in a Microsoft Word file. Together, these were more complete than the
web page. Ruth Becker updated this file and, with further revisions,
it is also available as part of Beams-doc-925. In addition, a list of
document troubles was created in ASCII (MI_Notes_Troubles.txt).
This was updated through April 2003. Most
issues and resolutions were transferred to the ProblemFixFall2003
worksheet of MI_Notes_Transfer_Audit_Nov2003.xls.
Summer 2003
Monica Neuffer was employed for the summer to assist in completing
this project. She filled a number of roles:
- A column for Beams-doc numbers was added to the spreadsheet and
she obtained the document number for each MI-Note which had a reserved
number in DocDB.
- Using the keywords column of the spreadsheet, she read documents
and selected keywords from the then available lists (multiple, at the time)
placing them in the keywords column of the spreadsheet, then adding them
to documents in DocDB along with selection of additional Topic/Subtopic
designations as was suggested by review of the document.
- Documents available online were reviewed and transferred as suitable.
- In the FileType column of the spreadsheet, she documented which files
were available with types which could be more easily transferred.
Some .pdf files were then transferred.
- Many documents were not available or were unsuitable as found.
These were obtained from the paper file copies, scanned and converted
to .html and transferred to the DocDB. Most of this was carried out
using a system provided by the Fermilab Library. Since that system
provided only black-and-white scanning, Beams-doc-343-v1 was scanned
on a color scanner and converted. About 70 documents were scanned
and inserted including some of hundreds of pages.
- Bruce Brown and Monica Neuffer examined the complete list (.doc) of
MI-Notes and sent e-mail reminders to authors or ask Ruth Becker to search
for electronic copies on the Main Injector Administrative Assistant computer.
When copies were provided (many were inserted into DocDB by the author), Monica
inserted them into DocDB.
- Due to inadequate communication, Monica inserted some html documents
without following all of the documented instructions and observing all
the known concerns. [These were addressed in Fall 2003 as they were noted.]
- As .pdf documents were identified with minor corrections required, Bruce
Brown used Adobe Acrobat to rotate pages, add in single re-scanned pages or
make other useful changes to the files before insertion into DocDB.
- In the second week of August, a few entirely new MI-Notes were
inserted into DocDB (through MI-0302). Bruce Brown then issued an
e-mail declaring that MI-Notes were a closed set and that further
documents on Main Injector or Recycler should be issued only as Beams
Documents without any MI-Note designation.
Fall 2003
Following the departure of Monica Neuffer, Ruth Dejerld was asked
to assist in this task. The following tasks were carried out with
her assistance:
- The completed MS Word list of MI-Notes (which started with Marion
Richardson) [attached to Beams doc 925] was checked against the
document database and a new version was created in which the titles
were replaced (in Word) with Links. Bruce edited this file, added
some text, removed the version part of the link to cause the link to
access the newest version of each document, and created a new MI-Notes
web page. After minor additional editing, this was installed on the
Beams Documents menu as the MI
Notes link. Minor further corrections have been made to the file
MI_Notes_Webpage.doc before being stored in Beams-doc-925.
- Ruth Dejerld then audited each MI-Note on this list, entering the
date in the Audit Date column of the MI Notes worksheet for
satisfactory documents. A list was generated of the 46 problems
observed (documents not entered at all, figures missing, missing
formulas and the like). Of these Bruce Brown identifies 16 as ones
which we cannot expect to improve (or perhaps consider good enough)
and 2 of which the authors need to submit the documents. The list of
issues and their resolution make up the first entries in the
ProblemFixFall2003 worksheet in the spreadsheet.
- To address problems, a variety of actions were taken including
locating equivalent documents which had been submitted as TM's and
scanned by the library, located .doc versions (already submitted or
not yet submitted) and convert to .pdf with Adobe Acrobat), or scan
from paper copies (about 10 additional documents), or use the .html
version from the existing MI-Notes website, address problems
in html version by re-extracting it and resubmitting after any
required editing. An ACSII file, Notes_on_DocDB_archiving.txt,
was maintained to document the various steps in saving/converting
these documents.
These activities were the major effort documented in Bruce Brown's web
pages of activities for Summer 2003 ( MagneData
2003 and Projects
and Progress for Monica Neuffer). Additional comments on the
document database and its use are available as Beams-doc-553 and
Beams-doc-919. A web page Notes
on Document Database Effort for Main Injector Group was created
early in the DocDB effort and it is also now available as a resource
concerning these activities.