The Booster Monitor application is designed for requesting device snapshots and calculating statistics on the stability of the data for each snapshot. The application also has the ability to monitor scalar accelerator devices although that is a secondary function and not as well tested at this time.
A statistical analysis of each point in the snapshot is performed to calculate the average, standard deviation, and standard deviation of the standard deviation. The analysis also performs a comparison of the currently calculated values to a set of baseline values which were determined earlier by the application. Optionally the statistics can be compared to a pre-determined reference set or to create a new reference set. If a variation of the current running statistics from the baseline or reference set is larger than a pre-set threshold, then the plot is flagged as being out of tolerance, a message set to the logger, and the baseline for the out of tolerance points is reset to the current running values.
The Booster Monitor uses the application framework developed by the Controls group of the Fermilab Beams division. Because this application needs to connect to a Data Acquisition Engine, the application can only be run on a node behind the firewall where the Data Acquisition engines are located. The application can be run locally on nova at a session prompt for debugging or development purposes, but that is not preferred way for normal use as it poses a great limitation on ease of use.
Instead it is recommended that the application be started through the Webstart mechanism. Webstart is not installed by default on all systems, nor can one expect that their browser is configured by default to use Webstart. The Webstart software comes bundled with Java JRE (starting with 1.2 of JRE) environment. Some system and browser combinations may automatically be configured to run WebStart once the Java JRE is installed. If however that is not the case for the combination of system and browser at hand, then the browser will need to be configured for Webstart.
Since the Java byte code as well as the any supporting Java libraries will be downloaded to the local machine, there is a requirement for free space on the system. Where this data is downloaded depends on the system in use and configuration generally beyond the control of the user. It is unlikely this will actually be a problem for the typical user, but the user should contact the system administrator if it does.
The first step is to open a web browsers and connect to the Application Index (http://www-bd.fnal.gov/appix/index.html) page. From there there are a few simple steps for finding and starting the Booster Monitor application.
Once Webstart has downloaded the files and started the application, the user will be presented with the a window for selecting a reference set. The Reference Set is optional and should only be used when a thorough understanding of the operation of the list of devices to monitor is achieved and a solid Reference Set has been defined. Premature or inappropriate use of the reference sets will put a significant load on the datalogger and is not an appropriate use of those resources. Unless use of a reference set is necessary the user should choose "Cancel". If using a reference set is appropriate, then click on the desired reference set in the scrollable list and then chose select. If the chosen reference set matches all device snapshot requests for the configuration, then the curves will be displayed. Otherwise no curves will be displayed indicating a mismatch between the reference set and the current request configuration.
Before the Booster Monitor can be start data taking the user must connect to a Data Acquisition Engine (DAE). In the bottom right hand corner of the main window there is a plug and socket widget. Clicking on that widget will open a window that allows the user to manage a connection to the DAE.
Once a connection to the DAE has been established, the user can start a run. The "Start" button is toward the bottom of the window. Clicking on the "Start" button will start a run. Assuming that the clock event signals requested are being generated the display will begin to the snapshot data as it is received. Additionally, a successful run start will disable any button that can only be used between runs.
Once a run has been started the "Stop" button will become enabled. This button can be used to "Stop" a run.
For further information on operational use and behavior of the Booster Monitor see the Operational Details section.
From a web browser inside the firewall go to the link http://www-bd.fnal.gov/appix/index.html. Note that the applet which displays the list of applications can be very slow in building the tree. Thus be patient when deciding if the browser is hung as it may take a few minutes.
Webstart will display its own download window which will show the progress of each library being downloaded to the local machine. After the downloading is complete then it will indicate it is starting the application and the splash window from the Application Framework will be displayed as the framework is initialized.
Connections to the Data Acquisition Engine are managed through the connection window. The window can be opened by clicking on the plug and socket widget in the lower right hand corner of the main window. The connection window allows the user to connect and disconnect from the DAE. In most cases the user will want to use the default location for the DAE listed on the "Connect To" line. To make a connection first select the "Connect To" checkbox and then click on "Ok" or "Apply". The former button will hide the window and make the connection while the later will keep the connection window visible and make the connection.
In order to complete a connection to the DAE the user will need to enter their kerberos principal and password. A kerberos login window will appear the first time the user attempts to connect to the DAE during the current session of the Booster Monitor. There are two text entry boxes in the window the one for the principal name will echo the characters typed while the one for the password will echo an asterisk for each character typed. The user should enter their kerberos principal and password and then click on "OK". If all goes well application will successfully connect to the DAE. If however the connection is refused due to an invalid kerberos principal and password combination, the application framework will terminate the session. This is unfortunate but at the moment a work around has not been found. Establishing a connection to the DAE can be quick or very slow so again patience is required.
The Booster Monitor provides a great deal of information in display format. Currently the Booster Monitor will display up to 16 separate snapshot plots in the main display. The application will automatically set each based on the order the snapshot requests appear in the configuration. Each plot has a button associated with it labeled "Plot0" through "Plot16" that can be used to change the request displayed. Clicking one of these buttons will open a window that allows the user to change the request being displayed. The user can if desired show the same request in every plot.
Each plot will display the snapshot data received along with the statistics calculated for the snapshot and the reference set if defined. A plot is updated when data for that associated request is received.
There are two counters near the bottom right hand corner of the Monitor and Deviations tabbed panes. The one on the left with the capital A is a counter for the number of Analog or Array data call-backs received. The one on the right with the capital S is a counter for the number of snapshot call-backs received. Each time a call-back with valid data is received the appropriate counter will be updated. These counters allow the user to see the how often data is received from the DAE.
In addition to the "Start" and "Stop" buttons there are two buttons related to changing the configuration called "Reconfig" and "ReadConfig", two buttons related to reference sets called "Save RefSet" and "Reload RefSet" and one button for changing the threshold.
There are three tabbed panes in the monitor application. The first two panes Monitor and Deviations tabbed panes show data plots for the average values and the average standard deviations respectively. A third tabbed pane, called the Editor pane, allows the user to modify the existing configuration and optionally save the modified version for future use.
The reassign plot window allows the user to change the snapshot request being plotted in the selected plot. The window title will indicate the plot number that the current window session will effect. Terminating the window will cancel the request. In order to trigger a change first select the new request in the scrollable window and then click the "Use Snapshot" button. Note that the application allows plotting of Analog or Array Accelerator devices if present in the configuration. These can be select in the scrollable window on the right hand side and will take effect if "Use Analog or Array" is selected.
The snapshot plot display not only plots the data and statistics, it also provides other information related to the plot. A border title around the display indicates whether the data is related to a snapshot request or an accelerator analog or array device request. The title also indicates if the central type of value plotted is an average value ("Plot") or an average standard deviation ("Sigma"). See sections on Monitor and Deviations tabbed panes. There is also a number associated with the plot so that the button for reassigning a plot can be matched to the display.
Next there is a title with three fields separated by a colon. The first field indicates the hex value of the clock event number while the second indicates the clock event name. The third field is the clock event display which is almost always 0. To the right of the title is a the text "FLAG". When any point in the snapshot request is out of tolerance the "FLAG" text color will change from green to red and the window background will be given a yellow tint.
Below the title line is a plot key entry for the name of the device for which the snapshot data has been generated.
The plot itself is automatically rescales the y axis to display the data within bounds. There are a series of curves displayed on each plot. The baseline average value is displayed in blue. The current average value is displayed in green. The standard deviation of the baseline average value multiplied by the threshold value added and subtracted from the baseline average value is displayed in red. If a reference set is used, then the reference set average value is displayed in yellow and in an analogous manner the reference set standard deviation added and subtracted from the reference average value is displayed in gray. Note in the above discussion the average value means the actual average value for each point when viewing in the Monitor tabbed pane, while it means the average standard deviation when viewing in the Deviations tabbed pane. In the later case the standard deviations then become the standard deviation of the standard deviation.
The user can pop up a larger version of any plot in a separate window by double-clicking on the desired plot anywhere in the white background region. The enlarged plot can be resized and more than one enlarged plot can be spawned at any time.
Essentially the same as the snapshot plot display for accelerator device array data. For analog data it behaves as a strip chart display with the various curves having the same meaning as in the other plots.
The monitor display tabbed is a collection of plots of snapshot data for the average value of the each point in the snapshot request. Each plot in the window can be for a different snapshot request. The display plots can be changed by clicking on the appropriate Plot button to reassign a plot display to another snapshot request. The pane also contains a set of run control and configuration configuration buttons.
The editor pane allows the user to modify the current configuration. The various fields are all keyed to the entries described by the configuration format. The required fields are placed in a red region in the upper left hand corner of the pane. Optional fields are placed in a blue region in the upper right hand corner. There are also regions for managing the actual requests with a pull-down list of requests currently defined. The basic functionality is the same for both snapshots and accdevices (analog or array) in the main editor window: add; modify; clone (an existing one); delete. Clicking on the "Add Snapshot" button for example will bring up the Snapshot Entry window. Similarly clicking the "Add AccDevice" button will bring up the AccDevice Entry window. These windows are for defining snapshot and analog/array requests respectively.
The user can start over with a different configuration by clicking on the "ReadNewConf" button and selecting a new configuration to read from disk.
To save a configuration go to the File menu and select SaveAs. Input a unique name in the text entry field and click on the "OK" button.
Changes made in the Editor pane will have no effect on the application configuration unless the application is explicitly reconfigured. This allows the user to design a completely different configuration for future use without interfering with the configuration of the current session. A useful feature for users capable of multi-tasking.
The "ReadNewConf" button from the Editor pane, or the "ReadConfig" button from the Monitor pane or the Deviations pane will open an tree view to select a new configuration file. Currently the configuration files are stored under the test sub branch of the test branch. Uncollapse the initial test branch and then click on the test test sub folder. Do not uncollapse the test sub folder, as confusing as it may seem, just click on the name test itself. This will cause a listing of configuration files not all of which are Booster Monitor related. Best to select one with BoosterMonitor in the name unless you really know what you doing. In the future it is hoped a separate area will be created for the Booster Monitor, but for now we can only do what the underlying infrastructure can support. Select the desired configuration file by clicking on it and it will appear in the label with the tag "file name". Clicking "ok" will cause the new configuration to be loaded if it is valid.
The "Reconfig" button on the the Monitor pane or the Deviations pane will read from memory any changes made in the the Editor pane and reconfigure the application according to that configuration as if it was read from disk. Only by clicking on this button will the changes made in the Editor pane have any effect on the actual configuration. This allows the user to be running one configuration while designing a completely different one for future use without any interference.
There are two reference set related buttons: "Save RefSet" and "Reload RefSet". The first allows the user to save the current baseline statistics to the reference cache (a database), thus defining a new reference set. The second allows a new reference set to be read from the cache.
Clicking on the "Save RefSet" button will bring up a window that asks for a unique name to tag the reference set with. If the user does not know if the name will be unique they can select "cancel" and check the currently defined reference sets using the "Reload RefSet" button. Note that the system will prepend a date tag to the user name to aid in selecting a unique name.
Clicking on the "Reload RefSet" button will bring up the a window with a scrollable list of reference sets. Select the desired reference set from the list and then click on "Select" to load the new reference set. If the chosen reference set matches all device snapshot requests for the configuration, then the curves will be displayed. Otherwise no curves will be displayed indicating a mismatch between the reference set and the current request configuration.
The threshold is a scaling factor applied to the baseline, and reference set if defined, standard deviation to determine the tolerance range for comparison with current average of a value and the standard deviation. For example a threshold of 1 would mean tolerances of plus or minus one standard deviation. Similarly a threshold of 7.5 would mean plus or minus 7.5 standard deviations. Clicking on the button with "Thresh" in the name will bring up a window for changing the threshold.
This entry window allows the user to create or modify a snapshot request. Required parameters are in a red region in the upper left hand corner while optional parameters are a blue region in the upper right hand corner. Actions options are advanced features and generally will not be used by the normal user. See the section on configuration format for details on parameters.
This entry window allows the user to create or modify a Accelerator device Analog or Array request. Required parameters are in a red region in the upper left hand corner while optional parameters are a blue region in the upper right hand corner. Actions options are advanced features and generally will not be used by the normal user. See the section on configuration format for details on parameters.
The configuration of requests has a main wrapper which supports a list of snapshots and Analog or Array requests. The main parameters are:
The Accelerator device requests are for analog or array type devices. These requests take the following parameters:
The Snapshot requests are for getting time sequence data for a device. These requests take the following parameters:
Actions are specially defined functions that can be called in addition to or instead of the default analysis. An action can be used to define new analysis algorithms such as curve fitting comparisons. Note that under webstart the user cannot locally define new actions to use. Instead these actions need to be predefined in the code release build. Once in the release however they can be used to perform special analyses. Developers working locally on a build system can define new action classes and test with them dynamically as long as they are on the CLASSPATH and extend the appropriate class. Check out the Action*.java examples in the boosterMonitor package code base. Actions take the following parameters in the configuration:
Java Webstart is a mechanism for downloading Java applications via a web browser and running the applications locally. Webstart not only downloads the application code, it also downloads the Java environment required for running the application and launches the application in the downloaded environment.
In order for the browser to properly handle a Webstart link the browser needs to be configured for the MIME type returned. Here is an example of how to configure a Netscape browser on nova to allow support of Webstart applications. In general one needs to know the absolute path to the javaws application.
The general statistics mechanism is based on acquiring data for each point in the request and calculating separate average, standard deviation, and standard deviation of the standard deviation. The first task is to develop a baseline for comparison purposes. The application also calculates a current set of statistics, again using a limited window size, for comparison purposes. When a deviation is found for a point beyond the tolerance level, then the baseline for that point is set to the current values and the event recorded to the logger and flagged on the display. If the comparison is against a reference set then no reset is triggered but a message is sent to the logger and the display is flagged.
There are two different methods for calculating the statistics. The first is a fifo method (the default). In the fifo method all data for the a number of data sets, defined as the window size, are kept for recalculating values. When a new data set arrives the old set set is dropped, the new one added and the quantities recalculated. This is the preferred mode of operation but it will not scale well when the window size becomes large.
A second method for calculating statistics attempts to be more clever. Instead of using a fifo it uses a sliding window scheme where a shadow set is always half the window size behind the current set. When the current set reaches above the window size limit, the shadow set is half that size and used as the new current window size. The shadow window then starts over from empty. Additionally the data sets are not kept for each point. Instead the values are calculated based on previous calculated values and new data. This is possible because the average for a set with one new point can be related to the old average plus a difference term. Expanding all calculations and replacing the difference in terms of know quantities allows each calculation to be updated more quickly. It also means that the amount of data stored remains constant as the window size is increased. This can be beneficial for large window sizes but should not be used for smaller ones.
Generating a baseline involves collecting a limited number of data values for each point in the request and then determining the statistical quantities mentioned previously. The number of values to collect is called the window size in the configuration. The baseline consists of an average value, a standard deviation and a standard deviation of the standard deviation for each point in a request. In other words each quantity for every point on the curve. The baseline gives the user an idea of short term stability of the system.
A reference set is a based on a collection of a limited number of data values for each point in the request and the determination of the statistical quantities mentioned previously. The number of values is called the window size. The reference set consists of an average value, a standard deviation and a standard deviation of the standard deviation for each point in a request. In other words each quantity for every point on the curve. The reference set gives the user an idea of the longer term stability of the system.
The Booster Monitor Project arose from a need to monitor a large number of operational measurements of devices in the Proton Source at various accelerator clock cycles. It was anticipated that the number of measurements that might be required could easily exceed the capabilities of the Data Acquisition Engine (DAE) and therefore a means of throttling requests was required. In the end it turned out that there were also potential hardware limitations on the number of measurements that could be simultaneously made in the front ends reinforcing the need for throttling. From early on there was recognition that the application could become useful beyond the realm of the Booster machine. Thus the software was designed as a general purpose multi-snapshot analysis monitor.
The operational measurements consist of one or more device snapshots or readings of scalar or array data from accelerator devices. These operations can be grouped according to the API available for making requests to the DAE. The first type is the accelerator device group which consists of analog and array reads of devices. The second type is for snapshots. Snapshots represent a time sequence (set by the sampling rate) of readings of the same scalar value in a device.
From an operational standpoint there are not a lot of differences between the various parameters needed to make a request and very little in the data returned. Still the structure for making requests to the DAE are very different. Thus a parallel approach for managing the two types of requests is well suited for the situation.
Because the number of requests can be large enough to overwhelm the DAE and the front end systems, a means of looping through all of the requests is necessary. Additionally there are a large number of different clock events that can be used as a trigger for data collection. Various clock events are related to different modes of operation of an accelerator. Thus at any given time some of these clock events may not be generated in the system depending on the current mode of operation for the accelerator. Requests made for clock events that are not currently being fired will just remain pending until the clock events start firing again. Because the application needs to loop through a list of requests, these requests should be grouped into separate tracks based on their clock event triggers. This will allow the looping process to continue for all requests with clock event triggers firing without being blocked by requests with missing clock event triggers.
Another parameter available in the request format, but not recommended for a setting other than 0, is the clock event delay. Support for this is included for flexibility and potential use by very select experts only due to the soft nature of how nonzero delays are treated in the front ends. Again due to the request API it follows that a further grouping based on clock event delay is desirable. Requests with the same clock event and delay can be bundled together and made as one request to the DAE. Because even with this level of sub grouping there might still be too many requests to the DAE and front ends, the ability to bundle only a subset of requests and loop over the entire list repeatedly is needed.
The general cycle of operation is to have separate tracks for in each request type, with each track representing a different clock event group. Each track will make one request to the DAE after a data callback from the DAE for that track has been received and processed. Thus there are a number of parallel loops in progress, one for each track, that act mostly independent of each other. Within each track requests up to a set limit will be bundled together which have the same clock event delay by looping over the requests in a clock event delay. Within each track when all requests for a given delay have been made for the current iteration cycle, the track will look at the next delay and start bundling those requests. Once all delays have been completed for a cycle the process starts over with the first delay again.
When data is received via callback from the DAE it needs to be analyzed and displayed. Each point in a curve (each distinct point of data returned in a request) should have an average, standard deviation and standard deviation of the standard deviation calculated for a finite number of cycles (called a window size). These statistical values should then be compared to other sets of data and discrepancies noted. It is desirable to have an initial baseline generated by the application which can be updated for any point that goes out of tolerance. This gives a short term idea on the stability of the data for a device. For a longer term idea on stability there is also the need for reference sets to be created and used for comparison. A reference set is just a baseline from some earlier time that was deemed by experts to be an ideal description of how the various devices should look when the accelerator is running well. In this case discrepancies are only noted and do not alter the reference set.
The basic design philosophy used is to use a tree structure of grouped functionality. At the top level the analog and array requests are grouped together and the snapshot requests are grouped in another group. Inside each of the top level groups things are grouped together based on clock event names (also referred to as clock event triggers above). Each clock event name grouping is then sub-grouped based on clock event delay. Every element inside a clock event delay grouping represents a request for a device and all statistics related information for the device. These device holders are then further sub divided into a separate set of statistics for each point in the device. Above the tree level the application handles the interaction with the DAE and acting as a middleman between the DAE, the data and request processing tree, and the display.
When the application is configured at start up, or reconfigured later, all of the requests are read in, processed, and used to populate the tree structure. This higher layer goes to the tree and asks for bundled requests which it then sends to the DAE. Data received back from the DAE is then passed back to the tree which will respond with information for the display. The layer then uses the information for communication with the display. Thus various tasks are grouped in layers designed for a more specific purpose.
The configuration is XML based and uses a validating parser. The engine for converting the XML into Java classes is fairly generic. The engine expects the Java class name will match the XML tag name. Furthermore, the engine expects the Java class to advertise and export all allowed attributes and members based its namesake XML tag. This structure thus allows changes to the XML/DTD to be made without any alterations to the engine itself. There would be changes in the mapped Java classes however. In this way the engine code can be considered a generic library and the mapped classes as user code. Using this model the same basic engine has been now used in at last 3 separate and diverse projects at Fermilab.
The reference sets are cached in a database. The structure of the database is not completely as one might expect as timing issues forced some large groupings to be made. Interaction with the cache for reading should display a list of current reference sets in reverse order from when they were cached. This allows the user to see the most recent reference sets earliest in the list. When attempting to save a reference set the name given for the reference set must be unique. For several reason a date string will be prepended to the name provided by the user and used as the actual reference set name.
Logging of out of tolerance incidents are done through the Controls group datalogger mechanism. Data can be retrieved via D44. For various reasons including advice from a datalogger expert the device names will be converted to uppercase and the first two characters replaced with "G_". The normal format of the data sent to the datalogger will be: