The EPICS archiver appliance offers a web UI for typical configuration tasks. This UI works best with a recent version of Firefox or Chrome.
Adding a PV to be archived.
To add a PV to the archiver,
Go to the home page of the archiver appliance.
Enter any number of PV's in the text area (one per line) and click on the Archive button.
If you wish to specify the sampling period, then click on the Archive (specify sampling period) button instead.
When adding a new PV to the cluster, the archiver appliance measures various parameters about the PV. This takes about 4-5 minutes during which the PV is in an Initial sampling state.
Once the PV's characteristics have been measured, the archiver appliance assigns the PV to an appliance in the cluster and the PV is in a Being archived state.
If the archiver appliance is not able to connect to a PV, the PV stays in the Initial sampling state until it is able to connect.
You an request a PV to be archived before it is available on the network. The request to archive the PV is stored in the persistent configuration until the archiver appliance can connect.
Once the archiver appliance is able to connect to the PV, it continues on the normal workflow (where various parameters about the PV are measured etc).
Using Archive (specify sampling period), you can additionally specify
Archival method: Whether the archiver will use CA monitors/scan threads to sample at the specified sampling period.
Conditional archiving PV: The archiving for this PV is performed only when the conditional archiving PV's .VAL field is non-zero.
Policy: The archiver appliance uses policies set up the administrator to configure how the PV is to be archived.
You can override this by choosing a policy yourself.
This lets your system administrator set up a set of policies and lets the user choose manually from among those policies.
To check the status of a PV.
Enter any number of PV's in the text area (one per line) and click on the Check Status button
You can also enter GLOB expressions like XCOR* and the archiver appliance will then display the status of all PV's that match the GLOB expression.
Note that the number of matched PV's can be quite large; for now, no attempt is made to restrict the number of entries in this particular request. This may change in the future based on user feedback.
Retrieving data using Archive Viewer and Matlab
To retrieve data using the Archive Viewer, see the Archive Viewer section.
To retrieve data using Matlab, see the Matlab section.
Retrieving data using other tools
The EPICS Archiver Appliance supports data retrieval in multiple formats/MIME types. These are some of the few formats supported today; more can easily be added as needed.
JSON - A generic JSON format that can be easily loaded into most browsers using Javascript.
CSV - Can be used for importing into Excel and other spreadsheets.
MAT - This is the file format used for interoperating with Matlab.
RAW - This is a binary format used by the Archive Viewer and is based on the PB/HTTP protocol.
TXT - A simple text format that is often helpful for debugging.
SVG - A XML format that can also be used as a SVG element in tools that support this format.
In general, getting data into a tool necessitates construction of a data retrieval URL as the first step.
A data retrieval URL looks something like so
http://archiver.slac.stanford.edu/retrieval is the data_retrieval_url element from your appliances.xml
/data/getData is the path into the data retrieval servlet and is fixed.
.json identifies the MIME-type of the returned data.
The remainder consists of parameters (all of which need to be URL encoded for correctness)
pv - This identifies the name of the PV for which data is requested.
from - This is the start time of the data in ISO 8601 format; specifically this format.
to - This is the end time of the data in the same format.
Here's an example of loading data into Python using JSON and then using Chaco to render a plot.
import numpy as np
from chaco.shell import *
import urllib2
import json
req = urllib2.urlopen("http://archiver.slac.stanford.edu/retrieval/data/getData.json?pv=test%3Apv%3A123&donotchunk")
data = json.load(req)
secs = [x['secs'] for x in data[0]['data']]
vals = [x['val'] for x in data[0]['data']]
plot(secs, vals, "r-")
xscale('time')
show()
Reports
The EPICS Archiver Appliance supports a wide variety of reports based on static and dynamic information.
These reports can also be accessed from within scripting languages like python and can be used to facilitate automation and integration.
Some reports that are currently available are
PV's that may not exist - This report lists all the PVs in the cluster that have never connected. PVs whose names have typos in them would be included in this report.
Currently disconnected PVs - This report lists all the PVs in the cluster that have connected in the past but are currently disconnected; perhaps because their IOCs have been turned off.
Top PVs by event rate - These reports contain the PVs sorted by their actual event rate (descending).
Top PVs by storage rate - These reports contain the PVs sorted by their actual storage rate (descending).
Recently added PVs - These reports contain the PVs sorted by the creation timestamp of their PVTypeInfos (descending). PVs that were recently added show up first.
Recently modified PVs - These reports contain the PVs sorted by the modification timestamp of their PVTypeInfos (descending). PVs that were recently modified show up first.
PVs by lost/regained connections - These reports contain the PVs sorted by the number of times we've lost/re-established the CA connection. This can be used to determine if an IOC is being overwhelmed (perhaps by all the archiving targeted at it).
PVs by last known timestamp - These reports contain the PVs sorted by the timestamp of the last event received.
PVs by dropped events from incorrect timestamps - The EPICS archiver appliance discards events that are too far into the future or too far in the past. These reports contain the PVs sorted by the number of events dropped because they had invalid timestamps.
PVs by dropped events from buffer overflows - The EPICS archiver appliance discards events when the sampling buffers (as estimated from the sampling period) become full. These reports contain the PVs sorted by the number of events dropped because the sampling buffers were full.
Metrics
The EPICS Archiver Appliance maintains a wide variety of metrics to facilitate in capacity planning and load balancing.
These metrics can be viewed in the Metrics page; a small subset of these metrics can be viewed across the cluster.
To view more details about the metrics on a particular appliance, click on that appliance in the list view.
Storage
The EPICS Archiver Appliance supports multiple stages for storing data and allows for configuration of data stores on a per PV basis.
The storage consumed across all PVs can be viewed in the Storage page.
Appliances
The EPICS Archiver Appliance maintains a days worth of CPU/heap usage statistics on a per appliance basis.
These statistics can be viewed on the Appliances page.
Integrations
The EPICS Archiver Appliance supports limited integration with existing Channel Archiver installations.
To import ChannelArchiver XML configuration files, click on the Choose Files button, select any number of ChannelArchiver XML configuration files and press Upload.
To proxy data from existing ChannelArchiver XML-RPC data servers, add the URL to the XML-RPC data server using the Add button.
The EPICS Archiver Appliance uses the archiver.names method to determine the PVs that are hosted by the ChannelArchiver XML-RPC data server and adds this server as a data source for these PVs.
