Core software (server)

RoboDIMM core software

The complete RoboDIMM software package consists of two executables, some configuration files and a number of external utilities.  One of the executables is the ‘Core System’ , and the other executable is the graphical user interface. The core system is in fact a server, where as the graphical user interface is the client.

Protocol

 

Within the network application layer, TCP/IP is used as the main data protocol. TCP/IP is very convenient for application interoperability, independent of platform, operating system and programming language. This means that one is able to control the RoboDIMM from anywhere on the Internet.

In theory, TCP/IP clients (read: user interfaces) can run on anything from a hand-held computer to a desktop machine. For now it’s used only to connect the user interface clients to the server, but it is easy to build a user interface on any other operating system.

The core system also accepts standard telnet-connections (but not at standard port 23), which are also based on TCP/IP.

In the next paragraphs we will discuss both the core system and clients in outline.

 

1.1              The core system (server)

 

The core system carries out all kinds of tasks, either automatically when in robotic mode, or on command in manual mode. Although calculating seeing estimates is its major goal, it is not the most complex task to perform. Most of the time the core system is interacting with the hardware peripherals and taking decisions based on their output.

The core system can be started on the command line by using a Linux-shell with proper rights, or it can automatically start by adding it to one of the Linux system startup files, for example ‘rc.local’ . The software stays active on the controlling terminal, showing status information and error messages, unless the operator intentionally uses a specific parameter on the command line to put the process in daemon-mode.

Once the process is running, it’ll wait for clients to connect via a TCP/IP socket, and it also performs a number of tasks by default:

1.    Synchronize the system clock (both CMOS hardware clock and Linux system clock) with a NTP server. Synchronising the time will ensure that all measurements, log entries and stored images will have a correct time stamp. If a permanent Internet-connection is available, a NTP server (configurable) will be used for this. If there’s no Internet access, a DCF-clock connected to the computer is used.

2.    If the software is started with the ‘watchdog-parameter’ then check whether old RoboDIMM instances are still running while showing no significant activity. If so, kill them, park the telescope in its home position and exit. The system might have crashed some time ago, and by parking the telescope we might prevent serious system damage to the mount. By regularly running the executable in combination with this ‘watchdog-parameter’ (for example in crontab), it is ensured that destructive run-away processes are virtually non-existent.

3.    Check whether the external database server (PostgreSQL) is still up and running. If not, start the service and continue.

4.    If the operator used the ‘robotic’ command line switch, the software will enter the ‘robotic mode’ by starting a separate thread within the software. This may also happen if the main configuration file contains parameters to force it into the robotic mode. Whenever the software has entered the robotic mode, it will begin to act completely autonomously, taking decisions on its own and without any intervention of an operator. For a more extensive description of this robotic mode, read paragraph 1.1.1 .

5.    If the system did not enter the ‘robotic mode’, then ‘manual mode’ is enabled. A thread is started in which the system will wait and process incoming requests of clients (through TCP/IP).  The system will react only in response to queries or commands from connected clients and does not take any actions on its own. For a more extensive description of the manual mode, see paragraph 1.1.2 .
Note: the loop in which incoming requests are processed will always be activated, regardless of the state of the core system (manual mode or robotic mode). Thus, incoming requests (e.g. commands that were typed manually) will always get processed, in robotic mode AND in manual mode. However, in robotic mode not all commands are accepted. For example, if the telescope is slewing of tracking in robotic mode, all manually given commands related to telescope movement are ignored.

1.1.1            Robotic mode

If the system entered ‘robotic mode’, the system will immediately try to begin its seeing measurements by starting a thread which repeatedly performs the following tasks:

1.    Read the set of all suitable target stars from the database. The database contains a collection of stars which are imported from the SAO database, using a limiting minimum and maximum magnitude (configurable).

2.    Calculate the position of the sun and check whether it is already dark enough to begin the measurements. If not, wait a minute and go to 2).

3.    It is dark. Use the most recent measurements from the Automatic Weather Station as well as the power supply sensors, intruder sensors and some other sensors to decide whether it is safe to open the dome and start the measurements. If it is not safe, park the telescope if necessary, close the dome if it was opened, and go to 2). For extra information about what’s safe and what’s not, see below.

4.    Open the dome if it is not already opened. Put the telescope in tracking mode.

5.    Select the most suitable target star to use in a new seeing measurement. Selection is based on the following criteria:

a.    Star must be within X degrees (configurable, 30 degrees or so) zenithal distance.

b.    System must be able to track the star for at least X minutes (configurable) without having to switch to another star due to low altitude.

c.    The distance between the target star and the moon or planets must be at least X degrees (configurable). Comets or other bright objects that are temporarily visible are ignored.

d.    From the stars that are left, the brightest star is chosen. However, if the magnitude in relation to the uptime*) of another star gets above a (configurable) limit, the star with longer uptime is chosen. For example, if 10% of the maximum uptime is equivalent to 0.1 magnitude (configurable), a star with magnitude 1.5 that can be tracked for 200 minutes gets priority above a star with magnitude 1.3 that can be tracked for 150 minutes.
*) the maximum uptime is the longest time that any target star could possibly be tracked with the telescope within the configured zenithal distance limit.

6.    If the selected target star turned out to be labeled as ‘temporarily excluded’ because the RoboDIMM could not find the star in a previous attempt (for example due to a partially clouded sky), the robot will select another target star to continue on.

7.    Move the telescope to the selected target star. If the telescope does not reach the intended position within X seconds (configurable), generate an error message (and mail it to the operator) and try another target. If that fails too, close the dome, park the telescope and switch to manual mode.

8.    Take a number (configurable) of full frame CCD exposures, and try to locate the target star in them. If no spots are found in any of the exposures, start a spiral scan:

a.    Try to find the star by creating a mosaic: move the telescope around in a spiral-like style. Assumed that the field of view of the CCD chip is X arc seconds, move the telescope with X*2/3 (configurable) arc second steps each time. After each step, take N exposures (configurable) to detect whether the target is already in view. Repeat this until the spiral size has exceeded a certain (configurable) limit or until at least X (configurable) spots are detected.

b.    If no star was found, go to step 5 and try to locate another target. It might be (partially) clouded, be the system will try X (configurable) other targets before giving up.

c.    If spots were found, center the spots within X (configurable) arc seconds of the CCD midpoint. If this fails, go to step 5.

d.    The target star is centered perfectly onto the CCD chip.

9.    Synchronize the CMOS hardware clock, the Linux system clock and the internal clock of the GTO1200 mount with a NTP server.

10.  Determine the difference between the RA/DEC coordinates reported by the telescope mount and the RA/DEC coordinates of the target object (= pointing error). Store that difference in the database so that it can be used to build up and train a telescope pointing model. Synchronize the Astro-Physics GTO1200 mount with the current object coordinates to adjust/realign the telescope’s position.

11.  Focus the camera. This can be done in one step, no iterative procedures are needed.

12.  Start one seeing measurement:

a.    Take a few full frame exposures and determine the average position of the spots.

b.    Use their positions to calculate the position and size of the CCD sub-frame. The size of that sub-frame is dependent of the distance between the spots and some configured minimum and maximum limits. The sub-frame is kept as small as possible to increase the download speed of the CCD exposures, therefore increasing the total number of seeing measurements per hour.

c.    Take a series of X exposures (configurable, about 200), downloading only the sub-frame that was determined.

d.    Extract the stars from all exposures, and calculate for each spot:

                                                              i.    Visual magnitude

                                                            ii.     Centroid

                                                          iii.      Peak pixel value

                                                          iv.      Peak pixel X and Y coordinates

                                                            v.      Sextractor Star class (see Sextractor manual)

                                                          vi.      Flux

                                                        vii.      FWHM

                                                      viii.      Elongation

                                                           ix.     Sextractor extraction flags (see Sextractor manual):

1.    "the object has neighbours, bright and close enough to significantly bias the MAG_AUTO photometry, or bad pixels"

2.    “the object was originally blended with another one"

3.    "at least one pixel of the object is saturated (or very close to)"

4.    "the object is truncated (too close to an image boundary)"

5.    "object's aperture data are incomplete or corrupted"

6.    "object's isophotal data are incomplete or corrupted"

7.    "a memory overflow occurred during deblending"

8.    "a memory overflow occurred during extraction"

e.    Filter out exposures that are deviating too much, which could cause inaccurate seeing measurements. All exposures/spots must meet the following criteria:

                                                              i.      The detected number of spots must be higher than X and lower than Y.

                                                            ii.      None of the extraction flags, reported by sextractor, must be enabled for all of the extracted spots. For example, if flag ‘4’ is reported (object is truncated), the spot may have moved to close to the CCD sub-frame boundary (due to improper telescope alignment, or high wind speeds, etc) , which would yield inaccurate centroids.

                                                          iii.      The elongation of each spot must not exceed X (configurable). The accuracy of centroid calculation will degrade as the shape of the spot gets more elongated  (e.g. due to wind vibrations causing star trails).

                                                          iv.      The star class, reported by sextractor, must lie between a minimum and maximum limit (configurable). This is used to filter out the effects of cosmic rays and other artifacts.

                                                            v.      The reported flux must be between a minimum X and a maximum Y.

                                                          vi.      The reported flux of each spot must not exceed the flux of the other spots within the same exposure, with a margin of X percent (configurable). 

                                                        vii.      The FWHM must not exceed X (configurable).

                                                      viii.      The distance between all spot pairs must be higher than X and lower than Y pixels.

                                                           ix.      The peak value for each spot must not exceed X, where X is calculated on basis of the CCD camera specifications and the telescope used (using the full well capacity and CCD gain).

f.     Reject those exposures not meeting the filter criteria.

g.    Count the total number of approved exposures that were left. If this number is below X, reject the complete seeing measurement and go to step 2.

h.    Store all calculated values and other statistics (size of the sub-frame, Alt/Az coordinates, centroids, peak values, flux etc) in the database. The software is able to recalculate all seeing FWHM’s on basis of the statistics stored in the database, even if the CCD exposures were not saved themselves.

i.      If configured so, store them (all or partially) in the database. This includes the rejected exposures, which enables the operator to trace back or debug the system. All exposures are stored in FITS format. Each FITS header contains as much information as possible, like an accurate timestamp, Alt/Az coordinates, absolute CCD sub-frame position, telescope and CCD properties, CCD plate scale, etc).

j.     Calculate the seeing FHWM and store it in the database, together  with a timestamp.

13.  If a successful seeing measurement was carried out, the focuser will be readjusted, using the statistics originating from the most recent seeing measurement.

14.  We reached the end of this loop, go to step 2.

On the background (only in robotic mode), all sensor output (AWS input, voltage sensors, etc) is continuously checked against a set of predefined limits. If any of these sensors does not meet the criteria, some kind of action is performed like closing the dome. 
Here is an abstract summary of criteria that the RoboDIMM software will use to decide whether it is safe to open the dome (or keep the dome open). All of the following cases are considered as unsafe and will cause a switch to “closed state” which is at least followed by logging the event, sending a warning signal to the operator, parking the telescope and closing the dome.:

1.    If the telescope is repeatedly unable to find its target stars, it will assume that something is wrong. The operator is warned (through an e-mail message and error messages in the log files) to take the appropriate actions. Possible causes:

a.    the telescope is severely misaligned, or

b.    one of the focusers has moved too far outwards or inwards, causing the stars/spots to become virtually undetectable on the CCD exposures. 

c.    it is (partially) clouded

2.    If rain is detected (reported by the AWS) the system will stop measurements. If the AWS has not reported rain for longer than X minutes, the system will reactivate itself again and restart the seeing measurements.

3.    When the wind speed, humidity level, dust level, air temperature, barometric pressure, solar meter or enclose temperatures exceed the configured minimum or maximum limits, the system will stop.

4.    When the software is detecting hardware problems, possibly indicating hardware damage, the system will stop. Some examples: rs232 time outs occurring on the telescope mount communication cable, CCD communication errors, wrong coordinates reported by the telescope mount  (too low, wrong syntax, etc).

5.    When the sensors are indicating that the power voltage or amperage is too high or low, the system will stop and the computer will shut down completely! The computer is automatically restarted when the voltage has reached normal levels.

6.    When the dome-sensors are indicating that the dome cannot be closed or opened (e.g. if the dome is covered by ice) the system will stop.

7.    When potential intruders are detected, the system will stop.

8.    When hard disk space is running out, multiple warnings are sent to the operator as the disk space is decreasing. If the hard disk is almost full, the system will send an error message and stop all activities. See section 2.6.2 in the System Specifications Document.

1.1.2            Manual mode / service state

In manual mode the software will not perform any action by itself. Only manually given commands (by the operator using telnet or the graphical user interface) will be interpreted and executed, but no security checks or measures are taken.

In manual mode the software can be used to service the system. Think of:

1.    Realigning the telescope. The software can assist with this.

2.    Parking the telescope

3.    Moving the telescope

4.    Read sensors and display their raw output data

5.    Database backup

6.    Test sub-functions and functionalities

7.    Debug the system.

8.    Manually focus the CCD camera

9.    Take CCD exposures in order to determine the plate scale,  noise levels, etc.

10.  Manually choose a target star to use for manually performed seeing measurements etc.

11.  Refill the database with a new set of possible target objects, imported from a SAO or BSC star catalogue.

12.  Manually extract centroids and other statistics from selected CCD exposures

13.  etc. etc.

In short, all major units/objects which are used by the robot to control the RoboDIMM system in order to do measurements (in ‘working state’) can also separately be called as commands when service mode is active.