Go to the first, previous, next, last section, table of contents, index.

Vision System Software

Software Overview

This section provides an overview of the vision system software. This includes relevant behavior of the system software running on the vision system's processor board. Topics covered include software startup behavior, how to download new software, and how to interact with the vision system specific software.

Startup Behavior

When the 68332 processor board starts up, it goes through several stages. Each stage can be recognized by the information it sends over the serial port.

Bootstrap Startup

First the bootstrap starts up and sends the following message at 38400 baud.

BOOTSTRAP <version number>

This message will always be at 38400 baud, regardless of what baud you are running at. Therefore, if you are running at 9600 baud it will appear as some garbage characters. Do not be alarmed by this.

Kernel Startup

Next the Kernel (the operating system on the processor board) starts up, sets the baud rate to be what it was last running at (in the case of the Pioneer vision software, this will be 9600 baud, otherwise it will be 38400 baud), checksums itself, and loads custom microcode. It prints a message like the following:

Checksum OK
Restoring all backups unless RAM programs are newer
Microcode loaded from 17f28 to fff000, 800 bytes

Anarchboot Multitasking 68332 Kernel Version 0.986
Copyright Anne Wright 1993

Compiled May  6 1996 at 15:20:43

The actual boot message will change with different versions of the kernel, but it will generally be in a similar form to that shown above.

Check for Programs to Run on Reset

Next it checks whether there are any programs which should be run on reset. If it detects any, it will give you an opportunity to keep them from running (see section Inhibiting Reset Run of Programs).

Checking for reset run programs
Control-c to inhibit reset run...10...9...8...7...6...5...4...3...2...1...
Library Initialization

If you did not prevent programs from being run on reset, it will start to run them at this time. First, it will initialize any libraries it finds. In the case of the software for both the stand alone and Pioneer vision system software, it should run the library `vislib-xx_x' (where `xx_x' refers to the version number of the vision library).

The vision library will beep, read the color training and persistent parameters which had been saved to the serial EEPROM on the vision system (see section Training the Vision System to See a Target Color), then beep again.

Initializing library vislib-26_2
Cognachrome Vision System
Software version 26.2   [beep]
Channel 0: Restored 393 bytes
Channel 1: Restored 1035 bytes
Channel 2: Restored 6 bytes
Restored a total of 1434 bytes
Restored protocol string: "$    %04xF:%04xR:%04xC#" from EEPROM [beep]
Application Program Startup

Finally it runs any application programs that are loaded. In the case of the stand alone software, the application program should be `vision-xx_x' (where `xx_x' refers to the version number). In the case of the Pioneer software, this should be `pioneer-xx_x'.

Running program vision-26_2 as process 99 
vestaboot.986>

If this sequence does not proceed according as described above, determine the first step at which the actual behavior diverged.

Inhibiting Reset Run of Programs

When the board is reset, the kernel will normally initialize libraries and run application programs automatically (see section Startup Behavior). However, there is an opportunity to prevent it from performing this step. This is particularly useful if you are writing your own software and are putting the serial port into raw mode, or if you are using the Pioneer software and wish to prevent the vision system from entering Saphira interaction mode (which runs with the serial port in raw mode).

Do the following to inhibit programs from being run on reset:

There is only a small window of opportunity, as pressing Control-C before or after the dots will not work. If you succeed at pressing Control-C in the appropriate interval, it will say:

Reset running of programs inhibited

If nothing appears when you press reset, check the power and serial connections, and check that you are using the correct serial port (see section Troubleshooting). If you get reasonable boot messages (see section Startup Behavior), but it does not print the above message, most likely there are no programs loaded. See section Downloading Programs to the Vision System.

Interaction Modes

The behavior of the vision system is governed by the interaction mode it is in at a given time. The mode determines what processing it will perform, what information it will send over its serial port, and what commands it will respond to or ignore. The following table provides an overview of these modes. They are described in more detail in later sections.

Tracking Upload Mode

In Tracking Upload Mode, the vision system processes and uploads tracking data. The format of the tracking data is configurable by setting the Protocol String (see section Protocol String). It will continue to upload data until it is stopped (see section Exiting Tracking Upload Mode).

TPUVIS Prompt Mode
In TPUVIS Prompt Mode, the vision system interactively accepts commands which can affect configuration or perform diagnostics. You should use ARC to interact with the vision system in this mode. The interaction is performed in a separate window from the tracking uploads.
Saphira mode
This mode is only available if you are using the Pioneer software. In Saphira mode, the vision system runs at 9600 baud and only responds to PSOS packets, which is the data format used by the 6811 board on the Pioneer and the Saphira robot control program (see section Saphira Packet Protocol). In this mode, the vision system will act as a bridge between Saphira and the robot processor, handle PSOS packets from the host directed to the vision system, and upload vision data to the host formatted as PSOS packets.

You must exit this mode in order to download programs or use regular ARC commands.

In all modes except Saphira Interaction Mode, the vision system interacts with the host over a multi-streamed protocol. This allows ARC to interact simultaneously with multiple processes on the board, and present each interaction as a separate window. It also allows ARC to communicate with the board to download programs, set the baud rate, etc. (see section Downloading Programs to the Vision System).

Downloading Programs to the Vision System

In any mode except Saphira mode, you can download programs using the regular download commands. If you are in Saphira mode, exit first before downloading (see section Exiting Vision System Saphira Interaction Mode). This section describes the minimal downloading instructions, which is all you should need. See the ARC User's Guide for more details. It is available at:
`http://www.newtonlabs.com/arc/arc_1.html'

The vision code is distributed in two formats:

S19 Files

Installation and upgrade packages for the vision system software contain the operating system, libraries, and application code (either stand alone or Pioneer). These are distributed as Motorola S-Record files (.s19 suffix) containing multiple executables.

Executable Files

These files are in a.out executable format, which is output by the compiler. If you are writing your own programs using the full ARC Development system, you will mainly be downloading files in this format.

WARNING: Downloading either kind of package to non-volatile memory only works for processor boards which have been specially modified to use EEPROM. All processor boards shipped with vision systems sold after March 31, 1996, are modified to use EEPROM. See section Processor Board Jumper Configuration, for more information.

Downloading S19 Packages

To download these packages to EEPROM, start ARC and use the kernelload command.

arc> kernelload packagename.s19

ARC will prompt you to turn off the board and hit RETURN. After you hit RETURN, turn the system back on. Downloading will begin automatically.

It is normal for ARC to pause while it reads the file, so be patient. However, if it does not start downloading after about 30 seconds, there is probably a problem. Double check that serial interaction works in general (see section Serial Port Troubleshooting).

These .s19 distribution files contain the bootstrap. This is necessary to allow the same file to work whether downloaded by ARC or a device programmer. As a result, when downloading using ARC, writes to the first 16K of EEPROM will print a message that they are being skipped so as not to overwrite the current bootstrap. This is normal, and is nothing to worry about.

The download will fill the entire EEPROM (to address 0x40000), so it takes a while.

When the download completes, press the vision system reset button. For vision systems installed in Pioneer robots, do not confuse the reset for the vision system and for the robot itself. The vision system reset button is the one labeled Boot Button on the vision interface panel(see section Pioneer Vision Interface Panel Diagram).

Downloading Executables

Executables output by the GCC cross compiler are in a.out format. If you are writing your own programs using the full ARC Development System, you will mainly be downloading your programs in this format.

To download these files:

Tracking Upload Mode

In Tracking Upload Mode, the vision system will upload tracking data over stream 28 on serial port A. The format of the tracking data is configurable by setting the Protocol String (see section Protocol String). It will continue to upload data until it is stopped (see section Exiting Tracking Upload Mode).

Entering Tracking Upload Mode

With Stand Alone Mode software installed
Press the RESET button.
With Pioneer software installed
Press RESET, exit Saphira Interaction Mode, press `r'.
General
From TPUVIS prompt, press `r'.

Exiting Tracking Upload Mode

Once in Tracking Upload Mode, do one of the following to exit:

TPUVIS Prompt Mode

Entering TPUVIS Prompt Mode

With Stand Alone Mode software installed
Do one of the following:
With Pioneer software installed
Press RESET, exit Saphira Interaction Mode.

Exiting TPUVIS Prompt Mode

Most other modes can be entered from TPUVIS Prompt Mode by pressing a key. See the instructions for entering the other modes for instructions.

With either the stand alone or Pioneer software, if you want to quit the program from the TPUVIS Prompt, press capital `Q'. If you do this, the program will no longer be running, and you will have to reset or type run filename at the arc> prompt to run it again.

TPUVIS Command Summary

From the `tpuvis>' prompt, you can retrain the three tracking channels and change the parameters used during tracking, as well as other miscellaneous operations. To execute a command, simply type its letter (don't press return). Case is important: some commands are lowercase, while some are uppercase.

Turning Tracking On and Off Summary

`r'
(remote control: `play') Start tracking.
`q'
(remote control: `stop') Stop tracking (and give `tpuvis>' prompt).

Training and Configuring Channels Summary

`a'
Set current tracking channel (`a', `b', or `c').
`s'
(remote control: `0') Clear current channel, and train on object in center of screen.
`j'
(remote control: `enter') Add to training of current channel, based on currently tracked objects.
`>'
(remote control: `channel up') Grow recognition region for current channel.
`<'
(remote control: `channel down') Shrink recognition region for current channel.
`S'
(remote control: `record') Save trained colors, protocol string, and persistents to EEPROM.
`R'
(remote control: `rewind') Restore trained colors and protocol string from EEPROM.
`f'
Configure signal filter.
`p'
Change computed parameters and serial transmission format.
`n'
Change min/max number of tracked objects per channel.
`c'
Go to advanced "colorplot" menu.
`A'
Adjust NTSC->RGB converter parameters.

Debugging and Miscellaneous Summary

`?'
List available commands.
`v'
Print version information.
`P'
Change persistent variable.
`O'
Track for one frame (for testing protocol string).
`m'
Benchmark frame acquisition and latency.
`l'
Track largest object in current channel, with human readable diameter, row, and column output.
`o'
Track largest object in current channel, with human readable diameter, row, column, and orientation output.
`+'
(remote control: `volume up') Turn crosshair display on.
`-'
(remote control: `volume down') Turn crosshair display off.
`b'
Show, in ARC/X windows graphics mode, the segments seen by the color thresholding circuitry for the current channel before processing.
`d'
Dump, in text, the segments seen by the color thresholding circuitry for the current channel before processing.
`T'
Configure the behavior of the system on startup.
`Q'
Kill the vision process.

TPUVIS Command Explanations

This section describes the individual `tpuvis>' commands in detail.

Turning Tracking On and Off Detail

`r'
(remote control: `play') Run color tracking. The amount of processing and format of output is governed by the protocol string. See section Protocol String.
`q'
(remote control: `stop') Stop tracking if running. In general, if something is running and you want the prompt back, you can type `q' to get it back.

Training and Configuring Channels Detail

`a'
This command changes the current channel--that is, the channel affected by the other training and configuration commands in this section. It takes a second one-character parameter specifying which is the new channel to use, `a', `b', or `c'. The channel most recently chosen is shown in square brackets in the prompt itself (when the prompt is `tpuvis [b]>', for example, it means channel `b' is the current channel).
`s' (lower case s)
(remote control: `0') Clear LUT and start with object in center. This is the most convenient way to change what color is being tracked on a given channel. Put a large object of the color you want to track in the center of the screen and type lower case `s'. This will clear out what used to be in the LUT and replace it with a region of the color cube containing the color it found in the center and around it in the region seen as belonging to the same object.

The border of the object is assumed to be where the distance in the color cube from the color in the center to another color exceeds the persistent value diffthresh (which defaults to 5000). The scale of diffthresh is pretty arbitrary, but when it gets bigger it is more generous about how different a color can be from the center color and still be accepted as belonging to the same object, and when it gets smaller it gets more stingy.

`j'
(remote control: `enter') Add tracked color to the LUT. Use this, after getting a reasonable lock on the color you want by using lower case `s', in order to add the colors resulting from different distance, lighting, or angle of the target object. The `j' command uses the blob tracking capability to find where the target object is, then takes a frame grab and adds the color it finds to the LUT.
`<' or `>'
(remote control: `channel down' or `channel up') Shrink or grow region in LUT. This command takes the region of the LUT which is thresholded high and shrinks or expands it by the persistent amount color_grow along all edges. color_grow defaults to 5. Increasing this number increases the effect of `<' and `>', and decreasing it decreases the effect.
`S' (capital)
(remote control: `record') Save the current values in the LUT for thresholding all three channels, the protocol string, and the persistents to the EEPROM. It is important to remember to do this after you are satisfied with the effect of retraining the color or changing the protocol string and before the next reset, as on reset the values last stored to the EEPROM are restored.
`R'
(remote control: `rewind') Restore the old values of the thresholding LUT and protocol string from the EEPROM for all three channels. This is useful when you've tried to change the settings and you aren't happy with it.
`f'
Change the amount of filtering done. The amount of filtering is the same for each channel; it cannot be set separately for the channels. 0 is the least filtering (none), while 3 is the most filtering. Filtering is only done in the horizontal direction. Filtering is typically used to make up for things like a cluttered environment, where small objects in the background might be incorrectly detected.

In typical systems the filtering is usually set to 1. The primary problem with increasing filtering is that it increases the size of the smallest object detectable by the system. (Sometimes making targets that are wide but short can help meet a constraint of small target but large filtering). The primary advantage of increasing the filtering is that the processing time per frame is proportional to the number of transitions in the thresholded image. Increasing the filtering typically reduces the number of transitions by removing the spurious ones, which decreases the processing time per frame, which speeds up the entire system. Therefore, it is best to keep the filter level as high as you can, given the size of the smallest object you want to be able to track.

Be sure to remember the capital `S' command to save to EEPROM if you want the new filtering level to persist when you turn off the board.

`p'
Change a channel's protocol string. The amount of processing, which data are output, and the format of the output are all governed by the protocol string. See section Protocol String, for a full explanation of the protocol string's format, and what it does.

When you execute this command, you will be given a choice of which protocol string to change, or if you want to modify the header: type `h' to change the header, or `a', `b', or `c' to change a particular channel. Be sure to try tracking out afterwards to see if your change had its desired effect--protocol strings can be tricky. You can use the capital letter `O' command to track a single frame; this is very useful for protocol string testing.

Remember that commands capital `S' and `R' save and restore the protocol strings as well as the colors being tracked. If you change the protocol string and are happy with it, be sure to use `S' to save it to EEPROM.

`n'
Change the minimum and maximum number of tracked objects per channel; see section Protocol String, for more explanation of this feature.
`c'
Go to the colorplot submenu. The colorplot submenu provides advanced commands for training colors and analyzing color training. Hopefully you will not need to use these features, as we have tried to provide all necessary operations at the top level menu.
`A'
Adjust the NTSC to RGB converter on the Sony daughterboard. There are four customizable parameters, each of which ranges from 0 to 255 (with a default of 127). Normally the default works well.

The four parameters are sony_apl, which adjusts the auto picture level (auto gain control); sony_sharpness, which adjusts the sharpness; sony_saturation, which adjusts the saturation; and sony_hue, which adjusts the hue. See section Important Persistents for more information.

Debugging and Miscellaneous Detail

`?'
This prints a list of the available commands. The list is likely to be more up to date than this document, so you may want to refer to it even if you don't need your memory refreshed.
`v'
This prints the software version that's running on the board.
`P'
This lets you change a persistent (see section Persistent Variables). First, it prompts you for the persistent to set; you can type `?' to get a list of persistents, or just hit RET to abort the command. Then, it prints the old value of the persistent, and asks for the new value; pressing just RET leaves it unchanged.
`O'
This command prints one frame of tracking information in the current window. It prints the same information as the `r' command (see section Protocol String), but only for one frame.
`m'
Benchmark tracking and latency. This command performs exactly the same processing as would be done by using the `r' command, but instead of uploading the data over serial it only takes statistics and prints the accumulated statistics at the end. See section Tracking Performance, for more information and examples.
`l'
Track largest blob with human readable output. This function tracks the diameter and center of the largest blob and prints it in human readable format like this. (Note that since a large amount of data is printed per frame, you may not receive 60 Hz tracking using this command).
Frame   70: N=     27, cg= ( 58,201)
Frame   71: N=     27, cg= ( 58,202)
Frame   72: N=     25, cg= ( 57,201)
Frame   73: N=     27, cg= ( 59,201)
Frame   74: N=     25, cg= ( 56,201)
`o'
Track largest blob with orientation and human readable output. This command is just like `l' but it also does angle and aspect ratio calculations. (Note that since a large amount of data is printed per frame, you may not receive 60 Hz tracking using this command).
Frame   83: N=     27, cg= ( 60,201), angle=  60, aspect ratio =  16
Frame   85: N=     28, cg= ( 57,202), angle=  53, aspect ratio =  16
Frame   87: N=     27, cg= ( 59,201), angle=  56, aspect ratio =  14
Frame   89: N=     27, cg= ( 58,202), angle=  48, aspect ratio =  15
`+'
(remote control: `volume up') Enable the crosshairs. All the default tracking commands set the position of the crosshairs to be the center of the largest item they are tracking.

Crosshairs are intended to be used primarily as a debugging aid. Because the timer processor generates the crosshairs in software, tracking performance will degrade somewhat when the crosshairs are on, so we recommend leaving crosshairs off during normal operation.

Also, you may notice unusual streaks to the right of the object being tracked while crosshairs are on. This doesn't indicate a problem in tracking, but is rather an artifact of minimizing crosshair's impact on tracking performace (the timer processor gives higher priority to recording the incoming tracking data, so it may be sluggish turning off the crosshair signal if there are a lot of incoming data points to record).

`-'
(remote control: `volume down') Disable the crosshairs. If the crosshairs are showing, this will make them disappear.
`b'
Acquire raw data with debugging enabled. This command prints the data for each blob individually and outlines it in a Tektronix window. This is nice for low level debugging, but it requires that you are running ARC under X in order to work.
`d'
Dump raw data to serial port. This is the lowest level debugging available. It prints the row, start column, and end column (in that order) for all the segments captured by the hardware:
84: 94 -> 108
85: 88 -> 110
`T'
Configure the behavior of the system on startup. A menu is printed with three choices:
`Q'
Kill the vision process.

Saphira Interaction Mode

This section is only relevant if you are using the Pioneer software (see section Pioneer Quick Start Guide). The commands described in this section are not available if you are using the stand alone software (see section Stand Alone Quick Start Guide).

When the Pioneer software starts up, it automatically enters Saphira interaction mode (see section Startup Behavior). In this mode, the vision system runs at 9600 baud and communicates using PSOS packets--the data format used by the 6811 board on the Pioneer and the Saphira robot control program (see section Saphira Packet Protocol). In this mode, the vision system will act as a bridge between Saphira and the robot processor, handle PSOS packets from the host directed to the vision system, and upload vision data to the host formatted as PSOS packets. It will not exit this mode unless it receives a PSOS packet instructing it to. See section Exiting Vision System Saphira Interaction Mode.

The following block diagram shows the communications between the components of a vision-equipped Pioneer.

The vision system has two serial ports. In Saphira mode, one communicates with the host and the other communicates with the 6811-based Pioneer controller board. The host will typically be running the Saphira host program (the graphical robot-control environment for the Pioneer written by Kurt Konolige). You may choose instead to run ARC with the `-saphira' flag on the host (see section Starting ARC for use with the Pioneer Vision System).

PSOS packets from the host contain an address field which tells the vision system whether it should handle the packet itself, or just pass it along to the Pioneer controller. For example, commands for controlling the motors or reading the sonars are passed along to the Pioneer controller unmodified. On the other hand, commands for configuring the vision system are handled locally and are not passed on to the Pioneer controller at all.

This section discusses the commands which Saphira can send the vision system. In normal use, you should not need to issue these commands directly, as Saphira will do it for you. However, in some cases you may wish to issue these commands to the vision system directly without using Saphira. See section Interacting Directly with the Vision System using ARC, for information on setting up to communicate with the vision system in Saphira mode from ARC.

WARNING: Be sure that ARC and the Saphira host program are not both trying to talk to the robot at the same time. Exit one before starting the other.

Entering Vision System Saphira Interaction Mode

With Stand Alone Mode software installed
NOT AVAILABLE
With Pioneer software installed
When the Pioneer software starts up, it automatically enters Saphira interaction mode (see section Startup Behavior). If you have exited Saphira mode using ARC and are currently in TPUVIS mode, you can use the TPUVIS `9' command to reenter Saphira mode.

Exiting Vision System Saphira Interaction Mode

There are several ways to make the vision system exit Saphira interaction mode:

In the first case, the vision code will not run at all. In all of the other cases, the vision system will now be in TPUVIS Prompt Mode (see section TPUVIS Prompt Mode). In either case, regular ARC commands will work at this point (the vision system's host serial port will no longer be communicating using PSOS packets).

Saphira Mode Channel Commands

These commands perform operations on the tracking channels. Replace `%' with `a', `b', or `c' to specify which channel to affect. These behave the same as their counterpart commands at the TPUVIS prompt. See section Training and Configuring Channels Detail, for details. These commands are case-sensitive, so be sure to match the case listed here.

`s%'
Train on color in center of given channel.
`j%'
Incrementally add to training for given channel.
`<%'
Shrink accepted color set for given channel.
`>%'
Expand accepted color set for given channel.
`a%'
Display tracking output of given channel on Video Out port.

Saphira Mode Frame Commands

The frame commands perform a 64x48 pixel frame grab and upload information for display. Currently the frames uploaded by these commands are displayed using the program xv. In order to use these, be sure you have installed xv, and that it is in your path. The default settings are for an 8-bit display. For xv to display a frame with acceptable quality on an 8-bit display, it must construct an appropriate 8-bit colormap. xv creates a window with a private colormap, so move the mouse inside the window to see what the frame really looks like (with the mouse outside, you're likely to see some pretty strange artifacts).

Replace `#' with `0', `1', or `2' for compression amount. Currently, the only compression performed is by reducing the bits per pixel. In the future, other compression means (such as JPEG) may be incorporated.

`0'
No compression, highest quality.
`1'
2x compression. Moderate quality.
`2'
4x compression. Low quality.

Replace `%' with `a', `b', or `c' to specify which tracking channel the command applies to.

These commands are case-sensitive, so be sure to match the case listed here.

`G#'
Grab and display a color frame
`E#'
Grab a frame, run edge detection, and display the result
`T#%'
Grab a frame and display the part included by the color tracking for the given channel

Saphira Mode Visual Sonar commands

The vision system can analyze frame grabs to produce visual sonars data. This method calculates the distance to the nearest visible obstruction, just as sonars caculate the distance to the nearest object which reflects sound. Visual sonars can have high spatial resolution without the problems of cross-talk to which regular sonars are prone. They can also be more directional, and their speed does not depend on the maximum range as regular sonars do. They can recognize some types of obstacles that sonars miss, but can be fooled by things like changes in carpet color which would not fool regular sonars. Therefore both visual and normal sonars are useful.

The distance measurement is based on the assumption an obstacle touches the floor. In order to work, the camera must be angled towards the floor, and the floor must be a relatively solid color. First the image is processed with an edge detection filter (see section Saphira Mode Frame Commands). Then the system scans up in columns from the bottom of the image to find the vertical position in the frame of the lowest obstruction.

Obstructions are assumed to contrast with the floor. While scanning up from the bottom, the first horizontal line in the column with a large enough value in the edge-filtered image is assumed to belong to the nearest obstacle. You can adjust the sensitivity of the obstacle avoidance by setting the persistent variable edge_thresh to a value between 1 and 255. 100 is the default. Lower values increase sensitivity, and higher values decrease sensitivity. (Persistents are viewed and set with the `P' command, described in the next section).

These commands display the visual sonars data using the plotting program gnuplot. A statically-linked version of gnuplot is included with ARC for this purpose. Be sure that it is installed and in your path to use these commands.

These commands are case-sensitive, so be sure to match the case listed here.

`D'
Grab a frame and display visual sonars data for that frame.
`DB'
Start continuously updating the visual sonars display.
`DE'
Stop updating the visual sonars display.

Saphira Mode Persistent Commands

Persistent variables provide a method of preserving calibration variables even if the system is restarted. It is possible to view and modify any persistent variables, not just the ones particular to the Pioneer software, using the following Saphira interaction mode command. See section Persistent Variables, for more information on how to use these persistent variables. Commands are case-sensitive, so be sure to match the case listed here.

`Ppersist[=val]'
Query or set the value of a persistent variable.

For example, to check the sensitivity of the edge detection, send `Pedge_thresh' in a PSOS packet (see section Saphira Packet Protocol). It will respond with a packet containing `edge_thresh = 100'.

Say that you now want to set the value to 200. Send `Pedge_thresh=200' in a packet. It will respond with `edge_thresh = 200 (was 100)'.

Miscellaneous Saphira mode commands

`S' (capital), `R'
Save or restore current color settings and persistents to EEPROM. See section Setting and Storing Persistents.
`v'
Print software version info
`q'
Exit Saphira mode, enter TPUVIS prompt mode.

Saphira Packet Protocol

When the vision system is in Saphira mode, it is controlled using PSOS packets. This is the data format used by the 6811-based robot controller board on the Pioneer and the Saphira robot control program. Here is the exact format of the packets that the vision system accepts and generates while in Saphira mode (see section Saphira Interaction Mode).

We assume that you're already familiar with the PSOS packet protocol as described in Pioneer 1 Software Manual.

The basic operation of the vision system in Saphira mode is as follows: whenever the host sends a packet to the vision system, if the vision system does not recognize the packet type, the packet is forwarded to the 6811. Whenever the 6811 sends a packet to the vision system, the vision system forwards the packet to the host, and follows it with a VISIONpac status packet. In this way, the vision system is almost totally transparent between the host and the 6811 (except for some latency).

Host-initiated Saphira Packet format

The vision system understands two packet formats from the host: VISION_COM and VISIONIO_COM. The former is subsumed by the latter, and should not be used (it is retained only for backward compatibility).

VISIONIO_COM packets contain length-prefixed ASCII strings, in the format generated by the Saphira function sfRobotComStr. These strings are just the Saphira commands described above (see section Saphira Interaction Mode).

#define VISIONIO_COM 100
2 bytes
sync chars
1 byte
packet length
1 byte
VISIONIO_COM (= 100)
1 byte
sfARGSTR (= 0x2B)
1 byte
string length
N bytes
string
2 bytes
checksum

Each VISIONIO_COM command responds with a VISIONIOpac packet when it is done. Mostly these are short status messages (suitable for displaying to the user); they are subject to change. For now, the main purpose of the return messages is handshaking; you should get exactly one VISIONIOpac for each VISIONIO_COM you send. Note that the VISIONIOpac string is not NUL-terminated.

#define VISIONIOpac 0x80
2 bytes
sync chars
1 byte
packet length
1 byte
VISIONIOpac (= 0x80)
1 byte
string length
N bytes
string
2 bytes
checksum

Robot-initiated Saphira Packet format

Each time the vision system forwards a status packet from the 6811 to the host (at approximately 10 Hz), it also sends a VISIONpac. The mode of each of the three color channels is set by using persistent variables (see section Persistent Variables). Each color channel can either be placed in a line-tracking mode or one of two blob-following modes. A VISIONpac contains some status information about the current line-tracking settings, followed by a tracking data block for each of the channels.

The format of the tracking data block depends on the tracking mode of the channel. The tracking mode of the three channels are set using the persistents pioneer_a_mode, pioneer_b_mode, and pioneer_c_mode. Currently, the possible values for these persistents are BLOB_MODE=0, LINE_MODE=1, and BLOB_BB_MODE=2.

#define VISIONpac 0x50
2 bytes
sync chars
1 byte
packet length
1 byte
VISIONpac (= 0x50)
2 bytes
current value of line_bottom_row persistent
2 bytes
current value of line_num_slices persistent
2 bytes
current value of line_slice_size persistent
2 bytes
current value of line_min_mass persistent
channel A tracking data
channel B tracking data
channel C tracking data
2 bytes
checksum

BLOB_MODE Tracking data

In BLOB_MODE, the center of the largest blob in image is calculated and uploaded. If there is nothing of the tracked color in the field of view, the data consists of all zeroes.

1 byte
BLOB_MODE (= 0)
4 bytes
area of largest blob
1 byte
X-coordinate of the center of gravity of the largest blob
1 byte
Y-coordinate of the center of gravity of the largest blob

BLOB_BB_MODE Tracking data

In BLOB_MODE, the center and bounding box of the largest blob in image is calculated and uploaded. If there is nothing of the tracked color in the field of view, the data consists of all zeroes.

In BLOB_BB_MODE, the data consists of

1 byte
BLOB_BB_MODE (= 2)
4 bytes
area of largest blob
1 byte
X-coordinate of the center of gravity of the largest blob
1 byte
Y-coordinate of the center of gravity of the largest blob
1 byte
X-coordinate of left edge of largest blob's bounding box
1 byte
X-coordinate of right edge of largest blob's bounding box
1 byte
Y-coordinate of top edge of largest blob's bounding box
1 byte
Y-coordinate of bottom edge of largest blob's bounding box

LINE_MODE Tracking data

To understand the LINE_MODE data chunk, we need to understand the processing the vision system does for LINE_MODE.

In LINE_MODE, the vision system divides its field of view into horizontal chunks called slices. Each slice has line_slice_size rows. The bottom slice (slice 0) starts at line_bottom_row and extends to line_bottom_row+line_slice_size. Each succeeding slice starts line_slice_size lines higher. There are line_num_slices slices in all.

In each slice, the vision system checks how many pixels in the slice are of the currently-tracked color for that channel; if the number is greater than line_min_mass, then the vision system believes it sees the line in that slice.

1 byte
LINE_MODE (= 1)
2 bytes
the number of the nearest slice in which the line can be seen
2 bytes
the number of slices in which the line can be seen
1 byte
X-coordinate of the line in the nearest slice in which it can be seen

The idea is to give you an idea of what the line is doing both nearby and in the distance. You can tell what the line is doing nearby so that you can turn in the correct direction. The information about how far away you can see the line lets you slow down for curves.

Special Saphira Packet formats

In addition to the above packet types, some of the Saphira commands return special-purpose packets. The `G#' and `T#%' commands respond with a color frame upload (a series of FRAME24pac's), the `E#' command responds with a grayscale frame upload (a series of FRAME8pac's), the `D' command responds with a DISTpac, and the `DB' command puts the system into a mode where it sends DISTpac's continually (at somewhat less than 5 Hz).

Frame Upload Saphira Packet formats

The color and grayscale upload packets are very similar. Each frame upload splits the frame into several pieces, with some number of rows in each piece, and sends one piece in each packet. The packet consists of the following:

#define FRAME24pac 0xb0
#define FRAME8pac  0xc0
2 bytes
sync chars
1 byte
packet length
1 byte
packet type: FRAME24pac (= 0xB0) or FRAME8pac (= 0xC0)
1 byte
frame height (height of the whole frame, before splitting)
1 byte
frame width
1 byte
beginning row number of this piece
1 byte
number of rows in this piece
1 byte
compression type (see below)
1 byte
number of bytes of pixel data in this packet
pixel data
2 bytes
checksum

Note that frame uploads use a packet size that's slightly larger than the 200 byte maximum size mentioned in the PSOS packet documentation; this is necessary to allow an entire 64-pixel row to be uploaded in a single packet.

Currently, the compression schemes are very simple; they just throw away low-order bits. There are three compression types defined:

0
no compression
1
keep only the top 4 bits of each byte
2
keep only the top 2 bits of each byte

The remaining bits are packed somewhat oddly in a FRAME24pac. The pixel data in a FRAME24pac always consists of a byte of red, a byte of green, and a byte of blue, in that order. In compression mode 0 (no compression), these are just the 1-byte values from the board. In compression mode 1, however, each of these bytes contains the bits from two adjacent pixels, and in compression mode 2, each byte contains the bits from four adjacent pixels. (If the frame width is not a multiple of 2 for mode 1, or 4 for mode 2, the frame is effectively padded on the right with zeroes.) When data from multiple pixels is packed into a byte, the leftmost pixel goes to the most significant bits in the byte. The same packing scheme is used in a FRAME8pac; of course, there's no color interleaving in that case.

Visual Sonar Upload Saphira Packet format

Visual sonar data is sent as a DISTpac packet. It has the following format:

#define DISTpac 0xd0
2 bytes
sync chars
1 byte
packet length
1 byte
DISTpac (= 0xD0)
1 byte
number of bytes of data
distance data
2 bytes
checksum

Each byte of the distance data is the y coordinate of the nearest obstacle in that direction. Thus, 0 means the obstacle is far away, and 47 means that the obstacle is very close. As a special case, 255 means that no obstacle was found at all in that direction. Currently, the vision system sends 8 byte DISTpac's, but this might be made configurable; it would be a good idea to make your code check the number of bytes of data to make sure it's what you're expecting.


Go to the first, previous, next, last section, table of contents, index.