habitat.uploader

Python interface to document insertion into CouchDB.

The uploader is a client for end users that pushes documents into a CouchDB database where they can be used directly by the web client or picked up by a daemon for further processing.

Classes

Extractor()	A base class for an Extractor.
ExtractorManager(uploader)	Manage one or more Extractor objects, and handle their logging.
UKHASExtractor()
Uploader(callsign[, couch_uri, couch_db, ...])	An easy interface to insert documents into a habitat CouchDB.
UploaderThread()	An easy wrapper around Uploader to make a non blocking Uploader

Exceptions

CollisionError
UnmergeableError	Couldn’t merge a payload_telemetry CouchDB conflict after many tries.

exception habitat.uploader.UnmergeableError

Couldn’t merge a payload_telemetry CouchDB conflict after many tries.

class habitat.uploader.Uploader(callsign, couch_uri='http://habitat.habhub.org/', couch_db='habitat', max_merge_attempts=20)

An easy interface to insert documents into a habitat CouchDB.

This class is intended for use by a listener.

After having created an Uploader object, call payload_telemetry(), listener_telemetry() or listener_information() in any order. It is however recommended that listener_information() and listener_telemetry() are called once before any other uploads.

flights() returns a list of current flight documents.

Each method that causes an upload accepts an optional kwarg, time_created, which should be the unix timestamp of when the doc was created, if it is different from the default ‘now’. It will add time_uploaded, and turn both times into RFC3339 strings using the local offset.

See the CouchDB schema for more information, both on validation/restrictions and data formats.

listener_telemetry(data, time_created=None)

Upload a listener_telemetry doc. The doc_id is returned

A listener_telemetry doc contains information about the listener’s current location, be it a rough stationary location or a constant feed of GPS points. In the former case, you may only need to call this function once, at startup. In the latter, you might want to call it constantly.

The format of the document produced is described elsewhere; the actual document will be constructed by Uploader. data must be a dict and should typically look something like this:

data = {
    "time": "12:40:12",
    "latitude": -35.11,
    "longitude": 137.567,
    "altitude": 12
}

time is the GPS time for this point, latitude and longitude are in decimal degrees, and altitude is in metres.

latitude and longitude are mandatory.

Validation will be performed by the CouchDB server. data must not contain the key callsign as that is added by Uploader.

listener_information(data, time_created=None)

Upload a listener_information doc. The doc_id is returned

A listener_information document contains static human readable information about a listener.

The format of the document produced is described elsewhere (TODO?); the actual document will be constructed by Uploader. data must be a dict and should typically look something like this:

data = {
    "name": "Adam Greig",
    "location": "Cambridge, UK",
    "radio": "ICOM IC-7000",
    "antenna": "9el 434MHz Yagi"
}

data must not contain the key callsign as that is added by Uploader.

payload_telemetry(string, metadata=None, time_created=None)

Create or add to the payload_telemetry document for string.

This function attempts to create a new payload_telemetry document for the provided string (a new document, with one receiver: you). If the document already exists in the database it instead downloads it, adds you to the list of receivers, and reuploads.

metadata can contain extra information about your receipt of string. Nothing has been standardised yet (TODO), but here’s an example of what you might be able to do in the future:

metadata = {
    "frequency": 434075000,
    "signal_strength": 5
}

metadata must not contain the keys time_created, time_uploaded, latest_listener_information or latest_listener_telemetry. These are added by Uploader.

flights()

Return a list of flight documents.

Finished flights are not included; so the returned list contains active and not yet started flights (i.e., now <= flight.end).

Only approved flights are included.

Flights are sorted by end time.

Active is (flight.start <= now <= flight.end), i.e., within the launch window.

The key _payload_docs is added to each flight document and is populated with the documents listed in the payloads array, provided they exist. If they don’t, that _id will be skipped.

payloads()

Returns a list of all payload_configuration docs ever.

Sorted by name, then time created.

class habitat.uploader.UploaderThread

An easy wrapper around Uploader to make a non blocking Uploader

After creating an UploaderThread object, call start() to create a thread. Then, call settings() to initialise the underlying Uploader. You may then call any of the 4 action methods from Uploader with exactly the same arguments. Note however, that they do not return anything (see below for flights() returning).

Several methods may be overridden in the UploaderThread. They are:

• log()
• warning()
• saved_id()
• initialised()
• reset_done()
• caught_exception()
• got_flights()
• got_payloads()

Please note that these must all be thread safe.

If initialisation fails (bad arguments or similar), a warning will be emitted but the UploaderThread will continue to exist. Further calls will just emit warnings and do nothing until a successful settings() call is made.

The reset() method destroys the underlying Uploader. Calls will emit warnings in the same fashion as a failed initialisation.

start()

Start the background UploaderThread

join()

Asks the background thread to exit, and then blocks until it has

settings(*args, **kwargs)

See Uploader‘s initialiser

reset()

Destroys the Uploader object, disabling uploads.

payload_telemetry(*args, **kwargs)

See Uploader.payload_telemetry()

listener_telemetry(*args, **kwargs)

See Uploader.listener_telemetry()

listener_information(*args, **kwargs)

See Uploader.listener_information()

flights()

See Uploader.flights().

Flight data is passed to got_flights().

payloads()

See Uploader.payloads().

Flight data is passed to got_payloads().

debug(msg)

Log a debug message

log(msg)

Log a generic string message

warning(msg)

Alike log, but more important

saved_id(doc_type, doc_id)

Called when a document is succesfully saved to couch

initialised()

Called immiediately after successful Uploader initialisation

reset_done()

Called immediately after resetting the Uploader object

caught_exception()

Called when the Uploader throws an exception

got_flights(flights)

Called after a successful flights download, with the data.

Downloads are initiated by calling flights()

got_payloads(payloads)

Called after a successful payloads download, with the data.

Downloads are initiated by calling payloads()

class habitat.uploader.ExtractorManager(uploader)

Manage one or more Extractor objects, and handle their logging.

The extractor manager maintains a list of Extractor objects. Any push() or skipped() calls are passed directly to each added Extractor in turn. If any Extractor produces logging output, or parsed data, it is returned to the status() and data() methods, which the user should override.

The ExtractorManager also handles thread safety for all Extractors (i.e., it holds a lock while pushing data to each extractor). Your status() and data() methods should be thread safe if you want to call the ExtractorManager from more than one thread.

uploader: an Uploader or UploaderThread object

add(extractor)

Add the extractor object to the manager

push(b, **kwargs)

Push a received byte of data, b, to all extractors.

b must be of type str (i.e., ascii, not unicode) and of length 1.

Any kwargs are passed to extractors. The only useful kwarg at the moment is the boolean “baudot hack”.

baudot_hack is set to True when decoding baudot, which doesn’t support the ‘*’ character, as the UKHASExtractor needs to know to replace all ‘#’ characters with ‘*’s.

skipped(n)

Tell all extractors that approximately n undecodable bytes have passed

This advises extractors that some bytes couldn’t be decoded for whatever reason, but were transmitted. This can assist some fixed-size packet formats in recovering from errors if one byte is dropped, say, due to the start bit being flipped. It also causes Extractors to ‘give up’ after a certain amount of time has passed.

status(msg)

Logging method, called by Extractors when something happens

data(d)

Called by Extractors if they are able to parse extracted data

class habitat.uploader.Extractor

A base class for an Extractor.

An extractor is responsible for identifying telemetry in a stream of bytes, and extracting them as standalone strings. This may be by using start/end delimiters, or packet lengths, or whatever. Extracted strings are passed to Uploader.payload_telemetry() via the ExtractorManager.

An extractor may optionally attempt to parse the data it has extracted. This does not affect the upload of extracted data, and offical parsing is done by the habitat server, but may be useful to display in a GUI. It could even be a stripped down parser capable of only a subset of the full protocol, or able to parse the bare minimum only. If it succeeds, the result is passed to ExtractorManager.data().

push(b, **kwargs)

see ExtractorManager.push()

skipped(n)

see ExtractorManager.skipped()