4. Extending PEPI

Generally, you’ll want to extend PEPI by creating new servers and adding support for new cameras. The client-side software should work with any server out-of-the-box, but there are still some features that need to be added to the client.

4.1. PEPI Theory

PEPI can be divided into the client-side and the server-side. As discussed, the client-side doesn’t really need to be extended–the server is where the interesting extensions can happen.

Servers can be divided into two components:

  • The actual server itself (the CameraServer)
  • Cameras connected to the server (the Camera)

We do not mandate that a CameraServer must take Camera objects, but it is strongly recommended unless you have a valid reason (e.g. very complicated hardware requirements). The provided Python implementations shows how this may be implemented.

4.1.1. Languages

PEPI is indifferent to which language you implement your server in, so long as it can be accessed over Apache Thrift. Thrift’s language bindings include:

  • C++
  • C#
  • Cocoa
  • D
  • Delphi
  • Erlang
  • Haskell
  • Java
  • OCaml
  • Perl
  • PHP
  • Python
  • Ruby
  • Smalltalk
  • ..plus others in the works or supported by third parties

Therefore, any of the above languages can be used to implement PEPI server-side components.

4.2. Writing New Servers

4.2.1. Interface Definition File

At the heart of PEPI is its interface definition file pepi.thrift. This defines the interface used to access servers and therefore specifies what functions you need to implement.

/*******************************************************************************
 * File: pepi.thrift
 * Author: Curtis West
 * -----------------------------
 * Interface definition file for Apache Thrift.
 ******************************************************************************/

/*******************************************************************************
Thrown when a requested image is not available on the server
*******************************************************************************/
exception ImageUnavailable {
    1: string message,
}

/*******************************************************************************
A Camera provides images from a physical camera in the form of RGB arrays.
*******************************************************************************/
service Camera {
  /* still
   * description: returns a still image capture from the camera at the currently
   *              set resolution
   * returns: multidimensional array of row, column, RGB representing the image
   */
   list<list<list<i16>>> still()

  /* low_res_still
   * description: gets a 640 x 480px still from this camera for previewing
   * returns: multidimensional array of row, column, RGB representing the image
   */
   list<list<list<i16>>> low_res_still()

  /* still
   * description: gets the maximum resolution supported by this camera
   * returns: a list of length 2 representing the resolution i.e. (x, y)
   */
   list<i16> get_max_resolution(),

  /* get_current_resolution
   * description: gets the current resolution of this camera
   * returns: a list of length 2 representing the resolution i.e. (x, y)
   */
   list<i16> get_current_resolution(),

  /* set_resolution
   * description: if supported, sets the resolution of the camera
   */
   oneway void set_resolution(1:i16 x, 2:i16 y)
}

/*******************************************************************************
A CameraServer serves as a wrapper around a camera and provides a number of
utility functions for managing the server and camera.
*******************************************************************************/
service CameraServer {
  /* ping
   * description: Used to ping the server.
   * returns: True, always
   */
  bool ping(),

  /* identify
   * description: Gets this server's unique identifier.
   * returns: String containing this server's identifier
   */
  string identify(),

  /* stream_urls
   * description: Gets a list of URL where the stream of this server's cameras
                  may be accessed, if they exist.
   * returns: List of strings containing URLs, or an empty list.
   */
  list<string> stream_urls(),

  /* ping
   * description: Shuts down this server. This does not need to be implemented,
   *              but the server must accept the function call.
   */
  oneway void shutdown(),

  /* start_capture
   * description: Captures a still from this server's camera(s) and stores in
                  internally under the given `data_code` for later retrieval.
   */
  oneway void start_capture(1:string data_code),

  /* retrieve_stills_png
   * description: Retrieves .png images that were captured using `start_capture`
                  under the specified `data_code` (if they exist), encoded as
                  PNGs.
   * throws: ImageUnavailable
   * returns: a list of strings, where each string contains one image encoded as
              a PNG file. Each string should be able to be dumped directly to
              the disk and still form a valid PNG file.
   */
  list<string> retrieve_stills_png(1:string with_data_code) throws(1:ImageUnavailable unavailable),

  /* retrieve_stills_jpg
   * description: Retrieves images that were captured using `start_capture`
                  under the specified `data_code` (if they exist), encoded as
                  JPEGs.
   * throws: ImageUnavailable
   * returns: a list of strings, where each string contains one image encoded as
              a JPEG file. Each string should be able to be dumped directly to
              the disk and still form a valid JPEG file.
   */
  list<string> retrieve_stills_jpg(1:string with_data_code) throws(1:ImageUnavailable unavailable),

  /* enumerate_methods
   * description: Returns a dictionary of the methods supported by this server.
                  This is currently not used for any function as of v3,
                  so you may choose to just return an empty dictionary,
                  but be aware that this may change in the future versions.
   * returns: A dictionary with:
               key: method name
               value: a list of argument names that the method takes
   */
  map<string, list<string>> enumerate_methods()
}

Depending on the language you choose to implement your new server/camera, the exact format of how you implement these functions will vary, but generally you’ll just write the functions exactly as listed (but in the syntax of your language).

From the perspective of writing a server implementation, there are no special requirements from Thrift; you don’t need to return Thrift types or use Thrift objects. Your server won’t even know its been called from Thrift (sometimes it won’t be). Instead, treat component implementations as handlers that are called when in response to a Thrift requests, with Thrift managing all the necessary type conversions and network transports.

4.2.2. Python’s BaseCameraServer

PEPI provides a minimal implementation of a CameraServer under the class BaseCameraServer.

In most cases, BaseCameraServer can be used without modification as long as you can provide the Camera you’d like to use. However, you may wish to override some methods to better suit your use case. For example, RaspPiCameraServer overrides the identifier() method to use the Raspberry Pi’s CPU serial number as the ID.

If your server is being implemented in another language, it is still beneficial to refer to this implementation to understand how certain operations are accomplished.

class BaseCameraServer(object):
    """
    BaseCameraServer is the minimal Python implementation of a CameraServer
    as defined in ``pepi.thrift``. CameraServers are used in with the
    Apache Thrift protocol to provide RPC mechanisms, allowing control
    of this server over RPC, if it is launched with Thrift.

    A CameraServer subclassing BaseCameraServer may override any of these
    methods to better reflect their use. However, care must be taken to
    ensure that the side effects of the subclass's methods do not affect
    other methods. For example, if you were to change the capture method
    to store images in a list for whatever reason, you would need to change
    the image retrieval methods.

    A CameraServer's use-case is to provide a server that controls a
    number of cameras to be controlled in a consistent manner. This allows
    for a client to seamlessly control all implementations of CameraServer's,
    over Thrift without needing to concern themselves with what cameras are
    attached, the procedure call names, etc.

    This BaseCameraServer implementation supports multiple connected cameras,
    that are transparent to the connecting client. When retrieving images, a
    list of encoded images are returned. The order of this list remains
    consistent across procedure calls.
    """

    StreamInfo = collections.namedtuple('StreamInfo', 'port, folder, streamer')
    STREAM_PORT = 6001

    def __init__(self, cameras, stream=True):
        # type: ([AbstractCamera], bool) -> None
        """
        Initialises the BaseCameraServer.

        :param cameras: a list of AbstractCamera objects
        :param stream: True to start streams for all cameras, False to not.
        """
        # self.cameras = CameraManager(cameras)
        self.cameras = cameras
        self._stored_captures = dict()
        self.streams = dict()
        self.identifier = str(uuid.uuid4().hex)

        if stream:
            StreamInfo = collections.namedtuple('StreamInfo', 'port, folder, streamer, capturer')
            for count, camera in enumerate(cameras):
                port_ = self.STREAM_PORT + count
                folder_ = tempfile.mkdtemp()
                streamer_ = MJPGStreamer(folder_, port=port_)
                capturer = CameraTimelapser(camera=camera, folder=folder_, interval=0.5)
                capturer.start()
                self.streams[camera] = StreamInfo(port=port_, folder=folder_, streamer=streamer_, capturer=capturer)

        def cleanup():  # pragma: no cover
            """
            Cleans up after this server by destroying connected cameras
            and their streams, and erasing the stored images.
            """
            logging.info('Cleaning up RaspPiCameraServer')
            self._stored_captures = None
            self.cameras = None
            self.streams = None
            logging.info('Cleanup complete for RaspPiCameraServer')
        atexit.register(cleanup)

    def ping(self):
        # type: () -> bool
        """
        Ping the server to check if it is active and responding.

        :return: True (always)
        """
        logging.info('ping()')
        return True

    def identify(self):
        # type: () -> str
        """
        Get the unique identifier of this server.

        :return: the server's unique identifier string
        """
        logging.info('identify()')
        return self.identifier

    @staticmethod
    def _current_ip():
        # type: () -> str
        return IPTools.current_ips()[0]

    def stream_urls(self):
        # type: () ->  [str]
        """
        Get the a list of URLs where the MJPG image stream of each camera
        connected to this server may be accessed.

        The order of the returned images is consistent, e.g. Camera #1, #2
        .., #x returned in that order.

        :return: a list of the stream URLs as a string
        """
        logging.info('stream_urls()')
        out_urls = []
        for _, stream_info in viewitems(self.streams):
            out_urls.append('http://{}:{}/stream.mjpeg'.format(self._current_ip(), stream_info.port))
        return out_urls

    def shutdown(self):
        # type: () -> None
        """
        Shutdown the server (i.e. power-off).

        Subclasses may choose to
        ignore calls to this function, in which case they should override
        this function to do nothing.

        :return: None
        """
        logging.info('shutdown()')
        os.system('shutdown now')

    def start_capture(self, data_code):
        # type: (str) -> None
        """
        Immediately starts the process of capturing from this server's Camera(s),
        and stores the captured data under the given unique data_code.

        Note: the received `data_code` is assumed to be unique. Subclasses may
        choose to implement better isolation methods, but this is not
        guaranteed nor required.

        :param data_code: the requested data_code to store the capture under
        :return: None
        """
        logging.info('start_capture(data_code: {})'.format(data_code))
        captures = []
        # TODO: parallelize capture from all cameras
        for camera in self.cameras:
            try:
                captures.append(Image.fromarray(np.array(camera.still(), dtype=np.uint8)))
            except (AttributeError, TypeError, ValueError) as e:
                logging.warn('Could not construct image from received RGB array: {}'.format(e))
                continue
        if captures:
            self._stored_captures[data_code] = captures
        logging.info('Stored_captured after start_capture(): {}'.format(self._stored_captures.keys()))

    def _retrieve_and_encode_from_stored_captures(self, data_code, encoding, quality):
        # type: (str, str, int) -> [str]
        try:
            image_list = self._stored_captures.pop(data_code)
        except KeyError:
            raise ImageUnavailable('No images are stored under the data_code "{}"'.format(data_code))
        else:
            out_strings = []
            for image in image_list:
                image_buffer = BytesIO()
                image.save(image_buffer, encoding, quality=quality)
                out_strings.append(image_buffer.getvalue())
            return out_strings

    def retrieve_stills_png(self, with_data_code):
        # type: (str) -> [str]
        """
        Retrieves the images stored under `with_data_code`, if they exist, and
        encodes them into a .png str (i.e. bytes).

        The order of the returned images is consistent, e.g. Camera #1, #2
        .., #x returned in that order.

        :param with_data_code: the data_code from which the image will be retrieved
        :raises: ImageUnavailable: when image requested with an invalid/unknown data_code
        :return: a list of strings with each string containing an encoded as a .png
        """
        logging.info('retrieve_stills_png(with_data_code: {})'.format(with_data_code))
        return self._retrieve_and_encode_from_stored_captures(with_data_code, 'PNG', quality=3)

    def retrieve_stills_jpg(self, with_data_code):
        # type: (str) -> [str]
        """
        Retrieves the images stored under `with_data_code`, if they exist, and
        encodes them into a .jpg str (i.e. bytes).

        The order of the returned images is consistent, e.g. Camera #1, #2
        .., #x returned in that order.

        :param with_data_code: the data_code from which the image will be retrieved
        :raises: ImageUnavailable: when image requested with an invalid/unknown data_code
        :return: a list of strings with each string containing an encoded as a .jpg
        """
        logging.info('retrieve_stills_jpg(with_data_code: {})'.format(with_data_code))
        return self._retrieve_and_encode_from_stored_captures(with_data_code, 'JPEG', quality=85)

    def enumerate_methods(self):
        # type: () -> [(str, str)]
        """
        Retrieves a map of the methods available on this server. This is useful
        for clients to verify the methods it can expect to be able to call
        if being called remotely.

        :return: dict of <method_name: [arguments]>
        """
        import inspect
        methods = inspect.getmembers(self, predicate=inspect.ismethod)
        output_dict = dict()
        for _tuple in methods:
            name, pointer = _tuple
            args = inspect.getargspec(pointer).args
            try:
                args.remove('self')
            except ValueError:  # pragma: no cover
                pass
            output_dict[name] = args

        return output_dict

4.3. Writing New Cameras

If you choose to use a Camera object with your server, then you should implement your camera according to the Camera interface.

In Python, AbstractCamera is provided as an abstract class with some implemented methods. It is an abstract class rather than a base class (like CameraServers) because it’s impossible to cater for all possible connected hardware.

The most important (and tricky) method in a Camera is it’s still() method that returns a multi-dimensional array of 0-255 RGB pixels (row, column, RGB). For example, MyConcreteCamera is implemented in Python and captures RGB images at a 4-by-3 pixel resolution:

>>> camera = MyConcreteCamera()
>>> image = camera.still()
>>> print(type(image))
<type 'numpy.ndarray'>
>>> print(image.shape)
(3, 4, 3)
>>> image
    array([[[244, 213,  53],  # Row 1, Col 1
            [141, 130, 195],  #        Col 2
            [229, 156,  94],  #        Col 3
            [204,  19, 191]], #        Col 4

           [[105, 202, 239],  # Row 2, Col 1
            [183, 109, 243],  #        Col 2
            [164, 190,   1],  #        Col 3
            [216, 191,  63]], #        Col 4

           [[160, 232, 240],  # Row 3, Col 1
            [ 86, 186, 252],  #        Col 2
            [ 19, 212, 221],  #        Col 3
            [253, 143,  29]]], dtype=uint8)  # Col 4

Most physical cameras don’t provide a RGB array. The easiest way to transform from JPG or PNG (preferred) files is to use a library such as Pillow (previously, PIL). In Python, we provide a utility class server.abstract_camera.RGBImage based on Pillow that can do some of these conversions for you.

from server import RGBImage
import numpy as np

class MyConcreteCamera(AbstractCamera):
   def __init__(self):
      self._camera = MyDSLRCamera()

   def still(self):
      png = self._camera.get_png()
      return np.array(RGBImage.fromstring(png))

Alternatively, if you wish to use Pillow directly:

from io import BytesIO

from PIL import Image
import numpy as np

class MyConcreteCamera(AbstractCamera):
   def __init__(self):
      self._camera = MyDSLRCamera()

   def still(self):
      png_buffer = BytesIO()
      png_buffer.write(self._camera.get_png())
      png_buffer.seek(0)
      image = Image.open(png_buffer)
      return np.array(Image.open(png_buffer))

4.3.1. Testing Your Camera Implementation

PEPI includes tests that you can run against your new camera implementation to see if it returns the correct values both natively in Python and over Apache Thrift. Note that this isn’t an exhaustive test of your camera implementation and how it handles errors etc., but instead just a test to check the correct values are returned.

To setup these tests against your server, you’ll need to define a few pytest fixtures that is used to “inject” your camera into the tests:

import pytest

from server.tests import AbstractCameraContract, AbstractCameraOverThrift
import MyConcreteCamera

class TestMyConcreteCamera(AbstractCameraContract):
   @pytest.fixture(scope="module")
   def camera(self):
      return MyConcreteCamera()

class TestDummyCameraOverThrift(AbstractCameraOverThrift):
 @pytest.fixture(scope="module")
 def local_camera(self):
     return MyConcreteCamera()

Refer to the Testing section for more details on testing in PEPI.

4.4. Raspi Server Implementation

A subclass of BaseCameraServer is provided with PEPI for use with Raspberry Pi’s, RaspPiCameraServer. This serves as a useful example on how to extend a language’s base implementation to customize functionality.

class RaspPiCameraServer(BaseCameraServer):
    """
    An implementation of a BaseCameraServer for a Raspberry Pi.
    """
    def __init__(self, cameras, stream=True):
        super(RaspPiCameraServer, self).__init__(cameras, stream)

        # Set identifier based on the RPi's CPU serial number
        try:
            with open('/proc/cpuinfo', 'r') as f:
                for line in f:
                    if line[0:6] == 'Serial':
                        self.identifier = line[10:26]
        except IOError:
            pass

As we have previously discussed, in most cases the procedures implemented in BaseCameraServer can be used for new servers. Simply subclass BaseCameraServer and your new server will inherit all of these implementations, which have been thoroughly tested and refined.

RaspPiCameraServer does exactly this: subclasses BaseCameraServer and overrides the identify() procedure to better suit its use case by using the CPU serial number to identify the server (to allow rapid deployment without manual setup on each server). This demonstrates just how easy extendnig servers are (at least in Python, but generally you’ll only need one base server to extend written for each language).