VOEventDB and Sustainable Software

Tim Staley / 4pisky.org

Hotwiring the Transient Universe V
Villanova, PA, Oct 2016

Intro

This talk covers our recently published archiving and query tool VOEventDB, and also uses examples from the development of VOEventDB to touch on the much broader topic of sustainable software.
Sustainable software essentially means making software reusable.
It’s a set of goals to bear in mind - no project is perfect, and there’s a cost/reward trade-off - there’s little point writing extensive documentation for your 10 line shell script.

VOEventDB, in brief

Context

• VOEvent is a standardised format for astronomical transient alerts.
• NASA-GCN have been transmitting alerts in this format for over 2 years.
• Previously, there was no public archive for alerts in this format.
• VOEvent standard has always referred to a ‘registry’ of ‘repositories’ - clear gap to fill.

VOEventDB: Spec

• Store raw VOEvent XML, provide XML content at a persistent URL
• Store a common subset of VOEvent metadata in regular database schema
• Make queries based on this common subset
• Including spatial (cone-search) and citation-based queries
• ‘RESTful’ web-API
• Python client-library for remote-queries
• Store raw XML - same data as you’d receive in real-time
• The VOEvent Schema has some flexibility, but there’s a core subset we expect to see in all packets - use that for filtering and searching.

Additional goals for VOEventDB: reusable, decentralized

• Agnostic about inputs and outputs
• Easy for any team to set up their own local repository

Implementation

• Postgres + SQLAlchemy
• Spatial queries powered by qc3 Postgres extension.
• Flask-powered RESTful interface
• Partially-autogenerated documentation.
• Extensive test-suite using pytest fixtures.

Extensive examples can be found at http://voeventdbremote.readthedocs.io/.

VOEventDB

• Provides a ‘turn-key’ queryable repository for transient alerts.
• Can be used as a remote service
• Or run your own
• Overview paper: arXiv:1606.03735

Packaging

• Make use of your packaging ecosystem
• Think about use of your code as a component
• Keep versioning information in your version control system! - automate package versioning

Documentation

• Minimum: description + example usage + install requirements
• Documentation goes stale - test your examples
• In Python, notebooks are a great format for this - try nbsphinx!

Deployment

• Docs are a start, but easily go out of date
• Automate!

Links:

• Slides: http://tiny.cc/voeventdb-talk
• 4 Pi Sky VOEvents web page: https://4pisky.org/voevents/
• 4 Pi Sky VOEvents overview paper: arXiv:1606.03735](https://arxiv.org/abs/1606.03735
• VOEventDB client-docs and tutorial: http://voeventdbremote.rtfd.io/
• VOEventDB server-docs: http://voeventdb.rtfd.io/
• NBSphinx: https://nbsphinx.readthedocs.io/
• Versioneer: https://github.com/warner/python-versioneer