[Proposal] Standardising messages with protocol buffers


#1

This proposal would help everyone follow a common data sharing standard for log generation and permission artefact verification. (and maybe other things in the future)

Problems with text based encoding (JSON, XML):

  1. Two similar JSON objects could be encoded in different ways with different hashes. This means they would generate different signatures. Something like: Test App to validate NPNT implementation
  2. Occupies too much space: JSON and XML data could be 5 to 10 times larger than protobufs. RPAS would generate tons of logs in the future which means more bandwidth to transmit this data.
  3. JSON format is not native to C and C++: JSON is amazing with JS or even Python but not with embedded systems. Parsing text could be very slow as compared to parsing binary.
  4. Difficult to update: Updating and versioning JSON schemas is very hard and one thing is sure that these message will be updated in the future.

Introducing Google’s Protocol Buffers:
Protocol buffers are Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML and JSON, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages. [credit: Google]
More on Protocol Buffers: https://developers.google.com/protocol-buffers

Benefit: Just invert all the problems I mentioned for JSON and XML.

  1. As protocol buffer support binary encoding, encodings are always deterministic and hence would always generate the same signature.
  2. Not just binary, it also support JSON and XML encoding if that’s needed. One can interoperate between binary, JSON and XML (and even other encodings as well)
  3. Language agnostic: Protobuf code is compiled in the native language and hence it could be used in any embedded system and same protocol would come handy for testing in python.

To give you an idea of how it looks, I have created a protocol for flight logs.

syntax = "proto3";

message Log {
    FlightLog flightLog = 1;
    bytes signature = 2;
}

message FlightLog {
    enum EntryType {
        UNKNOWN = 0;
        TAKEOFF_ARM = 1;
        GEOFENCE_BREACH = 2;
        TIME_BREACH = 3;
        LAND_DISARM = 4;
    }
    message LogEntry {
        EntryType entryType = 1;
        uint64 timestamp = 2;
        double longitude = 3;
        double latitude = 4;
        double altitude = 5;
        bytes crc = 6;
    }

    string permissionArtefact = 1;
    repeated LogEntry logEntries = 2;
    bytes previousLogHash = 3;
}

Sample JSON generated with the about protobuf

{
  "flightLog": {
    "permissionArtefact": "RYUsgYrOjpyyD3Qx3OjXgFAnSYl7GgxAWnkLf6T",
    "logEntries": [
      {
        "entryType": "TAKEOFF_ARM",
        "timestamp": "1579937423",
        "longitude": 77.231212,
        "latitude": 12.23232,
        "altitude": 1125.34232
      },
      {
        "entryType": "LAND_DISARM",
        "timestamp": "1579937423",
        "longitude": 77.261653,
        "latitude": 12.27253,
        "altitude": 0.34232
      }
    ]
  }
}

Protobuf for FlightLog and PermissionArtefact: https://github.com/pravahio/protocols/tree/rpas/protocols/rpas-india

I would be happy to release the bindings for python if someone is interested. We have been using them for our project.


#2

Sounds promising! Yes I would be interested. I also have a doubt. The artifact that we will receive from Digisky will be in JSON. When we download the Artifact , it will be in XML. Is this correct?


#3

Yes, you can do that but it’s not recommended because it’s an overkill (JSON -> binary -> XML)

Ideally, DigitalSky should use Protobufs to standardise messages and transmit binary data generated by protobufs. It should be left to the end user how they would like to use it (binary -> native class/structures, binary -> JSON or binary -> XML) [absolutely recommended]

If DigitalSky could not transmit binary for some reason, the platform should transmit JSON that follows a certain protobuf format (say FlightLog above) so that people on the other end could use protobuf libraries to understand the data.


#4

Made the code open. Check it out here: https://github.com/pravahio/py-rpas

How to use it:

from rpas.fl import FlightLog
from rpas.pa import RPASPermission

def main():

    pa = RPASPermission()

    # Generate random permission artefact
    permission_artefact = pa.request_permission()

    log = FlightLog(permission_artefact)

    # Update current location. FlightLog will log if
    # point is outside the boundary or if time limit expire.
    log.update_current_position(12.5463, 77.7635, 0.5443, FlightLog.State.Takeoff)
    log.update_current_position(12.6453, 77.7354, 5.2736) # Lat, Lon, Alt
    # ...
    log.update_current_position(12.9574, 77.2735, 0.0342, FlightLog.State.Land)
    
    # Get JSON representation of the logs
    json_logs = log.get_json_string()

    print(json_logs)

if '__main__' == __name__:
    main()

Suggestions are welcome. If someone wants to contribute to the code, just drop me a text. Python will be used for testing initially and later C and Java bindings would be build for flight controllers and GCS.