Iottly project technical insights

[two_third]

Contents

IOTTLY overall specifications Implementation details.
IOTTLY Brokers are server-side physical implementations of communication protocols.
IOTTLY DATA MODEL Model describes the fundamental entities involved in the design and management of IoT projects, and the relationships among such entities.
IOTTLY Reliable Message Queue All the building blocks of the IOTTLY Distribution will communicate using IOTTLY’s Reliable Message Queue.
IOTTLY Polyglot implements the interaction between IOTTLY Core and each IOTTLY broker.
IOTTLY History storage is the physical database where incoming messages from the devices are stored.
IOTTLY Trust is the integrated security layer responsible for authenticating and authorizing every single actor communicating within IOTTLY.
IOTTLY RESTful APIs are the way IOTTLY exposes its core functionalities to third-party applications and IOTTLY tools.
IOTTLY Message designer is an editor which helps project creators draft and annotate the messages which will be exchanged with the devices.
Device admin console provides real-time administrative control of the devices.
Config tools are the pages used to configure specific data for Organizations and Projects.
Ready-to-use control app IOTTLY provides a ready-to-use web Control App, available on both desktop and mobile devices.
Custom logic flows provide server side programming and execution of event driven logics

[/two_third]

[two_third]

Iottly overall specifications

[/two_third]
[one_third_last]

[/one_third_last]

[two_third]

The first release of IOTTLY will run on top of Linux

other operating systems will be supported in future releases

IOTTLY source code will be hosted on Github

IOTTLY overall architecture will follow the pattern of micro-services

each component will run as an independent service
services will communicate through IP protocols depending on the specific needs
docker images for each service and a docker-composer script for the overall distro will be managed

IOTTLY messages format will be JSON, including:

messages exchanged with devices (the JSON structure will be free and customizable)
internal messages
messages exchanged with third parties applications

Further implementation details will be specified in the following for each architectural component.

[/two_third]

[one_third_last]
[/one_third_last]

[two_third]

Iottly Brokers

[/two_third]
[one_third_last]

[/one_third_last]

[two_third]

IOTTLY Brokers are server side physical implementations of communication protocols.

IOTTLY will run multiple broker instances at the same time.

This will allow to support different protocols through the implementation of different brokers
It will also allow the community to contribute different implementations of the same protocol or new and different protocols

IOTTLY will use docker to run each broker instance in a sandboxed environment

different organizations or projects will have separated broker instances for scalability and security purposes
IOTTLY will take care of automating all the required docker management tasks

The first release of IOTTLY will ship MQTT and HTTP support

MQTT is a standard, ligthweight, bi-directional protocol which implements a powerful publish-subscribe paradigm
HTTP has to be regarded as a “fallback” protocol to support legacy devices; in this case commands will be implemented by means of polling

[/two_third]

[one_third_last]


implementation details
IOTTLY MQTT support will be based on Mosquitto 
Brokers sandbox environment will be based on 
Broker automation tasks will be developed on top of docker-py

[/one_third_last]

[two_third]

Iottly Data model

[/two_third]
[one_third_last]

[/one_third_last]

[two_third]

IOTTLY Brokers are server side physical implementations of communication protocols.

The IOTTLY Data Model describes the fundamental entities involved in the design and management of IoT projects, and the relationships among such entities.

Project creators will work in IOTTLY either privately or within the scope of an organization

The Project entity will support creators in taking care of the overall technical details:

the communication protocol with devices
the kind and properties of devices to be used
the kind of messages to be exchanged with devices and third parties applications

Hence devices belong to a project which in turn belongs to an organization (or to a private project creator)

devices can be declared as private or public with respect to the scope of the organization
public devices are accessible from any organization with different degree of controls (read and/or write)
furthermore, private devices can be shared among different organizations

The Application entity takes care of the interaction relationships of devices with third parties applications.

[/two_third]

[one_third_last]


implementation details
The physical RDBMS of choice will be 


Database access APIs will be developed with

[/one_third_last]

[two_third]

IOTTLY Reliable Message Queue

[/two_third]
[one_third_last]

[/one_third_last]

[two_third]

All the building blocks of the IOTTLY Distribution will communicate using IOTTLY’s Reliable Message Queue.

This leads to asynchronous and scalable interaction among each IOTTLY component.

Data model relationships will be enforced on top of the IOTTLY Reliable Message Queue in order to implement secure communication policies.

Queue model design will target:

data streams to enable event-driven manipulation of devices data
job orders to enable execution of on-demand tasks on data

Reliability of the queue resides on the consumer ability to acknowledge the producer about the job being correctly processed

From this point of view, for example, Polyglot behaves like a consumer when it has to send a command to a device, and it behaves like a producer when it has to dispatch an event received from a device, for further processing
the History Storage behaves like a consumer when it has to store a value received from a device, while being a producer when it is required to retrieve some data by the RESTful APIs

[/two_third]

[one_third_last]


implementation details
The choice of which Message Queue to adopt is still under investigation.

Disque is among the main options.
The queue model will be designed and constrained by means of an API each producer / consumer will have to derive from.

[/one_third_last]

[two_third]

Iottly Polyglot

[/two_third]
[one_third_last]

[/one_third_last]

[two_third]

IOTTLY Polyglot implements the interaction between IOTTLY Core and each IOTTLY broker.

Each broker instance will have a Polyglot client counterpart running in IOTTLY Polyglot.

IOTTLY Polyglot will define the interface and the API which will be implemented by each Polyglot client.

Each Polyglot client will have to properly register on the Message Queue to exchange messages from/to the message queue to/from the broker.

Polyglot Clients will be responsible of mapping the producer-consumer internal paradigm of IOTTLY onto each of the implemented brokers

for brokers natively supporting publish-subscribe (like MQTT) this will be a pass-through mapping
for other brokers the relative client will map internal IOTTLY communication paradigm to the specific protocol

[/two_third]

[one_third_last]


implementation details
MQTT Polyglot Client will be developed on top of 
HTTP Polyglot Client will be developed in python 
Interaction with the Message Queue will be specified by means of an API definition and making use of a proper Message Queue client.

[/one_third_last]

[two_third]

Iottly History storage

[/two_third]
[one_third_last]

[/one_third_last]

[two_third]

History storage is the physical database where the incoming messages from devices are stored.

A partitioned history space will be reserved to each project in Iottly

projects space will be limited by quotas defined at the Organization / Project levels

Device messages will be stored in raw JSON format

Raw data will follow retention policies based on Organization / Project History quotas

Iottly History Storage will automatically extract raw time-series from the more complex incoming JSON structures

Raw time-series will be stored in flat, per project, per dimension tables
Raw time-series exposure to RESTful APIs will be bounded in time
future releases of Iottly will support time-series replication into off-line big-data databases, such as HBase

A set of standard time aggregations (mean, max/min, median) will be performed on raw time-series and stored on History storage for long-time reporting of data.

[/two_third]

[one_third_last]


implementation details
The physical RDBMS of choice for History storage will be 

Native support for JSON


JSON flattening to time-series will be performed in real-time within the History storage job responsible for persisting raw JSON messages.



Workers will be implemented to perform time aggregations asynchronously

[/one_third_last]

[two_third]

IOTTLY Trust

[/two_third]
[one_third_last]

[/one_third_last]

[two_third]

IOTTLY Trust is the integrated security layer responsible for authenticating and authorizing every single actor communicating within IOTTLY.

Authorization logics and communication policies are derived from the data model

It acts on devices

enforcing devices authentication (specific of the particular protocol)
managing the communication encryption
preventing fake devices to impersonate real ones
preventing devices data leakage

It acts on RESTful APIs

enforcing APIs calls to be authenticated and verifying in real-time the caller effective rights
enforcing the relationships between applications, users and devices, thus preventing the possibility of a user to take control over a device he is not authorized to

Authentication tokens are managed at the Organization / Project level, thus preventing by design data leakages between organizations

Reliable device provisioning

When provisioned a device is required to know (fabric settings):

the organization IOTTLY Broker end-point
the IOTTLY project global device token

The first time it connects the device will start the handshaking to receive its unique auth. token
This procedure makes it easy to preserve device security while not requiring to hard code the specific auth token for each device

[/two_third]

[one_third_last]


implementation details
MQTT authentication is SSL based (both client and server auth)



APIs authentication is implemented by means of http headers tokens

[/one_third_last]

[two_third]

IOTTLY RESTful APIs

[/two_third]
[one_third_last]

[/one_third_last]

[two_third]

RESTful APIs are the way IOTTLY exposes its core functionalities to third-party applications and IOTTLY tools.

History APIs

they provide access to IOTTLY History storage (raw JSON messages, time-series raw and aggregated data)

Configuration APIs

they enable the automation of IoT management tasks, providing read-write access to IOTTLY data model

Commands APIs

they provide full control of devices, allowing applications and users to send remote commands to them

Real-time Webhooks APIs

applications may register specific callback to receive real-time events from devices

[/two_third]

[one_third_last]


implementation details
RESTful HTTPS APIs



Payload is JSON

[/one_third_last]

[two_third]

Iottly Message designer

[/two_third]
[one_third_last]

[/one_third_last]

[two_third]

The Message designer is an editor which helps project creators draft and annotate the messages which will be exchanged with the devices.

Coherently with its open nature, IOTTLY will be payload agnostic with respect to devices messages.

defining a wide message semantic would result in a naive attempt, always leaving out some cases

The Message designer easily allows to annotate JSON tags, binding them with specific semantic.

Thus messages will be interpreted in order to

properly being routed through the APIs
properly being represented on Device admin console
properly build the GUI command controls both in the Device admin console and in the ready to use sample app.

[/two_third]

[one_third_last]


implementation details
A dynamic JSON parser will be developed in python. Each project will have its own parser.

Parser binding with message semantic will be persisted on database at project level


Semantically meaningful functions will be dynamically mapped to JSON tags.



Message designer will generate such functions in python snippet which will then be compiled into the specific project parser.

[/one_third_last]

[two_third]

Iottly Device admin console

[/two_third]
[one_third_last]

[/one_third_last]

[two_third]

The Device admin console provides real-time administrative control of the devices.

customized on payload semantic

near-real time charts of time-series device data

Over the air firmware upgrades

Geo-location of devices

[/two_third]

[one_third_last]


implementation details
The back-end APIs are History, Commands and Webhooks APIs.



Client side MVC is developed with 



Responsive GUI is developed with Bootstrap 



Time-series charts are developed with Bokeh 



real-time: web sockets

[/one_third_last]

[two_third]

Iottly Config tools

[/two_third]
[one_third_last]

[/one_third_last]

[two_third]

Config tools are the pages used to configure specific data for Organizations and Projects.

Organizations

list of users (project creators, technicians, end users)
list or projects
list of end user third parties applications

Projects

Project protocol

list of available firmwares

global project device token
list of devices

Devices

deviceid and token

Applications

RESTful APIs tokens
Applications webhooks

[/two_third]

[one_third_last]


implementation details
The back-end APIs is Configuration APIs.



Client side MVC is developed with 



Responsive GUI is developed with Bootstrap

[/one_third_last]

[two_third]

Iottly Ready-to-use control app

[/two_third]
[one_third_last]

[/one_third_last]

[two_third]

IOTTLY provides a ready-to-use web Control App, available on both desktop and mobile devices:

to support rapid prototyping
to make IoT instant projects

The Control App is customizable:

Just drag a device message onto the App designer
then select a control (switch, value selector, bar)
choose an icon to complete your IoT App

[/two_third]

[one_third_last]


implementation details
The back-end APIs are Commands and Webhooks.



Client side MVC is developed with 



Responsive GUI is developed with Bootstrap 



real-time: web sockets



Graphical controls are binded to each message whithin the IOTTLY Project.

[/one_third_last]

[two_third]

Iottly Custom logic flows

[/two_third]
[one_third_last]

[/one_third_last]

[two_third]

Custom logic flows provide server side programming and execution of event driven logics:

When cross device logic is needed, server side is the right place to put it
For example if you want to send an e-mail based on some device message content.

IOTTLY will integrate a customized fork of Node-RED:

since IOTTLY lets you to customize device messages at the project level, a node-red instance will be run for each project
your custom device messages will appear as input nodes in the inputs pane
simply dragging a specific message node will make the flow trigger when the specific message is received without the hassle of writing js code to parse the message
flows will be persisted on IOTTLY database on a per project basis

[/two_third]

[one_third_last]


implementation details
Custom logic flows will be implemented by integrating Node-RED  into IOTTLY.
Single Node-red instances (client and server) will be run separately for each IOTTLY project, on top of 

Node-red instances automation tasks will be developed on top of docker-py

[/one_third_last]