Q3 - Progress Report
What did you do to reach this Milestone?
This quarter’s work has been primarily focused in the following areas: User Interface Initial Setup Wizard (for first-time configuration of nodes) Admin Panel (for administration of running nodes) Node Interoperability/’Grouping’ via VPN Drag and Drop Dataset Imports Continuous Integration & Automated Testing
Initial Setup Wizard
The initial setup wizard is a browser-accessible configuration page that is made available to board administrators after a board has been booted for the first time. It is accessed via a default WiFi SSID/password (for ARM deployments) or via a default primary network interface (for x86/cloud deployments). It provides the user with the following essential configuration options when the system is first booted: Fully Qualified Domain Name Admin username/password (Optional) Hotspot SSID/password
For both ARM and x86 deployments, the Admin Panel is accessible via the ‘admin’ option on the node’s landing page. These screens can only be accessed by an administrative user. The first admin user is created via the ‘Initial Setup Wizard’ described above. The panel provides the user with advanced configuration options for a running system:
Status - Gives an admin a view of the overall system status Network - Allows an admin to change advanced network options Services - Allows an admin to enable/disable bundled services Users - Allows an admin to add/remove/modify users on the system Discovery - Allows an admin to review/whitelist/blacklist discovered nodes VPN - Allows for creation/revocation of client keys and setup of VPN client via .ovpn file) Contact - Provides Liquid Investigations contact details for filing bug reports For an interactive demo of these UI components see demo video.
Node Interoperability/’Grouping’ via VPN
An important consideration for our Q3 milestone was the creation of interoperable ‘groups’ of nodes. Journalists around the world may want to access (either temporarily or permanently) the nodes of other journalists for collaborative purposes. As a solution for the grouping of nodes, we have integrated the OpenVPN client & server into the Liquid distribution. This gives users the option to easily and securely interconnect boards across potentially insecure networks (such as the internet).
Our VPN integration automates: Certificate Authority creation Server key generation General server configuration based on configuration panel input Client key generation Client key revocation Client config (.ovpn) file generation
Using the Admin Panel users are able to configure OpenVPN server in a simple, user-friendly way (as seen on demo video).
They can also generate client configuration files for upload to client nodes, allowing them to automatically connect to VPN servers. These client configuration files describe everything that is needed for a node to join a group, and can be distributed to collaborating journalists by (for example) USB thumb drive or encrypted email. Once in possession of the configuration file, it is simple for a node administrator to upload the file and join the group.
The Admin Panel also allows for an administrator to easily revoke the access keys in configuration files, allowing for removing nodes from a group once collaboration has finished.
Drag and Drop Dataset Imports
Excitingly, we have now integrated the Davros file sharing system with Hoover, our search tool. Users can now drag and drop datasets into the Davros web interface, where they will be identified and imported into Hoover, creating new searchable collections. This greatly simplifies the process of indexing new datasets, removing the need for a system administrator in the data import process.
Continuous Integration & Automated Testing
This quarter, we have implemented Continuous Integration as part of our development process. We have configured an instance of Jenkins (https://jenkins.liquiddemo.org) that supports image building and automated tests for both x86 and ARM architectures. Automated tests are run against both the software bundle and the first boot process.
For ARM builds, we are currently using 3 Odroid C2 boards and one Rock64 board as Jenkins build slaves tied to the master server. x86 builds are performed by virtual build slaves on a VPS. We have implemented a complete ‘build factory’ system which flexibly uses VMs to augment the build process (https://github.com/liquidinvestigations/factory).
Travis CI is also used for running backend API tests in isolation from the rest of the system, for faster development (https://travis-ci.org/liquidinvestigations/core).
Can you please describe the stage of your project and the achievements realised so far?
In Q1, we focused on evaluating various existing software services to incorporate into the liquid software bundle and determine software constraints for hardware resources (ARM), connectivity, security and usability. In Q1, we demonstrated how 2 ARM boards, running the liquid software bundle (Search and Annotations) can work together in a basic network setup (the first board was configured to start a WIFI access point and the second board was configured to accept VPN connections from other boards).
In Q2, we focused primarily on: Build infrastructure and application integration Node discovery Dockerization of applications OAuth and shared users between applications Initial implementation of a configuration API.
In Q2 we demonstrated how users can log in and use all of the bundled applications with their credentials.
Q3 has revolved mostly around UI and further configuration work. We have now a configuration wizard for new boards as well as a configuration panel for administration of running boards. We now support secure collaboration and ‘grouping’ of boards via VPN, drag-and-drop dataset imports, and we have refined our Continuous Integration infrastructure to support image building and automated testing of both x86 and ARM images.
At the close of Q3, we are comfortable saying that we have a feature-complete and working prototype of the system. We are ready to begin testing (both internal and end-user focused) in earnest, with an eye to refining and productizing this prototype.
During the quarter we extensively used the following communication streams and maintained clean project documentation following free and open source standards and licensing:
External task management:
Installation of the liquid software bundle:
What evidence can you provide to prove that you reached the Milestone?
Video demo of logging into WiFi and setting up an ARM board via the Initial Setup Wizard, of admin logging into the server and administrating the bundle via the Admin Panel and showing how two boards connect remotely via OpenVPN https://www.youtube.com/watch?v=7SRltbxBMSI
What were your biggest impediments so far?
Once again, acquiring ARM boards has proven difficult because of their limited availability in online stores. Our engineers work remotely from different locations (Bucharest, Leipzig) so shipping ARM boards for testing purposes to various locations brings delays.
In addition to sourcing ARM boards, sourcing Linux-compatible USB WiFi dongles proved also difficult.
Besides hardware order issues we had some minor planning setbacks. We didn’t factor in vacation time in the original planning, so we had some delays in execution, as most of our developers were away during the month of August. We’ve managed to get back on track. However, this caused some project stress that could have been easily avoided.
What caused problems?
Reiterating some of the aspects mentioned above:
ARM board acquisition/delivery logistics Sourcing Linux-compatible WiFi dongles that support Hotspot mode Running Jenkins build slaves on ARM boards proved more difficult than initially expected due to dropped connections. This seems to have been solved by abandoning Jenkins’ built-in SSH support in favour of system SSH support.
Any other things you want to mention here?
We have created a logo and overall visual identity for the product. You can find our project logos here: https://github.com/liquidinvestigations/liquidinvestigations/tree/master/logos. Using this visual identity, and in collaboration with our UI implementation team, our first task in November will be to perform a ‘facelift’ of the entire system, updating the overall product landing page and the front pages of each of the bundled applications.
In addition, we held a tech summit in Leipzig, Germany (Oct 19th and Oct 20) with members of our tech and project team, as well as tech people from various newsrooms in the EIC.network. The goal of this meeting was to introduce the Liquid Investigations project and perform demos of the product on different classes of hardware. We demonstrated the product on both ARM boards (ODROID-C2) and x86 mini-PCs (Intel NUC i5). After the demos, we distributed pre-installed ODROID-C2 devices to a number of the participants to take home for evaluation.
This meeting exposed us to different use cases pertaining to different newsrooms. We also gained new perspectives on how our toolset could be of use (i.e. Le Soir noticed that Davros, our secure file sharing system could improve their internal file distribution system, as most of the journalists are adverse to using PGP encrypted emails).
The meeting was also a great recruiting exercise for outside testers for the liquid tool bundle. We’ve selected testers from some of Europe’s biggest newsrooms (der Spiegel, MediaPart, El Mundo, Le soir). This is a great segue into Q4, which will be mostly about setting in place an efficient testing plan, testing the prototype with outside people, and collecting and implementing feedback to work towards a complete and refined product.