Q4 - Progress Report
What did you do to reach this Milestone?
This quarter’s work has been primarily focused on the following areas
- Single Sign-On (SSO)
- External Storage and Encryption
- User Interface improvements
- Riot/Matrix integration into the liquid bundle on ARM and x86 environments (Riot)
- User Acceptance Testing (UAT)
SSO SSO is one of the larger efforts in Q4. The SSO system currently stores login information for all the users. All the apps in the liquid bundle (Hoover search, Hypothes.is digital annotations, Davros file sharing, Dokuwiki and Riot chat) use now the OAuth2 protocol, to provide a seamless user experience, by reducing the number of times users are required to login when using each application. Users are now able to log in to all services using a single account name and password and that streamlines the research workflow, letting the user focus on typing research questions and sharing information with the other users instead of focusing to typing many login informations into many different applications ( for now guest or visitor accounts are not supported).
SSO and User Management The liquid-core app (https://github.com/liquidinvestigations/core) serves as the central user database in the bundle. It’s a Django app with django-oauth-toolkit which acts as an OAuth2 provider. The Admin UI provides an interface for the administrator to manage user accounts. The users are stored as plain Django users, so if needed, they can be managed via Django’s admin pages. This makes easy the adding of extra layers of security like 2-factor-authentication (2FA).
External Storage and Encryption By default, the image looks for the first external block device (USB stick, USB or internal hard disk, virtual hard disk).The system then formats and mounts the drive. It’s going to be used as external storage; all user data (documents, wiki pages, chat history, annotations) are going to be stored on it.
The block device that’s going to be mounted into the system can be overridden by setting a configuration file. Both Luks and VeraCrypt containers can be used, if installed and configured through the shell. The process can be automated and adapted to the organization’s secured storage workflow by hooking into the provided entry point, see the documentation (https://github.com/liquidinvestigations/setup) for more details.
Riot Integration of the Riot chat service UI (https://riot.im/app/) into Matrix chat, for both ARM and x86 environments. Riot chat has the advantage of coming with its own widgets, such as Etherpad for collaborative editing of documents, or Jitsi for VoIP, which alleviated our efforts to package other real-time collaborative editing tools and audio chat tools into the liquid bundle.
User Interface improvements Building on Q3’s UI work (setup wizard and admin panel) we’ve also done a complete ‘facelift’ of the front page as well as a visual harmonization of the UI of the integrated apps (Hoover, Hypothesis, DokuWiki, Matrix/Riot and Davros); Additionally we’ve done adjustments and bug-fixes to the admin panel. We will continue to improve the liquid UI as we get more usability feedback and input from our journalistic testers.
UAT Our formal test strategy included manual and automated testing, continuous integration and end-user testing. End-user testing was performed with the aid of couple journalistic networks (EIC.network, and theblacksea.eu). We’ve tested our prototype with 3 groups of journalists/researchers (between 2-4 people per group) and equipped them with the necessary hardware components. The testing period started at the beginning of January, 2018 until mid March, 2018. Not all the features were test-ready from the beginning so we had the same groups of journalists test at different dates. The liquid prototype was tested within a research effort that didn’t produce any journalistic output by the time of this report. Some of the journalist testers choose to remain anonymous, due to their affiliation to different journalistic networks and the nature of the investigations that they are part of. The overall test results were successful. The expected results were in most cases en par with the actual results.
Can you please describe the stage of your project and the achievements realized so far?
In Q1, we focused on evaluating various existing software services to incorporate into the liquid software bundle based on the needs identified with investigative journalism networks. We determined software constraints for hardware resources (ARM), connectivity, security and usability. In Q1, we demonstrated how 2 ARM boards, running the liquid software bundle (Search and Annotations) can work together in a basic network setup (the first board was configured to start a WIFI access point and the second board was configured to accept VPN connections from other boards).
See the video here.
In Q2, we focused primarily on:
- Build infrastructure and application integration
- Node discovery
- Dockerization of applications
- Shared users between applications
- Initial implementation of a configuration API.
In Q2 we demonstrated how users can log in and use all of the bundled applications with their credentials. See video here.
Q3 has revolved mostly around UI and further configuration work. We have now a configuration wizard for new boards as well as a configuration panel for administration of running boards. We now support secure collaboration and ‘grouping’ of boards via VPN, drag-and-drop dataset imports, and we have refined our Continuous Integration infrastructure to support image building and automated testing of both x86 and ARM images. See video here.
Q4 was mostly about refining the liquid prototype with features such as Single-Sign-On (SSO), encrypted HDD (as external storage) , UI improvements, Testing (both internal and end-user focused) and overall bug fixes.
At the close of Q4, we are comfortable saying that we have a feature-complete and working prototype of the system. We added functionalities based on user feedback and their workflow in real collaborative research situations (e.g. the bundle supports the automatic synchronization between Davros file sharing and Hoover search engine).
We are looking to conduct extensive stress/performance testing, once the liquid technology is deployed for a larger group of users, with an eye to refining and productizing this prototype.
Additionally we managed to maintain an extensive and clean project documentation, following free and open source standards and licensing:
- External task management
- Continuous Integration
- Installation of the liquid software bundle
What evidence can you provide to prove that you reached the Milestone?
Q4 video showing the entire flow of the liquid prototype, encompassing all functionalities:
A working demo for the liquid prototype can be found here: https://liquiddemo.org/
- Username: test
- Password: E6eRkjZGOYrvRcStQPFNonGTqBhTvY/P5t4czC6Z3Z+ODdF6uuDz4Q5viqQvOchr
What were your biggest impediments so far?
The lack of a higher budget for a larger tech team that can analyze, synthesize, implement and test faster, before the different modules of integrated software and hardware evolve to new versions. Same budget constraints slowed us down in our iterative co-design approach where we would ideally involve more researchers to test user interactions and system bugs. This impediment impacts the security architecture and the speed with we can actually match users eagerness to use our bundle in collaborative production.
See issues listed below.
What caused problems?
- ARM board acquisition/delivery logistics and lack of documentation support
- Sourcing Linux-compatible WiFi dongles that support Hotspot mode
- Low availability of the correct versions of dependencies on the ARM64 platform for the various packages that make up the toolkit
Any other things you want to mention here?
As previously mentioned, we hold a good position because several journalistic networks are co-designing this bundle. The liquid technology stems directly from within the journalistic community and is therefore going to be tested extensively by journalistic professionals and hopefully adopted fairly quickly. We envision the EIC.network to be one of the early adopters of the liquid investigations kit. The kit will enable EIC.network and it’s members to reduce their operating costs thus freeing up more resources to invest back into journalism.
In this respect we had another face to face tech meeting with members of the EIC network, where we demoed the capabilities of the liquid prototype. The meeting took place in Feb, 2018 within the Madeira Interactive Tech Institute (M-ITI.org). The immediate impact of that meeting is that during the month of April, 2018 - the liquid prototype will be deployed for evaluation within the EIC.network. Another place aligned for deployment and evaluation is to support journalists meeting in work groups at Data Harvest, the most important investigative journalism conference in Europe, at the end of May 2018.
Depending on the feedback and scalability we are hoping to have the liquid technology be production ready by summer 2018 - which will be a huge success for our project. User management and support, software updates (including additional apps in the bundle), bug fixing and stress testing are some of the segments we’ll be focusing in the nearby future.
Conclusion We think that one of the biggest achievements is that we created a flexible and modular prototype. Depending on each use-case, the liquid investigations software bundle can be stored on an x86 architecture: regular or smaller servers (intel NUC i5) or on ARM class devices (Odroid C2 or Rock/Pine64). The connectivity can be internet/or intranet depending on the threat model of the investigation. Hence, our prototype can serve both small and large collaborations and can facilitate permanent or ad hoc networks. The information exchange can happen outside the cloud servers and the same data can be stored on multiple liquid nodes, insuring a degree of decentralization of the liquid network. Thus, users retain access to source documents and all information exchanges, even when exiting the network.
Another very positive aspect of the liquid tool’s bundle, is that we managed to lower the financial investment for both the software (all the bundled tools are open source) as well as for the hardware (intel NUC ~ 500 EUR, ARM class ~50 EUR), making collaborative investigations affordable to journalists at all levels of the media system across all regions. The support we received from Google DNI, enabled us to bring to light a functional prototype. Next we will focus on further enhancements of the liquid bundle prototype as well as finding the necessary means to transform it into a market ready product.