More than two months ago, we argued the decision to close MutableSecurity. We described how MutableSecurity's journey looked like and highlighted the plans for the following period: open-sourcing the codebase and archiving/deleting MutableSecurity's assets on the Internet (repositories, packages, etc.).

As a follow-up, the purpose of this post is to describe the software architecture that we wanted to achieve for our commercial product. In addition, it will list the steps executed for the project's retirement, which could represent a model for other closed-source software initiatives that reached this stage.

Desired Software Architecture​

The outcome of our commercial product was to deliver a way to remotely manage and monitor the security solutions installed on the organization's hosts. The following sections will describe each element of the architecture.

Cybersecurity Solutions' Automation with Target Agents​

The target agents are Python 3 wrappers for the open-source package, mutablesecurity. To give visibility and control to the web dashboard, the administrator installs agents on each server that needs to be managed by MutableSecurity.

The functionality consists of the following steps:

1. Parses the configuration file that contains information such as:
   • API key to authenticate to the orchestration agent;
   • The address and port of the orchestration agent; and
   • The reporting period, in seconds.
2. Connects to the orchestration agent and authenticates itself using the provided API key.
3. Sets a ticker depending on the reporting period and execute, at each tick, the following operations:
   1. Deduces what cybersecurity solutions are installed on the server.
   2. For each solution, extracts the specific information and executes the tests.
   3. Combines the resulting information into a single data structure.
   4. Sends the generated data to the orchestration agent via HTTPS.

Data Aggregation with Orchestration Agents​

The orchestration agent is the aggregation service, built with Python 3. All target agents in the current network will connect to this type of agent. In this manner, it will be the single point in the reporting infrastructure with an Internet connection.

The operations that it executes are:

1. Parses the configuration file that contains information such as:
   • Bind address and port, where it will listen; and
   • Credentials to authenticate to Firebase.
2. Processes a host file, which contains, for each target agent which is allowed a connection, the following information:
   • An API key that needs to be presented by the agent on each connection;
   • A unique identifier; and
   • A description of the agent.
3. Authenticates to Firebase using the credentials from the configuration.
4. Sets up an HTTPS server to let the target agents connect.
5. Whenever a legitimate agent connects, store the reported information in the user-specific section of the database.

The host files mentioned above could be modified either manually or with a standalone script.

Storage with Firebase​

Firebase is used to store the reported data in a resilient cloud environment. Its non-structural aspect provides a malleability that is not present for structural (for example, SQL-backed) ones. This approach avoids issues imposed by fixed structures, but comes with an additional responsibility to ensure data's correctness.

In addition, we created access control lists in Firebase such that a user can write only to its allocated partition.

Data Visualization with Dash​

The visualization element in the distributed architecture is a web dashboard. Dash is built with React.js and Chakra UI. In the first iteration, it queries data from Firebase to:

• Show overview statistics about the security architecture;
• List the installed agents;
• List the automated solutions; and
• Monitor the details (configuration, metrics, and tests) of the automated solutions.

Alerting with Google Cloud Functions​

Another functional requirement of the architecture was alerting the administrators when certain events occur:

• Failed tests: Consider a scenario in which we've used MutableSecurity to install Suricata in the IDS mode. If the test checking the Suricata process fails, then the IDS no longer processes traffic and generates security alerts. This issue permits attackers to benefit from the lack of network analysis and visibility. Thereby, the administrator should be alerted as he needs to manually analyze why this happened and restore its functioning.
• Configuration change: In the same scenario, an attacker with access to the IDS may disable the automatic updates. Although the community feeds will publish IOCs (hashes, IP addresses, etc.) describing the attacker's behavior, the IDS will not be able to retrieve them. Thereby, the administrator should be noticed of any configuration change of the security solutions, despite the cases in which he makes the changes.

The alerting infrastructure consists of a Google Cloud function that is executed every time data is reported by an agent. If failed tests or configuration changes are detected, then an email is sent via SendGrid, having the above dynamic templates.

Periodically Deleting Data with Google Cloud Functions​

The last element of the architecture is a Google Cloud function, whose sole purpose is to periodically delete old data from Firebase. The configuration provides the retention period.

Retirement Checklist​

• Closing the issues and pull requests
• Deleting the accounts related to the project
• Revoking the access of third-party apps to the organization
• Checking the repositories with Gitleaks such that no active API key is disclosed
• Adding LICENSE.md and README.md files to all repositories
• Archiving all repositories
• Adding a banner to the website to mark the project as discontinued

Account Deletion​

The second step in the previous checklist involved, in our case, the deletion of:

• PyPi;
• DeepSource;
• Google Cloud Platform, including Firebase; and
• SendGrid accounts.

This automatically ensures that:

• The API keys will be destroyed. Even though they will become public in the open-source code, they cannot be used anymore.
• The packages cannot be downloaded anymore with pip install mutablesecurity.

Conclusion​

This wrap-up blog post described, from a technical perspective, how the retirement process looked like for MutableSecurity. This involved the publishing in the open-source of multiple repositories that were previously accessible only by the team members.

With this information in mind, the blog post is the last one in MutableSecurity's lifespan. With the previous thanks in mind, we want to end by hoping the information presented in this post and blog was helpful.

See you next time!

We are a part of our civilization’s ongoing evolution as brilliant brains seek out novel technology and large organizations, like corporations and NGOs, speed their acceptance. Although this concept initially looks utopian, reality is harsher: ideas also go through the same natural selection process that affects all living species competing for scarce resources.

Up until 2011, phones featuring a full-featured QUERTY keyboard were popular. With 50 million devices sold during that year, Blackberry experienced its highest sales volume. However, at the same time, smartphones with touchscreens have gained popularity. When Blackberry began to lose customers in 2012, Apple nearly doubled the number of iPhones it sold.

Startup ideas, for example, could die more abruptly than other ideas. One well-known rough estimation states that only one startup out of every ten succeeds. This occurs as a result of a lack of market demand, insufficient funding, unfit teams, or faulty products.

The key takeaway is that all ideas have a finite life span and either get ingrained in society or perish. Or, in the words of Frank Herbert, that “every revolution carries within it the seeds of its own destruction”.

MutableSecurity in a Nutshell​

Everything began at the University Politehnica of Bucharest in the fall of the previous year. We were a team of seven students that were passionate about cybersecurity and came together for a semester project. The concept for MutableSecurity, an automated platform for cybersecurity solutions, emerged. We intended to make it as simple as possible for IT and security professionals to install, configure, test, and monitor the tools they require to safeguard their IT infrastructures.

To cut a long story short, the following memorable moments occurred during the course of the story:

• Entering startup programs such as Innovation Labs, StepFWD, and Startup School;
• Winning the Innovation Labs IdeaJAM because of our talented pitcher;
• Launching an open source, well documented project;
• Developing a distributed reporting architecture, with agents and a visualization dashboard;
• Testing our software into an international cybersecurity exercise, CWIX;
• Presenting the idea on DefCamp; and
• Meeting a lot of great people.

Coming to an End​

We regretfully announce the closure of MutableSecurity, but we have far higher hopes for the future. After all these months of work, the primary concern we had in the most recent period was not whether we could (technically) complete the product, but whether we still wanted to.

We were particularly worried about the validation. Although we conducted informal interviews and discussions with potential customers, the perception of a constantly contracting total addressable market persisted.

The technology stack appears to be continuously abstracted: services that were before controlled at the operating system level (on which MutableSecurity resided) are now moving into containers that are managed by solutions like Kubernetes. Additionally, working with low-level notions is further abstracted via serverless functions and no-code solutions. It does not mean that the operating system is no longer important, but it is already managed at scale by cloud providers or managed service providers using de facto configuration management standards such as Ansible.

The principles below can be used to describe further conclusions we came to from this experience:

1. Engage end consumers in conversation, even if they are corporate representatives.
2. Select technologies that are future-proof, eventually from current industry standards.
3. Pick programming languages with strong packaging and testing features. If you do the reverse, you’ll stop testing your code and release buggy software, or spend too much time creating deployable packages.
4. Distinguish between the opinions of well-known individuals to whom you pitch the idea and those of your target audience. The former are nice and inspiring, but they don’t provide validation.

What’s Next for MutableSecurity​

Despite our decision, MutableSecurity is not being shut down immediately today. The three months’ worth of plans are:

1. Publishing a stable version of the source code (currently closed source) for agents and Dash, the online visualization dashboard; and
2. Deleting the package from PyPi and archiving the repositories at the end of a three-month maintenance phase, during which we will resolve reported bugs.

We’ll continue to use LinkedIn and Twitter as our primary social media platforms for posting updates on the above-mentioned actionables. We make available the credentials needed to access Dash so that you can get a brief idea of how MutableSecurity was intended to function.

Lastly​

We’d want to end by thanking all the wonderful folks we met over this time ❤️. We know that without the help we received (crash courses, workshops, direct feedback, advices, and suggestions), our journey would not have been as beautiful as it was.

The last two days, our team was present at DefCamp, the largest cybersecurity conference from Central and Eastern Europe.

Presentation​

Beside talking to (and drinking beers with) awesome hackers, we had the opportunity to showcase our open-source project in a presentation called "MutableSecurity: Life's too short. Automate everything!".

The following subjects we're discussed:

• The economic perspective on the cybersecurity context;
• How MutableSecurity can be of assistance;
• A demonstration; and
• The integration of security mechanisms to our codebase.

Demo​

The presentation was accompanied by a scenario taking into account a small IT outsourcing company's production infrastructure, which consisted of an nginx web server with enabled remote access through password-based SSH.

Two separate viewpoints are presented in the video below. The first is designated with a red border, which is the hacker one. Its objectives are to scan the server ports, bruteforce the SSH credentials using a known username, and exploit of the local file inclusion vulnerability. A Python 3 script is used to automate the final two steps.

The second viewpoint has purple borders and shows how the company's IT or security engineer can use MutableSecurity to protect its server. The following security solutions were configured, installed, and tested:

• teler to detect web attacks;
• Fail2ban to detect SSH bruteforce and block it in addition to the web attacks; and
• Vector to send logs from teler and Fail2ban to a cloud-based Loki instance.

Linux has been considered one of the most secure OS and, along the history, there have been a minimal number of attacks. Quite recently there has been a change, for instance a new stealthy malware targeting Linux, called Shikitega. There are a few key features to this malware, such as using Metasploit's Mettle to maximize the control of infected machines, exploiting vulnerabilities to execute crypto miners, and using polymorphic encoder to make it difficult for antivirus engines to detect it.

This being said, we will explain in this blog post how to install and configure an open-source anti-malware software for Linux, ClamAV. It is developed by Cisco Talos, one of the largest commercial threat intelligence teams in the world. By having the signatures database updated by Talos, it is able to detect a multitude of threats and offer a quite decent security solution.

Requirements​

By having, at least, Python 3.9 installed, you can simply type pip install mutablesecurity, and you will have access to our open-source solution. There you will be able to use a few solutions, including ClamAV. Make sure you are not in the root directory before continuing with the next steps.

Installing ClamAV​

Now that you have mutablesecurity installed on your machine, the next step is configuring ClamAV.

Simply run mutablesecurity. You will be able to see a detailed guide about how to use the solution, but for now, this guide will help and walk you through it.

                  _        _     _      __                      _ _
      /\/\  _   _| |_ __ _| |__ | | ___/ _\ ___  ___ _   _ _ __(_| |_ _   _
     /    \| | | | __/ _` | '_ \| |/ _ \ \ / _ \/ __| | | | '__| | __| | | |
    / /\/\ | |_| | || (_| | |_) | |  ___\ |  __| (__| |_| | |  | | |_| |_| |
    \/    \/\__,_|\__\__,_|_.__/|_|\___\__/\___|\___|\__,_|_|  |_|\__|\__, |
         Seamless deployment and management of cybersecurity solutions |___/

Usage: mutablesecurity [OPTIONS]

Options:
  -r, --remote TEXT               Connect to remote in the
                                  USERNAME@HOSTNAME:PORT format. If ommited
                                  (besides the remote list parameter), the
                                  operations are executed locally.
  -l, --remote-list FILE          Connect to a series of remote hosts
                                  specified in a file, in the
                                  USERNAME@HOSTNAME:PORT format. If ommited
                                  (besides the remote host parameter), the
                                  operations are executed locally.
  -k, --key FILE                  SSH key to use when connecting to the remote
                                  host
  -s, --solution [CLAMAV|LETS_ENCRYPT|TELER|FAIL2BAN]
                                  Solution to manage
  -o, --operation [INIT|INSTALL|GET_INFORMATION|SET_INFORMATION|TEST|GET_LOGS|UPDATE|UNINSTALL|EXECUTE]
                                  Operation to perform
  -i, --identifier TEXT           Information to modify. Available only with a
                                  value (--value)
  -v, --value TEXT                New value of the information. Available only
                                  with an identifier (--identifier).
  -a, --arguments TEXT            Arguments to be passed to an action, in the
                                  "key=value" format
  --verbose                       Increase in the logging volume
  --feedback                      Show feedback form
  -h, --help                      Useful information for using MutableSecurity
                                  or about a solution

Quick observation here. The code blocks you are about to see are from the current version, there will be improvements and more solutions added as the times goes by.

Type mutablesecurity -s CLAMAV. There you will be able to see all the information you need to know about ClamAV, furthermore, -s means solution.

$ mutablesecurity -s CLAMAV
                  _        _     _      __                      _ _
      /\/\  _   _| |_ __ _| |__ | | ___/ _\ ___  ___ _   _ _ __(_| |_ _   _
     /    \| | | | __/ _` | '_ \| |/ _ \ \ / _ \/ __| | | | '__| | __| | | |
    / /\/\ | |_| | || (_| | |_) | |  ___\ |  __| (__| |_| | |  | | |_| |_| |
    \/    \/\__,_|\__\__,_|_.__/|_|\___\__/\___|\___|\__,_|_|  |_|\__|\__, |
         Seamless deployment and management of cybersecurity solutions |___/


Full name: ClamAV

Categories:
- Antimalware
- Host Protection

Description:
Clam AntiVirus (ClamAV) is a free software, cross-platfom antimalware toolkit able to detect many types of malware, including viruses. ClamAV includes a
command-line scanner, automatic database updater, and a scalable multi-threaded daemon running on an anti-virus engine from a shared library. FreshClam is a
virus database update tool for ClamAV. ClamAV Daemon checks periodically for virus database definition updates, downloads, installs them, and notifies clamd to
refresh it's in-memory virus database cache.

References:
- https://www.clamav.net/
- https://github.com/Cisco-Talos/clamav
- https://docs.clamav.net/Introduction.html

[...]

Before you can install the solution, there is one more step you need to do, and that is to initialize the configuration that will be used for installation:

$ mutablesecurity -s CLAMAV -o INIT
$ cat root@localhost_clamav.yaml
quarantine_location: /opt/mutablesecurity/clamav/quarantine/
scan_day_of_month: '*'
scan_day_of_week: MON
scan_hour: '0'
scan_location: /
scan_log_location: /opt/mutablesecurity/clamav/logs/logs.txt
scan_minute: '0'
scan_month: '*'

The first command will create a configuration file on the local machine. After running the command, the file in local directory (called, in our case, root@localhost_clamav.yaml) should be looking like in the previous code block.

In contrast, if you want to deploy ClamAV on a target host, or multiple target hosts, please refer to the documentation on our website or by typing mutablesecurity.

Each of them can take a special value, except quarantine_location and scan_log_location. They control how the crontab will act. Please look into the documentation, or simply type mutablesecurity -s CLAMAV for all the information regarding the values. Also, after the INSTALL, it can be changed using SET_INFORMATION.

Since the scan will take place each Monday at 12:00 AM, you might want to change it. An option would be to make it daily at 5:00 AM. To do that, use vim or any editor you prefer and change scan_day_of_week to "*" and scan_hour to "5". That should suffice.

That being said, you can keep the default configuration for now and just run mutablesecurity -s CLAMAV -o INSTALL to install the solution. Be aware, it will take a bit of time.

Using the Newly Installed ClamAV​

Testing​

At this point, the solution is installed. To see if the solution is fully operational, run:

$ mutablesecurity -s CLAMAV -o TEST.
❓ Password:
💻 Connection root@localhost
✅ The operation was successfully executed!
┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━┓
┃ Identifier      ┃                                       Description                                       ┃    Type     ┃ Value ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━┩
│ active_database │                 Checks if the ClamAV virus database service is active.                  │ OPERATIONAL │ True  │
├─────────────────┼─────────────────────────────────────────────────────────────────────────────────────────┼─────────────┼───────┤
│ eicar_detection │ Creates a EICAR-STANDARD-ANTIVIRUS-TEST-FILE and checks if ClamAV is able to detect it. │  SECURITY   │ True  │
├─────────────────┼─────────────────────────────────────────────────────────────────────────────────────────┼─────────────┼───────┤
│ internet_access │                           Checks if host has Internet access.                           │ REQUIREMENT │ True  │
├─────────────────┼─────────────────────────────────────────────────────────────────────────────────────────┼─────────────┼───────┤
│ ubuntu          │                        Checks if the operating system is Ubuntu.                        │ REQUIREMENT │ True  │
└─────────────────┴─────────────────────────────────────────────────────────────────────────────────────────┴─────────────┴───────┘

Manipulating the Configuration​

To see all the information about the installed solution, run the GET_INFORMATION operation:

$ mutablesecurity -s CLAMAV -o GET_INFORMATION
❓ Password:
💻 Connection root@localhost
✅ The operation was successfully executed!
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Identifier                  ┃         Description         ┃  Type   ┃          Properties          ┃        Default Value        ┃            Value             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ daily_infected_files_detec… │  Total number of infected   │ INTEGER │      METRIC, READ_ONLY       │                             │              2               │
│                             │    files detected today     │         │                              │                             │                              │
├─────────────────────────────┼─────────────────────────────┼─────────┼──────────────────────────────┼─────────────────────────────┼──────────────────────────────┤
│ quarantine_location         │   The location where the    │ STRING  │          MANDATORY,          │ /opt/mutablesecurity/clama… │ /opt/mutablesecurity/clamav… │
│                             │   infected files will be    │         │     WITH_DEFAULT_VALUE,      │                             │                              │
│                             │     moved to after the      │         │        CONFIGURATION,        │                             │                              │
│                             │  on-demand/crontab scans.   │         │   NON_DEDUCTIBLE, WRITABLE   │                             │                              │
│                             │ Select a directory in which │         │                              │                             │                              │
│                             │  the quarantine will take   │         │                              │                             │                              │
│                             │ place if you would like to  │         │                              │                             │                              │
│                             │           change.           │         │                              │                             │                              │
├─────────────────────────────┼─────────────────────────────┼─────────┼──────────────────────────────┼─────────────────────────────┼──────────────────────────────┤
│ scan_day_of_month           │   The day (1-31, or * for   │ STRING  │          OPTIONAL,           │              *              │              *               │
│                             │ any) of the month when the  │         │     WITH_DEFAULT_VALUE,      │                             │                              │
│                             │   crontab scan will take    │         │        CONFIGURATION,        │                             │                              │
│                             │            place            │         │   NON_DEDUCTIBLE, WRITABLE   │                             │                              │
├─────────────────────────────┼─────────────────────────────┼─────────┼──────────────────────────────┼─────────────────────────────┼──────────────────────────────┤
│ scan_day_of_week            │  The day (0-6, SUN-SAT, 7   │ STRING  │          OPTIONAL,           │             MON             │             MON              │
│                             │ for Sunday or * for any) of │         │     WITH_DEFAULT_VALUE,      │                             │                              │
│                             │  the week when the crontab  │         │        CONFIGURATION,        │                             │                              │
│                             │    scan will take place     │         │   NON_DEDUCTIBLE, WRITABLE   │                             │                              │
├─────────────────────────────┼─────────────────────────────┼─────────┼──────────────────────────────┼─────────────────────────────┼──────────────────────────────┤
│ scan_hour                   │  The hour (0-23, or * for   │ STRING  │          OPTIONAL,           │              0              │              0               │
│                             │ any) when the crontab scan  │         │     WITH_DEFAULT_VALUE,      │                             │                              │
│                             │       will take place       │         │        CONFIGURATION,        │                             │                              │
│                             │                             │         │   NON_DEDUCTIBLE, WRITABLE   │                             │                              │
├─────────────────────────────┼─────────────────────────────┼─────────┼──────────────────────────────┼─────────────────────────────┼──────────────────────────────┤
│ scan_location               │   The location where the    │ STRING  │          MANDATORY,          │              /              │              /               │
│                             │   on-demand/crontab scans   │         │     WITH_DEFAULT_VALUE,      │                             │                              │
│                             │  will take place.Select a   │         │        CONFIGURATION,        │                             │                              │
│                             │ different directory if you  │         │   NON_DEDUCTIBLE, WRITABLE   │                             │                              │
│                             │    would like to change.    │         │                              │                             │                              │
├─────────────────────────────┼─────────────────────────────┼─────────┼──────────────────────────────┼─────────────────────────────┼──────────────────────────────┤
│ scan_log_location           │     The location of the     │ STRING  │          MANDATORY,          │ /opt/mutablesecurity/clama… │ /opt/mutablesecurity/clamav… │
│                             │  generated logs after the   │         │     WITH_DEFAULT_VALUE,      │                             │                              │
│                             │      on-demand/crontab      │         │        CONFIGURATION,        │                             │                              │
│                             │ scans.Chose a file in which │         │   NON_DEDUCTIBLE, WRITABLE   │                             │                              │
│                             │ the logs will be stored if  │         │                              │                             │                              │
│                             │  you would like to change.  │         │                              │                             │                              │
├─────────────────────────────┼─────────────────────────────┼─────────┼──────────────────────────────┼─────────────────────────────┼──────────────────────────────┤
│ scan_minute                 │ The minute (0-59, or * for  │ STRING  │          OPTIONAL,           │              0              │              0               │
│                             │ any) when the crontab scan  │         │     WITH_DEFAULT_VALUE,      │                             │                              │
│                             │       will take place       │         │        CONFIGURATION,        │                             │                              │
│                             │                             │         │   NON_DEDUCTIBLE, WRITABLE   │                             │                              │
├─────────────────────────────┼─────────────────────────────┼─────────┼──────────────────────────────┼─────────────────────────────┼──────────────────────────────┤
│ scan_month                  │  The month (1-12, JAN-DEC,  │ STRING  │          OPTIONAL,           │              *              │              *               │
│                             │   or * for any) when the    │         │     WITH_DEFAULT_VALUE,      │                             │                              │
│                             │   crontab scan will take    │         │        CONFIGURATION,        │                             │                              │
│                             │            place            │         │   NON_DEDUCTIBLE, WRITABLE   │                             │                              │
├─────────────────────────────┼─────────────────────────────┼─────────┼──────────────────────────────┼─────────────────────────────┼──────────────────────────────┤
│ total_infected_files_detec… │  Total number of infected   │ INTEGER │      METRIC, READ_ONLY       │                             │              2               │
│                             │   files detected overall    │         │                              │                             │                              │
├─────────────────────────────┼─────────────────────────────┼─────────┼──────────────────────────────┼─────────────────────────────┼──────────────────────────────┤
│ version                     │      Installed version      │ STRING  │      METRIC, READ_ONLY       │                             │ 0.103.6+dfsg-0ubuntu0.22.04… │
└─────────────────────────────┴─────────────────────────────┴─────────┴──────────────────────────────┴─────────────────────────────┴──────────────────────────────┘

As earlier said, you can change the values of the table above (including crontabs) from the CLI, without using any text editor. An example is to change the scan_day_of_week to "*", in order to scan every day at 12:00 AM. To do that, use SET_INFORMATION:

$ mutablesecurity -s CLAMAV -o SET_INFORMATION -i scan_day_of_week -v "*”

-i is the information identifier (the first column) parameter, and -v represents the parameter to change the old value (the last column) with a new one.

By using the GET_INFORMATION operation, we can see that the value is changed:

[...]
├───────────────────────────────┼─────────────────────────────────┼─────────┼─────────────────────────────────┼─────────────────────────────────┼─────────────────────────────────┤
│ scan_day_of_week              │  The day (0-6, SUN-SAT, 7 for   │ STRING  │  OPTIONAL, WITH_DEFAULT_VALUE,  │               MON               │                *                │
│                               │   Sunday or * for any) of the   │         │ CONFIGURATION, NON_DEDUCTIBLE,  │                                 │                                 │
│                               │ week when the crontab scan will │         │            WRITABLE             │                                 │                                 │
│                               │           take place            │         │                                 │                                 │                                 │
├───────────────────────────────┼─────────────────────────────────┼─────────┼─────────────────────────────────┼─────────────────────────────────┼─────────────────────────────────┤
[...]

Viewing Logs​

There are also logs. To see all the logs generated by ClamAV, Freshclam, and the on-demand scans, you can type:

mutablesecurity -s CLAMAV -o GET_LOGS -i logs

Keep in mind that it contains all logs, and you can scroll down to see everything, from configuration to identified infected files, or when the scans took place.

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Identifier                              ┃                                                                 logs                                                                  ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Description                             │                                              The logs generated by ClamAV and FreshClam                                               │
└─────────────────────────────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

[...]

----------- SCAN SUMMARY -----------
Known viruses: 8638130
Engine version: 0.103.6
Scanned directories: 1
Scanned files: 1
Infected files: 1
Data scanned: 0.00 MB
Data read: 0.00 MB (ratio 0.00:1)
Time: 26.547 sec (0 m 26 s)
Start Date: 2022:10:01 17:21:52
End Date:   2022:10:01 17:22:18

Scanning for Malware​

First off, we download a malware file from MalwareBazaar website and decompress the archive using the infected password. The PE format (Windows-specific) will protect the machine from being infected, as it cannot be executed natively.

To remove this file using ClamAV, we run the command below:

$ mutablesecurity -s CLAMAV -o EXECUTE -i start_scan -a scan_location=/home/ubuntu

If we check the end of the logs, we see how ClamAV successfully identified the threat:

/home/ubuntu/e08f17b42e47bf973bf866f167a49b931b99fe1b50bc820078ac2644bd8c209c.exe: Win.Ransomware.Packer-7473772-1 FOUND
/home/ubuntu/e08f17b42e47bf973bf866f167a49b931b99fe1b50bc820078ac2644bd8c209c.exe: moved to
'/opt/mutablesecurity/clamav/quarantine/e08f17b42e47bf973bf866f167a49b931b99fe1b50bc820078ac2644bd8c209c.exe'

----------- SCAN SUMMARY -----------
Known viruses: 8638370
Engine version: 0.103.6
Scanned directories: 5327
Scanned files: 26700
Infected files: 1
Data scanned: 703.96 MB
Data read: 606.18 MB (ratio 1.16:1)
Time: 2127.701 sec (35 m 27 s)
Start Date: 2022:10:02 14:06:59
End Date:   2022:10:02 14:42:27

Also, the file was moved to the quarantine folder. Sometimes ClamAV finds false positives, therefore, you should delete it manually if the file is a malware, exactly like this example.

Updating​

ClamAV can be updated manually by running the command below. Most of the time, it will say that it is at its newest version.

$ mutablesecurity -s CLAMAV -o UPDATE
❓ Password:
💻 Connection root@localhost
🛑 ClamAV is already at its newest version.

The Talos database has a process attached to it called freshclam. This process can be seen at the TEST operation, and the identifier is called active_database. Freshclam is in charge of updating the database automatically.

Uninstall ClamAV​

To uninstall the solution, simply run mutablesecurity -s CLAMAV -o UNINSTALL. Everything tied to ClamAV will be deleted.

Conclusion​

Despite the fact that ClamAV does not offer full protection for a server, it is a nice open-source weapon in the Linux security arsenal. This blog post has shown how to install, configure and scan with ClamAV.

Until next time, take care!

This blog post will explain the approach we used in MutableSecurity to add minimal, non-intrusive application monitoring, for both crash reporting and usage monitoring. Despite the examples from our codebase, that is mostly Python-based, the principles used may be applied with ease to other programming languages.

Why Metrics are Important​

The quote “What gets measured gets improved” is mostly used to highlight the importance of having quantitative measures of performance about a system. Whether it was said by the influential management consultant Peter Drunker back in the previous century, there are some situations in which these numeric metrics help us to better understand a system's functioning. In the software field, examples may be finding out how many users you have, how they use your program and how the product behaves regarding performance.

In addition, there may be another piece of the puzzle left: the survivorship bias. Simply said, history is written by the victorious. Applied to the software industry, we can say that we tend to judge the overall experience of our products by interacting with the users or customers (via feedback forms, interviews, etc.) that are active today, not those who abandoned the journey due to encountered issues (for example, bad UX practices, bugs and errors). But we may learn about the latter category by implementing passive feedback loops.

Feedback Loops​

For open source projects, we can have multiple places from which we can learn from and about our users.

There are active forms, in which the user can deliberately contact the project's developers to share an impression, request a feature, or report a security problem. They can create GitHub issues or, more privately, contact us through in-app forms or emailing to our addresses.

On the other hand, we have passive data collection. The user interacts as normal with the application, but he deliberately allows the collection of usage data: which UI elements he interacted with, difficulty of finding out a desired page and so on.

We can consider the app stores downloads too, but they are too opaque. For example, we could not know if the software downloaded through GitHub or PyPi (a Python package repository) was actually run. Or if the user only downloaded it, but found it hard to understand the workflow. To consolidate this argument, think about the Python ecosystem: bots (like Snyk's ones) are scanning the published packages in order to find vulnerabilities.

The Privacy Dilemma​

But there's the catch 22: as software developers, we need to think profoundly about the privacy of our users. We can't collect all possible data. In the past years, due to many privacy issues events such as keylogging social media platforms and huge data leaks, people got more conscious about what data are collected by companies and how they are used afterwards. It can be said, for sure, that the trust of users was damaged.

Our Approach for Application Monitoring​

To give a bit of context, we created a platform to automatically deploy and manage cybersecurity solutions. At the first step, we published on GitHub a CLI tool to achieve these goals. The hardest part, now, is to determine what happens next after the download from PyPi is complete. Does the user deploy a specific solution? Or does he encounter an error and uses the software only once?

We reached a solution to these issues. We implemented a minimal, non-intrusive application monitoring system for MutableSecurity:

1. Collecting usage data with Firebase Realtime Database and a serverless function, deployed on Google's cloud (Google Cloud Platform or simply GCP)
2. Integrating an error tracking platform, Sentry
3. Giving the user a method to opt out
4. Documenting the whole monitoring process.

The following sections will describe each of them with a bird's eye view. All of them are exemplified with Python snippets from our codebase and screenshots.

Usage Monitoring​

1. Collect the data: We implemented a base abstract class named Metric. Each collected metric should inherit it and overwrite the IDENTIFIER member (that defines the key used to identify the information when placed in Firebase) and the _get method (that extracts the information from the current host). When a new metric is defined, the __init_subclass__ method is used to automatically register it (by storing a reference in a list) in the DataCollector class, that deals with collecting all the metrics values.

2. Send the collected data: The Monitor class is then used to retrieve all the metric values and POST them to our serverless function.

3. Retrieve the data and store it inside Firebase: The serverless function from Google Cloud Platform is configured to run in a Python environment, with a secret that is used to store the service account's private key. It only takes the data from the HTTP request and stores it inside Firebase Realtime Database with the pyrebase4 package.

4. Check Firebase for the collected usage data: In our case, the data looks similar to the screenshot below.

   

Crash Reporting​

1. Add the SDK: After setting up an account, install the Sentry SDK for Python, sentry_sdk.
2. Initialize the SDK: In the source code, call the init method of Sentry's SDK.

3. Trigger a crash: Just place a division by zero (for example, 1/0) operation between some lines of code that are certainly executed. Be sure to remove it afterwards.

4. Find the crash in Sentry's dashboard: Sentry should list the triggered error. Alongside it, there are further details such as stack traces and runtime information.

   

   

Opting Out​

Implements a logic to let the user opt out of the monitoring data. This can be achieved by adding a new aspect in the configuration. In MutableSecurity, we skip the logic presented in the Usage Monitoring section if the user sets a field in the configuration file.

Be Transparent Regarding the Monitoring​

If you have read everything until this point, you are conscious about the benefits some data may have. Namely, to learn more about the users you want to help with your software. As the software user - developer relationship is one of a partnership, the main principle is trust, and it needs to be built and maintained. Also, it needs to be transparent about:

1. Why you collect metrics at all?
2. What metrics you collect?
3. If a user wants to see the implementation, what files from your codebase are relevant?
4. How can you opt out of sending any usage/crash data?

These questions may be answered with a page of your documentation or as a separate view inside your production software. You can see examples on Homebrew's and MutableSecurity's websites.

Conclusions​

Having feedback loops is important for a software developer. The data may shift the focus from some functionality considered relevant to others that are actually used in the wild. This blog post explained the reasoning behind collecting data, some handy principles to keep in mind, and a Python implementation we developed for MutableSecurity. For further information about the development in the open source community (including metrics), I recommend following the GitHub's articles on opensource.guide.

Until next time!

The MutableSecurity team will publish high-quality content for IT professionals on topics such as the use of cybersecurity solutions, DevSecOps techniques, and best practices for technologies used in the development of MutableSecurity.

Follow us on LinkedIn and Twitter to receive updates about new content!