Installing and Configuring Agents

This section describes the installation of Foglight for Cassandra. Follow the steps in the order specified below:

Configuring Cassandra

In order to allow full monitoring of Cassandra, the agent will require a user with sufficient privilege to execute system queries. For JMX monitoring, remote JMX connections must also be enabled on each node.

Database User Setup

Using a superuser role or a role with user creation permissions, connect via cqlsh to a node on the cluster and create a new user for the Foglight Agent as follows:

Cassandra >= 2.2

For versions of Cassandra >= 2.2 or DSE >= 5.0, create a user as follows:

CREATE ROLE IF NOT EXISTS <username> WITH PASSWORD = '<your-password>' AND LOGIN = true AND SUPERUSER = false;

There are then two ways to set permissions. The simpler way is to use the following single permission:

GRANT SELECT ON ALL KEYSPACES to <username>;

It is also possible to use more specific permissions:

GRANT SELECT ON TABLE system.local to <username>;

GRANT SELECT ON TABLE system.peers to <username>;

GRANT SELECT ON TABLE system_traces.events to <username>;

GRANT SELECT ON TABLE system_traces.sessions to <username>;

GRANT SELECT ON TABLE system_auth.roles to <username>;

Using specific permissions, there is one version-dependent permission. For versions of Cassandra >= 3.0:

GRANT SELECT ON TABLE system_schema.tables to <username>;

For versions of Cassandra >= 2.2 but less than 3.0:

GRANT SELECT ON TABLE system.schema_columnfamilies to <username>;

Using the more specific permissions on any version >=2.2 use the following if running DSE:

GRANT SELECT ON TABLE dse_perf.node_slow_log to <username>;

Cassandra < 2.2

For versions of Cassandra < 2.2 or DSE <= 4.8:

CREATE USER IF NOT EXISTS <username> WITH PASSWORD '<your-password>' NOSUPERUSER;

GRANT SELECT ON ALL KEYSPACES TO <username>;

JMX Setup

Enable Remote JMX Connections

Enable remote JMX authentication as per the instructions at either of the following links:

JMX Remote Files

Only a single user needs to be added to the jmxremote.password and jmxremote.access files for Foglight monitoring and administration. The following is the full (template) content of a sample jmxremote.password file:

<your-JMX-username> <your-JMX-password>

The following is the full (template) content of a sample jmxremote.access file:

<your-JMX-username> readwrite

There are two JMX access permissions to choose between:

  • readonly – For monitoring only.
  • readwrite – For monitoring as well as administration, which is currently the ability to set the per-node trace probability.

Configuring an SSL Connection

The below instructions cover common steps used to configure a TLS/SSL connection from the Cassandra Agent client. A full treatment of TLS/SSL keys, certificates, and certificate authorities (CA) is beyond the scope of this document. The following instructions assume familiarity with TLS/SSL concepts and tools. Client and certificate authority certificates must be available prior to proceeding.

In order to use SSL, your Cassandra cluster must include SSL support and allow SSL connections. There are various configurations options for client connections. Refer to the Cassandra documentation and verify that the current Cassandra server configuration parameters support the desired authentication.

The Foglight agent, in its capacity as a database client, requires access to a private key, its signed certificate, and the signing CA’s certificate. The client key and certificate must be imported into a keystore, and the CA certificate must be imported into a separate truststore.

One example method for generating a JKS keystore for use with Foglight utilizes openssl and keytool. Set the key and certificate filenames, alias name, and keystore password as appropriate.

openssl pkcs12 \\

\-export \\

\-in ${CERT_NAME}.crt \\

\-inkey ${CERT_NAME}.key \\

\-name $CERT_NAME \\

\-out temp-keystore.p12 \\

\-passout pass:${KEYPASS}

keytool -importkeystore \\

\-srckeystore temp-keystore.p12 \\

\-srcstoretype PKCS12 \\

\-srcstorepass $KEYPASS \\

\-destkeystore keystore \\

\-deststoretype JKS \\

\-deststorepass $KEYPASS

Regardless of how the keystore is constructed, it must list the client certificate as a ‘PrivateKeyEntry’, indicating that it also contains the private key, not just the signed certificate.

Separately, import the CA certificate into a truststore:

keytool -importcert \\

\-keystore truststore \\

\-alias $CA_NAME \\

\-file ${CA_NAME}.crt \\

\-keypass $TRUSTPASS \\

\-storepass $TRUSTPASS \\

\-storetype JKS \\

\-noprompt

Next, edit the baseline.jvmargs.config file in the <FglAM-install-root>/state/default/config directory and add the following parameters with file paths and passwords appropriate for your system.

vmparameter.0 = "-Djavax.net.ssl.keyStore=/path/to/keystore";

vmparameter.1 = "-Djavax.net.ssl.keyStorePassword=changeit";

vmparameter.2 = "-Djavax.net.ssl.trustStore=/path/to/truststore";

vmparameter.3 = "-Djavax.net.ssl.trustStorePassword=changeit";

Escape any quotes with a backslash (‘\’). On an Agent Manager installed on Windows, use forward slashes in the file paths, like so:

vmparameter.0 = "-Djavax.net.ssl.keyStore=\\"C:/path/to/keystore\\"";

Then, restart the FglAM and continue with the agent configuration, setting the “Use TLS/SSL?” option in the Agent Properties to true for the driver and/or JMX connections, depending on the Cassandra server configuration.

Creating and Configuring Agents

Agents can be created in one of two ways:

The Agent Installer Wizard simplifies the agent creation and configuration process and can be accessed from the Databases dashboard. For advanced configuration or modification of agent properties post-creation, use the Agent Status dashboard.

Using the Agent Installer Wizard

Foglight for Cassandra offers a graphical, intuitive interface for creating and configuring agents. This method serves as an alternative to Foglight’s default approach, which involves creating agents and editing their properties through the Agent Status dashboard.

With Foglight for Cassandra, you can use a wizard that provides a unified entry point for adding database instances and configuring them for monitoring.

To run the instance installation wizard:

  1. On the Welcome page, click Monitor Databases or click Databases in the navigation pane. On a fresh installation, no instances are listed for Cassandra in the Databases dashboard.
  2. On the Databases dashboard, click Monitor and select Cassandra. The Agent Installer wizard opens.
  3. The first card - Agent Deployment – has two fields:
    1. Agent Name – Provide a name for the agent that will be created. This is not canonical and should be representative of the database instance that this agent will monitor.
    2. Agent Manager - Choose the agent manager on which the agent should run. Considerations for this may include physical or virtual locality to the monitored instance, allocated resources, or grouping with other agents of the same type or monitored environment. If the agent package has not been deployed to this Agent Manager yet, it will be installed when the first agent of this type is created.
  4. The second card – Agent Properties – requires a basic set of parameters for connecting to and monitoring the database instance. A full explanation of these properties is available in the Configuring Agent Properties section of this document.
  5. The third card – Agent Summary – displays a review of the configuration that will be created and an option allowing the agent to be activated after creation. If the configuration looks good, click the Finish button to start the process.
  6. When the process completes, a results screen will appear showing the results of agent creation. If the agent was not created, follow the instructions on the results screen. If successful, the database instance should appear in the Databases table within a few minutes.

The database instance gets created and appears in the list of all instances.

If the agent is created successfully but data does not appear, navigate to Administration > Agents > Agent Status and click the icon in the Log File column for the agent you created. In most cases, the reason for the failure will be obvious. You can also refer to the Foglight for Cassandra Installation and Troubleshooting document for common errors and solutions. If the solution requires reconfiguring the agent properties, follow steps 7-10 of the Using the Agent Status Dashboard section.

Using the Agent Status Dashboard

The Agent Status page can be used to create new agents and configure and manage existing agents. To access the page from the navigation pane, click Administration > Agents > Agent Status.

To create a new agent instance:

Deploy the Cassandra agent package to the FglAM before creating the agent if it has not been deployed yet. Click Deploy Agent Package on the Agent Status or Agent Managers page to perform this.

  1. Navigate to Administration > Agents > Agent Status.
  2. Click Create Agent.
  3. Select the hosts to which you want to deploy agent packages.

    Considerations for this may include physical or virtual locality to the monitored instance, allocated resources, or grouping with other agents of the same type or monitored environment.

  4. Click Next.
  5. Select the Cassandra type. Then, select the Specify Name radio button and enter agent instance name. Click Next.
  6. Click Finish.
  7. Once the agent has been created, click the checkbox next to the Cassandra agent.
  8. Click Edit Properties.
  9. Select Modify the default properties for this agent.
  10. Edit the agent properties for the Cassandra agent instance:
  1. Click the Activate button.

To modify the properties of an existing agent, go to step 7, deactivate the agent, update the configuration, and then reactivate the agent.

Configuring Agent Properties

This is a full list and explanation of the configurable properties of the Foglight for Cassandra agent. The Agent Installer Wizard provides access to the essential subset of available properties. To modify other properties or modify the agent configuration after creation, use the Agent Status dashboard.

This section includes the following key areas:

Setting DB Connection

The agent requires a connection to the cluster in order to gather information about the cluster and data structure. The agent may be referred to other nodes than the one specified.

  • IP or Hostname – Host where Cassandra node is running. Default is “localhost”. (e.g.<hostname> or <IP address>)
  • Port – The CQL native transport port for the Cassandra node. Default is 9042.
  • Username – User that can connect to the Cassandra node.
  • Password – Password of the user that can connect to the Cassandra node.
  • Use TLS/SSL? – Whether the database connection should be made over SSL.

Setting JMX Connections

The agent requires JMX access to individual nodes in the cluster to gather most operational metrics.

  • Node Connections – IP/Hostname for each node in the cluster to be monitored must be added to this list. Values for the columns Port, Username, and Password can be specified for each node if they differ. If no value is provided for these, values will be taken from the default fields under JMX Connections (see below). Check “Use as Host Alias?” to indicate that the value provided for “IP or Hostname” should be used as an alias for the discovered hostname for the respective node. Secondary Property Lists are global and can be shared between agents. To create a new property list for a different cluster, clone an existing list and then edit and save it as the selected list for that agent.
  • Default Port – JMX listening port for the Cassandra node. This value will be used as the default value in the Node Connections list if no other value is provided. Default is 7199.
  • Default Username – JMX User that can connect to the Cassandra node. This value will be used as the default value in the Node Connections list if no other value is provided.
  • Default Password – Password of the JMX User that can connect to the Cassandra node. This value will be used as the default value in the Node Connections list if no other value is provided.
  • Use TLS/SSL? – Whether the JMX connection should be made over SSL.

Setting Collection Periods

The Collection Periods field list in the agent properties is used to set the sample frequencies. A collection can be turned off by setting its period to 0. The defaults are set based on the type of data being collected for relevancy.

  • Availability – Controls the query interval for node availability collections.
  • Cluster and Node – Controls the query interval for cluster and node collections over the database driver.
  • Keyspace and Table – Controls the query interval for keyspace and table collections over the database driver.
  • JMX Cluster – Controls the query interval for cluster metrics over JMX.
  • JMX Keyspace – Controls the query interval for keyspace metrics over JMX. If Max Tables is set to zero keyspace metrics are not collected.
  • JMX Table – Controls the query interval for table metrics over JMX. If Max Tables is set to a positive value, metrics controlled by this period are collected for the largest tables only. If Max Tables is equal to zero this period is not used. If Max Tables is set to -1 this period is used to collect metrics from all tables.
  • JMX Table Size – Controls the query interval for table size metrics over JMX. If Max Tables is set to a positive value, metrics controlled by this period are collected for all tables for the purpose of determining the largest tables on the cluster. If Max Tables is less than or equal to zero this period is not used.
  • Slow Query – Controls the query interval for collection of the DSE Slow Query log.
  • Trace – Controls the query interval for collection of the trace log. Note that tracing must be enabled on the Cassandra instance(s) separately for data to be collected.
  • JVM – Controls the query interval for the JVM collections over JMX.

Setting Options

  • Cluster Alias - The name under which the Cassandra cluster will be known in Foglight. Normally this field should be left blank, in which case the cluster’s actual name is used. In environments where two or more clusters have the same name, setting aliases for at least all but one of the clusters will prevent their monitoring data from merging and/or overwriting each other in Foglight.
  • Host Aliases - Aliases can optionally be added for Cassandra nodes in case hostnames as discovered by the agent differ from those used for the Hosts monitor. This Secondary Property List is a pairing of the uniquely identifying Host ID (a UUID string) and the desired hostname alias. There are two ways to determine a node’s Host ID for use in the Aliases list. For a node that has already been monitored for some time, the Host ID can be found on the Nodes Table of the Cassandra Clusters dashboard (the Host ID column is hidden by default, so it will need to be made visible via the table’s “Show columns” dialog). Host IDs can also be queried directly from a specific Cassandra instance with the following CQL:
SELECT host_id FROM system.local WHERE key='local';
  • Contact Points - Additional contact points for the database driver to assist with making an initial connection if the node specified by the main contact point is not available.
  • Max Traces/Queries - The maximum number of traces or slow query entries to collect per their respective collection periods. Limiting the number here prevents the FMS from being overwhelmed by data (particularly from traces) on very active instances.
  • Max Tables - The number of tables for which to collect full metrics. If set to 0, no keyspace or table metrics are collected. Must be a non-negative value.

Removing Monitored Databases

  1. Go to the Databases dashboard.
  2. Select the databases you wish to remove.
  3. Click the Settings button, then click ok.

Note: Doing this will remove the monitoring agents as well as the historical data already collected. If you wish to delete only the agents, you can do that on the Administration > Agents > Agent Status page. Because the Databases dashboard only shows databases which are being actively monitored, you will only be able to view these databases by going directly to the Cassandra dashboard.

Roles

Two roles, Cassandra User and Cassandra Administrator, are installed with the cartridge. Viewing Cassandra dashboards requires that a user be assigned one of these or have the core Administrator role. The Set Trace Probability function on the Traces Sample dashboard requires the Cassandra Administrator role.