Basic Cluster Configuration

In this guide, we will show you how to configure a basic Storm Streaming Server cluster. To create a cluster, you actually need only one server instance, as it can include the ClusterManager, an Origin application, and an Edge application. This configuration can be an excellent starting point for expanding our infrastructure, where adding additional instances will be seamless.

However, in this guide, we will use 3 instances to better illustrate the division of roles and tasks of each component.

Basic Configuration

In our example configuration, we assume having 3 Storm Streaming Server instances. The first one (IP: 192.168.0.2) is an Origin server and will also have the Cluster Manager active. The other two servers will have Edge-type applications defined (IP: 192.168.0.3 and 192.168.0.4).

ClusterManager (192.168.0.2)

Typically, the ClusterManager controller is not enabled; to activate it, we must edit the preferences.xml file in the config directory.

Sample configuration:

                        
<ClusterManager enabled="true">
  <Identity>Mother</Identity>
  <IPWhiteList></IPWhiteList>
  <Secret>qwerty*1234556</Secret>
</ClusterManager>
                    

Explanation:

IdentityIdentifier for the mother, so it is possible to recognize which instance we are dealing with. We recommend using naming like "Primary", "Secondary".
IPWhiteListAccess to the ClusterManager can be restricted to a list of defined IP addresses. If this field is empty, no restrictions apply. If we want to define an IP list that the server checks before each connection, simply enter the IP addresses separated by commas, e.g., "192.168.0.2, 192.168.0.3" (whitespace doesn't matter).
SecretThis is the password that all applications logging into the Cluster Manager must present.
Table 1. ClusterManager field explanation table

Origin Applications (192.168.0.2)

Below is an example configuration for an Origin-type application. The most important block here is StreamingClusterSettings, which defines the connection with the ClusterManager. It is possible to define more than one connection, but each must be assigned a priority (higher the number, higher the priority).

Sample configuration:

                        
<Application name="origin" type="origin">
  <StreamingClusterSettings>
    <SupervisorList>
      <ClusterManager host="192.168.0.2" port="8080" isSSL="true">
        <Priority>1</Priority>
        <Secret>qwerty*1234556</Secret>
      </ClusterManager>
      <PeerAccessSettings host="192.168.0.2" port="1935" isSSL="false"/>
    </SupervisorList>
    <CopyOnPublish>true</CopyOnPublish>
  </StreamingClusterSettings>
  ...
</Application>
                    
ClusterManager:hostHost or IP address where the ClusterManager controller is located.
ClusterManager:portPort on which the WebSocket protocol of the ClusterManager controller is available (usually 8080 or 443).
ClusterManager:isSSWhether to use SSL communication with the ClusterManager.
PriorityImportance of this connection compared to parallel connections.
SecretPassword with which the application will log into the ClusterManager (must be the same).
PeerAccessSettings:hostPublic host or IP address of this application, so other applications know where to connect.
PeerAccessSettings:portPublic port of this application, so other applications know where to connect (indicate a port with an active RTMP protocol).
PeerAccessSettings:isSSLWhether the connection should use SSL.
Table 2. Origin/Edge field explanation table

You can learn more about specifics configurations in: Origin Application Guide.

Edge Applications (192.168.0.3 & 192.168.0.4)

Below is an example configuration for an Edge-type application - it's very similar to Origin application with just a few additional parameters.

Sample configuration:

                        
<Application name="edge" type="edge">
  <StreamingClusterSettings>
    <SupervisorList>
      <ClusterManager host="192.168.0.2" port="8080" isSSL="false">
        <Priority>1</Priority>
        <Secret>qwerty*1234556</Secret>
      </ClusterManager>
    </SupervisorList>
    <CopyOnPublish>true</CopyOnPublish>
  </StreamingClusterSettings>
  ...
</Application>
                    
ClusterManager:hostHost or IP address where the ClusterManager controller is located.
ClusterManager:portPort on which the WebSocket protocol of the ClusterManager controller is available (usually 8080 or 443).
ClusterManager:isSSWhether to use SSL communication with the ClusterManager.
PriorityImportance of this connection compared to parallel connections.
SecretPassword with which the application will log into the ClusterManager (must be the same).
CopyOnPublishWhenever stream from an Origin application should be immediately copied to this Edge application, otherwise it'll wait for a first request from a viewer.
Table 3. Origin/Edge field explanation table

You can learn more about specifics configurations in: Edge Application Guide.

Redundancy

As mentioned earlier, a single server instance can only support a single ClusterManager controller. However, for each Edge and Origin application, it is possible to define connections with multiple such controllers located on different servers. This ensures that in case of a failure of the main instance, the cluster will continue to operate without interruption. When the main ClusterManager regains functionality, all applications will reconnect to it.

Sample configuration:

                        
<Application name="edge" type="edge">
  <StreamingClusterSettings>
    <SupervisorList>
      <ClusterManager host="192.168.0.2" port="8080" isSSL="false">
        <Priority>2</Priority>
        <Secret>qwerty*1234556</Secret>
      </ClusterManager>
      <ClusterManager host="192.168.0.10" port="8080" isSSL="false">
        <Priority>1</Priority>
        <Secret>1234567qwe</Secret>
      </ClusterManager>
    </SupervisorList>
    <CopyOnPublish>true</CopyOnPublish>
  </StreamingClusterSettings>
  ...
</Application>
                    
Next Step

You may want to learn more about Origin Application and Edge Application.

Support Needed?

Create a free ticket and our support team will provide you necessary assistance.