In order to explain the design of a system that takes advantage of the power of Qnet by performing distributed processing, consider a multiprocessor hardware configuration that's suitable for a typical telecom box.
This configuration has a generic controller card and several data cards to start with. These cards
are interconnected by a high-speed transport (HST) bus.
The controller card configures the box by communicating with the data cards, and establishes/enables
data transport in and out of the box (i.e., data cards) by routing packets.
The typical challenges to consider for this type of box include:
- Configuring the data cards
- Configuring the controller card
- Replacing a data card
- Enhancing reliability via multiple transport buses
- Enhancing reliability via multiple controller cards
You need several pieces of software components (along with the hardware) to build your distributed system.
Before going into further details, you might like to review the following sections from Using Qnet for Transparent Distributed Processing chapter in the QNX Neutrino User's Guide:
- Software components for Qnet networking
- Starting Qnet
- Conventions for naming nodes
Configuring the data cards
Power up the data cards to start procnto and qnet in sequence. These data cards need a minimal amount of flash memory to store the QNX Neutrino image.
In the buildfile of the data cards, you should link the directories of the data cards to the controller cards as follows:
[type=link] /bin = /net/cc0/bin [type=link] /sbin = /net/cc0/sbin [type=link] /usr = /net/cc0/usr
where cc0 is the name of the controller card.
Assuming that the data card has a console and shell prompt, try the following commands:
$ ls /net
You get a list of boards running QNX Neutrino and Qnet:
cc0 dc0 dc1 dc2 dc3
Or, use the following command on a data card:
$ ls /net/cc0
You get the following output (i.e., the contents of the root of the filesystem for the controller card):
. .inodes mnt0 tmp .. .longfilenames mnt1 usr .altboot bin net var .bad_blks dev proc xfer .bitmap etc sbin .boot home scratch
Configuring the controller card
Configure the controller card in order to access different servers running on it—either by the data cards, or by the controller card itself. Make sure that the controller card has a larger amount of flash memory than the data cards do. This flash memory contains all the binaries, data and configuration files that the applications on the data cards access as if they were on a local storage device.
Call the following API to communicate with the mqueue server by any application:
mq_open("/net/cc0/dev/mqueue/app_q", ....)
A simple variation of the above command requires that you run the following command during initialization:
$ ln -s /net/cc0/dev/mqueue /mq
Then all applications, whether they're running on the data cards or on the controller card, can call:
mq_open("/mq/app_q", ....)
Similarly, applications can even utilize the TCP/IP stack running on the controller card.
Enhancing reliability via multiple transport buses
Qnet provides design choices to improve the reliability of a high-speed transport bus, most often
a single-point of failure in such type of telecom box.
- loadbalance—no interface specified
- preferred—specify an interface, but allow failover
- exclusive—specify an interface, no failover
These selections allow you to control how data will flow via different transports.
In order to do that, first, find out what interfaces are available, by using the ifconfig command at the prompt of any card. For this example, we'll assume that the HST 0 and 1 (hs0 and hs1) interfaces are available.
| Use this command: | To select this transport: |
|---|---|
| ls /net/cc0 | Loadbalance, the default choice |
| ls /net/cc0~preferred:hs0 | Preferred. Try HST 0 first; if that fails, then transmit on HST 1. |
| ls /net/cc0~exclusive:hs0 | Exclusive. Try HST 0 first. If that fails, terminate transmission. |
You can have another economical variation of the above hardware configuration:
This configuration has asymmetric transport: a High-Speed Transport (HST) and a reliable and economical Low-Speed Transport (LST). You might use the HST for user data, and the LST exclusively for out-of-band control (which can be very helpful for diagnosis and during booting). For example, if you use generic Ethernet as the LST, you could use a bootp ROM on the data cards to economically boot—no flash would be required on the data cards.
With asymmetric transport, use of the QoS policy as described above likely becomes even more useful. You might want some applications to use the HST link first, but use the LST if the HST fails. You might want applications that transfer large amounts of data to exclusively use the HST, to avoid swamping the LST.
Redundancy and scalability using multiple controller cards
The reliability of such a telecom box also hinges on the controller card, which is a critical component and certainly a potential SPOF (single point of failure). You can increase the reliability of this telecom box by using additional controller cards.
For example, you could add another controller card for redundancy:
Once the (second) controller card is installed, the challenge is in the determination of the primary controller card. This is done by the software running on the controller cards. By default, applications on the data cards access the primary controller card. Assuming cc0 is the primary controller card, Use the following command to access this card in the /cc directory:
ln -s /net/cc0 /cc
The above indirection makes communication between data card and controller card transparent. In fact, the data cards remain unaware of the number of controller cards, or which card is the primary controller card.
Applications on the data cards access the primary controller card. In the event of failure of the primary controller card, the secondary controller card takes over. The applications on the data cards redirect their communications via Qnet to the secondary controller card.
You can also scale your resources to run a particular server application using additional controller cards. For example, if your controller card (either a SMP or non-SMP board) doesn't have the necessary resources (e.g., CPU cycles, memory), you could increase the total processor and box resources by using additional controller cards. Qnet transparently distributes the (load of) application servers across two or more controller cards.