Gateway Performance Testing User Guide
Litmus Gateway Performance Testing measures how much data collection a Litmus Edge (LE) gateway sustains. It runs a Modbus simulator as the data source, deploys DeviceHub devices to your gateway through Litmus Gateway Functionality Testing, and counts the messages the gateway publishes to a NATS server while it samples the gateway's CPU and memory usage. You run it from a browser and export the results as a report.
The tool answers one question: at how many devices and tags does your gateway stop keeping up? A run produces a table of device and tag combinations with the expected message rate, the actual message rate, the delivery success rate, and the CPU and memory cost at each combination.
Feature overview
- Automated sweep: A devices-by-tags grid that deploys, measures, and records each combination without supervision, with a duration estimate before you start and live progress while it runs.
- Manual NATS counter: A message counter for a scenario you deployed yourself, which records the run at a message target or when you stop it.
- Custom DeviceHub page: A builder for Modbus TCP scenarios of up to 100 devices with up to 1 million tags per device.
- Bundled Modbus simulator: A Modbus TCP server with 65,536 holding registers refreshed once per second, spread across 10 ports so polled tags always see changing data.
- Environment checks: Reachability indicators for the simulator, Gateway Functionality Testing, and NATS, each showing the endpoint it checked.
- Live resource sampling: Gateway CPU and memory usage captured once per second during every measurement, recorded as the value at the end of the window, the average, and the peak.
- Success-aware counting: Separate tallies for successful polls and failed polls, so a gateway that publishes failures does not read as throughput.
- Gateway cleanup: A scan and a clean for everything the testing tools created on the gateway.
- Persistent run history: Completed runs stored on a volume, kept across container restarts.
- Report exports: An .xlsx report in the Litmus gateway performance format, and a self-contained HTML dashboard.
- Themes: A light and dark mode switch in the header.
Before you begin
Important: Run the containers on a separate machine from the gateway under test, not on the gateway itself. Running them on the gateway competes with the gateway for CPU and memory, which skews the results you are trying to measure.
Before you begin, make sure you have the following:
- LE installed on the gateway: download the LE ISO from the Litmus Central Portal and install it following the ISO Installation on an Industrial PC (IPC) guide.
- The docker-compose.yml file: download it from the Litmus Central Portal. It pins the versions it shipped with, so if you also install from image tar files, take them from the same release.
- Docker Engine 20.10+ on Linux, or Docker Desktop 4.x+ with the Windows Subsystem for Linux 2 (WSL2) backend enabled on Windows.
- Docker Compose v2 (the docker compose command, from the docker-compose-plugin package on Linux).
- Host architecture: amd64/x86_64 (Intel/AMD).
- Free host ports: 1502 through 1511 for the Modbus simulator, 4007 for the Gateway Functionality Testing UI, 4008 for the Performance Testing UI, and 4222 for NATS.
- Network access:
- Access to litmusedge.azurecr.io to pull the images (first run and every update). The pull is anonymous, so no registry sign-in is needed. Without it, install from image tar files instead. See Installing without registry access.
- A route from the gateway back to this host's LAN IP address on ports 1502 through 1511 and 4222. The gateway polls the simulator and publishes to NATS, so it dials in to this machine rather than the other way around.
- Outbound HTTPS from the host to the target LE gateway.
- LE gateway credentials: a username and password, or a bearer token, for the gateway you want to test. You enter these in Gateway Functionality Testing, not in this tool.
- A wired network between the host and the gateway. A congested or wireless link caps the measured rate before the gateway does, which makes the result a measure of your network instead of your gateway.
Running the containers
Put docker-compose.yml in a directory of its own and run:
This pulls the images and starts four services: the Modbus simulator, Gateway Functionality Testing, a NATS server, and this measurement tool. Once they are up:
- Open http://localhost:4008 for Performance Testing.
- Open http://localhost:4007 for Gateway Functionality Testing, where you connect the gateway.
Reusing a Gateway Functionality Testing instance you already run
The compose file bundles Gateway Functionality Testing, which owns the connection to your gateway. If you already run it somewhere else, skip the bundled copy so the two instances do not compete for port 4007 and for the gateway:
Use host.docker.internal for an instance on this machine, or a LAN IP address for one elsewhere. Either way it has to serve on port 4007.
Your instance also has to be version 1.1.0 or newer. That release added the endpoint the sweep uses to grow tag counts on devices that already exist. Against an older instance, the first tag tier of every device count fails.
Installing without registry access
If the host cannot reach litmusedge.azurecr.io, download two image tar files from the Litmus Central Portal and load both before you start the stack:
- The performance testing tar file, which holds the measurement tool and the Modbus simulator.
- The Gateway Functionality Testing tar file, which holds the tool that owns the connection to your gateway. Take the version your docker-compose.yml pins, or skip this file if you reuse an instance you already run.
The images arrive tagged exactly as the compose file expects, so nothing needs editing.
Connecting to your gateway
Performance Testing drives your gateway through Gateway Functionality Testing, so connect there first. The Litmus Gateway Functionality Testing user guide covers that tool in full.
To connect to your gateway:
- Open http://localhost:4007 in a browser.
- Enter the gateway's IP address or hostname in Gateway address. Enter the IP address or hostname only, because https is assumed.
- Select an authentication method (Username / password, API token, or First login) and fill in the matching fields.
- Leave Allow self-signed certificate selected if the gateway uses a self-signed certificate.
- Click Connect.
- Return to http://localhost:4008 and confirm the connection pill in the header shows the gateway as connected.
The Environment card at the top of the Performance Testing home page shows whether the tool reaches the Modbus Simulator, Gateway Testing, and NATS, along with the exact endpoint it checked. All three need to be up before you measure anything.
Running an automated sweep
To run an automated sweep:
- On the home page, find the Automated sweep: devices x tags card.
- In Simulator host, as seen from the LE, enter this machine's IP address on the network the gateway can reach.
- Leave Base port at 1502, the first port of the simulator's range.
- In Poll (s), enter how often each tag is polled, in seconds. This is the divisor in the expected rate: devices x tags / poll.
- Under Device counts, select the device counts to test.
- Under Tags per device, select the tag counts to test. Counts of H and above are slow to deploy and show a warning.
- In NATS host, as seen from the LE, enter the same machine IP address you used in step 2. Leave it blank to reuse the NATS host already saved in Gateway Functionality Testing.
- In Warmup (s), enter how long to idle after the devices start reporting, before measuring. The default of 15 gives newly added devices time to ramp up.
- In Window (s), enter the length of the measured window. The default of 10 matches the report's per-10-second column. Warmup and window each accept up to 120 seconds.
- Click Start sweep.
Tip: Read the duration estimate before you start. A sweep of five device counts across three tag tiers takes roughly 18 minutes, and adding the 100 device count or a tag tier above 10000 can push a run into hours.
The sweep runs one device count at a time, with device counts on the outside and tag counts on the inside, both ascending. For each device count it:
- Cleans the gateway, deploys the NATS connector, and builds that many devices at the lowest tag tier.
- Measures a checkpoint, then grows the tags on those same devices to the next tier and measures again, until the tiers run out.
Measuring a checkpoint means waiting for the first message to arrive, waiting for the rate to approach the expected rate, idling for the warmup period, then counting messages for the length of the window while sampling the gateway's CPU and memory usage once per second. Each checkpoint records one row in the Test runs table.
Moving to the next device count forces a clean, because the tag tiers have to restart at the bottom and registers can only be added to a device, never taken away.
A checkpoint the gateway cannot sustain records a zero row and the sweep carries on. While the sweep runs, the card shows the current phase, a progress bar, the live message rate, and the failed poll count. Click Stop to end the sweep early. Rows already recorded are kept.
Running a single measurement by hand
Use the manual path when you want to measure one specific configuration, or a scenario you deployed yourself, instead of a grid.
Deploying a scenario from the Custom DeviceHub page
To deploy a scenario by hand:
- Click Custom DeviceHub in the header.
- In Simulator host, as seen from the Litmus Edge, enter this machine's IP address, and leave Base port at 1502.
- In Devices (max 100), enter how many DeviceHub devices poll the simulator. This is what drives the load.
- In Tags/device (max 1M), enter how many tags each device polls. Tags map to holding registers from address 0.
- In Poll (s), enter the polling interval in seconds.
- Check the summary at the top of the page, which shows the total tag count and the expected message rate.
- Click Save + Deploy to build the scenario and deploy it to the gateway, or Save scenario to save it without deploying.
Devices are named gw-test-custom-* so the Gateway Functionality Testing cleaner removes them later. Deployment time tracks the device count at roughly 2.4 seconds per device, not the tag count. Tags are effectively free up to about 10000 per device, above which they add real time.
Deploy the NATS connector from the Integrations test in Gateway Functionality Testing before you count anything. Without it, the gateway polls the simulator but publishes nowhere.
Counting messages with the manual NATS counter
To count messages against a deployed scenario:
- On the home page, find the Manual NATS counter card.
- Leave NATS host blank to use the NATS server in this stack, or enter the host this tool should connect to. This is where the tool listens, not where the gateway publishes.
- In Port, enter the NATS port. The default is 4222.
- In Subject, enter the topic the gateway's NATS connector publishes to. The Integrations test deploys the connector with gw-test.
- In Target msgs, enter how many successful messages to count before the run records itself. Enter 0 to count until you click Stop.
- Under Deployed scenario (sets the expected rate), confirm Devices, Tags/device, and Poll (s) match what is deployed on the gateway. These fields deploy nothing. They only compute the expected rate and the success percentage for the recorded row, and they prefill from your last Custom DeviceHub build or sweep checkpoint.
- Click Start.
The card shows the live message rate, the count in the last 10 seconds, the running total, the failed poll count, the elapsed time, and the gateway's CPU and memory usage. When the target is reached, the run records itself along with the CPU and memory usage at that exact moment, plus the average and peak over the run. Click Stop to record a run early.
You cannot start the manual counter while a sweep is running.
Reading your results
Every finished run lands in a table on the home page. Sweep checkpoints go in the Test runs table. Manual counter runs go in the separate Manual counter runs table below it, because they measure to a message target rather than over a fixed window.
The Test runs table has the following columns:
- Time (UTC): When the measurement window ended.
- Devices: How many DeviceHub devices were deployed.
- Tags/dev: How many tags each device polled.
- Poll (s): The polling interval.
- Expected msg/s: The theoretical rate, calculated as devices x tags / poll.
- Messages: Successful messages counted during the window.
- Elapsed (s): The measured window length.
- Actual msg/s: Messages divided by elapsed time.
- Success %: Actual rate as a percentage of the expected rate.
- CPU % and CPU peak: Gateway processor usage during the window, at the end and at its highest.
- RAM % and RAM peak: Gateway memory usage during the window, at the end and at its highest.
Note: A message counts toward the rate and the target only if its payload reports that the poll succeeded, with "success": true. Messages carrying "success": false, or no flag at all, are counted separately as failed polls and shown next to the totals. A gateway that keeps publishing while its reads fail therefore reads as a shortfall rather than as throughput.
Three results are worth interpreting carefully:
- A success rate well above 100% usually means devices from an earlier run are still deployed and publishing. Clean the gateway and measure again.
- A row of zeros with a note means no successful messages arrived within the timeout. That checkpoint is above what your gateway sustains, which is the finding you are looking for.
- A row whose note warns that the NATS counter lost its connection undercounts by an unknown amount, so it is not a measurement of your gateway's ceiling. The counter reconnects on its own and the sweep continues, but run that combination again before you read anything into it.
Exporting and submitting your report
The Test runs card has three actions:
- Export xlsx: Downloads the run history in the Litmus gateway performance report format. The file contains a gateway details block (make, model, processor, memory, and LE version, all pulled live from the connected gateway), one row per run, summary figures, and charts.
- Export HTML report: Downloads a self-contained HTML dashboard of the same data, useful for sharing without a spreadsheet application.
- Clear: Deletes the run history.
To submit your report:
- Run the sweep, or the manual measurements, that cover the configurations you want on record.
- Click Export xlsx and check the gateway details block at the top of the file. If any field is blank, confirm the gateway is still connected in Gateway Functionality Testing and export again.
- Email the .xlsx file to [email protected]. Send one gateway model per email and use the subject line Gateway performance report: <your company> - <gateway make and model>. In the body, include:
- Your name, company, and contact email address.
- The gateway's make, model, and part number.
- The LE version you tested.
- The device and tag counts you covered, and any you skipped.
- Wait for Litmus to confirm receipt by email. A member of the Litmus industrial team then reviews the results and follows up from their own email address.
You can send the performance report on its own, or attach it to a gateway self-certification submission. For the self-certification process and the functionality report it uses, see the Gateway Functionality Testing User Guide.
Cleaning up the gateway
The Environment card has two cleanup controls:
- Scan LE test leftovers: Lists the devices, connectors, and processors the testing tools created on the gateway.
- Clean LE: Removes them.
Caution: Clean LE removes every device and connector the testing tools created on the gateway, including the NATS connector. Run it between test campaigns rather than mid-sweep, because the sweep manages its own cleanup.
An automated sweep cleans the gateway at the start of each device count, and again at the end of the run. Clean by hand when you deployed a scenario from the Custom DeviceHub page, or when a run ended unexpectedly and left devices behind.
Removing the tools from your host
Clean the gateway first, then stop the stack. To stop the containers and keep your run history:
To remove the containers, the images, and the volumes in one step:
The -v flag deletes the volumes, which means your run history and saved scenarios go with them. Export any report you want to keep before you run it. Both commands act only on this stack and leave unrelated containers and images on the host untouched.
Updating
The compose file pins the version it shipped with, so docker compose pull on its own does not move you to a newer release. Download the docker-compose.yml for the version you want from the Litmus Central Portal, replace your copy, then run:
To run a different version without replacing the compose file, set PERF_VERSION in a .env file next to docker-compose.yml, or inline for a single run:
PERF_VERSION pins the measurement tool and the simulator together. GWT_VERSION pins the Gateway Functionality Testing version separately, and needs to stay at 1.1.0 or newer for the sweep to grow tag counts in place.
Troubleshooting
No messages arrive. In most cases one of the host fields holds localhost. The gateway is a separate machine, so localhost points it at itself. Put this machine's LAN IP address in Simulator host, as seen from the LE and in NATS host, as seen from the LE. If the addresses are correct, confirm the NATS connector is deployed from the Integrations test in Gateway Functionality Testing, and confirm no firewall blocks ports 1502 through 1511 or 4222 between the gateway and this host.
The actual rate falls far short of the expected rate. Check the failed poll count next to the totals. A high failed count means the gateway is publishing but its reads are failing, which points at the gateway or the network rather than at the tool. A low failed count with a low rate means the gateway is not keeping up, which is a legitimate result.
A sweep or run disappeared after a restart. Completed rows persist, but a measurement in flight does not, because it is held in memory. The gateway keeps polling and publishing after the restart, so clean the gateway and start the sweep again.
Port 4007 is already in use. Another Gateway Functionality Testing instance is running. Skip the bundled copy as described in Reusing a Gateway Functionality Testing instance you already run.
A deployment with a very high tag count takes a long time. Tag counts of 100000 and above add minutes per device, because tags are created in batches of 1,000 and the gateway slows down as its tag table grows. This is expected. Let it finish, or select a lower tag tier.
The environment indicators show a service as down. Run docker compose ps to confirm all four services are up, and docker compose logs to see what a failed service reported.
Known Issues
NATS message counts are unreliable above about 50,000 messages per second: the measured rate plateaus at roughly 50–60k msg/s regardless of load, so success % at higher rates understates what the gateway actually published.