Gateway Performance Testing User Guide

litmus gateway performance testing measures how much data collection a litmus edge (le) gateway sustains it runs a modbus simulator as the data source, deploys devicehub devices to your gateway through litmus gateway functionality testing, and counts the messages the gateway publishes to a nats server while it samples the gateway's cpu and memory usage you run it from a browser and export the results as a report the tool answers one question at how many devices and tags does your gateway stop keeping up? a run produces a table of device and tag combinations with the expected message rate, the actual message rate, the delivery success rate, and the cpu and memory cost at each combination feature overview automated sweep a devices by tags grid that deploys, measures, and records each combination without supervision, with a duration estimate before you start and live progress while it runs manual nats counter a message counter for a scenario you deployed yourself, which records the run at a message target or when you stop it custom devicehub page a builder for modbus tcp scenarios of up to 100 devices with up to 1 million tags per device bundled modbus simulator a modbus tcp server with 65,536 holding registers refreshed once per second, spread across 10 ports so polled tags always see changing data environment checks reachability indicators for the simulator, gateway functionality testing, and nats, each showing the endpoint it checked live resource sampling gateway cpu and memory usage captured once per second during every measurement, recorded as the value at the end of the window, the average, and the peak success aware counting separate tallies for successful polls and failed polls, so a gateway that publishes failures does not read as throughput gateway cleanup a scan and a clean for everything the testing tools created on the gateway persistent run history completed runs stored on a volume, kept across container restarts report exports an xlsx report in the litmus gateway performance format, and a self contained html dashboard themes a light and dark mode switch in the header before you begin important run the containers on a separate machine from the gateway under test, not on the gateway itself running them on the gateway competes with the gateway for cpu and memory, which skews the results you are trying to measure before you begin, make sure you have the following le installed on the gateway download the le iso from the litmus central portal https //portal litmus io and install it following the iso installation on an industrial pc (ipc) docid\ bldxzrjtg sgsxjgyt2sk guide the docker compose yml file download it from the litmus central portal it pins the versions it shipped with, so if you also install from image tar files, take them from the same release docker engine 20 10+ on linux, or docker desktop 4 x+ with the windows subsystem for linux 2 (wsl2) backend enabled on windows docker compose v2 (the docker compose command, from the docker compose plugin package on linux) host architecture amd64/x86 64 (intel/amd) free host ports 1502 through 1511 for the modbus simulator, 4007 for the gateway functionality testing ui, 4008 for the performance testing ui, and 4222 for nats network access access to litmusedge azurecr io to pull the images (first run and every update) the pull is anonymous, so no registry sign in is needed without it, install from image tar files instead see installing without registry access a route from the gateway back to this host's lan ip address on ports 1502 through 1511 and 4222 the gateway polls the simulator and publishes to nats, so it dials in to this machine rather than the other way around outbound https from the host to the target le gateway le gateway credentials a username and password, or a bearer token, for the gateway you want to test you enter these in gateway functionality testing, not in this tool a wired network between the host and the gateway a congested or wireless link caps the measured rate before the gateway does, which makes the result a measure of your network instead of your gateway running the containers put docker compose yml in a directory of its own and run docker compose up d this pulls the images and starts four services the modbus simulator, gateway functionality testing, a nats server, and this measurement tool once they are up open http //localhost 4008 for performance testing open http //localhost 4007 for gateway functionality testing, where you connect the gateway reusing a gateway functionality testing instance you already run the compose file bundles gateway functionality testing, which owns the connection to your gateway if you already run it somewhere else, skip the bundled copy so the two instances do not compete for port 4007 and for the gateway gwt host=host docker internal docker compose up d scale gateway testing=0 use host docker internal for an instance on this machine, or a lan ip address for one elsewhere either way it has to serve on port 4007 your instance also has to be version 1 1 0 or newer that release added the endpoint the sweep uses to grow tag counts on devices that already exist against an older instance, the first tag tier of every device count fails installing without registry access if the host cannot reach litmusedge azurecr io , download two image tar files from the litmus central portal and load both before you start the stack the performance testing tar file, which holds the measurement tool and the modbus simulator the gateway functionality testing tar file, which holds the tool that owns the connection to your gateway take the version your docker compose ym l pins, or skip this file if you reuse an instance you already run docker load < litmus gateway performance 1 0 0 tar gz docker load < litmus gateway functionality 1 1 0 tar gz docker compose up d the images arrive tagged exactly as the compose file expects, so nothing needs editing connecting to your gateway performance testing drives your gateway through gateway functionality testing, so connect there first the litmus gateway functionality testing user guide covers that tool in full to connect to your gateway open http //localhost 4007 in a browser enter the gateway's ip address or hostname in gateway address enter the ip address or hostname only, because https is assumed select an authentication method ( username / password , api token , or first login ) and fill in the matching fields leave allow self signed certificate selected if the gateway uses a self signed certificate click connect return to http //localhost 4008 and confirm the connection pill in the header shows the gateway as connected the environment card at the top of the performance testing home page shows whether the tool reaches the modbus simulator , gateway testing , and nats , along with the exact endpoint it checked all three need to be up before you measure anything running an automated sweep to run an automated sweep on the home page, find the automated sweep devices x tags card in simulator host, as seen from the le , enter this machine's ip address on the network the gateway can reach leave base port at 1502, the first port of the simulator's range in poll (s) , enter how often each tag is polled, in seconds this is the divisor in the expected rate devices x tags / poll under device counts , select the device counts to test under tags per device , select the tag counts to test counts of h and above are slow to deploy and show a warning in nats host, as seen from the le , enter the same machine ip address you used in step 2 leave it blank to reuse the nats host already saved in gateway functionality testing in warmup (s) , enter how long to idle after the devices start reporting, before measuring the default of 15 gives newly added devices time to ramp up in window (s) , enter the length of the measured window the default of 10 matches the report's per 10 second column warmup and window each accept up to 120 seconds click start sweep tip read the duration estimate before you start a sweep of five device counts across three tag tiers takes roughly 18 minutes, and adding the 100 device count or a tag tier above 10000 can push a run into hours the sweep runs one device count at a time, with device counts on the outside and tag counts on the inside, both ascending for each device count it cleans the gateway, deploys the nats connector, and builds that many devices at the lowest tag tier measures a checkpoint, then grows the tags on those same devices to the next tier and measures again, until the tiers run out measuring a checkpoint means waiting for the first message to arrive, waiting for the rate to approach the expected rate, idling for the warmup period, then counting messages for the length of the window while sampling the gateway's cpu and memory usage once per second each checkpoint records one row in the test runs table moving to the next device count forces a clean, because the tag tiers have to restart at the bottom and registers can only be added to a device, never taken away a checkpoint the gateway cannot sustain records a zero row and the sweep carries on while the sweep runs, the card shows the current phase, a progress bar, the live message rate, and the failed poll count click stop to end the sweep early rows already recorded are kept running a single measurement by hand use the manual path when you want to measure one specific configuration, or a scenario you deployed yourself, instead of a grid deploying a scenario from the custom devicehub page to deploy a scenario by hand click custom devicehub in the header in simulator host, as seen from the litmus edge , enter this machine's ip address, and leave base port at 1502 in devices (max 100) , enter how many devicehub devices poll the simulator this is what drives the load in tags/device (max 1m) , enter how many tags each device polls tags map to holding registers from address 0 in poll (s) , enter the polling interval in seconds check the summary at the top of the page, which shows the total tag count and the expected message rate click save + deploy to build the scenario and deploy it to the gateway, or save scenario to save it without deploying devices are named gw test custom so the gateway functionality testing cleaner removes them later deployment time tracks the device count at roughly 2 4 seconds per device, not the tag count tags are effectively free up to about 10000 per device, above which they add real time deploy the nats connector from the integrations test in gateway functionality testing before you count anything without it, the gateway polls the simulator but publishes nowhere counting messages with the manual nats counter to count messages against a deployed scenario on the home page, find the manual nats counter card leave nats host blank to use the nats server in this stack, or enter the host this tool should connect to this is where the tool listens, not where the gateway publishes in port , enter the nats port the default is 4222 in subject , enter the topic the gateway's nats connector publishes to the integrations test deploys the connector with gw test in target msgs , enter how many successful messages to count before the run records itself enter 0 to count until you click stop under deployed scenario (sets the expected rate) , confirm devices , tags/device , and poll (s) match what is deployed on the gateway these fields deploy nothing they only compute the expected rate and the success percentage for the recorded row, and they prefill from your last custom devicehub build or sweep checkpoint click start the card shows the live message rate, the count in the last 10 seconds, the running total, the failed poll count, the elapsed time, and the gateway's cpu and memory usage when the target is reached, the run records itself along with the cpu and memory usage at that exact moment, plus the average and peak over the run click stop to record a run early you cannot start the manual counter while a sweep is running reading your results every finished run lands in a table on the home page sweep checkpoints go in the test runs table manual counter runs go in the separate manual counter runs table below it, because they measure to a message target rather than over a fixed window the test runs table has the following columns time (utc) when the measurement window ended devices how many devicehub devices were deployed tags/dev how many tags each device polled poll (s) the polling interval expected msg/s the theoretical rate, calculated as devices x tags / poll messages successful messages counted during the window elapsed (s) the measured window length actual msg/s messages divided by elapsed time success % actual rate as a percentage of the expected rate cpu % and cpu peak gateway processor usage during the window, at the end and at its highest ram % and ram peak gateway memory usage during the window, at the end and at its highest note a message counts toward the rate and the target only if its payload reports that the poll succeeded, with "success" true messages carrying "success" false , or no flag at all, are counted separately as failed polls and shown next to the totals a gateway that keeps publishing while its reads fail therefore reads as a shortfall rather than as throughput three results are worth interpreting carefully a success rate well above 100% usually means devices from an earlier run are still deployed and publishing clean the gateway and measure again a row of zeros with a note means no successful messages arrived within the timeout that checkpoint is above what your gateway sustains, which is the finding you are looking for a row whose note warns that the nats counter lost its connection undercounts by an unknown amount, so it is not a measurement of your gateway's ceiling the counter reconnects on its own and the sweep continues, but run that combination again before you read anything into it exporting and submitting your report the test runs card has three actions export xlsx downloads the run history in the litmus gateway performance report format the file contains a gateway details block (make, model, processor, memory, and le version, all pulled live from the connected gateway), one row per run, summary figures, and charts export html report downloads a self contained html dashboard of the same data, useful for sharing without a spreadsheet application clear deletes the run history to submit your report run the sweep, or the manual measurements, that cover the configurations you want on record click export xlsx and check the gateway details block at the top of the file if any field is blank, confirm the gateway is still connected in gateway functionality testing and export again email the xlsx file to gatewaytesting\@litmus io send one gateway model per email and use the subject line gateway performance report \<your company> \<gateway make and model> in the body, include your name, company, and contact email address the gateway's make, model, and part number the le version you tested the device and tag counts you covered, and any you skipped wait for litmus to confirm receipt by email a member of the litmus industrial team then reviews the results and follows up from their own email address you can send the performance report on its own, or attach it to a gateway self certification submission for the self certification process and the functionality report it uses, see the gateway functionality testing user guide docid\ qajxt7nulxzkombrafqly cleaning up the gateway the environment card has two cleanup controls scan le test leftovers lists the devices, connectors, and processors the testing tools created on the gateway clean le removes them caution clean le removes every device and connector the testing tools created on the gateway, including the nats connector run it between test campaigns rather than mid sweep, because the sweep manages its own cleanup an automated sweep cleans the gateway at the start of each device count, and again at the end of the run clean by hand when you deployed a scenario from the custom devicehub page, or when a run ended unexpectedly and left devices behind removing the tools from your host clean the gateway first, then stop the stack to stop the containers and keep your run history docker compose down to remove the containers, the images, and the volumes in one step docker compose down rmi all v the v flag deletes the volumes, which means your run history and saved scenarios go with them export any report you want to keep before you run it both commands act only on this stack and leave unrelated containers and images on the host untouched updating the compose file pins the version it shipped with, so docker compose pull on its own does not move you to a newer release download the docker compose yml for the version you want from the litmus central portal, replace your copy, then run docker compose pull docker compose up d to run a different version without replacing the compose file, set perf version in a env file next to docker compose yml , or inline for a single run perf version=1 0 0 docker compose up d perf version pins the measurement tool and the simulator together gwt version pins the gateway functionality testing version separately, and needs to stay at 1 1 0 or newer for the sweep to grow tag counts in place troubleshooting no messages arrive in most cases one of the host fields holds localhost the gateway is a separate machine, so localhost points it at itself put this machine's lan ip address in simulator host, as seen from the le and in nats host, as seen from the le if the addresses are correct, confirm the nats connector is deployed from the integrations test in gateway functionality testing, and confirm no firewall blocks ports 1502 through 1511 or 4222 between the gateway and this host the actual rate falls far short of the expected rate check the failed poll count next to the totals a high failed count means the gateway is publishing but its reads are failing, which points at the gateway or the network rather than at the tool a low failed count with a low rate means the gateway is not keeping up, which is a legitimate result a sweep or run disappeared after a restart completed rows persist, but a measurement in flight does not, because it is held in memory the gateway keeps polling and publishing after the restart, so clean the gateway and start the sweep again port 4007 is already in use another gateway functionality testing instance is running skip the bundled copy as described in reusing a gateway functionality testing instance you already run a deployment with a very high tag count takes a long time tag counts of 100000 and above add minutes per device, because tags are created in batches of 1,000 and the gateway slows down as its tag table grows this is expected let it finish, or select a lower tag tier the environment indicators show a service as down run docker compose ps to confirm all four services are up, and docker compose logs to see what a failed service reported