Validation

The Energy Framework has a fairly extensive set of validation tests, for which the unit tests also serve as good example code.

The tests were also run under valgrind and there are no reported memory errors or leaks in Energy Framework modules.

valgrind --leak-check=yes --show-reachable=yes

The validation tests are in contrib/power/tests. Each directory contains a README describing the tests and a directory valid/ containing reference output.

Most omnetpp.ini files define several runs. In order to match the *.vec output in the directory valid/, it's necessary to run the tests individually. (This is because vector ID's are assigned sequentially across a multi-run execution.) In order to match the omnetpp.sca output in the directory valid/, it's also necessary to run the all the tests in order. This is because omnetpp.sca accumulates data over multiple runs.

One way to do this easily is via the unix shell (may need to be modified for different shells). For example:

cd tests/AFTest
for i in 1 2; do (see # of runs in .ini or README)
./AFTest -r $i
done

The output can be compared with the reference output in the same way:

for f in *.vec *.sca; do
diff $f valid/$f
done

By default, the tests run in Cmdenv. In most cases, the output .vec files are omnetpp_$i.vec; in a few cases more mnemonic names are given.

The scripts scripts/convertvec.py script is used to convert .vec files to a directory of per-host time series data. For 3-way comparisons between BatteryModule2.0, the compatibility modules, and the Energy Framework Nic80211, the time series data is least formatted and easiest to compare. See framework documentation and README's for details.

The tests in deviceTest, deviceTestMulti, deviceTestAccts, and deviceTestAll are primarily systematic unit tests.

The tests in AFOrig, AFTest, and 80211Test use BurstApplLayer and 80211Nic to do a 3-way comparison of the output of the Battery Module2.0, the compatibility module deviceAF, and the new
Nic80211Battery.

More information about each test directory is included below:

Device Test

Fundamental test case, used for basic unit testing. The Host includes only the Battery and the DutyCycleSimple device.

The tests go through various relations between events and parameters (i.e. CURRENT messages at duty cycle boundary, resolution interval updates of battery capacity, host failure). Also checks various combinations of data collection parameters. See below and omnetpp.ini for details.

Run each test individually in order.

./deviceTest -r $i

Compare output files with the valid/ directory. Note that Run 13 and 14 do not have an output vector, only omnetpp.sca.

for f in *.vec *.sca ; do
diff $f valid/$f
done

DETAILS

Run 1 - resolution < cycle, aligned to cycle boundary, battery fails at cycle boundary

Run 2 - resolution < cycle, not aligned to cycle boundary, battery fails at cycle boundary

Run 3 - resolution >> cycle (not a sensible configuration), note that failure is detected late

Run 4 - initial capacity < 1.0, battery fails mid cycle

Run 5 - sim-time ends before battery fails, not aligned to resolution

Run 6 - sim-time ends before battery fails, aligned to resolution

Run 7 - time series with publishTime only, time > resolution

Run 8 - time series with publishTime only, time < resolution

Run 9 - time series with publishDelta only, delta large

Run 10 - time series with publishDelta only, delta small

Run 11 - check estimate vector

Run 12 - check estimate vector, initial capacity < 1.0 (matches residual)

Run 13 - turn off detail and time series

Run 14 - turn off time series

DeviceTestMulti

Similar to deviceTest, but the Host now has two DutyCycleSimple devices, each with different parameters.

Run each test individually in order.

./deviceTest -r $i

Compare output files with the valid/ directory.

for f in *.vec *.sca ; do
diff $f valid/$f
done

DETAILS

The 2nd device has period half of the 1st device.

Run 1 - 1st and 2nd devices turn on at the same time, battery fails

Run 2 - 2nd device turns off while 1st device is on

Run 3 - 1st device turns off while 2nd device is off, battery fails

Run 4 - 2nd device turns on while first device is off

DeviceTestAccts

Similar to deviceTest, but uses the two-phase DutyCycle device, which assigns each of the two ON periods to a different activity. Both wakeups are assigned to a third activity.

Run each test individually in order.

./deviceTest -r $i

Compare output files with the valid/ directory.

for f in *.vec *.sca ; do
diff $f valid/$f
done

DETAILS

Run 1 - sim-time < lifetime

Run 2 - sim-time > lifetime

DeviceTestAll

Combines deviceTestMulti and deviceTestAll. Uses three instances of the DeviceDuty and DeviceDutySimple devices, one of which has no continuous CURRENT draw, only discrete ENERGY draw from the Battery.

Run each test individually in order.

./deviceTest -r $i

Compare output files with the valid/ directory.

for f in *.vec *.sca ; do
diff $f valid/$f
done

DETAILS

Run 1 - just the one run

AFOrig

This test uses the original BatteryModule2.0 written by Anna Förster at http://www.inf.unisi.ch/phd/foerster/downloads.html.

Running the executable generates an omnetpp.sca containing the time spent in each state and a directory battery/, which contains files with time series of residual capacity for each host.

./AFOrig

There seem to be two issues with the original Battery Module2.0 code. The first is that the sum of the times listed in omnetpp.sca for each node does not match the sim-time-limit. It also appears that the per-host time series data is written to files numbered -1 to 18, rather than 0 to 19. These issues are not present in the compatibility modules tested in ../AFTest.

The output is intended to be compared to the Energy Framework's compatibility modules in directory AFTest.

AFTest

Tests the DeviceAF and BatteryStatsAF modules. These modules use the Energy Framework to provide compatibility with the BatteryModule2.0

Because the deviceAF operates like the BatteryModule2.0, its output should be the same as the AFOrig (the original BatteryModule2.0). Because deviceAF is part of the EnergyFramework, its output can also be compared with the Nic80211Battery, which is the Energy Framework's more detailed model.

Run each test individually in order, i.e.

./AFTest -r $i

The .vec and .sca files should match the corresponding files in the valid/ directory. The .vec outputs are named omnetpp_AFOrig.vec and omnetpp_80211.vec, indicating which other test they are to be compared with.

for f in *.vec *.sca ; do
diff $f valid/$f
done

DETAILS

Run 1 - compare output to the BatteryModule2.0 in tests/AFOrig

Note that different omnetpp.ini parameters are needed to get equivalent behavior because the Energy Framework defines battery capacity as mA-hr at a nominal voltage and uses mW-s as its base unit.

The Energy Framework uses OMNET++ scalar and vector data collection.

Comparing omnetpp.sca and tests/AFOrig/omnetpp.sca shows that the timeInTx match, but the timeInRx differ by either 0 or 1 second. This is due to an issue in BatteryModule2.0 (note that in AFOrig the total time in all states does not match the total simulation time).

To generate compatible time series data use the script:

../../scripts/convertvec.py -i omnetpp_AFOrig.vec -o batteryAFOrig

There is a minor issue in the BatteryModule2.0 where the time series data for each of the 20 hosts are written to files numbered -1 to 18, rather than 0 to 19. The output in AFOrig/battery/node(i).power should match batteryAFOrig/node(i-1).power.

Run 2 - compare output to the 802.11 NIC of the Energy Framework

The omnetpp.sca files do not match at all, as the 802.11 NIC records additional data and uses a quite different format. The Tx times are the same. Because Nic80211Battery distinguishes between Rx and Idle, so its necessary to compare sums.

The convertvec.py script can be used to generate time series data:

../../scripts convertvec.py -i omnetpp_80211.vec -o battery80211 -p 4

The output is comparable (but not identical) to the Nic80211Battery of the energy framework, because it uses the Nic, rather than radio state for its information and generates a quite different sequence of floating point operations. . The argument -p 4 reduces the output to 4 significant digits (which is really still too many given the crude battery model).

for f in battery80211/*.power
do

diff $f ../80211Test/$f
done

80211Test

Tests of the implementation of the Nic80211Battery for the Energy Framework. These are the primary tests of the Energy Framework.

The first seven runs are simple unit tests. The last three are more complex, see details below.

Run each test individually in order, i.e.

./80211Test -r $i

The .vec and .sca files should match the corresponding files in the valid/ directory. Output of run 8 can also be compared with output from the AFTest compatibility modules. Output of run 9 and 10 can be compared with each other.

for f in *.vec *.sca ; do
diff $f valid/$f
done

Note that in some cases, the battery consumption parameters are unrealistically chosen to implement a test case. See
networks/80211Battery for a an 802.11 simulation.

DETAILS

Run 1-3 are simple two-host single-message tests that check that the correct CURRENT messages are generated.

Run 1 - nodes are in range: each host sends and receives one BCAST and one REPLY

Run 2 - nodes are out of range: each host transmits one BCAST and receives nothing

Run 3 - nodes are error range: each host transmits one BCAST and receives an ERROR

Manually, it is easiest to check that the times are correct. For reference these are (from instrumented Mac80211):

broadcast 0.00086

rts       0.00028
cts       0.00024
data      0.00086
ack       0.00025

Run 4-7 - are simple three-host bursty traffic scenarios that check battery depletion and host failure notification by sending very long bursts and comparing the total number of responses to the burst. All runs show that in/outbound frames are properly halted at the NIC; runs 5 and 7 also show that the appl layer cancels pending burst.

Run 4 - all bursts run to completion
Run 5 - host 2 fails halfway through 1st burst (host 0's)
Run 6 - host 2 fails halfway through 2nd (own) burst
Run 7 - high idleCurrent; hosts fail in quiescent network

Run 8 - compare output to tests/AFTest

The point of run 8 is to compare the output Nic80211Battery with the output of the compatibility modules from tests/AFTest (run 2). The output should also match the output in the valid/ directory.

Note that the omnetpp.vec output for run 8 is written to
omnetpp_80211.vec.

These cannot be compared directly, since the Nic80211Battery records additional data. In the omnetpp.sca file, the Tx times are the same, but because the NIC80211Battery distinguishes between Rx and Idle, the sum of these times must be compared to the Rx time in the AFTest output.

Because the test is configured to use the same current for idle and recv, the total energy consumed at each node should be the same.

The convertvec.py script can be used to generate time series data:

../../scripts/convertvec.py -i omnetpp_80211.vec -o battery80211 -p 4

Each file battery_80211/node(i).power can then be compared with the corresponding file AFTest/battery_80211/node(i).power. The data still display a few variations at 4 significant digits. This is because the battery-aware 802.11 module generates a quite different sequence of floating point operations as it depletes the battery. (Given the very crude battery model, it is clear that meaningful data is much less than 4 significant digits in any case.)

for f in battery80211/*.power ; do
diff $f ../AFTest/$f
done

Run 9-10 - realistic hardware parameters and sanity check

Run the burstApplLayer with burst sizes 10 and 50. It can be seen directly in omnetpp.sca that each host uses approximately 4 1/2 times more energy for SEND and RECV in the latter case. The exact ratio varies due to different RNG sequences for traffic, collisions, overlapping broadcasts, etc.

To compare the two cases graphically using gnuplot, generate output from the vector files

convertvec.py -i omnet.10.vec -o out_10 -c "#"

convertvec.py -i omnet.50.vec -o out_50 -c "#"

The -c "#" is the comment character for gnuplot (so that it will ignore column headers in the output). Then plot the capacity curves for some pair of hosts, say host 3.

gnuplot
> plot "out_10/node3.power", "out_50/node3.power"

The plots are time (x-axis) vs capacity (y-axis).

There are clear differences between the two plots, in addition to the obviously greater energy consumption of the larger burst.

The disparity between the publishTime (250ms) and the resolution (1s) is visible (it's not really a sensible configuration; the one in networks/80211Battery is more reasonable). When the channel is idle, there are up to four data points published between updates. This is particularly obvious at the end, once both bursts are complete, but there are also some quiescent periods in the middle of the lower traffic scenario. It's also clear that the longer burst also takes a little longer to complete.