Skip to main content

nifectl abtest Command Reference

Manage A/B tests on an application.

Overview​

The abtest commands manage an A/B test: two versions of an app (variant A and variant B) run side by side and traffic is split between them. They mirror the A/B Testing tab in the nife.io dashboard.

Deploy the app with --deployment_strategy ab-testing first. Then use these commands to watch the results, change the traffic split and finish the test.

Subcommands​

SubcommandDescription
abtest statusShow the current A/B test status
abtest resultsShow the A/B test results
abtest trafficChange the traffic split between variant A and variant B
abtest configureSet the test name, duration, success metrics and starting split
abtest declare-winnerSend all traffic to the winning variant and end the test
abtest endEnd the A/B test without declaring a winner
nifectl abtest <subcommand> [flags]

Options​

-a, --app string App name to operate on (env: NIFE_APP)
-c, --config string Path to an app config file or directory containing one (env: NIFE_APP_CONFIG)
-h, --help Help for abtest

Global Options​

-t, --access-token string Nife API Access Token
-j, --json JSON output format
-V, --verbose Verbose / debug output

abtest status​

Show the current A/B test status.

Display whether an A/B test is running, the version, health and pod count of variant A and variant B, and the current traffic split, mirroring the A/B Testing tab in the nife.io dashboard.

nifectl abtest status [flags]

Options:

-a, --app string App name to operate on (env: NIFE_APP)
-c, --config string Path to an app config file or directory containing one (env: NIFE_APP_CONFIG)

abtest results​

Show the A/B test results.

Compare variant A and variant B: users, conversions, conversion rate, average latency and error rate, plus the leading variant and how statistically significant the difference is.

Use --watch to keep refreshing until you press Ctrl+C.

nifectl abtest results [flags]

Options:

-a, --app string App name to operate on (env: NIFE_APP)
-c, --config string Path to an app config file or directory containing one (env: NIFE_APP_CONFIG)
--watch Keep refreshing the results until you press Ctrl+C
--interval int Seconds between refreshes when using --watch (default 10)

abtest traffic​

Change the traffic split between variant A and variant B.

Set the percentage of traffic sent to each variant. Give --traffic-a or --traffic-b (the other one gets the rest). Prompts if you give neither.

nifectl abtest traffic [flags]

Options:

-a, --app string App name to operate on (env: NIFE_APP)
-c, --config string Path to an app config file or directory containing one (env: NIFE_APP_CONFIG)
--traffic-a int Percentage of traffic sent to variant A (0-100). Variant B gets the rest.
--traffic-b int Percentage of traffic sent to variant B (0-100). Variant A gets the rest.

Examples:

nifectl abtest traffic -a myapp --traffic-a 70
nifectl abtest traffic -a myapp --traffic-a 70 --traffic-b 30

abtest configure​

Set the test name, duration, success metrics and starting split.

Save the settings of an A/B test. The app must already be deployed with --deployment_strategy ab-testing. Give at least one flag.

nifectl abtest configure [flags]

Options:

-a, --app string App name to operate on (env: NIFE_APP)
-c, --config string Path to an app config file or directory containing one (env: NIFE_APP_CONFIG)
--test-name string Display name for the test
--duration-days int How many days the test should run
--success-metric strings A metric that decides success (repeatable), e.g. conversion
--traffic-a int Starting percentage of traffic sent to variant A (0-100). Variant B gets the rest.
--traffic-b int Starting percentage of traffic sent to variant B (0-100). Variant A gets the rest.

Examples:

nifectl abtest configure -a myapp --test-name "New checkout" --duration-days 14 \
--success-metric conversion --success-metric latency --traffic-a 80

abtest declare-winner​

Send all traffic to the winning variant and end the test.

Route 100% of traffic to variant A or variant B and end the test. The losing variant is deleted unless you pass --keep-loser.

Like the dashboard, this is blocked until the results are statistically significant (0.95). Use --force to skip that check.

nifectl abtest declare-winner <a|b> [flags]

Options:

-a, --app string App name to operate on (env: NIFE_APP)
-c, --config string Path to an app config file or directory containing one (env: NIFE_APP_CONFIG)
--keep-loser Keep the losing variant running instead of deleting it
--force Declare the winner even if the results are not statistically significant yet
-y, --yes Accept all confirmations

Examples:

nifectl abtest declare-winner b -a myapp

abtest end​

End the A/B test without declaring a winner.

End the test. Traffic stays at the current split and both variants keep running. Use declare-winner instead to pick a variant.

nifectl abtest end [flags]

Options:

-a, --app string App name to operate on (env: NIFE_APP)
-c, --config string Path to an app config file or directory containing one (env: NIFE_APP_CONFIG)
-y, --yes Accept all confirmations

Best Practices​

  • Run nifectl abtest results --watch to follow the numbers live; press Ctrl+C to stop.
  • declare-winner is blocked until the result is statistically significant (0.95), just like the dashboard. Use --force only when you have decided without it.
  • The losing variant is deleted when you declare a winner. Add --keep-loser to keep it.
  • Use abtest end to stop the test without choosing a winner — both variants keep running at the current split.

See Also​

Summary​

Nifectl Abtest brings the whole A/B testing flow — configure, compare, shift traffic and pick a winner — to the command line.