cucumber features --guess - cucumber

I'm setting up my Authlogic user session features.
And I'm in some kind of confusion.
When I run:
cucumber features
I get some red errors
Scenario: User signs in successfully # features/user_sessions.feature:21
Given a user exists with login: "sachin", password: "secret" # features/step_definitions/pickle_steps.rb:4
When I go to the login page # features/step_definitions/webrat_steps.rb:15
And I login as "sachin" with password "secret" # features/step_definitions/user_session_setps.rb:5
Then I should see "Login successful" # features/step_definitions/webrat_steps.rb:123
And I should be logged in # features/user_sessions.feature:26
Ambiguous match of "I should be logged in":
features/step_definitions/pickle_steps.rb:55:in `/^((?:(?:I|myself|me|my)|(?:(?:a|an|another|the|that) )?(?:(?:(?:(?:first|last|(?:\d+(?:st|nd|rd|th))) )?(?:user))|(?:(?:user)(?::? "(?:[^\"]|\.)*"))))) should (?:be|have) (?:an? )?((?:duplicable|readonly|nil|store[_ ]full[_ ]sti[_ ]class|new[_ ]record|equal|present|eql|marked[_ ]for[_ ]destruction|valid[_ ]password|valid[_ ]with[_ ]callbacks|logged[_ ]in|valid[_ ]without[_ ]callbacks|respond[_ ]to[_ ]without[_ ]attributes|valid|logged[_ ]out|respond[_ ]to|instance[_ ]variable[_ ]defined|admin|blank|changed|tainted|unserializable[_ ]attribute|locking[_ ]enabled|has[_ ]role|instance[_ ]of|partial[_ ]updates|kind[_ ]of|attribute[_ ]present|is[_ ]a|frozen|invalid|acts[_ ]like|method[_ ]exists|has[_ ]attribute|disable[_ ]perishable[_ ]token[_ ]maintenance|is[_ ]haml|id|created[_ ]at|updated[_ ]at|login|crypted[_ ]password|password[_ ]salt|persistence[_ ]token|login[_ ]count|last[_ ]request[_ ]at|last[_ ]login[_ ]at|current[_ ]login[_ ]at|last[_ ]login[_ ]ip|current[_ ]login[_ ]ip|roles|first[_ ]name|last[_ ]name|perishable[_ ]token|email))$/'
features/step_definitions/user_session_setps.rb:13:in `/^I should be logged in$/'
You can run again with --guess to make Cucumber be more smart about it
(Cucumber::Ambiguous)
features/user_sessions.feature:26:in `And I should be logged in'
Failing Scenarios:
cucumber features/user_sessions.feature:7 # Scenario: User is not signed up
cucumber features/user_sessions.feature:14 # Scenario: User enters wrong password
cucumber features/user_sessions.feature:21 # Scenario: User signs in successfully
And as cucumber suggests, and when I run with --guess option:
cucumber features --guess
All come green and passes.
Is this the expected behavior or error??

AFAIK, if you have 2 different step definitions that are the same, Cucumber does not know which one to run. Using the --guess flag forces Cucumber to pick the most likely one. The cucumber rake tasks use the --strict flag, and will fail if you have ambigious steps.
Essectially, you need to consider the 2 steps (one in pickle steps and one in user_session_steps) that are conflicting and make a decision as to which one to use, then delete the other one.

Consider these step definitions:
Given /Three (.*) mice/ do |disability|
# some code
end
Given /Three blind (.*)/ do |animal|
# some other code..
end
And a plain text step:
Given Three blind mice
Cucumber can’t make a decision about what Step Definition to execute, and will raise a Cucumber::Ambiguous error telling you to fix the ambiguity.
Guess mode
Running the plain text step will match the Regexp of both step definitions and raise Cucumber::Ambiguous. However,
if you run Cucumber with --guess, it will guess that you were aiming for the step definition with 2 match groups.
There is ranking logic that gets invoked when the option is turned on:
The longest Regexp with 0 capture groups always wins.
The Regexp with the most capture groups wins (when there are none with 0 groups).
If there are 2+ Regexen with the same number of capture groups, the one with the shortest overall captured string length wins.
If there are still 2+ options then an Ambiguous error is raised.
So if you try --guess with the mice above, Cucumber will pick /Three blind (.*)/, because “mice” is shorter than “blind”.
More details here

for $ rake to work properly, specify cucumber's guess mode
in lib/tasks/cucumber.rake, specify t.cucumber_opts = "--guess":
# ...
namespace :cucumber do
Cucumber::Rake::Task.new({:ok => 'test:prepare'}, 'Run features that should pass') do |t|
t.binary = vendored_cucumber_bin # If nil, the gem's binary is used.
t.fork = true # You may get faster startup if you set this to false
t.profile = 'default'
t.cucumber_opts = "--guess" # <------- add this line -------<<<
end
# ...
end
# ...

Related

How to run cucumber scenario's based on Test Case ID that is appended with the Scenario name?

I wanted to run Cucumber Feature file based on the Test case ID that scanerio name contains.
I know we can use #CucumberOptions 'features' tag and specify the line number to execute e.g "src/test/resources/Folder/myfile.feature:7:12"
This will run scenarios at line 7 and 12. But i wanted to run based on the TC ID.
Below is the feature file code
#Run
Feature: Login Functionality
Scenario: First Test Case(TC.No:1)
Given I perform action 1
Scenario: Second Test Case(TC.No:2)
Given I perform action 2
Scenario: Third Test Case(TC.No:3)
Given I perform action 3
Scenario: Fourth Test Case(TC.No:4)
Given I perform action 4
Scenario: Fifth Test Case(TC.No:5)
Given I perform action 5
All the scenario's are in a single feature.
For the feature file code above i wanted some way through which i can execute based on TC Id. E.g I only want to execute TC1,TC2 and TC5( TC id's picked up from scenario names).
There is a property file that contains the TC Id's to be executed. My code should read the file and then execute only those TC id's.
This can help me in reducing the number of automation TC's to be run.
Is it possible?
You can use the name property of #CucumberOptions or use the '-n' option if you are using the cli option. It also supports regular expressions.
To run TC.No:1 and TC.No:4 use something like this
#CucumberOptions(name = { "TC.No:1|TC.No:4" })
or
#CucumberOptions(name = { "TC.No:1","TC.No:4" })
You can get more details at this link.
As you are reading the ids from a file, the second option is the best. Use the cucumber.api.cli.Main class main() method to execute the features. You can create the options dynamically. Refer to this post.
CLI reference docs.
Not familiar with cucumber-jvm.
But, here is the general logic which should work (based on my ruby Cucumber knowledge)
In the hook, you can write the logic to under before method to get the scenario name scenario.name and then extract the TC.No. Compare the TC.No and skip if it's not part of your list.
Here is the link which will give information how to skip the scenario (use this class in the before method)
https://junit.org/junit4/javadoc/4.12/org/junit/AssumptionViolatedException.html
However, the best practice is to use the tags, it would have been easy if you had #TCId-xx tag. Still you can write a simple program that will scan all the feature files and update the scenarios with the tag based on the TC.No in the scenario name.

How to implement different data for cucumber scenarios based on environment

I have an issue with executing the cucumber-jvm scenarios in different environments. Data that is incorporated in the feature files for scenarios belongs to one environment. To execute the scenarios in different environemnts, I need to update the data in the features files as per the environment to be executed.
for example, in the following scenario, i have the search criteria included in the feature file. search criteria is valid for lets say QA env.
Scenario: search user with valid criteria
Given user navigated to login page
And clicked search link
When searched by providing search criteria
|fname1 |lname1 |address1|address2|city1|state1|58884|
Then verify the results displayed
it works fine in QA env. But to execute the same scenario in other environments (UAT,stage..), i need to modify search criteria in feature files as per the data in those environments.
I'm thinking about maintaing the data for scenarios in properties file for different environments and read it based on the execution environment.
if data is in properties file, scenario will look like below. Instead of the search criteria, I will give propertyName:
Scenario: search user with valid criteria
Given user navigated to login page
And clicked search link
When searched by providing search criteria
|validSearchCriteria|
Then verify the results displayed
Is there any other way I could maintain the data for scenarios for all environments and use it as per the environment the scenario is getting executed? please let me know.
Thanks
I understand the problem, but I don't quite understand the example, so allow me to provide my own example to illustrate how this can be solved.
Let's assume we test a library management software and that in our development environment our test data have 3 books by Leo Tolstoy.
We can have test case like this:
Scenario: Search by Author
When I search for "Leo Tolstoy" books
Then I should get result "3"
Now let's assume we create our QA test environment and in that environment we have 5 books by Leo Tolstoy. The question is how do we modify our test case so it works in both environments?
One way is to use tags. For example:
#dev_env
Scenario: Search by Author
When I search for "Leo Tolstoy" books
Then I should get result "3"
#qa_env
Scenario: Search by Author
When I search for "Leo Tolstoy" books
Then I should get result "5"
The problem here is that we have lots of code duplication. We can solve that by using Scenario Outline, like this:
Scenario Outline: Search by Author
When I search for "Leo Tolstoy"
Then I should see "<number_of_books>" books
#qa_env
Examples:
| number_of_books |
| 5 |
#dev_env
Examples:
| number_of_books |
| 3 |
Now when you execute the tests, you should use #dev_env tag in dev environment and #qa_env in QA environment.
I'll be glad to hear some other ways to solve this problem.
You can do this in two ways
Push the programming up, so that you pass in the search criteria by the way you run cucumber
Push the programming down, so that your step definition uses the environment to decide where to get the valid search criteria from
Both of these involve writing a more abstract feature that does not specify the details of the search criteria. So you should end up with a feature that is like
Scenario: Search with valid criteria
When I search with valid criteria
Then I get valid results
I would implement this using the second method and write the step definitions as follows:
When "I search with valid criteria" do
search_with_valid_criteria
end
module SearchStepHelper
def search_with_valid_criteria
criteria = retrieve_criteria
search_with criteria
end
def retrieve_criteria
# get the environment you are working in
# use the environment to retrieve the search criteria
# return the criteria
end
end
World SearchStepHelper
Notice that all I have done is change the place where you do the work, from the feature, to a helper method in a module.
This means that as you are doing your programming in a proper programming language (rather than in the features) you can do whatever you want to get the correct criteria.
This may have been answered elsewhere but the team I work with currently tends to prefer pushing environmental-specific pre-conditions down into the code behind the step definitions.
One way to do this is by setting the environment name as an environment variable in the process running the test runner class. An example could be $ENV set to 'Dev'. Then #Before each scenario is tested it is possible verify the environment in which the scenario is being executed and load any environment-specific data needed by the scenario:
#Before
public void before(Scenario scenario) throws Throwable {
String scenarioName = scenario.getName();
env = System.getenv("ENV");
if (env == null) {
env = "Dev";
}
envHelper.loadEnvironmentSpecificVariables();
}
Here we set a 'default' value of 'Dev' in case the test runner is run without the environment variable being set. The envHelper points to a test utility class with the method loadEnvironmentSpecificVariables() that could load data from a JSON, csv, XML file with data specific to the environment being tested against.
An advantage of this approach is that it can help to de-clutter Feature files from potentially distracting environmental meta-data which can impact the readability of the feature outside of the development and testing domains.

Protect user credentials when connecting R with databases using JDBC/ODBC drivers

Usually I connect to a database with R using JDBC/ODBC driver. A typical code would look like
library(RJDBC)
vDriver = JDBC(driverClass="com.vertica.jdbc.Driver", classPath="/home/Drivers/vertica-jdbc-7.0.1-0.jar")
vertica = dbConnect(vDriver, "jdbc:vertica://servername:5433/db", "username", "password")
I would like others to access the db using my credentials but I want to protect my username and password. So I plan save the above script as a "Connections.r" file and ask users to source this file.
source("/opt/mount1/Connections.r")
If I give execute only permission to Connections.r others cannot source the file
chmod 710 Connections.r
Only if I give read and execute permission R lets users to source it. If I give the read permission my credentials will be exposed. Is there anyways we could solve this by protecting user credentials?
Unless you were to deeply obfuscate your credentials by making an Rcpp function or package that does the initial JDBC connection (which won't be trivial) one of your only lighter obfuscation mechanisms is to store your credentials in a file and have your sourced R script read them from the file, use them in the call and them rm them from the environment right after that call. That will still expose them, but not directly.
One other way, since the users have their own logins to RStudio Server, is to use Hadley's new secure package (a few of us sec folks are running it through it's paces), add the user keys and have your credentials stored encrypted but have your sourced R script auto-decrypt them. You'll still need to do the rm of any variables you use since they'll be part of environment if you don't.
A final way, since you're giving them access to the data anyway, is to use a separate set of credentials (the way you phrased the question it seems you're using your credentials for this) that only work in read-only mode to the databases & tables required for these analyses. That way, it doesn't matter if the creds leak since there's nothing "bad" that can be done with them.
Ultimately, I'm as confused as to why you can't just setup the users with read only permissions on the database side? That's what role-based access controls are for. It's administrative work, but it's absolutely the right thing to do.
Do you want to give someone access, but not have them be able to see your credentials? That's not possible in this case. If my code can read a file, I can see everything in the file.
Make more accounts on the SQL server. Or make one guest account. But you're trying to solve the problem that account management solves.
Have the credentials sent as command arguments? Here's an example of how one would do that:
suppressPackageStartupMessages(library("argparse"))
# create parser object
parser <- ArgumentParser()
# specify our desired options
# by default ArgumentParser will add an help option
parser$add_argument("-v", "--verbose", action="store_true", default=TRUE,
help="Print extra output [default]")
parser$add_argument("-q", "--quietly", action="store_false",
dest="verbose", help="Print little output")
parser$add_argument("-c", "--count", type="integer", default=5,
help="Number of random normals to generate [default %(default)s]",
metavar="number")
parser$add_argument("--generator", default="rnorm",
help = "Function to generate random deviates [default \"%(default)s\"]")
parser$add_argument("--mean", default=0, type="double", help="Mean if generator == \"rnorm\" [default %(default)s]")
parser$add_argument("--sd", default=1, type="double",
metavar="standard deviation",
help="Standard deviation if generator == \"rnorm\" [default %(default)s]")
# get command line options, if help option encountered print help and exit,
# otherwise if options not found on command line then set defaults,
args <- parser$parse_args()
# print some progress messages to stderr if "quietly" wasn't requested
if ( args$verbose ) {
write("writing some verbose output to standard error...\n", stderr())
}
# do some operations based on user input
if( args$generator == "rnorm") {
cat(paste(rnorm(args$count, mean=args$mean, sd=args$sd), collapse="\n"))
} else {
cat(paste(do.call(args$generator, list(args$count)), collapse="\n"))
}
cat("\n")
Sample run (no parameters):
usage: example.R [-h] [-v] [-q] [-c number] [--generator GENERATOR] [--mean MEAN] [--sd standard deviation]
optional arguments:
-h, --help show this help message and exit
-v, --verbose Print extra output [default]
-q, --quietly Print little output
-c number, --count number
Number of random normals to generate [default 5]
--generator GENERATOR
Function to generate random deviates [default "rnorm"]
--mean MEAN Mean if generator == "rnorm" [default 0]
--sd standard deviation
Standard deviation if generator == "rnorm" [default 1]
The package was apparently inspired by the python package of the same name, so looking there may also be useful.
Looking at your code, I'd probably rewrite it as follows:
library(RJDBC)
library(argparse)
args <- ArgumentParser()
args$add_argument('--driver', dest='driver', default="com.vertica.jdbc.Driver")
args$add_argument('--classPath', dest='classPath', default="/home/Drivers/vertica-jdbc-7.0.1-0.jar")
args$add_argument('--url', dest='url', default="jdbc:vertica://servername:5433/db")
args$add_argument('--user', dest='user', default='username')
args$add_argument('--password', dest='password', default='password')
parser <- args$parse_args
vDriver <- JDBC(driverClass=parser$driver, parser$classPath)
vertica <- dbConnect(vDriver, parser$url, parser$user , parser$password)
# continue here
Jana, it seems odd that you are willing to let the users connect via R but not in any other way. How is that obscuring anything from them?
I don't understand why you would not be satisfied with a guest account that has specific SELECT-only access to certain tables (or even views)?

How to ensure Cucumber doesn't print skipped steps?

I have a long Cucumber scenario with only a few steps implemented, and many steps are pending. In my set up, Cucumber skips all pending steps, but still prints all the names of those skipped tests in the output, and that makes it tedious to scroll up to find the step where the scenario failed.
How can I make sure the skipped steps are not printed in the output? Ideally, I want the output to show all passing steps and the last failed step.
I think there is no out of the box formatter that does exactly that. You would need to create a custom one but perhaps you could use the progress formatter. It prints information about the failed steps, green dots for the passing steps and dashes for the skipped ones:
>> cucumber --format progress
..............................F-
(::) failed steps (::)
"Error messages here"
Failing Scenarios:
cucumber features/dashboard.feature:13 # Scenario: Dashboard data is loaded
4 scenarios (1 failed, 3 passed)
26 steps (1 failed, 1 skipped, 24 passed)
0m41.604s
Also you can combine different formatters. One to give its output in the console and others to output its format in different files for later inspection. for example:
$ cucumber -f progress -f pretty --out results.txt -f rerun --out rerun.txt

How can I make cucumber run all the steps (not skip them) even if one of them fails?

I am using Cucumber with RubyMine, and I have a scenario with steps that verify some special controls from a form (I am using cucumber for automation testing). The controls don't have anything to do with each other, and there is no reason for the steps to be skipped if one in front of them fails.
Does anyone know what configurations or commands should I use to run all the steps in a scenario even if they all fail?
I think the only way to achieve desired behavior (which is quite uncommon) is to define custom steps and catch exceptions in it yourself. According to cucumber wiki step is failed if it raises an error. Almost all default steps raise error if they can't find or interact with an element on the page. If you'll catch this exceptions the step will be marked as passed, but in rescue you can provide custom output. Also I recommend you to carefully define exceptions you want to catch, I think if you're Ok if selenium can't find an element on the page rescue only from ElementNotFound exceptions, don't catch all exceptions.
I've seen a lot of threads on the Web about people wanting to continue steps execution if one failed.
I've discussed with Cucumber developers: they think this is a bad idea: https://groups.google.com/forum/#!topic/cukes/xTqSyR1qvSc
Many times, scenarios can be reworked to avoid this need: scenarios must be split into several smaller and independent scenarios, or several checks can be aggregated into one, providing a more human scenario and a less script-like scenario.
But if you REALLY need this feature, like our project do, we've done a fork of Cucumber-JVM.
This fork let you annotate steps so that when they fail with a determined exception, they will let let next steps execute anyway (and the step itself is marked as failed).
The fork is available here:
https://github.com/slaout/cucumber-jvm/tree/continue-next-steps-for-exceptions-1.2.4
It's published on the OSSRH Maven repository.
See the README.md for usage, explanation screenshot and Maven dependency.
It's only available for the Java language, tough: any help is welcome to adapt the code to Ruby, for instance. I don't think it will be a lot of work.
The question is old, but hopefully this will be helpful. What I'm doing feels kind of "wrong", but it works. In your web steps, if you want to keep going, you have to catch exceptions. I'm doing that primarily to add helpful failure messages. I'm checking a table full of values that are identified in Cucumber with a table having a bunch of rows like:
Then my result should be:
| Row Identifier | Column Identifier | Subcolum Identifier | $1,247.50 |
where the identifiers make sense in the application domain, and name a specific cell in the results table in a human-friendly way. I have helpers that convert the human identifiers to DOM IDs, which are used to first check whether the row I'm looking for exists at all, then look for the specific value in a cell in that row. The default failure message for a missing row is clear enough for me (expected to find css "tr#my_specific_dom_id" but there were no matches). But the failure message for checking specific text in a cell is completely unhelpful. So I made a step that catches the exception and uses the Cucumber step info and some element searching to get a good failure message:
Then /^my application domain results should be:$/ do |table|
table.rows.each do |row|
row_id = dom_id_for(row[0])
cell_id = dom_id_for(row[0], row[1], row[2])
page.should have_css "tr##{row_id}"
begin
page.should have_xpath("//td[#id='#{cell_id}'][text()=\"#{row[3].strip.lstrip}\"]")
rescue Capybara::ExpectationNotMet => exception
# find returns a Capybara::Element, native returns a Selenium::WebDriver::Element
contents = find(:xpath, "//td[#id='#{cell_id}']").native.text
puts "Expected #{ row[3] } for #{ row[0,2].join(' ') } but found #{ contents } instead."
#step_failures_were_rescued = true
end
end
end
Then I define a hook in features/support/hooks.rb like:
After do |scenario|
unless scenario.failed?
raise Capybara::ExpectationNotMet if #step_failures_were_rescued
end
end
This makes the overall scenario fail, but it masks the step failure from Cucumber, so all the step results are green, including the ones that aren't right. You have to see the scenario failure, then look back at the messages to see what failed. This seems kind of "bad" to me, but it works. It's WAY more convenient in my case to get the expected and found values listed in a domain-friendly context for the whole table I'm checking, rather than to get a message like "I looked for "$123.45" but I couldn't find it." There might be a better way to do this using the Capybara "within" method. This is the best I've come up with so far though.

Resources